Traditional databases organize data by structure. Rows, columns, schemas, keys. You query them with exact logic: "Give me records where age > 30 and city = 'New York.'" Binary. True or false. Exact or not exact. But what if you want to ask a different question: "Give me records most similar to this thing." That's what vector databases solve.
A vector database stores high-dimensional vectors (lists of numbers) and retrieves them by similarity. You embed your documents into vectors. You embed a query into a vector. You ask the database: "Return the 10 vectors most similar to my query vector." The database computes similarity (usually cosine distance in high-dimensional space), ranks the results, returns the closest matches. This is semantic search. This is how RAG works under the hood.
Vector databases include Pinecone, Weaviate, Qdrant, Milvus, Chroma, and others. They vary in deployment (managed cloud vs. self-hosted), scalability, query speed, and pricing. A small deployment might use an in-memory vector store. A production system might use a dedicated vector database handling millions of vectors. The tradeoff is complexity versus capability.
The performance bottleneck is distance calculation. If you have 1 million vectors and each vector has 1,536 dimensions, calculating distance between a query and all 1 million vectors requires matrix math. This gets expensive fast. Vector databases solve this with approximate nearest neighbor (ANN) search. Instead of calculating exact distances to all vectors, ANN algorithms use data structures (like hierarchical navigable small-world graphs) to approximate which vectors are likely closest, then compute exact distances for a candidate set. Trade some accuracy for massive speed gains. Good ANN implementations get you 99%+ of results 100x faster.
Vector databases also introduce new operational challenges. You need to decide embedding model consistency. If you switch embedding models, all your stored vectors become incompatible with new queries. Reembedding a massive corpus is expensive. Scaling vector databases to billions of vectors requires sharding and distributed systems. Query latency becomes critical because every RAG request hits the vector database.
Cost gets tricky too. Managed vector database services charge per-operation or per-query. A chatbot system with thousands of queries daily multiplies these costs. Self-hosted vector databases require infrastructure and maintenance. The economics depend heavily on query volume and vector count.
Vector databases also don't solve the retrieval quality problem. If your embedding model is weak, if your embeddings aren't aligned with your queries, if your vectors are poorly distributed in semantic space, the best vector database won't help. You get back the wrong results efficiently. This is why embedding model selection matters so much.
Hybrid search is becoming standard because pure vector similarity isn't always sufficient. A query like "red cars manufactured in 2020" has both semantic (color, purpose) and structured (year, type) components. Hybrid search combines vector similarity with traditional keyword search and filters, getting better results than vector-only search. This adds complexity but improves accuracy significantly.
The vector database landscape is still evolving. New databases emerge constantly. Features like SIMD optimization, GPU acceleration, and distributed indexing improve performance. Standards for vector operations haven't fully settled yet. Teams should expect the landscape to shift and plan for vector database migration in their architecture.
Why It Matters
Vector databases are the infrastructure enabling semantic search and RAG at scale. Without them, retrieval-augmented systems can't efficiently find relevant information across large datasets. They transform from "slow brute force search" to "fast, scalable semantic retrieval." For enterprise applications handling large document collections, knowledge bases, or user-generated content, vector databases are essential infrastructure. They enable real-time semantic search, similarity clustering, and recommendation systems that would be impossible with traditional databases.
| Store | Category | Strongest for | Main cost |
|---|---|---|---|
| pgvector | Extension on Postgres | Staying in one database | Falls behind well past a million vectors |
| Pinecone | Managed vector service | Removing operational burden | Price, and a second system |
| Qdrant | Vector, self or managed | Filtered search at scale | Another service to run |
| Weaviate | Vector with hybrid search | Keyword and semantic in one place | Operational complexity |
| Milvus | Vector, self-hosted | Very large collections | Real infrastructure work |
| Chroma | Embedded vector store | Prototypes and local development | Not built for production scale |
| Redis | Operational store with vector search | Session state and vectors together | Memory-resident cost |
| Neo4j / FalkorDB | Graph | Multi-hop relationship questions | A second system, and a query language |
Example
An e-commerce platform has 500,000 product listings with attributes, descriptions, and images. Traditional databases support exact search ("blue shirt, size M, under $50") fine. But customers ask semantic questions: "Something similar to this jacket but for summer" or "Show me products similar to what I bought last time." A vector database embeds product descriptions and customer queries, enabling semantic search. The system finds conceptually similar products even if they use different terminology, dramatically improving discovery and sales.