The retrieval pipeline is the machinery that powers information retrieval. It's used in RAG systems, search systems, and any AI application that needs to find relevant information from large collections. A good retrieval pipeline can dramatically improve the quality of AI responses.
The pipeline starts with document preprocessing. Raw documents (PDFs, web pages, emails) are converted to text, cleaned (removing noise), and structured. This is harder than it sounds. A PDF might have multiple columns, figures, complex formatting. Extracting clean text is non-trivial.
Next is chunking. A document is split into smaller pieces. The question of how to chunk is deceptively important. If you chunk too small (100 words per chunk), context gets lost. If you too large (10,000 words per chunk), you can only fit a few in the prompt. Different document types need different chunking strategies. Code documents chunk differently from prose documents.
Embedding is the next step. Each chunk is converted to a vector (a list of numbers representing its meaning). Embeddings enable semantic similarity search: you can find chunks with meaning similar to the query, even if they use different words.
The vector database stores embeddings efficiently. Similarity search in a million embeddings needs to be fast (subsecond). Vector databases use specialized indexes (like approximate nearest neighbor search) to achieve this.
At query time, the query is embedded using the same embedding model. The embedding is searched against stored embeddings, retrieving the most similar chunks. These chunks are then ranked (in case there are more chunks than fit in the prompt) and retrieved.
Query expansion helps. Instead of searching for the user's exact query, you might expand it: generate multiple variations and search for all of them. This catches documents that match variations of the query.
Re-ranking improves results. An initial fast retriever narrows down candidates. Then a slower, more sophisticated ranking step orders them. This is faster than ranking all documents with the sophisticated method, but better than just using the fast method.
Different retrieval methods have different strengths. Keyword search is good for exact matches. Semantic search is good for meaning matches. Graph-based search is good when you care about relationships between documents. Good pipelines use multiple methods and combine results.
Evaluation is critical. You need to measure: what percentage of queries retrieve the documents needed to answer them? This tells you whether your retrieval pipeline is working.
There's also the cold start problem. A new document gets added. When should it be embeddings? Immediately, so search returns it? Or batched, to maintain consistency and efficiency? Depending on the system, the answer differs.
Personalization in retrieval is increasingly common. Different users might have different relevant documents for the same query. A personalized retriever might boost documents from the user's team, or documents the user has interacted with before.
Why It Matters
Retrieval is the bottleneck for RAG quality. If retrieval fails to find relevant information, the model can't generate good responses. Investing in retrieval quality directly improves AI system quality.
Example
An enterprise knowledge base has 100,000 documents. A user searches for "approval process for equipment purchase." A basic keyword search finds 50 documents, most not relevant. A sophisticated retrieval pipeline: expands the query to variations, searches semantically, boosts internal policy documents, re-ranks by relevance, returns the 5 most relevant documents which directly answer the question.