Use RAG when the answer needs evidence
Retrieval-augmented generation pulls source material into a model's context so answers can draw from documents available at request time. That works well when the answer lives in internal docs, tickets, policies, code, or product data that changes over time.
RAG is a poor fit when nobody needs citations, the source data already fits in the prompt, the task is creative, or the underlying documents are a mess. It adds real complexity, and that complexity buys nothing unless the problem calls for evidence retrieval.
Deciding whether to use it
A few questions sort this out quickly:
- Does the answer depend on private or frequently changing facts? Retrieve those facts at request time because the model can't know them on its own.
- Does the user need to see where each claim came from? Retrieval gives the model source material it can cite.
- Is the relevant information a small slice of a large corpus? Retrieval supplies that slice without stuffing the full corpus into the prompt.
- Do different users have access to different sources? The retrieval layer must enforce each user's access scope, which makes the system harder to build and audit.
If none of those conditions applies, put the relevant context directly in a well-written prompt. It will usually outperform a retrieval pipeline while staying cheaper and easier to debug.
A useful rule of thumb: use training to shape behavior and RAG to supply evidence. Once the right evidence reaches the model, retrieval has done its job; remaining errors belong in prompting or training.
Failure points in the pipeline
When a RAG system gives bad answers, teams often start by swapping in a bigger model, tuning the prompt, or changing the temperature. Those changes only help when generation is the failed layer, so check the upstream pipeline first.
Most failures start upstream. The document wasn't parsed correctly. The chunk boundaries landed in the wrong place. Metadata was missing, so the filter couldn't do its job. The right chunk existed in the index but ranked below junk because the query hit on surface-level keyword overlap. The model then receives an evidence packet that can't support a reliable answer.
The pipeline has distinct layers, and each one can fail independently:
- Ingestion. The source document wasn't extracted properly: tables got flattened, headings disappeared, OCR garbled the text, or the file was never ingested at all.
- Chunking. The text was cut at arbitrary character counts that ignored section boundaries. A paragraph that answers the question got split across two chunks, and neither half is useful alone.
- Indexing and metadata. The chunk made it into the index without the metadata needed for filtering: no version tag, no tenant ID, no date. Or worse, the metadata is wrong.
- Retrieval. The right chunk is in the index but doesn't surface in the top results. Embedding search can miss the relevant chunk when the query uses different vocabulary. Sparse search can miss a conceptual match when the document and query don't share the same terms.
- Generation. The model received good evidence and still produced a bad answer: it hallucinated a citation, ignored a relevant chunk, or merged conflicting sources without saying so.
Generation failures need changes to prompting, training, or generation logic. Ingestion, chunking, indexing, metadata, and retrieval failures need data or systems fixes; a model upgrade leaves them untouched.
Chunking shapes what retrieval can find
Set chunk boundaries around complete units of meaning. A 500-character window that splits a policy statement in the middle is worse than a 1200-character chunk that keeps the full section together.
Different content types need different rules. Split policy documents at headings and subsections. Keep table headers with every slice and never cut through a row. Split code at function or logical-block boundaries while preserving imports and type signatures. For ticket threads, use issue boundaries or related turn groups; fixed word counts cut through the conversation.
Every chunk needs to carry enough metadata to be cited and filtered: source ID, section path, version, date, permissions. If a retrieved chunk can't be traced back to an exact location in an exact version of a source document, the citation is meaningless and debugging is guesswork.
A good test: pick ten real questions. Pull up the top chunks the retriever returns. Can a person answer the question from those chunks alone, without opening the original document? If not, the chunking or retrieval needs work before anything else changes.
Evaluate retrieval before you evaluate answers
The most common RAG evaluation mistake is scoring the final answer before checking whether the retriever found the right evidence. If the evidence is wrong, any correct answer was accidental and won't hold up on the next query.
Write retrieval eval cases before tuning anything. Each case has a question, an expected source, and expected chunk IDs. Run the retriever. Check whether those chunks appear in the top results. If they are missing, fix retrieval before evaluating downstream stages.
A minimum eval set should include:
- Known-answer questions where you can name the source and section.
- Questions that use exact terms: IDs, error messages, file paths, product codes. These test sparse search.
- Paraphrased questions that don't reuse document wording. These test embedding search.
- Questions where the evidence is missing and the correct answer is "I don't know."
- Questions where two sources conflict and the system needs to pick the authoritative one.
- Access-control cases that compare a user who should see the answer with a user who shouldn't.
Twenty hand-written cases are a reasonable starting point. Keep them simple and cover the question types that real users will ask.
Hybrid retrieval
Embedding search handles conceptual similarity. Sparse keyword search handles exact tokens and identifiers.
If someone asks "what's the PTO policy for new hires," embedding search handles that fine. A query about an exact identifier such as CONN_REFUSED_4412 is better handled by sparse keyword search because embedding search will probably miss the token. Most real-world RAG workloads include both types of queries, so the retriever needs both capabilities.
Hybrid retrieval combines the two candidate sets, deduplicates by chunk ID, and optionally reranks. Bad score blending can bury the best result from either branch. Log candidates from each branch separately so you can tell whether the miss came from embedding search, sparse search, or the merge.
Debug in pipeline order
When a RAG system gives a wrong answer, trace the failure through the pipeline in order:
- Is the source document in the corpus at all?
- Was the relevant section extracted correctly? Compare extracted text to the original.
- Does any chunk contain enough context to answer? Find the expected text in the chunk manifest.
- Is the chunk in the index? Check both sparse and embedding indexes.
- Does the chunk appear in retrieval candidates before reranking?
- Does it survive reranking and make it into the evidence packet?
- Did the model use it correctly in the final answer?
Stop at the first layer that failed. Fix that layer. Rerun. A bad PDF parser needs a parser fix; changing the model or prompt wastes time.
Common failure patterns
Stale versions outrank current ones. If the index has three versions of a document and no authority or freshness metadata, the retriever picks whichever version scores best on surface similarity. Add version precedence rules and deduplicate before generation.
Exact identifiers disappear. Embedding models don't handle rare tokens, product codes, or error strings well. Add sparse search or a metadata filter path for these queries.
The model cites sources that don't support the claim. This usually means the prompt allows unsupported synthesis, or the system assigns citations after writing the answer. Require claim-level citation alignment and test it.
The model answers when it shouldn't. Without an explicit abstention policy and missing-evidence examples, models will fill in gaps from training data. That's a hallucination with a citation pasted on, which is worse than no answer at all.
What a debuggable system tracks
For every request, log the full path from query to answer:
- The query as submitted and after any normalization.
- Filters applied: user, tenant, date range, access scope.
- Candidate chunk IDs and scores from each retrieval branch.
- Selected chunks after reranking and diversity rules.
- The evidence packet sent to the model.
- The prompt version and model version.
- The final answer and which chunks were cited.
- Latency for each stage.
These records make each failure traceable to its pipeline layer and provide a baseline for measuring every change.