The failure mode nobody sees
A broken database query throws. A broken retrieval pipeline returns something — and the model, being cooperative, writes a fluent answer on top of it. This is the defining operational risk of retrieval-augmented systems: failures are silent, well-formatted, and indistinguishable from success without evaluation.
Similarity is not relevance
Embedding similarity measures whether two passages occupy the same neighbourhood of meaning. Relevance measures whether a passage answers the question asked. These correlate, but not tightly enough to build on directly.
A production retrieval path needs three stages, not one:
- Recall — retrieve generously across multiple strategies (dense embeddings, keyword/BM25, and structured metadata filters)
- Rerank — score the candidate set against the actual query with a model whose only job is ordering
- Select — pass forward the smallest set that fully answers, not the largest set that fits
Most systems ship stage one and stop. Adding stage two typically produces a larger accuracy gain than upgrading the generation model.
Chunking is a schema decision
Chunk boundaries are treated as a tuning parameter — a token count picked once and forgotten. They are closer to a schema decision. A chunk should be the smallest unit that remains true when read alone. A contract clause severed from its definitions section is not a smaller fact; it is a different and incorrect one.
Practically, this means chunking along document structure — sections, clauses, rows, headings — and carrying the parent context with each chunk rather than relying on the model to reconstruct it.
Bigger context windows did not solve this
Long-context models made it possible to stuff a hundred passages into a prompt. They did not make it advisable. Padding a prompt with weakly-relevant material measurably degrades answer quality while increasing cost and latency. Precision still wins; the window size just raised the ceiling on how expensively you can be imprecise.
Evaluate retrieval separately from generation
The single highest-leverage practice: build a fixed set of real questions with known correct sources, and score retrieval on its own — did the right passage appear, and at what rank? This isolates the layer that actually broke. Teams that only evaluate final answers spend their time tuning prompts to compensate for a ranking problem.
The architectural takeaway
Retrieval belongs behind a defined interface with its own tests, its own metrics, and its own failure semantics — not inlined into feature code as "the part before the model call." Treated as infrastructure, it becomes debuggable. Treated as glue, it stays silently wrong.