When Not to Use RAG
Retrieval-augmented generation has become the default answer to “we have documents and we want a chatbot.” Defaults are useful. They’re also how teams end up owning infrastructure that solves a problem they didn’t have.
Four cases where the honest recommendation is: don’t build it.
1. The corpus is small and stable
If everything the assistant needs to know fits comfortably inside the model’s context window, and the documents change rarely, retrieval is a lossy optimisation.
Consider what you get by not retrieving. There is no chunking strategy to get wrong. No embedding model to choose, evaluate, and later migrate off. No index to keep in sync. No top-k to tune. No class of failure where the right paragraph existed and the retriever didn’t find it — which is, in practice, the most common way RAG systems produce wrong answers.
You pay for it in tokens per request, and with prompt caching a stable prefix is usually the cheapest part of the bill.
The threshold. This flips on volume, not on corpus size alone. Take your corpus token count, multiply by requests per day, and compare against what retrieval would cost you: a much smaller per-request prompt, plus vector storage, plus the embedding of each query, plus the amortised engineering. At low query volume, stuffing wins by a wide margin. At high volume the token term dominates everything else and retrieval wins by a wide margin. The arithmetic is worked through in RAG versus long context.
Rule of thumb: corpus that fits in context, changes monthly or less, and gets queried in the hundreds per day rather than the tens of thousands — put it in the prompt. Revisit at 10× volume.
2. The question maps to structured data
A large share of “we need RAG” requirements are a database query wearing a costume.
“What’s the status of order 4471?” “How many seats are left on my plan?” “When does my contract renew?” These have exact answers that live in a table. Embedding them as prose and retrieving by semantic similarity is a strictly worse way to reach a fact you can look up by primary key — worse on accuracy, worse on latency, worse on cost, and worse on the ability to say “no such order” instead of returning something that looks nearby.
The right architecture here is tool calling: the model translates the user’s question into a function call, your code runs the query, and the result comes back as structured data. The model formats; the database is authoritative.
The tell. If you can write the SQL that answers a question, retrieval is the wrong mechanism for it. This is the case with the highest ratio of misapplied RAG to justified RAG, because “our data is in a database and we want a chat interface” sounds like the same problem as “our data is in documents and we want a chat interface.” It isn’t.
Hybrid is common and fine. Structured queries for facts, retrieval for policy and prose. What matters is not routing the structured questions through the vector index because it was the thing you built.
3. The users know the exact term they want
Semantic search is powerful precisely because it doesn’t require exact matches. That is also its weakness.
If your users search by part number, error code, ticket ID, statute reference, or drug name, dense retrieval will cheerfully return things that are similar to what they asked for. For a query like ERR_4419, similar is not a feature. It’s the failure mode. Embedding models compress meaning, and rare high-precision tokens are exactly what compression discards.
Keyword search — BM25, or your database’s full-text index — handles these correctly, costs a fraction as much, returns in single-digit milliseconds, and has the enormous operational advantage of being explainable when it’s wrong.
The threshold. If most queries contain an exact identifier, start with keyword search and add a semantic layer only if you observe real recall failures on natural-language phrasings. If queries are mostly conversational, dense retrieval first. If genuinely mixed, hybrid — but build the keyword half first, because it’s cheaper and it’s the half that anchors precision.
4. Nobody owns the corpus
This is the failure nobody puts in an architecture diagram, and it kills more RAG systems than any technical decision.
A retrieval system’s answers are only as current as its index. Which means someone has to notice when the returns policy changes and make sure the index reflects it. Someone has to decide what happens to the three superseded copies of that policy still sitting in the corpus. Someone has to re-run the eval when the documentation team restructures the wiki. Someone has to care, on an ongoing basis, about the quality of a pile of documents that no individual team is measured on.
If you can’t name that person during the design meeting, the system has a known expiry date. It will work at launch, drift for six months, and then someone will get a confidently wrong answer from a 2024 policy document and trust in the whole thing will evaporate in a week.
The alternative isn’t a different architecture — it’s a smaller scope. Pick the subset of the corpus that does have an owner. A retrieval system over 40 well-maintained pages beats one over 4,000 pages nobody curates, on every dimension that matters, including the one where users keep using it.
The decision in one pass
| Situation | Build instead |
|---|---|
| Small, stable corpus; low query volume | Put the documents in the prompt |
| Answers live in tables | Tool calling against the database |
| Queries contain exact identifiers | Keyword/BM25 search, semantic layer only if needed |
| No corpus owner | Narrow the scope to what is owned |
| Large or changing corpus, natural-language queries, real ownership | RAG — this is what it’s for |
That last row is not a small category. Retrieval is the right answer for a genuinely large class of problems: corpora too big for a context window, documents that change faster than a fine-tune cycle, and questions phrased in the user’s words rather than the document’s. The point is not that RAG is overused in principle. It’s that it’s chosen by default rather than by argument, and four of the five rows above have a cheaper, more accurate, more maintainable answer.
Before you commit, price the whole thing rather than the prototype — what a RAG system actually costs to run — and check that your latency target survives the extra hop: latency budgets for retrieval.