RAG Versus Long Context: The Cost Arithmetic
Context windows got long enough that “just put the documents in the prompt” became a real architecture rather than a joke. That changed the RAG decision from a technical question into an arithmetic one.
The arithmetic is not hard. Most teams skip it anyway, because the RAG-versus-long-context debate is usually conducted in adjectives.
The two cost shapes
Long context. You send the corpus (or the relevant slice of it) with every request. Cost per request scales with corpus size. Cost per day scales with corpus size × requests. No infrastructure.
Retrieval. You embed the corpus once, store the vectors, and send a small prompt per request. Per-request cost is roughly flat regardless of corpus size. You pay a fixed-ish monthly cost for storage and a small per-request cost for the query embedding — plus engineering time, which is the term everyone leaves out.
So: long context is linear in corpus size and free of infrastructure; retrieval is nearly flat in corpus size and carries a floor. They cross. Where they cross depends on three numbers.
The three variables
- Corpus size in tokens (
C). Not pages — tokens. A rough working estimate is around 750 tokens per page of dense prose, and you should measure rather than assume if the number matters. - Requests per day (
R). - Input price per million tokens (
P). Yours, from your contract.
Everything else is second-order until you’re at scale.
The comparison
Long context, daily input cost:
C × R × P / 1,000,000Retrieval, daily input cost:
K × R × P / 1,000,000 + (storage/day) + (query embedding/day)where K is the tokens you actually send — retrieved chunks plus instructions — typically a few thousand, and crucially independent of C.
The token saving per request is (C − K). Storage and query-embedding costs are small at most realistic scales; the term that actually decides the comparison is engineering.
A worked example
Every price below is hypothetical and illustrative. Substitute your own; the structure is what transfers.
Assume:
- Corpus: 400 pages ≈ 300,000 tokens
- Retrieval sends 4,000 tokens per request
- Input price: $3.00 per million tokens (hypothetical)
- No prompt caching, initially
| Requests/day | Long context/day | Retrieval tokens/day | Difference/month |
|---|---|---|---|
| 100 | $90 | $1.20 | ~$2,660 |
| 1,000 | $900 | $12 | ~$26,640 |
| 10,000 | $9,000 | $120 | ~$266,400 |
At 1,000 requests a day, that difference pays for a substantial amount of engineering, and the recommendation is obvious. At 100 requests a day, ~$2,700/month is still real money — but now compare it against building and maintaining the retrieval pipeline, and against the fact that a retrieval system can be wrong in ways a full-context prompt cannot.
Now change one input. Suppose the corpus is 40 pages, not 400:
- Corpus ≈ 30,000 tokens
- 100 requests/day → $9/day, about $270/month
That’s below the noise floor of most engineering budgets. Build retrieval for that and you have spent weeks of engineering to save the cost of a couple of lunches, and acquired a permanent maintenance obligation in exchange.
Prompt caching changes the numbers, not the shape
If your corpus is a stable prefix and your provider supports prompt caching, the long-context column drops substantially — cached input is billed at a fraction of the standard rate, with the exact discount and TTL varying by provider and changing over time.
This matters enormously for the small-and-stable case and barely at all for the large-and-changing case:
- Stable corpus → high cache hit rate → long context gets much cheaper → crossover moves right, sometimes by an order of magnitude.
- Corpus changes hourly → cache invalidated constantly → you’re paying near-full price and also can’t take advantage of the one thing that made stuffing viable.
If you’re evaluating long context, model it with caching or you’ll overstate its cost badly. If your corpus changes frequently, model it without.
Cost is not the only axis
Three things the arithmetic doesn’t capture, and one of them usually decides the question.
Quality. Long context removes the retrieval failure mode entirely: the model always has the material. But performance across very long contexts is uneven — material in the middle of a long prompt tends to be used less reliably than material at the beginning or end, an effect widely enough observed to have a name (lost in the middle). Retrieval, when it works, hands the model a short, dense, relevant prompt, which is easier to reason over. Which is more accurate is genuinely task-dependent and you should measure it on your own eval set rather than trusting anyone’s general claim.
Latency. Long prompts take longer to process — time-to-first-token grows with input length. Retrieval adds a round trip (embedding + search, commonly tens to low hundreds of milliseconds) but then generates from a much shorter prompt. At large C, retrieval is usually faster end to end. See latency budgets for retrieval.
Ceiling. Beyond some corpus size, long context stops being an option at all. When C exceeds the window, the decision is made for you.
The recommendation
Default to long context when: the corpus fits in the window with room to spare, changes rarely enough to cache well, and query volume is low enough that the monthly token bill is smaller than the monthly cost of owning a pipeline. Below roughly a thousand requests a day on a corpus in the tens of thousands of tokens, this is nearly always the right call, and it is the call teams most often get wrong in the other direction.
Default to retrieval when: the corpus is large or growing, changes often, or query volume is high enough that the token difference dominates. Above roughly ten thousand requests a day on a corpus in the hundreds of thousands of tokens, it isn’t close.
In between — and the band is wide — do the arithmetic above with your own C, R, and P, then add the maintenance figure from what a RAG system actually costs to run. If the annual saving from retrieval is less than the fully loaded cost of a month of engineering time, don’t build it yet. Ship long context, instrument volume, and revisit when the numbers move.
One practical note: those defaults are cheap to reverse in one direction and expensive in the other. Starting with long context and migrating to retrieval later means building the pipeline you would have built anyway, with the advantage of real query logs to build the eval set from. Starting with retrieval and discovering you didn’t need it means you have already paid.
Also worth asking before either: whether retrieval is the right shape at all.