What a RAG System Actually Costs to Run
Most RAG cost estimates are wrong in the same direction, for the same reason: they price the prototype. Embedding the corpus, storing the vectors, and running inference are the three numbers that are easy to look up, and together they’re usually the smaller half of the bill.
Here’s the full list, in the order you’ll encounter it, with the structure of each term. Every unit price below is hypothetical and illustrative — model and infrastructure pricing changes on a scale of months and varies by region, tier, and contract. Substitute yours; the arithmetic holds.
Build costs (one-time, mostly)
1. Initial embedding
corpus_tokens × embedding_price_per_million / 1,000,000Embedding models are cheap — typically one to two orders of magnitude cheaper per token than generation. For a 300,000-token corpus at a hypothetical $0.10/M, that’s three cents. For a 300-million-token corpus, thirty dollars.
This line item is almost never the problem, and it consumes a disproportionate share of planning discussion because it’s the easiest to calculate.
2. Pipeline engineering
Ingestion, parsing, chunking, the retrieval service, prompt assembly, evaluation harness, observability. For a competent team on a normal corpus, a working prototype takes days and a production system takes weeks — longer if the documents are PDFs, much longer if they’re scanned PDFs.
At a fully loaded engineering cost that’s real money and it’s usually the largest single number in the build column. Price it in weeks of a named person’s time, not in “we’ll spike it.”
Run costs (recurring, predictable)
3. Vector storage
vector_count × dimensions × bytes_per_dimension × replication_overheadA million chunks at 1,536 dimensions and 4 bytes each is roughly 6 GB of raw vectors, before index structures and replicas — call it two to three times that in practice.
Managed vector databases price on some combination of stored vectors, provisioned compute, and queries; self-hosted prices on the instance. Either way this is a monthly cost in the tens to low hundreds of dollars for most corpora, and it scales with corpus size rather than traffic. It’s rarely the dominant term until you’re well past a million chunks.
4. Query embedding
queries_per_day × ~50 tokens × embedding_priceEffectively free. At a hypothetical $0.10/M and 10,000 daily queries, that’s five cents a day. Listed only so you can stop thinking about it.
5. Generation
requests × (retrieved_tokens + prompt_tokens + output_tokens) × respective_pricesThis is the dominant run cost at any meaningful volume, and it’s driven mostly by how many chunks you pass. At a hypothetical $3/M input and $15/M output, 10,000 daily requests with 4,000 input and 500 output tokens is $120/day input plus $75/day output — around $5,850/month.
Note what that implies: cutting from 20 chunks to 8 cuts your largest run cost by more than half, and on most corpora it improves answer quality. The single highest-leverage cost optimisation in a RAG system is passing fewer, better chunks — which requires knowing which chunks are better, which requires an eval set.
6. Reranking
If you retrieve 50 and rerank to 8, you’re paying a cross-encoder or a reranking API per candidate per query. Small per unit, multiplied by candidates × queries. Usually a few percent of the generation bill; occasionally more if you rerank aggressively at high volume.
Maintenance costs (recurring, unpredictable, usually unbudgeted)
This is the half that gets left out, and it’s the half that determines whether the system still works in a year.
7. Reindexing
Documents change. Each change means re-parsing, re-chunking, re-embedding, and upserting. The compute is trivial. The operational cost is not: you need change detection, an update pipeline, a deletion path for retired documents, and a way to notice when any of that silently stopped running.
Full reindexes happen more often than anyone plans for — you’ll change chunking strategy at least once, and you’ll change embedding models when a better one appears. Both mean re-embedding the entire corpus and rebuilding the index. Budget for one full reindex per year as a baseline; the compute is cheap and the coordination isn’t.
8. Evaluation
An eval set is not a one-time artifact. It needs to be built (human time, per question, and the questions have to be real), maintained as the corpus changes, and extended every time a failure is diagnosed. If you use LLM-as-judge, you’re also paying inference per eval run, which adds up if you run it in CI.
The failure mode here is not overspending. It’s skipping it entirely and losing the ability to tell whether any change helped — at which point every subsequent optimisation is guesswork. Methodology for this lives on what-is-rag.net.
9. Corpus ownership
The line item with no invoice. Someone has to decide what’s in the corpus, remove what’s superseded, notice when three versions of a policy are all live, and answer for quality. This is a fraction of a person, permanently, and it’s the cost most likely to be assigned to nobody. When it is, the system’s answers degrade quietly until trust collapses all at once.
10. Incident time
Retrieval systems fail in ways that generate support tickets: a wrong answer that came from a stale document, an embarrassing citation, a question that should have been refused. Each one costs investigation time — and the investigation is only fast if you built tracing in step 2.
Two worked scenarios
Hypothetical prices throughout, monthly, excluding one-time build.
Small internal tool — 50k-token corpus, 200 requests/day, 2,000 input and 300 output tokens per request:
| Line | Monthly |
|---|---|
| Storage | ~$25 |
| Generation | ~$63 |
| Query embedding | ~$0 |
| Infrastructure subtotal | ~$88 |
| Maintenance (est. 4 hrs/month of engineering) | the larger number |
The infrastructure is a rounding error. The decision is entirely about whether four hours a month of someone’s attention is available — and whether you needed retrieval at all at that corpus size (probably not).
Production support assistant — 5M-token corpus, 20,000 requests/day, 4,000 input and 500 output tokens:
| Line | Monthly |
|---|---|
| Storage (~50k chunks, managed) | ~$100 |
| Generation | ~$11,700 |
| Reranking | ~$300 |
| Reindexing compute | ~$20 |
| Infrastructure subtotal | ~$12,100 |
| Maintenance (est. 0.3 FTE) | substantial, and unavoidable |
Here generation is 97% of the infrastructure bill. Every optimisation conversation should start and end with chunk count and output length, and the eval set that tells you how far you can cut.
The shape to remember
- Embedding and storage are cheap. They dominate the planning conversation and almost never dominate the bill.
- Generation dominates infrastructure, and it’s controlled by how many tokens you pass, not by how clever your index is.
- Maintenance dominates total cost at low volume and stays significant at high volume. It’s the term that decides whether the project succeeds.
Price all three columns before deciding. Pricing only the first is how a system gets approved on a number that turns out to be 10% of the real one — and it’s why some systems shouldn’t be built at all.