Latency Budgets for Retrieval
“It feels slow” is not a requirement. A latency budget is: a target number, allocated across stages, with a named owner for each stage that exceeds its allocation.
Most teams never write one, discover the system is slow after launch, and then optimise whichever stage someone happens to have an opinion about. Usually the wrong one.
Set the target first
The target is a product decision, not an engineering one, and there are really three regimes:
- Under ~300 ms — feels instant. Not achievable for a generated answer; achievable for retrieval-only surfaces like search-as-you-type.
- Under ~1 s to first token — feels responsive. A reasonable target for a streaming chat interface, and the one most conversational products should aim at.
- Under ~3 s to first token — tolerable when the user has been given a reason to wait. Beyond that, abandonment climbs.
Two things follow from this that change how you optimise.
Streaming means time-to-first-token is the metric that matters, not total completion time. A response that starts in 800 ms and finishes in 6 s feels fast. One that starts in 4 s and finishes in 4.5 s feels broken. Everything before the first token is dead air; everything after is reading time.
Optimise the tail, not the mean. Your p50 is what you demo. Your p95 is what users complain about. A 700 ms mean with a 6 s p95 is a slow product, and every stage below has a tail that’s worse than its average.
Where the time goes
A typical RAG request, in order. Ranges are order-of-magnitude and vary enormously with implementation — measure yours.
| Stage | Typical | Notes |
|---|---|---|
| Request handling, auth | 1–10 ms | Ignorable |
| Query rewriting (if LLM-based) | 200–800 ms | An entire extra model call |
| Query embedding | 10–100 ms | Network-bound if it’s an API call |
| Vector search | 5–50 ms | Fast; scales sub-linearly with corpus |
| Keyword search (if hybrid) | 5–30 ms | Parallelisable with the vector half |
| Reranking | 50–500 ms | Depends on candidate count and model |
| Prompt assembly | 1–10 ms | Ignorable |
| Generation TTFT | 300–2000 ms | Grows with input length |
Two stages dominate, and neither is the one people expect.
Vector search is not your bottleneck. It’s usually the fastest substantive stage in the pipeline. The amount of engineering attention spent on index tuning relative to its share of the latency budget is the single biggest misallocation in this space. If your search is taking 300 ms, you have a configuration or network problem, not an algorithmic one.
LLM-based query rewriting is the most expensive thing you can add, because it’s a full model round trip that must complete before retrieval can even start. It’s a serial dependency on the critical path, and it frequently costs more than everything downstream of it combined.
A worked budget
Target: 1,000 ms to first token, at p95.
| Stage | Allocation |
|---|---|
| Query embedding | 80 ms |
| Retrieval (hybrid, parallel) | 60 ms |
| Reranking | 150 ms |
| Assembly + overhead | 30 ms |
| Generation TTFT | 600 ms |
| Guardrail check | 80 ms |
| Total | 1,000 ms |
Now notice what doesn’t fit. There is no room for LLM-based query rewriting — adding it at 400 ms puts you 40% over budget on its own. If you want it, something else has to go, and the honest options are dropping reranking or accepting a 1.4 s target.
That’s what a budget is for. Not to make things fast, but to force the trade-off to be explicit before it’s discovered in production.
The cuts, in order of what they cost you
Free — take these first
Run retrieval branches in parallel. Vector and keyword search have no dependency on each other. Sequential hybrid search is a pure waste of the shorter branch’s duration.
Co-locate. Vector DB and application in the same region. Cross-region round trips add 50–150 ms per hop and there’s no compensating benefit.
Reuse connections. Persistent HTTP connections to your embedding and generation providers. TLS handshakes on every request are a real and completely unnecessary cost.
Cache query embeddings. Real query distributions are heavily repetitive. A small cache keyed on the normalised query removes the embedding stage entirely for a meaningful fraction of traffic.
Stream. If you aren’t streaming, do that before optimising anything else. It changes the metric that matters and is usually a day of work.
Cheap — small quality cost
Cut chunk count. The largest lever on generation TTFT, because input length drives prompt processing time. Going from 20 chunks to 8 is a substantial latency win and a substantial cost win, and on most corpora it doesn’t hurt quality — sometimes it helps. Verify on your eval set before and after.
Rerank fewer candidates. 50 → 20 roughly halves reranking time. Measure the recall you lose at your final k.
Use a smaller reranker. Cross-encoders vary by an order of magnitude in speed. The largest one is rarely worth its latency.
Expensive — real trade-offs
Drop reranking entirely. Saves 50–500 ms. Costs precision, which means you need more chunks to maintain recall, which costs generation time. Sometimes a net win, sometimes a net loss. Measure end-to-end, not stage-by-stage.
Use a faster generation model. The biggest single lever, and the one with the most quality risk. Worth testing on your eval set rather than assuming — for tasks where retrieved context does the heavy lifting, smaller models often close much of the gap.
Drop query rewriting. Saves the most. Costs recall on ambiguous, short, and multi-turn queries — which may be a large share of your real traffic. Before dropping it, check what fraction of queries it actually changes; if it’s rewriting 15% of queries meaningfully, consider running it conditionally.
Structural
Precompute for known queries. If a head of common questions covers a large share of traffic, cache complete answers with an appropriate TTL and serve them in single-digit milliseconds. This is by far the largest available win and it’s usually available.
Speculative retrieval. Start retrieval on the partial query while the user is still typing. Complexity in exchange for hiding the entire retrieval phase.
Show progress. Not a latency fix, a perception fix, and a legitimate one. “Searching your documents…” makes 2 s feel purposeful rather than broken.
The comparison that surprises people
At large corpus sizes, retrieval is often faster than long context, not slower.
Long context means processing a very large input before the first token can be produced, and prompt processing time grows with input length. Retrieval adds a round trip of maybe 150–250 ms but then generates from a prompt an order of magnitude shorter.
So the latency axis and the cost axis point the same direction at scale — both favour retrieval on large corpora, both favour stuffing on small ones. That’s convenient, and it means the cost arithmetic usually settles the latency question too. Prompt caching complicates this: a cached prefix can be processed much faster than an uncached one, which pulls the long-context option’s latency down along with its cost.
Measure before cutting
Trace every stage with timestamps, in production, and look at p50, p95, and p99 separately. The stage that dominates your p95 is frequently not the one that dominates your p50 — an external API with an occasional 3 s tail can be invisible in the average and responsible for every complaint you get.
Then optimise the stage that owns the tail. Optimising the mean of a stage that was never the problem is the default behaviour in the absence of a trace, and it’s why so much index-tuning effort produces no user-visible change at all.