Production & Operations
Latency budgets, caching, cost control, incremental indexing, index migrations, observability, access control, multi-tenancy, and indirect prompt injection
Production & Operations
TL;DR
A prototype answers questions. A production system answers them fast, cheaply, safely, and repeatedly, over documents that keep changing, for users who are not allowed to see the same things. The gap between the two is almost entirely engineering that has nothing to do with retrieval quality — and it is where most RAG projects stall.
| Property | Value |
|---|---|
| Level | Advanced |
| Reading time | ~30 minutes |
| Prerequisites | Evaluation & Metrics |
| You will understand | Latency, cost, freshness, observability, and the security model RAG requires |
Part 1 — Latency
Where the time goes
Representative figures for a hybrid-search pipeline with reranking:
| Stage | Typical | Share | Notes |
|---|---|---|---|
| Query embedding | 20–80 ms | small | Network-dominated; local models cut this |
| Vector search | 10–50 ms | small | Scales with index size and recall settings |
| Keyword search | 5–30 ms | small | Usually parallel with vector search |
| Reranking | 50–200 ms | moderate | Grows linearly with candidate count |
| Generation | 500–3,000 ms | dominant | Grows with input and output tokens |
| Total | 0.6–3.4 s |
Generation is 70–90% of your latency. Optimizing vector search from 40 ms to 20 ms is invisible to users. Streaming the first token, shortening the context, or choosing a faster model are the changes people actually feel. Profile before optimizing — the intuition that "search must be the slow part" is almost always wrong.
Switch the optimizations on and off and watch both bars react:
Where the time and money actually go
Switch optimizations on and off
Turn on “make vector search 3× faster” by itself and watch the totals barely move — you saved 27 ms out of roughly 1,800. Now turn on streaming and halve the context instead. Generation is 70–90% of both bars, so that is the only place worth optimizing.
What actually helps
| Technique | Effect | Cost |
|---|---|---|
| Stream tokens | Perceived latency drops enormously; first token in ~300 ms | Complicates post-generation verification |
| Parallelize retrieval | Dense and sparse search concurrently — free time back | Trivial |
| Shrink the context | Fewer input tokens is faster and cheaper and often more accurate | Requires good reranking |
| Smaller/faster model | Large win where the task is extraction rather than reasoning | Quality risk — measure it |
| Cache | Repeated queries return in milliseconds | Staleness; see below |
| Skip retrieval when unnecessary | Adaptive routing removes whole stages | A classifier to maintain |
Part 2 — Caching
Real traffic is extremely repetitive. Caching is the highest-return optimization in production RAG.
Cache layers, cheapest hit first
Exact-match cache
Semantic cache
Embedding cache
Retrieval cache
| Layer | Invalidate when | Danger |
|---|---|---|
| Exact-match | Source documents change | Serving a superseded answer |
| Semantic | Same, plus threshold drift | Near-miss hits — the danger case below |
| Embedding | Only when the embedding model changes | Nearly none; safe and very effective |
| Retrieval | Index updates | Stale chunk sets after re-indexing |
Semantic caching is where the bodies are buried. "Can I return an item after 30 days?" and "Can I return an item after 60 days?" are highly similar vectors with opposite answers. Set the similarity threshold conservatively (0.95+), exclude queries containing numbers, dates, negations, or entity names from semantic hits, and log every hit so you can audit what was served. A wrong cached answer is indistinguishable from a wrong model answer to the user, and far harder for you to trace.
Part 3 — Cost
Where the money goes
For a system serving 100,000 queries per month over 1M chunks — illustrative, but the proportions hold:
| Component | Driver | Typical share |
|---|---|---|
| Generation | Input + output tokens per query | 60–85% |
| Vector storage | Vectors × dimensions × replicas | 5–20% |
| Embedding (queries) | One per query, cheap | 2–5% |
| Embedding (indexing) | One-off per chunk, recurring on model change | Spiky |
| Reranking | Candidates scored per query | 3–10% |
Levers, ordered by return
| Lever | Saving | Trade-off |
|---|---|---|
| Cut context size | Large — input tokens are most of the bill | Needs good reranking to stay accurate |
| Cache aggressively | 30–60% on repetitive traffic | Staleness risk |
| Route by complexity | 40%+ — simple queries skip expensive paths | A classifier to maintain |
| Smaller model for easy queries | 10–20× per-query on that slice | Measure quality per route |
| Quantize vectors | 4–32× storage reduction | Small recall loss |
| Cap output tokens | Bounds worst case | Truncation if set too low |
The instinct is to economize on the vector database because its bill is legible and recurring. The money is in generation tokens. Halving average context size beats any storage optimization you can make — and usually improves accuracy at the same time, since you removed noise.
Part 4 — Keeping the Index Fresh
Documents change. An index that does not is a system that confidently serves last quarter's policy.
Incremental updates
Change-driven indexing
Deleting superseded chunks is the most commonly missed step. Update a document without removing its old chunks and both versions now live in the index, competing. Retrieval will sometimes surface the outdated one, producing an answer that is correct according to a document that no longer exists. Store a stable document_id on every chunk and delete by it before every re-insert.
| Requirement | Approach |
|---|---|
| Deletions | Propagate hard deletes immediately — a deleted document that keeps answering questions is a serious incident |
| Access changes | Permission revocation must reach retrieval-time filters, not just the source system |
| Freshness SLA | State it explicitly ("index lags source by at most 15 minutes") and monitor it as a metric |
| Recency metadata | Store valid_from / superseded_by so the model can prefer current versions |
Migrating the embedding model
Changing embedding models means re-embedding everything — vectors from different models are mutually meaningless, and a mixed index silently returns garbage.
Blue/green index migration
Never re-embed in place. The old index is your rollback, and you will want it.
Part 5 — Observability
What to log per query
| Field | Why you will need it |
|---|---|
| Query, and rewritten query | Reproduce the request exactly |
| Retrieved chunk IDs + scores | The single most useful debugging field — was the answer even retrievable? |
| Post-rerank IDs + scores | Distinguishes retrieval failure from reranking failure |
| Final assembled prompt | Or a hash of it, if storage is a concern |
| Model, version, parameters | Behaviour changes across model versions |
| Per-stage latency | Locates regressions |
| Token counts and cost | Per-query cost attribution |
| Cache hit/miss and layer | Distinguishes a bad answer from a bad cached answer |
| Refusal flag | Refusal rate is a leading indicator of retrieval decay |
Trace-level logging of retrieved chunk IDs is non-negotiable. When a user reports a wrong answer, the first question is always "was the correct chunk in the context?" Without that log line you cannot answer it. You will then debug the prompt for what is really a retrieval bug — the most common wasted week in RAG operations.
Alerts worth having
| Signal | Why it matters |
|---|---|
| Refusal rate rising | Retrieval is degrading, or the index broke |
| Mean retrieval score falling | Query distribution shifted, or the index is stale |
| Cache hit rate collapsing | Often the first visible sign of an index rebuild gone wrong |
| p95 latency | Averages hide the users who are actually suffering |
| Cost per query drifting | Context bloat creeps in silently |
| Indexing lag | Your freshness SLA, as a monitored number |
Part 6 — Security
RAG has a security model that generic LLM applications do not, because it puts your data in the prompt.
Access control
Filter at retrieval time, never after. Retrieving documents a user cannot see and asking the model to ignore them is not access control — the content is already in the prompt, and any prompt-level instruction can be worked around. Permissions must be a hard filter applied inside the search query.
| Requirement | Implementation |
|---|---|
| Per-user permissions | Store ACL metadata on every chunk; pass the user's groups as a mandatory filter |
| Multi-tenancy | Tenant ID as a mandatory pre-filter, or a separate index/namespace per tenant |
| Permission changes | Re-sync ACL metadata on change — a stale ACL is a data leak |
| Cache isolation | Include the identity/tenant in every cache key. A shared cache across tenants leaks answers between customers |
| Audit trail | Log which chunks were shown to whom; regulated domains will require it |
The cache-key mistake is worth stating twice, because it is easy to make and severe. If your semantic cache is keyed on the query text alone, tenant A's question can return an answer built from tenant B's documents. Every cache key must include the tenant and, where documents are user-scoped, the identity.
Indirect prompt injection
The distinctive RAG threat: the attack arrives inside a retrieved document. Anyone who can write to a source you index — a wiki page, a support ticket, an uploaded PDF, a crawled web page — can plant instructions. Your pipeline then places them into the prompt for you.
| Defence | Detail |
|---|---|
| Treat retrieved text as data, not instructions | Delimit it clearly and state in the system prompt that content inside the context block is reference material, never commands |
| Control who can write to the corpus | The real fix. Ingesting untrusted user-generated content is the root exposure. |
| Scan at ingestion | Flag imperative patterns aimed at models in incoming documents |
| Constrain the output | Structured output and no tool access from the RAG path limits what a successful injection can achieve |
| Never grant the RAG path privileged tools | If retrieved text can trigger an email send or a database write, injection escalates from misinformation to action |
Data handling
| Concern | Practice |
|---|---|
| PII in the corpus | Detect and redact at ingestion, before embedding — vectors are hard to un-ring |
| PII in queries | Users paste sensitive data into questions; scrub before logging |
| Right to erasure | Deleting a document must delete its chunks and its vectors and its cached answers |
| Residency | Embeddings sent to a hosted API leave your boundary — self-host where that matters |
Part 7 — Reliability
Every external dependency will fail. Decide in advance what happens.
| Failure | Graceful degradation |
|---|---|
| Embedding API down | Fall back to keyword-only search — degraded but functional |
| Vector DB unavailable | Serve from cache; refuse honestly rather than answering ungrounded |
| Reranker times out | Return the un-reranked top-k; note the degradation in logs |
| LLM rate-limited | Queue, retry with backoff, or fail over to a secondary provider |
| Nothing retrieved | Refuse. Never silently fall through to an unground answer |
The worst failure mode in this table is the last one. A system that answers from training data when retrieval returns nothing looks healthy on every dashboard while producing exactly the ungrounded answers RAG exists to prevent. Make empty retrieval an explicit, logged, refusing path.
Production Readiness Checklist
Before you call it production
Evaluation runs in CI against a stored baseline
RecommendedWithout this, regression is invisible until users find it.
Retrieved chunk IDs logged for every query
RecommendedThe difference between debugging in minutes and debugging in days.
Permissions enforced as retrieval-time filters
RecommendedPost-hoc filtering and prompt instructions are not access control.
Cache keys include tenant and identity
RecommendedPrevents cross-tenant answer leakage through the cache.
Empty retrieval refuses explicitly
RecommendedCloses the silent-ungrounded-answer path.
Document deletion removes chunks, vectors, and cached answers
RecommendedRequired for correctness and for erasure obligations.
Index migration is blue/green with a rollback path
RecommendedRe-embedding in place has no undo.
Refusal rate and p95 latency alerting
RecommendedLeading indicators of decay, rather than lagging user complaints.
Concept Checks
Check yourself
You cut vector search from 45 ms to 15 ms. Will users notice?
No. Generation is 500–3,000 ms of a roughly 1–3 second request, so 30 ms is around 1% of total latency — well inside run-to-run noise. The changes users feel are streaming the first token, shortening the context, or a faster generation model. This is the standard trap: search latency is the most measurable number in the pipeline, which makes it feel like the most important one.
Why is semantic caching riskier than exact-match caching?
Because it deliberately serves an answer computed for a different question, betting that similarity implies interchangeability. That bet fails precisely where it matters: "30 days" and "60 days", "can I" and "can I not" are near-identical vectors with opposite correct answers. Exact matching has no such failure — the same string genuinely is the same question. Hence high thresholds, exclusions for numbers and negations, and logging every semantic hit.
A document was updated. Its old chunks remain indexed. What breaks?
Both versions now compete in retrieval, and similarity search has no notion of which is current — it ranks by meaning, and the outdated chunk may well score higher. The system then produces a confident, correctly-cited answer from a document that no longer exists, with nothing in the output indicating anything is wrong. Delete by stable document ID before every re-insert.
Why can't you enforce permissions by telling the model to ignore unauthorized documents?
Because the content is already in the prompt by then. The model has already received it. It may leak into the answer through paraphrase, it sits in your logs, and other text in the context can override any instruction you gave. Access control has to be a hard filter inside the search query, so unauthorized chunks are never candidates in the first place.
Why does indirect prompt injection matter more in RAG than in a plain chatbot?
Because RAG creates an automatic path from documents into the prompt. The attacker never needs access to the chat. They only need write access to something you index, like a wiki page or a support ticket. Your pipeline then dutifully retrieves and inserts their text. The primary defence is controlling who can write to the corpus; the secondary is ensuring the RAG path holds no privileged tools, so a successful injection can mislead but cannot act.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Generation dominates latency | 70–90% of the wall clock; optimize there or nowhere |
| Streaming | The largest perceived-performance win available |
| Cache layers | Exact, semantic, embedding, retrieval — each with its own invalidation |
| Semantic cache risk | Similar queries with opposite answers; high thresholds and audit logs |
| Generation dominates cost too | Context size is the main bill; storage optimizations are rounding errors |
| Incremental indexing | Hash, re-chunk, delete old chunks by document ID, upsert |
| Blue/green migration | Never re-embed in place; keep the old index for rollback |
| Log retrieved chunk IDs | The one log line that makes debugging tractable |
| Retrieval-time ACL filters | The only real access control; post-hoc filtering is not |
| Cache keys with tenant | Prevents cross-tenant leakage |
| Indirect prompt injection | The attack lives in a retrieved document; control corpus writes |
| Refuse on empty retrieval | Never silently fall back to ungrounded generation |
Next
You know how to run it. Now learn to diagnose it when it misbehaves: Failure Modes & Debugging.
Evaluation & Metrics
Measuring RAG properly — recall@k, precision@k, MRR, nDCG, faithfulness, answer relevance, golden datasets, LLM-as-judge, and online signals
Failure Modes & Debugging
A systematic method for diagnosing bad answers — symptom to root cause to fix, across parsing, chunking, embedding, retrieval, reranking, and generation