Caching & Cost Optimization
Exact-match and semantic caching mechanics, why threshold tuning is the whole game, and what to actually track to catch a cost regression
Caching & Cost Optimization
TL;DR
Exact-match caching skips the model entirely for identical repeated requests; semantic caching extends that to requests that mean the same thing even when worded differently — but only if the similarity threshold is tuned against real near-miss examples, not guessed. Tracking cost per request and cache hit rate together is what turns "costs went up" into "here's specifically why."
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~17 minutes |
| Prerequisites | Data & Model Drift |
| You will understand | How to build both caching layers correctly, and what to monitor to catch a cost regression early |
Exact-Match Caching
The simplest layer: hash the request, check whether that exact hash has been served before.
import hashlib, json
def cache_key(request: dict, model_version: str) -> str:
payload = json.dumps({"request": request, "model_version": model_version}, sort_keys=True)
return hashlib.sha256(payload.encode()).hexdigest()The model version belongs in the cache key, not just the request content. Without it, deploying a new model version can silently keep serving stale, previous-model-version responses from cache — a cache hit that should have been a miss, because the thing that changed (the model) isn't part of what the key represents. Anything that should invalidate a cached response belongs in the key.
Semantic Caching
Semantic caching extends this to requests that are worded differently but mean the same thing: embed the incoming request, search cached requests for one above a similarity threshold, and serve its cached response if found.
def get_cached_response(query: str, threshold: float = 0.92):
query_embedding = embed(query)
match, score = vector_index.search_nearest(query_embedding)
if match and score >= threshold:
return match.cached_response
return NoneThe threshold is the entire product, and it needs real test cases, not a guessed constant. "What's the refund window?" and "what's the return window?" should share a cache entry — genuinely the same question, different wording. "What's the refund window?" and "how do I cancel my refund request?" should not — superficially similar vocabulary, completely different intent, and a shared cache entry here means confidently serving the wrong answer.
Build the threshold decision as an actual test set, not a single number chosen by feel:
should_match = [
("What's the refund window?", "What's the return window?"),
("How do I reset my password?", "I forgot my password"),
]
should_not_match = [
("What's the refund window?", "How do I cancel my refund request?"),
("Where's my order?", "Where's my refund?"),
]
# Sweep candidate thresholds against both lists; pick the value that
# keeps every should_not_match pair below it and every should_match pair above it.A threshold that's too loose returns confidently wrong cached answers — worse than a cache miss, because a miss just costs a normal model call, while a false cache hit actively serves misinformation with full apparent confidence.
Routing to a Cheaper Model
The third major cost lever — sending easy requests to a small, cheap model and reserving an expensive one for genuinely hard requests — is covered in depth in the Small Language Models track's production guidance, including the escalation-trigger design and why tracking the escalation rate matters. This page treats it briefly and focuses on the observability side below, since that's what makes routing's savings verifiable rather than assumed.
What to Actually Track
| Metric | Why it matters |
|---|---|
| Cost per request | The headline number, but only useful broken down further |
| Cache hit rate | Falling hit rate with stable traffic is an early warning something changed |
| Cost by route/model | Separates "the cheap path got more expensive" from "more traffic shifted to the expensive path" |
A rising cost trend combined with a falling cache-hit-rate is a specific, diagnosable signal, not just "costs went up." It usually means one of two things: the prompt template grew (making exact-match hits less likely even for logically identical requests), or the traffic mix shifted toward genuinely novel queries (fewer questions repeat, so there's less for a cache to catch). Tracking hit rate alongside cost turns a vague budget alarm into an actual lead on what changed.
Concept Checks
Check yourself
Why does the model version need to be part of the exact-match cache key, not just the request content?
Because a model version change is exactly the kind of thing that should invalidate a cached response — without it in the key, a new model deployment can silently keep serving cached responses generated by the previous model version, turning what should be a cache miss into a stale hit that never gets corrected.
Why is a false semantic-cache hit worse than a cache miss?
A miss just costs one ordinary model call — no different from having no cache at all for that request. A false hit actively serves a wrong cached answer with full apparent confidence, for a query the cache incorrectly judged similar enough — which is a worse outcome than paying for a fresh model call, not just an equally bad one.
What does a rising cost trend combined with a falling cache-hit-rate suggest, that a rising cost trend alone doesn't?
It points toward two specific, checkable causes: the prompt template got longer or changed (making previously-identical requests no longer hash-match), or the traffic mix shifted toward more novel, less-repeated queries. Cost alone just says "spending increased" — pairing it with hit rate gives an actual lead on which upstream change to investigate, rather than starting from nothing.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Exact-match caching | Hash the request (including model version) and check for a prior identical hit |
| Semantic caching | Embed and search for a similar past request above a similarity threshold |
| Threshold tuning | Needs real should-match / should-not-match test pairs, not a guessed constant |
| False hits are worse than misses | A miss costs a call; a false hit serves a confidently wrong cached answer |
| Routing to a cheaper model | Covered in depth in the SLM track's production guidance |
| Cost + hit-rate together | Diagnoses why cost changed, not just that it did |
Next
Caching and routing change behavior — the next page covers how to prove a change like that actually helped, rather than assuming it did: A/B Testing & Experimentation.
Data & Model Drift
Data drift vs concept drift, how each is actually detected, and designing a retraining trigger that responds to the right signal
A/B Testing & Experimentation
Why deploying a new model isn't proof it's better, traffic splitting mechanics, statistical significance, and when a bandit beats a fixed split