A/B Testing & Experimentation
Why deploying a new model isn't proof it's better, traffic splitting mechanics, statistical significance, and when a bandit beats a fixed split
A/B Testing & Experimentation
TL;DR
Deploying a new model is not the same as proving it's better — that requires a controlled comparison against real traffic, with enough sample size that the result isn't just noise. A fixed A/B split gives a clean, interpretable answer; a multi-armed bandit adapts traffic toward the better arm as evidence accumulates, trading interpretability for lower cost while the test runs.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | Caching & Cost Optimization |
| You will understand | How to design a real experiment, read its result honestly, and choose between a fixed split and a bandit |
Why a Deploy Isn't Proof
A new model can look better in offline evaluation and still underperform in production — the held-out set never perfectly represents live traffic, and some regressions only show up under real usage patterns. The only way to know for certain is a controlled comparison where both versions see genuinely comparable traffic at the same time.
Traffic Splitting Mechanics
import hashlib
def assign_variant(user_id: str, split: float = 0.5) -> str:
digest = hashlib.sha256(user_id.encode()).hexdigest()
bucket = int(digest, 16) % 100
return "treatment" if bucket < split * 100 else "control"Hashing the user ID (rather than assigning randomly on every request) gives consistent assignment — the same user always lands in the same variant for the duration of the test.
Inconsistent assignment corrupts the measurement, not just the user experience. If a user flips between variants across requests, their behavior can't be cleanly attributed to either one, and effects that only appear over a session (trust building, task completion across multiple turns) get diluted or erased in the data. Hash-based assignment on a stable identifier is what keeps the comparison clean.
Statistical Significance, Practically
The core question: is the observed difference between control and treatment a real effect, or could it plausibly be noise from a small sample?
| Term | Working definition |
|---|---|
| Null hypothesis | The assumption that there's actually no real difference between the two variants |
| p-value | Roughly, how surprising the observed data would be if the null hypothesis were true — a small p-value (conventionally < 0.05) is evidence against "no real difference" |
| Sample size / test duration | How much data is needed before a result is trustworthy — too little, and even a real effect is indistinguishable from noise; too little the other way, and even no effect can produce an impressive-looking but meaningless difference |
A small sample can show a large, completely meaningless difference. Ten users in each arm, with treatment scoring 70% and control 50%, sounds like a strong result — but with that few observations, a couple of unrelated outliers can produce exactly that gap by chance. The fix isn't a smarter interpretation of a small sample; it's committing to a sample size or duration before looking at results, based on the smallest effect size that would actually matter for the decision.
A Worked Test Design
Experiment: model-v12 (treatment) vs model-v11 (control)
Metric: ticket resolution accuracy (human-reviewed, weekly sample)
Split: 50/50, consistent per-user assignment
Minimum sample: 2,000 tickets per arm (chosen before starting,
based on the smallest accuracy improvement worth shipping)
Stopping rule: analyze only after the minimum sample is reached —
not continuously, to avoid stopping early on a lucky streakDeciding the sample size, the metric, and the stopping rule before the test starts is what keeps the result honest — choosing them after looking at early data is how teams talk themselves into a result that isn't really there.
Multi-Armed Bandits: The Adaptive Alternative
Fixed A/B split vs bandit
Fixed A/B split
RecommendedClean, interpretable, easy to explain in a report — "we ran a controlled test, here's the result, here's the confidence." The right choice when the test's purpose is a clear causal answer for a decision, not just live optimization.
Multi-armed bandit
Shifts traffic toward the better-performing arm as evidence accumulates during the test itself, rather than waiting for a fixed sample before acting. Reduces the cost of running the underperforming arm at full volume for the whole test — appropriate when the cost of exposure to a bad variant is high and a crisp causal report isn't the actual goal.
A bandit still has to send some traffic to the apparently-losing arm — that's the exploration/exploitation tradeoff. Fully committing to the current leader too early risks missing that the "losing" arm was actually better and just unlucky early on. A bandit balances exploiting the current best guess against continuing to explore the alternative enough to be confident in that guess.
Concept Checks
Check yourself
Why does inconsistent per-request variant assignment corrupt an experiment's result, not just annoy users?
Because effects that build over a session — trust, task completion across multiple interactions — can't be cleanly attributed to either variant if a single user experiences both during the test. The measurement itself becomes contaminated, since the outcome data no longer cleanly reflects "what happens when a user consistently gets variant A" versus "variant B."
A test with 10 users per arm shows treatment at 70% and control at 50%. Why is this weak evidence despite the large-looking gap?
With such a small sample, a couple of unrelated outlier results can produce a gap that size purely by chance — the observed difference is well within what noise alone could plausibly produce. The fix is deciding an adequate sample size before starting, based on the smallest effect size that would actually matter, rather than trusting an impressive-looking gap from too few observations.
When would a fixed A/B split be preferable to a multi-armed bandit, even though the bandit reduces exposure to a worse variant?
When the goal is a clean, interpretable causal answer for a decision or report — a fixed split gives a straightforward "we tested this, here's the confidence in the result" that's easy to communicate and defend. A bandit optimizes in-flight performance during the test itself, which is valuable when exposure cost is high, but it produces a messier basis for a crisp causal claim than a pre-registered fixed split does.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| A deploy isn't proof | Offline evaluation doesn't guarantee production performance; a controlled test does |
| Consistent assignment | Hash on a stable ID so the same user always sees the same variant |
| Null hypothesis / p-value | The working vocabulary for "is this difference real or noise" |
| Decide sample size and stopping rule upfront | Deciding after looking at data invalidates the test |
| Fixed split | Clean, interpretable causal answer for a decision |
| Multi-armed bandit | Adapts traffic toward the better arm as evidence accumulates, trading interpretability for lower exposure cost |
Next
With serving, monitoring, drift, cost, and experimentation covered for models generally, the next page turns to the operational problems specific to LLMs: LLM Gateway & Routing.
Caching & Cost Optimization
Exact-match and semantic caching mechanics, why threshold tuning is the whole game, and what to actually track to catch a cost regression
LLM Gateway & Routing
Why LLM systems need a centralized operational layer, and how routing, reliability, and observability work inside one