CI/CD for ML
What ordinary software CI/CD checks, what a model validation gate adds, and blue-green vs canary deploys for a model change
CI/CD for ML
TL;DR
Ordinary CI/CD validates that code works. ML CI/CD has to add a second check that has no software equivalent: does the new model actually perform as well as, or better than, the one currently in production — because a model can pass every code test and still be worse. Blue-green and canary deploys extend that discipline into the release itself, routing real traffic to the new version gradually or provably before committing fully.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~19 minutes |
| Prerequisites | Orchestration & Scaling |
| You will understand | What a model validation gate does, and how to release a model change safely |
What Ordinary CI/CD Checks, and What's Missing
Standard software CI/CD answers one question well: does this code do what it's supposed to, without breaking anything that used to work? Unit tests, integration tests, linting — all of it verifies the code. None of it can tell you whether a newly trained model is actually a good model, because a model is data-dependent in a way ordinary code isn't: it can be syntactically and structurally perfect and still have learned something worse than what it's replacing.
A model deploy that only passes code CI is not validated — it's just not obviously broken. The pipeline needs a separate, explicit check for model quality, or a regression can ship with a perfectly green build.
The Validation Gate
def validation_gate(candidate_model, production_model, eval_set):
candidate_score = evaluate(candidate_model, eval_set)
production_score = evaluate(production_model, eval_set)
if candidate_score.overall < MINIMUM_ACCEPTABLE_SCORE:
raise DeploymentBlocked(f"Candidate scored {candidate_score.overall}, below floor")
for slice_name, slice_score in candidate_score.by_slice.items():
if slice_score < production_score.by_slice[slice_name] - REGRESSION_TOLERANCE:
raise DeploymentBlocked(f"Regression on slice '{slice_name}'")
return DeploymentApproved(candidate_score)Two checks worth separating: an absolute floor (never deploy anything below a minimum acceptable score, regardless of what's currently live) and a relative regression check per slice (don't deploy something that's worse than the current model on any individual, tracked segment of the data — even if the overall average looks fine). The second check matters because an aggregate score can improve while a specific important slice quietly gets worse, exactly the kind of regression an overall-average-only gate would miss.
Blue-Green vs Canary
Two ways to release a model change safely
Blue-green — two full environments, cut over at once
Recommended"Blue" is the currently live environment; "green" is a full copy running the new model version, validated before receiving any real traffic. Once green passes validation, traffic cuts over all at once — and blue stays running, untouched, as an instant rollback target if something unexpected shows up. Best when you want a clean, fast, all-or-nothing switch with an immediate way back.
Canary — a small slice of real traffic first
A small percentage of real traffic (5%, say) routes to the new version while the rest continues to the old one; the percentage increases gradually as live metrics confirm the new version is behaving well. Best when you want to observe genuine production behavior on a limited blast radius before committing further — catches issues a static validation set might miss, since it's exposed to real, current traffic.
A realistic pipeline for a model change
Automated Rollback Triggers
Deciding what triggers an automatic revert — rather than waiting for a human to notice — is part of designing the pipeline, not an afterthought:
| Signal | Why it's a good rollback trigger |
|---|---|
| Error rate spike on the new version specifically | A clear, fast, unambiguous signal something is structurally wrong |
| Latency regression beyond a threshold | Catches performance issues a validation set (which doesn't measure production latency) can't |
| A live quality proxy dropping (e.g. user-reported thumbs-down rate, conversion rate) | The closest thing to real-time model quality, when available |
Not every signal should trigger an automatic rollback — a slow, gradual accuracy drift over weeks is a drift problem better handled by monitoring and a planned retrain, not a same-day automated revert. Reserve automated rollback for fast, unambiguous signals; route slower degradation to human review.
Concept Checks
Check yourself
A candidate model passes every code test and its overall accuracy is slightly higher than the current production model. Should it deploy automatically?
Not necessarily — an improved overall average can still hide a regression on a specific, important slice of the data. A proper validation gate checks per-slice performance against the current production model, not just the aggregate score, precisely because averages can mask a real, localized quality drop that matters just as much as an overall one.
Why might a team choose canary over blue-green for a model change, even though blue-green offers a cleaner instant rollback?
Because canary exposes the new version to genuine, current production traffic at a small, controlled scale before committing further, which can surface issues a static validation set never would — real user behavior, edge-case inputs, or interactions with other live systems. Blue-green's all-at-once cutover is faster and just as reversible, but it commits the new version to 100% of traffic the moment it switches, with no gradual, traffic-informed confidence-building step first.
Why shouldn't every quality signal trigger an automatic rollback?
Because some signals are fast and unambiguous (an error rate spike) while others are slow and require judgment (a gradual accuracy drift from drift over weeks) — automatically reverting on a slow signal can mask a problem that actually needs a proper retrain rather than a same-day revert to an older, also-imperfect model. Reserving automatic rollback for fast, clear signals and routing slower ones to human review avoids both under-reacting to real incidents and over-reacting to normal drift.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| What ML CI/CD adds | A model validation gate — code CI alone can't tell you if a model is actually better |
| Absolute floor vs relative regression | Never deploy below a minimum score, and never deploy a per-slice regression, even if the average improves |
| Blue-green | Two full environments, all-at-once cutover, instant rollback target |
| Canary | Gradual real-traffic exposure before full commitment |
| Automated rollback | Reserved for fast, unambiguous signals — not slow drift, which needs human judgment |
| The realistic pipeline | Code tests → build → validation gate → canary → monitor → rollout or rollback |
Next
With a safe release process in place, the next question is knowing exactly which model version is running at any moment: Model Registry & Versioning.
Orchestration & Scaling
Kubernetes basics, HPA and KEDA in depth, how batching interacts with autoscaling, and why GPU workloads scale differently
Model Registry & Versioning
What a model registry actually stores, how models move through stages, and why "which model is in production" must be a queryable fact