Model Serving Patterns
REST, batching, and async serving in depth — plus the health-check mistake almost every team makes at least once
Model Serving Patterns
TL;DR
Serving a model means wrapping it in something that turns requests into predictions reliably — and the right shape (synchronous REST, batched, or async/queue-based) depends entirely on whether you're optimizing for latency or throughput. The single most common mistake: a /health endpoint that checks the process is alive but never actually confirms the model can predict anything.
| Property | Value |
|---|---|
| Level | Beginner |
| Reading time | ~18 minutes |
| Prerequisites | What Is MLOps? |
| You will understand | The main serving patterns, when to use each, and how to write a health check that actually means something |
A Real REST Service
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
model = None # loaded at startup
class PredictRequest(BaseModel):
features: list[float]
class PredictResponse(BaseModel):
prediction: float
model_version: str
@app.on_event("startup")
def load_model():
global model
model = load_model_from_registry(stage="production")
@app.post("/predict", response_model=PredictResponse)
def predict(payload: PredictRequest):
if model is None:
raise HTTPException(503, "Model not loaded")
return PredictResponse(prediction=model.predict(payload.features), model_version=model.version)Pydantic models on both request and response give you free input validation and a documented, typed contract — malformed input is rejected before it ever reaches the model, with a clear error instead of a confusing downstream exception.
The Health Check Mistake
Two health checks that look similar and aren't
Shallow: 'is the process alive'
@app.get("/health")
def health():
return {"status": "ok"}This returns ok even if model is None, even if the model failed to load at startup, even if a downstream dependency (a feature store, a vector database) is completely unreachable. An orchestrator sees a healthy service and keeps routing real traffic to it.
Real: exercises the actual dependencies
Recommended@app.get("/health")
def health():
if model is None:
raise HTTPException(503, "Model not loaded")
if not feature_store.ping():
raise HTTPException(503, "Feature store unreachable")
return {"status": "ok", "model_version": model.version}Now a failed model load or an unreachable dependency actually shows up as unhealthy — which is what lets an orchestrator stop routing traffic to this instance and restart or replace it, instead of quietly serving errors.
A health check that always returns 200 is worse than no health check at all. It gives every downstream system — load balancers, orchestrators, on-call dashboards — false confidence that everything is fine, which delays detection of a real outage rather than preventing one.
Choosing a Serving Pattern
Synchronous vs batched vs async/queue-based
Synchronous REST — one request, one prediction
RecommendedThe default. Simple, easy to reason about, low latency per request. Right choice whenever a person or another service is waiting on the response and predictions are cheap enough individually.
Batched — accumulate, then predict once
Groups several requests together (by a time window or a batch-size threshold, whichever hits first) into a single model call. Adds a small amount of latency to each individual request, in exchange for much better hardware utilization — the right trade when throughput matters more than any single request's latency, especially on a GPU.
Async / queue-based — decouple the request from the prediction
The caller submits a job and gets an ID back immediately; a worker pulls from the queue, runs (possibly slow) inference, and the result is fetched or pushed later. Right choice for inference that takes real time (seconds to minutes) and shouldn't block whoever made the request.
Batching, mechanically
Requests arrive: r1 (t=0ms) r2 (t=8ms) r3 (t=15ms) r4 (t=42ms)
Batch window: 20ms, or batch_size=8, whichever comes first
→ batch closes at t=20ms with [r1, r2, r3] (r4 starts the next batch)
→ one model call on the batch of 3
→ results split back out to each callerEvery request in that batch waited up to 20ms longer than it would have alone — the cost of batching, paid by every request, in exchange for the GPU running one larger, far more efficient call instead of three small ones.
gRPC, Briefly
REST over HTTP/JSON is the default because it's simple and universally supported. gRPC trades that simplicity for lower latency and a strongly-typed contract (via Protocol Buffers) — genuinely worth it for high-volume service-to-service calls inside your own infrastructure, where every millisecond and every byte of serialization overhead compounds across millions of calls. It's usually not worth the added tooling complexity for a public-facing API or a low-volume internal service, where REST's simplicity and ubiquity win.
Cold Start
The first request to a freshly started instance pays the cost of loading the model into memory — for a large model, this can be seconds, sometimes tens of seconds, far longer than any individual prediction should take.
| Mitigation | How it helps |
|---|---|
| Keep-warm instances | Never scale to zero; keep a minimum number of replicas always loaded and ready |
| Lighter model formats | A smaller or quantized model loads faster — see the loading-time discussion in On-Device & Edge Deployment for the same idea applied to memory-mapped formats |
| Readiness vs liveness separation | Mark an instance "not ready" until its model is fully loaded, so an orchestrator doesn't route traffic to it mid-load |
Concept Checks
Check yourself
Why is a health check that always returns 200 worse than having no health check at all?
Because it actively signals that everything is fine to every system relying on it — load balancers keep routing traffic, on-call dashboards show green — while the actual failure (a model that never loaded, an unreachable dependency) goes completely undetected. No health check at all at least leaves the question open; a fake one actively misleads.
A service needs to handle occasional inference jobs that take up to two minutes each. Which serving pattern fits, and why not synchronous REST?
Async/queue-based serving fits — the caller submits a job and polls or gets notified later, rather than holding a connection open for two minutes. Synchronous REST would mean the caller (and likely a load balancer's request timeout) is blocked for the full duration, which is exactly the failure mode queue-based serving exists to avoid for long-running inference.
Why does batching add latency to every individual request rather than only to some?
Because a request has to wait until its batch closes — either the time window elapses or the batch-size threshold is hit — before the model call happens at all, and every request in that batch is held for the same closing event. That's the batching trade made explicit: a small latency cost paid by every request, in exchange for one larger, more efficient model call instead of many small ones.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| A real health check | Confirms the model is loaded and dependencies are reachable, not just that the process is alive |
| Synchronous REST | The default — simple, low per-request latency |
| Batching | Groups requests for better throughput, at the cost of added per-request latency |
| Async/queue-based | Decouples request from response for long-running inference |
| gRPC | Lower latency, typed — worth it for high-volume internal service-to-service calls |
| Cold start | Model-loading time on a fresh instance; mitigated with keep-warm replicas and readiness checks |
Next
With a model actually serving requests, the next question is how to package it reproducibly: Containerization & Packaging.
What Is MLOps?
The gap between a working notebook and a reliable production system, how MLOps differs from DevOps and DataOps, and five concrete failure stories
Containerization & Packaging
A full worked multi-stage Dockerfile, GPU-enabled images, image size discipline, and why floating version tags are a production risk