Orchestration & Scaling
Kubernetes basics, HPA and KEDA in depth, how batching interacts with autoscaling, and why GPU workloads scale differently
Orchestration & Scaling
TL;DR
One container is a demo; production needs many, started, stopped, and scaled automatically as load changes, with failed instances replaced without anyone paging on-call at 3am. Kubernetes is the standard orchestrator, HPA scales on resource or custom metrics, and KEDA extends that to event-driven signals HPA's defaults don't cover — but GPU-backed services need a fundamentally more conservative scaling policy than stateless CPU ones.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~19 minutes |
| Prerequisites | Containerization & Packaging |
| You will understand | The core Kubernetes concepts needed for ML serving, how autoscaling actually decides to scale, and why GPU workloads need different treatment |
Why Orchestration Exists
A single running container has no story for what happens when it crashes, when traffic doubles, or when you need to roll out a new version without downtime. Orchestration automates exactly those three things: keeping the desired number of healthy instances running, adjusting that number to match load, and replacing anything that fails — without a human doing it by hand.
Kubernetes, Just Enough
| Concept | What it is |
|---|---|
| Pod | The smallest deployable unit — one or more containers running together, sharing network and storage |
| Deployment | A declaration of "I want N replicas of this Pod running," which Kubernetes continuously reconciles toward |
| Service | A stable network identity/load balancer in front of a Deployment's Pods, so callers don't need to track individual Pod IPs as they come and go |
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-service
spec:
replicas: 3
template:
spec:
containers:
- name: api
image: my-model-service:1.4.2
resources:
requests: { cpu: "500m", memory: "1Gi" }
limits: { cpu: "1", memory: "2Gi" }This is the object autoscaling actually adjusts — when a Horizontal Pod Autoscaler decides to scale, it's changing the replicas field on exactly this Deployment.
HPA: The Decision Loop
The Horizontal Pod Autoscaler watches a target metric (commonly CPU or memory utilization, or a custom metric via an adapter) and adjusts replica count to keep it near a target value.
One HPA evaluation cycle
Sample the target metric
e.g. average CPU utilization across all current Pods, on a regular interval
Compare to the target
Desired replicas = current replicas × (current metric / target metric), rounded up
Apply cooldown
Wait out a stabilization window before scaling down, to avoid reacting to a brief dip
Adjust replica count
Update the Deployment's replicas field; new Pods start or excess Pods terminate
Aggressive scale-down policies cause thrashing. Scaling down the instant load dips, then scaling back up seconds later when it returns, wastes the time and resources spent starting new Pods repeatedly — and for a service with any meaningful cold-start cost, actively hurts latency for whoever's request lands during that gap. A cooldown/stabilization window exists specifically to smooth over brief, noisy dips instead of reacting to every one.
KEDA: Scaling on What HPA's Defaults Don't See
HPA's built-in metrics (CPU, memory) are a poor proxy for load in a lot of ML serving shapes — a queue-based inference service can have near-idle CPU while its queue backs up for minutes, because the bottleneck is downstream (a slow model call), not the CPU handling incoming requests.
KEDA scales on external, event-driven signals instead: queue depth, request rate from a message broker, or any custom metric source.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: inference-worker-scaler
spec:
scaleTargetRef:
name: inference-worker
triggers:
- type: redis
metadata:
listName: inference-jobs
listLength: "10" # scale up when the queue exceeds 10 pending jobsThe concrete case where this matters most: a queue-based async inference service (see Model Serving Patterns) where CPU usage stays flat because workers are mostly waiting on a slow model call, not computing. Scaling on queue depth catches the real bottleneck; scaling on CPU would never trigger at all.
Batching Changes the Scaling Picture
A service that batches requests (see Model Serving Patterns) has spikier, less predictable resource utilization than one handling requests one at a time — CPU/GPU usage jumps when a batch closes and runs, then drops until the next one, rather than tracking incoming request rate smoothly. Setting a scaling threshold as if the service had smooth utilization can produce a policy that reacts to batch-timing noise rather than genuine load changes — worth accounting for when choosing the target metric and stabilization window for a batching service specifically.
GPU Workloads Scale Differently
Aggressive scale-to-zero is a real latency risk for GPU-backed, latency-sensitive services. A fresh GPU Pod has to be scheduled onto a node with an available GPU, start the container, and load model weights — often tens of seconds before it can serve its first request. A CPU-style policy that scales down to zero replicas during quiet periods means the next request pays that entire cold-start cost.
| Mitigation | Effect |
|---|---|
| Minimum warm replica count | Never scale below N replicas, even at zero measured traffic, so there's always at least one instance ready |
| Pre-warming on a schedule | Scale up ahead of a known traffic pattern (e.g. business hours) rather than reactively |
| Smaller/faster-loading model formats | Reduces the cold-start penalty itself, complementing (not replacing) a warm-replica floor |
Concept Checks
Check yourself
Why does HPA apply a cooldown before scaling down, but typically scale up more aggressively?
Because scaling down too eagerly in response to a brief dip causes thrashing — Pods terminate, then new ones have to start again seconds later when load returns, wasting the cost of both. Scaling up quickly protects against under-provisioning during real load, which is the more costly failure (dropped or slow requests), so the asymmetry is deliberate: be cautious about removing capacity, be quick about adding it.
A queue-based inference worker shows flat, low CPU usage even when its job queue is backing up for minutes. Why won't HPA's default CPU-based scaling catch this, and what would?
Because the workers are spending most of their time waiting on a slow downstream model call, not computing — CPU utilization simply isn't a proxy for the actual bottleneck in this shape of service. KEDA scaling on queue depth directly targets the real signal (how much work is backed up) instead of a metric that happens not to move when the real problem is occurring.
Why is scale-to-zero a bigger risk for a GPU-backed service than a stateless CPU one?
Because a fresh GPU Pod has a much larger cold-start cost — scheduling onto a GPU-available node, starting the container, and loading model weights can take tens of seconds, compared to a CPU service that typically starts in a couple of seconds. Scaling to zero means the very next request after a quiet period pays that entire cold-start penalty, which is why a minimum warm-replica floor is the standard mitigation for latency-sensitive GPU services specifically.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Pod / Deployment / Service | The unit that runs, the declaration of how many should run, and the stable network front for them |
| HPA | Scales replica count based on a target metric, with a cooldown to prevent thrashing |
| KEDA | Scales on event-driven signals (queue depth, request rate) that CPU/memory metrics miss |
| Batching's effect on scaling | Spikier utilization patterns need thresholds tuned for that shape, not smooth-traffic assumptions |
| GPU cold start | Tens of seconds to schedule, start, and load weights — much larger than a CPU service's |
| Warm replica floor | The standard mitigation against scale-to-zero's cold-start cost for latency-sensitive GPU services |
Next
With the service scaling reliably, the next question is how a new version of the model gets there safely: CI/CD for ML.