Latency Budgets for Real-Time ML Serving
Model quality gets the demos. Serving latency gets the pager at 2 a.m.
When inference sits on a user-facing path — search ranking, fraud scoring, live sentiment — the system isn't judged by val accuracy alone. It's judged by whether p99 stays inside a budget while throughput climbs.
Start from the SLO, not the GPU
A useful serving design begins backwards:
- What is the p99 latency users or downstream services can tolerate?
- What QPS must we sustain at that tail?
- How much batching delay can we afford to amortize compute?
Everything else — model format, instance type, queue depth — follows from those three numbers.
Where milliseconds disappear
| Stage | Typical culprit |
|---|---|
| Preprocessing | Tokenization, feature lookups, serialization |
| Queue wait | Under-provisioned workers, bursty traffic |
| Inference | Cold starts, oversized models, sync GPU calls |
| Postprocessing | JSON assembly, logging, synchronous DB writes |
The model forward pass is often not the dominant cost. I've seen pipelines where 60% of tail latency was queueing and preprocessing — fixable without touching weights.
Batching is a product decision
Dynamic batching improves throughput but adds delay variance. For a chat UI, waiting 40ms to fill a batch might be fine. For a fraud check at checkout, it might not.
# Illustrative: cap batch wait so tail latency stays bounded
max_batch_wait_ms = 10
max_batch_size = 32
The right knobs depend on whether you're optimizing for throughput per dollar or worst-case user wait — and those goals conflict more often than teams admit.
Observability that actually helps
Export per-stage latency histograms, not just a single "inference time" metric. When p99 spikes, you need to know if it's the model, the queue, or the feature store — without redeploying with printf debugging.
The takeaway
ML systems engineering is the work of making a model reliably fast under load, not just correct on a static eval set. Latency budgets are how you make that work legible to the rest of the team — and to future you, when traffic doubles overnight.