All essays
#ML-Systems

Why ML Pipelines Belong on the Stream

February 20, 202612 min readBy Caden Chuang

Most ML tutorials end at training. Production starts when data never stops moving — news feeds, clickstreams, sensor ticks — and the model is one stage in a longer pipe.

Batch notebooks don't survive that world. Streaming infrastructure does, not because streams are fashionable, but because backpressure and replay are first-class problems in real systems.

The batch/stream gap

Training wants large, shuffled, static datasets. Online systems want ordered, deduplicated, fresh events. The gap between those shapes is where most "we'll batch it nightly" plans break:

  • Duplicates inflate inference cost and skew metrics.
  • Stale features silently degrade model quality.
  • Spikes overwhelm a synchronous API with no queue to absorb them.

A stream processor — Kafka, Pulsar, whatever — gives you a durable log with consumer groups, which is the closest thing we have to a universal integration layer for ML infra.

Split compute by language, join by topic

A pattern I keep returning to:

  • Go (or Rust) for I/O-bound plumbing: fetch, dedup, normalize, fan-out. Cheap concurrency, predictable memory.
  • Python for model inference: ONNX, PyTorch, tokenizers — ecosystems that already exist.

They should not call each other directly. They meet at a topic contract:

raw events → processing stage → model-ready events → inference workers → storage / alerts

The schema between stages is the API. Version it like you would a REST endpoint.

Two-stage filtering saves real money

Run cheap filters before the model:

  1. Dedup by content hash.
  2. Drop irrelevant categories.
  3. Only then invoke FinBERT, an LLM, or whatever is expensive.

Consumer lag per stage becomes your backpressure signal. If the inference group falls behind, you know exactly where — and you can scale workers or shed load without guessing.

Replay is underrated

Streams let you reprocess history when the model changes. That turns "we need to retrain and hope" into "we replay last week's events against v2 and compare outputs."

For ethics and safety work, replay is how you regression-test behavior on real traffic shapes — not just curated eval sets.

Closing thought

ML systems and infrastructure isn't separate from ML research. It's the substrate that decides whether your model ever runs reliably, fairly, and at a cost you can sustain. Streams aren't the only answer — but they force you to design like data never stops, which is the job.