Infrastructure · Note
Three things I learned from recent inference papers
Serving performance is mostly a story about memory movement, scheduling, and the shape of demand.
Three ideas keep recurring in the inference systems papers I have been reading.
1. Memory movement dominates
Generating one token requires repeatedly reading model weights and key-value cache state. Arithmetic is only part of the cost; moving bytes through the memory hierarchy often determines throughput.
2. Batching changes the economics
Batching requests lets one weight read serve several sequences, improving accelerator utilization. But waiting to form a batch adds latency. Systems are continuously trading response time against throughput.
3. Demand shape matters
Two workloads with the same average request rate can behave differently if one has long prompts, bursty arrivals, or unpredictable output lengths. Scheduling policy must understand the work, not just count requests.
The larger lesson is that model serving is not a smaller version of training. It is a queueing and memory-management problem with a model in the middle.