This week’s Paper Club is all about inference. For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade. We cover the full lifecycle of an inference request, from the moment it hits the API down to the kernel running on the GPU; why the gap between the best and worst provider serving the same open model is bigger than the gap between open and closed models; how a compiler built on search rather than heuristics turns GPU hours into tokens; and how SGLang manages memory for the agent era.
Chapters:
00:00 – Why inference matters more than training
08:24 – Three kinds of inference innovation
12:14 – Dan Fu: the lifecycle of an inference request
42:18 – Jeff Morgan: not all open model tokens are the same
1:04:31 – Joe Fioti: compiling models with search, not heuristics
1:30:30 – Lianmin Zheng: SGLang and memory for the agent era
Apply to Y Combinator: https://www.ycombinator.com/apply
Work at a startup: https://www.ycombinator.com/jobs
