How GPT, Claude, and Gemini are actually trained and served – Reiner Pope

Summary

Reiner Pope gives a blackboard lecture on how frontier LLMs are trained and served, using roofline math to derive latency/cost trade-offs from batch size, context length, sparsity, and memory bandwidth. He explains mixture-of-experts parallelism, rack-scale NVLink/scale-up vs scale-out networking, and why memory bandwidth is the binding constraint on long-context AI. He then uses public API prices to infer memory bottlenecks and storage tiers, and estimates models are heavily over-trained relative to Chinchilla. The discussion points to advantages for Nvidia rack-scale systems, Google's earlier scale-up lead, and persistent memory/HBM tightness.

• Roofline analysis shows batching amortizes weight loads; optimal batch size is roughly 300 × sparsity, around 2,000-3,000 tokens per forward pass.
• NVIDIA Blackwell NVL72's rack-scale all-to-all NVLink matches MoE expert parallelism and makes larger scale-up domains a major unlock.
• Google had large scale-up domains earlier, which may explain Gemini's early lead in pre-training larger models.
• Memory bandwidth, not just capacity, is the binding AI bottleneck; HBM improvement looks limited and context lengths have plateaued.
• API pricing reveals prefill is cheaper than decode due memory bandwidth bottleneck; cache pricing tiers suggest flash and spinning disk are used for KV cache storage.
• Cost equalization suggests frontier models are trained with far more tokens than Chinchilla optimal, implying heavy AI compute demand.