RL Systems Mind the Gap: Matching Trainer and Generator Throughput
Kimbo Chen
· SemiAnalysis
· June 16, 2026 at 17:32
· ⏱ 32 min read
| Read on Substack ↗
Summary
RL training efficiency is governed by matching inference (generator) and training (trainer) throughput; async techniques like PipelineRL introduce policy staleness as a trade-off. Experiments show generation-bound systems (trainer idling) dominate, making inference efficiency and sandbox scalability critical. The article's TCO analysis implies hosted solutions like Tinker (private) are more cost efficient than open-source frameworks.
•Dario Amodei described RL performance scaling log-linearly with training duration, similar to pre-training scaling.
•Claude Opus 4.8 scores 69.2% on SWE-bench Pro and 74.6% on Terminal-Bench 2.1, driven by RL training.
•GRPO (Group-Relative Policy Optimization) is the open-source standard; it samples multiple completions per prompt and reinforces rollouts above group average reward.
•PipelineRL enables in-flight weight updates to overlap trainer and generator, but introduces policy staleness capped by a budget (e.g., 16 steps in experiments).
•In a Qwen3-235B case study, the trainer consumed 2.75 sample/s with 30% idle time, while the generator delivered 1.95 sample/s on 3x the compute.
•In a GLM-5 case study, 55% of problems had 100% solve rate (zero advantage), trainer waited 74% of wall-clock time, and average tool calls tripled from 20 to 51.
•Sandbox scaling at 960 concurrent rollouts caused initialization dead errors and 1-hour spin-up latency, forcing a reduction to 96.
•slime's partial rollout saves straggler rollouts to a replay buffer, but creates environment state-level policy staleness that can corrupt training signals.