The data black hole at the center of AI

Summary

Dwarkesh Patel argues that AI progress has come mainly from massive expansion of training data and compute rather than from improved sample efficiency. He compares human and model sample efficiency, showing models require far more tokens and demonstrations to learn skills. He highlights the booming market for expert data labeling and RL environments, the conditional multi-trillion-dollar robotics opportunity if sample efficiency improves, and implications for open-source catch-up, self-driving, and white-collar automation.

• AI progress is driven by more and better data plus scaling compute, not by large gains in sample efficiency.
• The human vs AI sample-efficiency gap is roughly a millionfold or more.
• Expert data labeling and RL environments are a billions-to-deca-billions revenue industry.
• Open models lag frontier models by about four months because data can be distilled from public APIs.
• Human-level sample efficiency would be transformational for robotics, such as Unitree humanoids.
• Scaling laws imply larger models cannot close the sample-efficiency gap.
• Labs are monetizing common white-collar tasks despite inefficient training.