Scaling laws are the bedrock of modern deep learning. The observation is deceptively simple: a model's training loss decreases predictably along a power-law curve as we scale up model size, dataset size, and compute. This empirical finding has become the guiding principle for allocating hundreds of millions of dollars in AI research. Yet, the history of scaling laws is a story of valuable insights, sharp disagreements, and surprising sensitivity to seemingly trivial details.
The foundational work established that generalization error decays as a power law across various domains, from image classification to language modeling. The early findings by Hestness et al. and Rosenfeld et al. showed that model improvements shift the error curve (offset) but do not seem to change the slope (exponent), which appeared to be a property of the problem domain itself.
The Kaplan-Chinchilla Divide: A Tale of Two Scaling Regimes
The debate that truly popularized scaling laws was the disagreement between the 2020 Kaplan et al. paper from OpenAI and the 2022 Chinchilla paper from DeepMind. Both used the same underlying power-law framework to answer the same question: given a fixed compute budget, how should we allocate resources between model size and training data?
Kaplan et al. concluded that model size should grow much faster than dataset size, recommending a 5.5x increase in parameters for every 10x increase in compute. Their logic suggested it was more efficient to train a very large model and stop before convergence than to train a smaller model to full convergence. The Chinchilla paper, using a much larger experiment spanning over 400 models, arrived at a very different answer. They found the optimal scaling relationship is roughly equal: for every doubling of model size, you should also double the training tokens. This led to the famous demonstration of the Chinchilla model, which was four times smaller than Gopher but trained on four times more data, outperforming its larger counterpart.
Why Did They Disagree So Much?
The disagreement didn't stem from a fundamental difference in theory but from fitting nuance. Kaplan et al. extrapolated from models up to 1.5B parameters, while the Chinchilla experiments reached beyond 16B. In the small-parameter regime, the number of embedding parameters is a non-negligible fraction of the total, affecting how you count model size. A subsequent analysis showed that the Kaplan coefficient (0.73) and the Chinchilla coefficient (0.5) are not as contradictory as they seem. The effective exponent changes as you move to larger scales, and the Kaplan fit turns out to be a local approximation that converges to the Chinchilla estimate as models grow larger.
The Theoretical Mystery Behind the Power Law
The empirical power law is well-established, but its theoretical underpinnings are still debated. One early hypothesis posits that language modeling is equivalent to regression on a low-dimensional data manifold. More parameters allow for a finer partition of this manifold, reducing error. A more recent "quantized knowledge" theory suggests that knowledge is learned in discrete chunks whose frequency distribution follows a power law. The model learns common skills first and rare skills later, resulting in the smooth decay.
Data-Constrained Scaling: When the Well Runs Dry
Classic scaling laws assume an effectively infinite supply of unique, high-quality tokens. As model sizes skyrocket, the industry is hitting a "data wall," and training runs now rely on multiple epochs. This dramatically complicates the simple power-law relationship.
Research has shown that repeating data is not a simple zero-sum trade-off. Hernandez et al. observed a double-descent phenomenon where test loss can worsen and then improve as the fraction of repeated data grows. The Muennighoff et al. study on data-constrained scaling introduced the concept of "effective data," where each repetition decays the token's value by a fraction, a "half-life" parameter. Their fit suggests that with data constraints, it is actually more efficient to allocate resources to more epochs rather than more model parameters.
The Fragility of the Fit: A Call for Caution
Despite its profound importance, scaling law fitting is surprisingly fragile. A follow-up analysis re-ran the Chinchilla method 3 and found that issues like rounding loss values and improper loss scaling in the optimizer could lead to significantly different estimates. A toy simulation demonstrates that rounding loss precision, adding milli-level noise, or changing the region of the data used for the fit can all change the apparent scaling exponent.
This sensitivity is a critical cautionary tale. The scaling laws are an immensely powerful tool, but they are an empirical approximation. The best strategy for allocating compute is still a matter of careful experimental design and constant re-evaluation. The true lesson of the scaling law debate is not the perfect exponent, but the importance of rigorous, reproducible, and thoughtfully interpreted empirical science in AI development.