Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple: training loss decreases predictably as we scale up model size, dataset size, and compute, following a power-law curve. This predictability makes scaling laws highly valuable in practice, allowing researchers to fit them on small runs and extrapolate to estimate the requirements for much larger models.
The Early Days
Long before scaling laws were a mainstream concept, researchers were investigating the predictability of generalization error with scale. Amari, Fujita, and Shinomoto (1992) derived four types of learning curves using a Bayesian approach. Later, Hestness et al. (2017) provided one of the earliest empirical studies, showing that generalization error scales as a power law across factors like data size. They made a key observation: while model improvements shift the error curve, they don't seem to affect the power-law exponent. The slope of the power law appears to be a property of the problem domain, not the model architecture.
Rosenfeld et al. (2020) pushed this further by modeling error as a joint function of both model size and data size across diverse architectures and optimizers. They built prediction models in the form of simple parametric functions that could extrapolate to larger regimes.
The Kaplan Era
Kaplan et al. (2020) popularized scaling laws in the language modeling community. They found that cross-entropy test loss scales as a power law with model size, dataset size, and training compute across many orders of magnitude. Key findings included:
- Larger models are more sample-efficient.
- For a fixed compute budget, it's more efficient to train a very large model and stop early than to train a smaller model to convergence.
This last point was the most influential and, in hindsight, the most contested. Kaplan et al. concluded that model size should grow faster than dataset size: for a 10x increase in compute, they suggested scaling the model size by ~5.5x but training tokens by only ~1.8x.
The Chinchilla Correction
The Chinchilla paper (Hoffmann et al., 2022) arrived at a different answer[reference:12]. Through a more careful experimental design scanning over 400 models, they found that the optimal model size and training tokens should scale roughly in tandem. Instead of "grow the model faster than the data," Chinchilla showed that for every doubling of model size, you should also double the number of training tokens.
The claim that most large models at the time were undertrained was supported by a famous demonstration: under the same compute budget as Gopher (a 280B model trained on 300B tokens), they trained Chinchilla (a 70B model trained on 1.4T tokens), which outperformed Gopher across the board.
Why the Disagreement?
Why did these two seminal papers disagree so much? Kaplan et al. experimented mostly on smaller models, while Chinchilla's experiments reached more than 10x larger scales. When extrapolating in log-log space, a small difference in the fit can result in large differences.
Another factor is parameter counting. In the small-parameter regime, embedding parameters are a non-negligible fraction of the total, and counting them or not matters. Follow-up analysis showed that as model size gets larger, the Kaplan estimate converges to the Chinchilla estimate.
The Data Wall
Classic scaling laws assume effectively infinite, unique data with no repetition. As model sizes grow significantly, we are running out of enough high-quality unique tokens. Some arguments about how long scaling in AI can continue are centered on whether we are hitting a "data wall."
Muennighoff et al. (2023) studied the impact of data repetition across roughly 400 experiments. They modeled how a token's value decays as it is repeated and found that we should allocate more resources to more epochs rather than more model parameters when data is constrained. More recently, Lovelace et al. (2026) introduced an explicit overfitting penalty term that grows with both the number of repetitions and how over-parameterized the model is relative to the unique data available.
The Tricky Art of Fitting
Despite its clean form, scaling law fitting can be surprisingly sensitive to seemingly trivial procedural choices, like how you count parameters or round precision. The disagreement between Kaplan and Chinchilla is one example. Another is a follow-up analysis showing that Chinchilla's method 3 results were slightly off due to issues like a high loss scale in the minimizer and rounding of reported exponents.
The lesson is clear: scaling laws are a powerful tool, but they must be used carefully. They are only as good as the fits they're based on, and the assumptions underlying them—like a fixed architecture, optimizer, and data mix—must hold. In a world where the only constant is change, that's a big if.