Scaling laws are the bedrock of modern deep learning. They describe how training loss decreases predictably with model size, dataset size, and compute. The observation is simple: it follows a power-law curve, appearing as a straight line on a log-log plot.
This predictability makes them invaluable. Labs use them to extrapolate from small runs to estimate the requirements for billion-dollar training runs. But as the famous disagreement between Kaplan et al. and the Chinchilla paper shows, a small difference in the fit can change the recommendation dramatically.
The Great Scaling Debate
In 2020, Kaplan et al. suggested that for a 10x increase in compute, you should scale the model size by ~5.5x and the data by ~1.8x. The Chinchilla paper (2022) overturned this. They argued that for every doubling of model size, you should also double the data. This disagreement was not due to physics, but due to methodology.
Kaplan et al. counted non-embedding parameters in their fit. Chinchilla counted total parameters. When you account for embedding parameters, the gap narrows. More importantly, Kaplan extrapolated from a regime where embeddings dominated, leading to a steeper exponent.
The Sensitivity of Fits
Fitting a scaling law is fragile. The number of decimal places you round a loss value to matters. The optimizer used for the fit matters (Chinchilla Method 3 used L-BFGS-B and had a bug that led to premature termination). Even the range of data you choose to fit matters, as the exponent can shift.
Data Repetition
The classic scaling laws assume unique tokens. We are now hitting a "data wall". Researchers like Muennighoff et al. have studied data-constrained scaling. They found that repeated tokens lose value, which requires an adjustment to the effective data size.
Conclusion
Scaling laws are a guide, not a physical law. They are powerful but brittle. A careful practitioner must be wary of the assumptions in their fit and the limitations of their data.