ISLAMABAD, June 28 — A new research paper offers a careful look at how deep learning models improve as they get larger, a phenomenon known as scaling laws. The findings, published online this week, provide a framework for understanding the relationship between model size, data, and computing power. The paper, titled “Scaling Laws, Carefully,” reviews a body of work that suggests training loss decreases predictably as model size, dataset size, and compute power increase.
This relationship follows what researchers call a power-law curve, which appears as a straight line on a log-log plot. The author notes that scaling laws are among the most critical empirical findings in deep learning.
The predictability they offer is highly valuable in practice. A common workflow, the paper explains, is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models. This is not a new idea.
The paper points out that the predictability of generalization error with scale had already been investigated before scaling laws became a mainstream concept. Early work by Amari and colleagues in 1992 derived four types of learning curves using a Bayesian approach and the annealed approximation.
Although their theoretical setup was based on a simplified binary classification task, the paper suggests it pointed in a useful direction for building empirical ML loss prediction models. The paper covers several key areas. It discusses early days of ML loss predictability, scaling laws in the data-infinite region, and the work of Kaplan and colleagues.
It also examines the Chinchilla scaling laws, which offer methods for fixing model sizes and varying token budgets, using IsoFLOP profiles, and parametric fitting. One section of the paper is dedicated to reconciling the differences between the Kaplan and Chinchilla findings. The author also explores why the power law holds and examines scaling laws in the data-limited region.
The paper does not shy away from the trickiness of fitting scaling laws in reality, and includes a toy simulation to illustrate the challenges. For researchers and practitioners working on large language models and other deep learning systems, these findings offer a practical guide.
The encouraging part is that these scaling relationships provide a way to plan experiments and allocate computing resources more efficiently.
