ISLAMABAD — The math behind deep learning’s biggest successes may have been hiding in plain sight since 1992. A new paper published this week traces the roots of modern scaling laws — the rules that let engineers predict how larger AI models will behave — back to work by Amari and colleagues more than three decades ago. That early research derived four types of learning curves using a Bayesian approach and an approximation method called the annealed approximation.
The setup was simple: a binary classification task. Nothing like today’s billion-parameter transformers.
Yet the paper, titled “Scaling Laws, Carefully,” argues that early work pointed in a useful direction for building empirical loss prediction models. The idea that generalization error could be predicted with scale existed before scaling laws became a mainstream concept. That matters because scaling laws are now among the most critical empirical findings in deep learning.
The central fact is straightforward. Training loss decreases predictably as model size, dataset size, and compute power increase.
That relationship follows a power-law curve. On a log-log plot, it looks like a straight line. Engineers exploit this every day.
A common workflow, the paper explains, is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models. That predictability is highly valuable in practice. It saves money.
It saves time. It tells a team whether a planned model is feasible before they spend millions of dollars on training.
The paper covers several key areas. It discusses the early days of ML loss predictability. It examines scaling laws in the data-infinite region.
It reviews the work of Kaplan and colleagues, which became a foundational reference in the field. It also looks at the Chinchilla scaling laws, which offer methods for fixing model sizes and varying token budgets.
Those methods use IsoFLOP profiles and parametric fitting. Chinchilla is notable because it challenged the assumption that bigger models always need more data. Instead, it showed that for a fixed compute budget, there is an optimal ratio of model size to training tokens.
That insight reshaped how labs allocate resources. The paper does not present new experiments. It synthesizes.
It organizes. It offers a framework for understanding the relationship between model size, data, and computing power.
That framework is badly needed. The field moves fast. Papers pile up.
Results get cited without context. “Scaling Laws, Carefully” tries to slow things down and get the story straight.
The author reviews a body of work that suggests training loss decreases predictably. Not perfectly. Predictably.
That distinction matters. Power laws are not guarantees. They are empirical regularities.
They hold across many architectures and datasets, but not every deviation is noise. Sometimes a model underperforms because the scaling law breaks.
Sometimes it breaks because the training setup is wrong. The paper does not resolve that ambiguity. It maps the terrain.
One consequence is clear: the field has moved from discovery to engineering. Scaling laws were once a curiosity.
Now they are a tool. Teams run small experiments, fit a curve, and bet millions on the extrapolation. That works.
It also carries risk. Extrapolation is only as good as the model behind it. The paper acknowledges this implicitly by emphasizing the word “carefully” in its title.
The early work from 1992 is a reminder that the problem is older than the current boom. Amari and colleagues derived four types of learning curves using a Bayesian approach.
Their theoretical setup was simplified. But it pointed in a useful direction. The new paper suggests that direction was correct.
The math was ahead of the hardware. Now the hardware has caught up.
{“@context”: “https://schema.org”, “@type”: “NewsArticle”, “headline”: “Researchers Trace Modern AI Scaling Laws to 1992 Math Paper”, “description”: “ISLAMABAD — The math behind deep learning’s biggest successes may have been hiding in plain sight since 1992. A new paper published this week traces…”, “datePublished”: “2026-06-28T14:13:00+00:00T09:30:00+00:00”, “author”: {“@type”: “Person”, “name”: “Li Wei”}, “publisher”: {“@type”: “Organization”, “name”: “InfoPulse Today”, “url”: “https://newsfeed.infopulsetoday.com/”}}
