SAN FRANCISCO, July 28 — For the researchers and engineers who build the generative AI tools that millions now use daily, a quiet reckoning is underway. The deep learning models that power everything from chatbots to image generators have grown breathtakingly large, but the costs—financial and environmental—are starting to force a hard conversation about whether the era of simply scaling up has a future. Deep learning, a subset of machine learning built on artificial neural networks with multiple layers, is the technical backbone of today’s generative AI systems.
The breakthrough moment came in 2017, when Google researchers published the paper “Attention Is All You Need,” introducing the transformer architecture. That architecture now forms the basis of models like GPT—short for Generative Pre-trained Transformer.
OpenAI’s GPT-3, released in June 2020, represents the scale that defined the early 2020s: 175 billion parameters, trained on roughly 570 gigabytes of text data. To put the computational effort in perspective, training GPT-3 required an estimated 3.14 × 10^23 floating-point operations. The cloud compute bill came to over $4 million for a single training run.
And the energy cost was staggering. According to a 2020 study by Emma Strubell and colleagues, training a large model like GPT-3 emitted 552 metric tons of CO2 equivalent.
That is not a one-time figure: it is the carbon footprint of a single round of training, before the model ever answers a user’s query.
The Price of Progress
But training is only part of the story. Once deployed, these models continue to consume resources. Processing a single ChatGPT query costs roughly $0.01 per request.
Multiply that across millions of daily users, and the ongoing operational expense becomes enormous. The industry’s belief in ever-larger models is rooted in what OpenAI researchers called the “scaling law” hypothesis.
In a 2020 paper by Kaplan and colleagues, they proposed that model performance improves predictably as you increase parameters, data, and compute. Yet the skeptics are growing louder. Some, including Doctorow, have questioned whether scaling will continue to yield meaningful gains or whether diminishing returns will set in.
Some recent studies suggest exactly that: the rate of improvement is slowing, even as the required resources balloon.
Diminishing Returns?
The question now facing the field is whether the enormous resources required can be sustained. If monetization does not keep pace with the rising costs, the scaling law may hit an economic ceiling. For startups and even well-funded labs, the price tag of a single training run—millions of dollars in cloud compute, plus the environmental toll—means that the gamble on bigger models becomes harder to justify.
For the people building these systems, the human side of the equation is becoming impossible to ignore. Developers must weigh the promise of marginal performance gains against a carbon footprint measured in hundreds of tons of CO2 per run.
Users, meanwhile, may not see dramatically better results from each new generation of models. What comes next is not yet clear. Some researchers are exploring more efficient architectures, smaller models trained on higher-quality data, or new ways to reduce inference costs.
But the path forward is no longer simply about adding more layers and more parameters. The scaling law, once treated as an article of faith, is now a subject of debate—and the outcome will shape not only the technology but the people who rely on it.
