SAN FRANCISCO, July 13 — The numbers behind the latest generation of OpenAI’s language models are staggering, but a new analysis of the company’s work offers a concrete look at the sheer scale required to build these systems. The practical upshot, in plain terms, is that each leap forward in artificial intelligence now demands a leap in computing power, energy, and cold, hard cash. It started with GPT-3, released by OpenAI in June 2020.
That model, with its 175 billion parameters, was trained on 570 GB of text data drawn from sources including Common Crawl, WebText2, Books1, Books2, and Wikipedia. The company’s own 2020 paper pegged the total compute used at 3.14e23 FLOPs, costing an estimated $4.6 million.
To get that done, OpenAI ran the training on 10,000 Nvidia V100 GPUs over several months. Think of it like building a skyscraper. GPT-3 was a tall building, but the next one, GPT-4, is a whole new city block.
GPT-4 arrived in March 2023, and it brought a fundamental architectural shift. OpenAI has disclosed that it uses a “mixture of experts” architecture, which essentially means the model can activate only the relevant parts of its brain for a given task, rather than firing every neuron at once.
The company has not disclosed the model’s total parameter count, leaving the industry to guess at its true size. What is clearer is the cost. Industry analysts estimate the training bill for GPT-4 landed somewhere between $100 million and $200 million.
The compute requirements were immense: likely using 25,000 Nvidia A100 GPUs running for 90 to 100 days. That is more than double the GPU count and many times the duration of the GPT-3 training run.
Data and energy at scale
For data, OpenAI moved beyond the public web scraping that fed GPT-3. The company licensed data from Microsoft and other sources to train GPT-4. This shift toward commercial data partnerships reflects a growing reality: the open internet may no longer be sufficient to feed the largest models.
There is also an environmental cost baked into these figures. A 2020 study by Patterson and colleagues calculated that training GPT-3 alone emitted 552 metric tons of carbon dioxide.
The emissions for GPT-4, given its far larger compute footprint, would be substantially higher, though no comparable public study has yet been released for that model.
What to watch next
The trajectory is clear. Each generation of these models requires an order-of-magnitude increase in compute, data, and investment. The question now is whether the industry can sustain this pace, or whether the next breakthrough will require a fundamentally different approach to building AI.
For now, the race is only getting more expensive.
Sources
{“@context”: “https://schema.org”, “@type”: “NewsArticle”, “headline”: “Training Data and Compute: The Massive Scale Behind GPT-3 and GPT-4”, “description”: “SAN FRANCISCO, July 13 — The numbers behind the latest generation of OpenAI’s language models are staggering, but a new analysis of the company’s work…”, “datePublished”: “2026-07-13T09:30:00+00:00”, “author”: {“@type”: “Person”, “name”: “Erika Johnson”}, “publisher”: {“@type”: “Organization”, “name”: “InfoPulse Today”, “url”: “https://newsfeed.infopulsetoday.com/”}, “citation”: [{“@type”: “CreativeWork”, “url”: “https://en.wikipedia.org/wiki/GPT-3”}]}
