LONDON, July 8 — The quiet, unglamorous work of assembling the data that powers large language models has become one of the most consequential — and legally fraught — activities in artificial intelligence. As the models themselves grow more capable, the technical reality of what goes into them, and what is left out, is attracting scrutiny from regulators, publishers, and the courts. The sheer scale of the undertaking is difficult to overstate.
GPT-3, the model that demonstrated the potential of modern LLMs, was trained on 570 GB of text. That corpus drew from Common Crawl — a nonprofit that has been crawling the web since 2008, providing free datasets — contributing 410 billion tokens.
WebText2 added 19 billion tokens, Books1 contributed 12 billion, Books2 added 55 billion, and Wikipedia chipped in 3 billion tokens. The training process itself required 10,000 NVIDIA V100 GPUs running over several weeks. Lambda Labs estimated the cloud compute cost at $4.6 million.
For GPT-4, SemiAnalysis estimated the training cost at between $100 million and $200 million. Under the hood, data quality remains the detail that matters.
Training data inevitably includes biases, misinformation, and copyrighted material — problems that no amount of compute power can automatically fix. The consequences have been legal as well as technical. The New York Times sued OpenAI and Microsoft in December 2023 for copyright infringement.
Getty Images sued Stability AI in February 2023. These cases, as ever, turn on what was scraped, from where, and whether the terms of use permitted it.
Filtering the firehose
Data filtering techniques have become a critical part of the pipeline. Deduplication, often using the MinHash algorithm, removes near-identical passages that could bias the model toward repetition. Toxicity filtering, using tools like the Perspective API, attempts to screen out harmful content.
Quality filtering employs classifiers trained on Wikipedia to identify text that resembles encyclopedic entries, on the theory that higher-quality source material produces better models. The environmental cost of training is also under scrutiny.
Strubell et al. (2019) found that training a 1.5 billion parameter model emits 626,000 pounds of CO2 equivalent. That figure predates the largest current models, and the energy demands have only grown as training runs have scaled up.
Synthetic solutions
Future trends point toward two approaches that may reshape how training data is sourced. Synthetic data generation uses models like GPT-4 to create training data for smaller models, potentially reducing reliance on scraped web content. Data curation, formalised in benchmarks like DataComp introduced in 2023, treats the selection and weighting of training data as a problem to be solved systematically, rather than a matter of brute-force collection.
The technical reality is that data, not architecture, increasingly determines what these models can and cannot do. As the lawsuits proceed and the costs mount, the quiet work of curation may prove as important as the flashier work of training.
{“@context”: “https://schema.org”, “@type”: “NewsArticle”, “headline”: “The High Cost of Training Data and Compute Behind LLMs”, “description”: “LONDON, July 8 — The quiet, unglamorous work of assembling the data that powers large language models has become one of the most consequential —…”, “datePublished”: “2026-07-08T09:30:00+00:00”, “author”: {“@type”: “Person”, “name”: “Sebastian Smith”}, “publisher”: {“@type”: “Organization”, “name”: “InfoPulse Today”, “url”: “https://newsfeed.infopulsetoday.com/”}}
