MILAN, September 13 — The architecture that powers every large language model in commercial use today was sketched out in a single research paper nearly a decade ago, and the industry has been building on it ever since. Google researchers published “Attention Is All You Need” in 2017, introducing the transformer — the neural network design that made modern AI possible. The shift is in the mechanism.
The transformer’s self-attention layer lets a model weigh the relevance of every word in a sequence against every other word at once, rather than reading text strictly in order the way earlier recurrent neural networks did. That parallelism is what made it practical to train on vastly larger datasets.
Everything since — ChatGPT, Gemini, Claude, Llama — traces back to that one design choice.
How the models actually learn
Training happens in stages. First comes pretraining: the model is fed enormous volumes of text — books, websites, code, articles — and repeatedly performs next-token prediction, guessing the next word or word fragment in a sequence. Through billions of such guesses, adjusted by backpropagation and gradient descent, the model tunes billions of internal numerical parameters.
GPT-3, released by OpenAI in 2020, carried 175 billion parameters. Then comes fine-tuning and alignment.
Human reviewers rank model outputs, and reinforcement learning from human feedback nudges the model toward helpful, harmless and honest responses. ChatGPT launched in November 2022 and became one of the fastest-adopted consumer software products in history. Rival systems followed from Google, Anthropic and Meta.
What these systems are not
LLMs are not databases. They store no verbatim record of their training text; they encode statistical patterns. That is why they generate fluent prose and code — and why they hallucinate, confidently producing false facts, fake citations or invented quotes.
Researchers have documented that hallucination rates vary widely by task and model, and that models can be more reliable on some benchmarks than others. Other limits are structural.
Context windows are fixed, though they have grown from a few thousand tokens in early models to over a million in some 2024-era systems. Training data cutoff dates leave models unaware of recent events. And jailbreak prompts can bypass safety guardrails.
Where the technology is heading
The field is advancing on several fronts at once: multimodal models that process images, audio and video alongside text; reasoning models that generate intermediate step-by-step chains of thought before answering; smaller, more efficient models that run on phones and laptops; and agentic systems that use LLMs to plan and execute multi-step tasks such as browsing the web or writing and running code. Regulation is catching up. The European Union’s AI Act, adopted in 2024, is the world’s first comprehensive AI law.
Copyright lawsuits from authors and news organizations over training data continue to mount, alongside questions about the energy and water consumption of data centers and the risk of misuse for disinformation. The core mechanics remain what they were in 2017: massive text corpora, transformer networks, next-token prediction and human feedback.
Understanding that pipeline is the key to understanding both the genuine capabilities and the persistent failure modes of systems now woven into everyday life. Watch this space — the trend underneath is not slowing.
