The gap between what AI chips can calculate and how fast they can feed themselves data has widened to 25,000 times over the last quarter-century, according to Fractile CEO Walter Goodwin. Floating-point throughput has scaled roughly a million-fold since 2000 while memory bandwidth has managed only about 40-fold, making the wire between processor and DRAM the hardest constraint on large language model inference today.
The 25,000x Gap Between Compute and Memory
Goodwin frames the problem in warehouse terms: the DRAM is a warehouse, the processor an office, and the connection between them is a loading dock served by a few thousand copper wires. Those wires cannot shrink further without hitting physical limits, so the office must keep sending runners back and forth for every box of data. The result is that a frontier model generating at 80 tokens per second is limited not by the chip’s math capability but by how fast it can pull weights and activations from memory.
Memory bandwidth, measured in bytes per second, is the rate at which a processor can read from or write to semiconductor memory. In practice, sustained bandwidth falls short of theoretical peaks, and the STREAM Triad benchmark has become a standard way to measure what a system actually delivers under realistic access patterns.
Why Token Generation Depends on Bandwidth, Not FLOPs
The key-value cache illustrates the shift. KV cache avoids repeated computation during generation by storing intermediate results, but it forces the system to retain and repeatedly access growing amounts of data as context windows expand. Longer contexts, higher concurrency, multimodal models, and agentic workflows all increase the pressure on memory capacity and bandwidth simultaneously. Adding arithmetic throughput delivers limited benefit if accelerators spend more time waiting for data than computing on it.

SemiEngineering notes that AI inference now requires balancing memory capacity, bandwidth, latency, power, data movement, utilization, and cost across compute, memory, interconnect, and software. Improving one dimension often creates pressure elsewhere. High-bandwidth memory close to the accelerator delivers tremendous bandwidth, but its capacity, power, and cost are constrained. Larger-capacity memory may be more economical but cannot match the bandwidth or latency. Moving data among tiers consumes bandwidth, power, and time.
Industry Responses: Vera, TPU 8i Max, UCIe 4.0
Nvidia’s Vera CPU addresses the bottleneck directly. Two Vera sockets sustained over 2 terabytes per second on STREAM Triad, 2.3 times the best DDR5 server tested. Even when reading the other socket’s memory over NVLink-C2C, Vera’s cores pulled more bandwidth than any x86 socket gets from local memory, with local latency 13 percent lower than on the EPYC 9175F.
Google’s TPU 8i Max takes a different approach: it combines two 8i chips into one package to double memory bandwidth for massive context windows. The inference-focused TPU 8i already uses a Boardfly topology with a maximum network-hop diameter of seven, compared to sixteen hops in the training-oriented torus design, reducing latency for real-time reasoning. The 8i Max extends that logic by putting more memory behind each compute unit.
The UCIe Consortium unveiled the UCIe 4.0 specification with a new UCIe.mem protocol for memory expansion. The protocol delivers 2.5 terabytes per second per millimeter of chip shoreline at about 94 percent efficiency, with up to 80 percent better energy efficiency overall. By separating memory control functions from the compute device while using a standardized die-to-die interconnect, UCIe.mem enables memory subsystem disaggregation so memory can scale independently of the compute chiplet.
From Compute-Centric to Memory-Centric Infrastructure
The shift is structural. For two years the industry bought GPUs on the assumption that more compute equals faster AI. Goodwin’s argument is that the bottleneck has moved to the physical interface between processor and memory — a copper-wire wall that more FLOPs cannot fix. The next phase of the AI buildout will be won by whoever moves data fastest, not whoever does the most math.
Sources
- Fractile CEO Walter Goodwin: The AI Chip Bottleneck Is Memory Bandwidth, Not Compute — BigGo Finance
- Memory bandwidth
- NVIDIA Vera CPU Review: 2TB/s of Memory Bandwidth and a Real Challenger to x86
- Google TPU 8tx Pairs AMD EPYC Turin Hosts with Confidential Computing while TPU 8i Max Doubles Memory Bandwidth
- Beyond Bandwidth: AI Inference Is Reshaping The Memory Wall
- UCIe Consortium Tackles the Memory Wall with New 4.0 Chiplet Spec – HPCwire
Photo: Wikideas1 (CC0)





