NEW YORK, Sept. 8 — The biggest bottleneck in the AI era may no longer be computing power, but the humble act of moving data from one place to another. A new analysis published by MIT Technology Review, produced in partnership with Micron, argues that data movement has emerged as the most pressing infrastructure constraint as AI inference becomes the dominant enterprise workload. The piece, headlined “Architecting memory and storage in the AI era,” makes the case that organizations can no longer optimize compute, memory, storage, and networking in silos.
Inference workloads — the continuous process of a model generating answers for users — are geographically distributed, highly sensitive to response time, and place sustained pressure on infrastructure in ways that differ from earlier training-centric deployments.
AI is not one workload
“We tend to think of AI as a single workload, and it’s not. It’s thousands, it’s millions, it’s billions of different workloads,” said Jim McGregor, founder and principal analyst at Tirias Research, in the article. McGregor stressed that data centers must now support continuous, distributed, and increasingly real-time AI services, each with different requirements from a system-level perspective.
The practical upshot for enterprises: a one-size-fits-all infrastructure plan is no longer viable. Modern AI techniques are driving the shift.
Retrieval-augmented generation, or RAG, requires systems to constantly scan massive databases to generate accurate responses. That demands immediate access to data rather than raw compute alone — a fundamental change from traditional application behavior.
Bottlenecks migrate
“The biggest thing we’re doing right now is moving data from one place to another and making sure that we can use it effectively,” McGregor said. He cautioned that bottlenecks migrate across layers, which means performance per watt, latency, memory bandwidth, storage throughput, and networking all have to be balanced together. “You have to architect all four together to be efficient, and that’s the challenge,” he said.
The analysis warns against overbuilding infrastructure for peak conditions, a costly trap for enterprises racing to deploy AI. Instead, it concludes that the organizations gaining the most will be those with the clearest understanding of how to align infrastructure with actual workload demands — treating memory and storage as strategic assets rather than background hardware.
The article is part of a sponsored feature in partnership with Micron, and it offers no specific product names or market projections. What it does offer is a framing shift: in plain terms, think of AI infrastructure like a highway system where the roadways matter as much as the engines on the trucks. As inference workloads multiply, the roadways are where the traffic jams are forming.
