SAN FRANCISCO, July 13 — The chips that power the latest wave of artificial intelligence are not just engineering marvels. For the people building and deploying these systems, they have become a scarce and expensive necessity. At the heart of large language model inference — the process of actually running a trained AI to generate text, answer questions, or analyze data — sit specialized graphics processing units.
The industry standard has settled on two workhorses from Nvidia: the A100, released in 2020 with 80 GB of video memory, and the newer H100, launched in 2022 with 80 GB of HBM3 memory. These are not your average consumer graphics cards. ## The cost of thinking Running a model like GPT-3, which has 175 billion parameters, comes with a clear price tag.
OpenAI’s pricing in 2023 put inference at about $0.02 per 1,000 tokens. For its more advanced GPT-4, the cost rises to $0.03 per 1,000 input tokens and $0.06 per 1,000 output tokens. A token is roughly a word or piece of a word, so generating a few paragraphs adds up fast.
Speed matters, too. On a single A100, GPT-3 generates roughly 30 tokens per second.
That is fast enough for a conversational chatbot, but it highlights how much compute is required for anything real-time. Memory is another bottleneck. A model like LLaMA-65B, with 65 billion parameters, needs 130 GB of VRAM to run — that means two A100s strapped together.
Without enough memory, the model simply will not load. ## Making it fit There are tricks to squeeze these giants onto fewer chips. A technique called quantization, which reduces the precision of the numbers the model uses, can cut memory requirements dramatically. Eight-bit quantization, for instance, reduces memory consumption by 75 percent.
That can mean the difference between needing a cluster of GPUs and fitting a model onto a single card. For smaller models, the hardware demands drop considerably.
LLaMA-7B, a 7-billion-parameter model, can run on a consumer-grade Nvidia RTX 4090. That opens the door for developers and researchers to experiment locally, without renting cloud time. ## Where the compute lives Most serious AI work happens in the cloud. Microsoft Azure hosts OpenAI’s models.
Google Cloud runs PaLM. Amazon Web Services powers Anthropic’s systems.
Each of these providers has stacked their data centers with the latest GPUs, and demand has been relentless. Energy consumption is another hidden cost. A 2023 study by Luccioni and colleagues found that GPT-3 inference uses about 1.5 kWh per 1,000 queries.
Across millions of queries a day, that adds up to a meaningful environmental footprint. The race is on to build more efficient hardware and software, but for now, the GPU remains the engine of the AI revolution. What happens next — whether models get smaller, chips get cheaper, or both — will determine who gets to build the future.
Sources
{“@context”: “https://schema.org”, “@type”: “NewsArticle”, “headline”: “LLMs Face Inference Costs, Hardware, and Latency Hurdles”, “description”: “SAN FRANCISCO, July 13 — The chips that power the latest wave of artificial intelligence are not just engineering marvels. For the people building and…”, “datePublished”: “2026-07-13T09:30:00+00:00”, “author”: {“@type”: “Person”, “name”: “James Roberto”}, “publisher”: {“@type”: “Organization”, “name”: “InfoPulse Today”, “url”: “https://newsfeed.infopulsetoday.com/”}, “citation”: [{“@type”: “CreativeWork”, “url”: “https://en.wikipedia.org/wiki/Graphics_processing_unit”}]}
