SANTA CLARA, September 25 — Nvidia says it can run large language model inference inside encrypted hardware while giving up very little speed, a claim the company laid out in a technical blog post published September 22. The post, titled “Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing” and authored by Lorna Aine, describes how Nvidia’s Confidential Computing technology lets LLM inference run on trusted hardware using memory-encrypted confidential virtual machines, confidential GPUs, and encrypted NVIDIA NVLink. The pitch is straightforward.
Data stays sealed even while a model is generating text. The engineering problem behind it is not.
Why encryption slows inference down
Inference frameworks such as NVIDIA TensorRT LLM are tuned for speed, and running them inside a confidential computing environment breaks the assumptions they were built on. Memory movement, timing, scheduling, and multi-GPU communication all behave differently once encryption is in the path. Nvidia says that introduces performance overhead unless the runtime adapts, and that holding onto high performance requires co-optimizing the inference framework and the confidential computing environment together.
The company describes CC-aware adaptations in TensorRT LLM for secure data movement, kernel autotuning, and multi-GPU communication on NVIDIA Blackwell GPUs. Some of the constraints are specific.
Host-to-device transfers in B200 confidential computing pass through a software encrypted bounce buffer, so TensorRT LLM selects pageable memory instead of pinned memory and moves repeated token readback to an asynchronous worker. The kernel autotuner switches from CUDA events to the GPU %globaltimer for tactic measurements under confidential computing, because timing signals otherwise become unstable. And NVLS multicast is unavailable in B200 confidential configurations, so frameworks should detect NVLS availability and choose communication algorithms that minimize latency for the given message size and topology.
The numbers from eight B200s
In a controlled comparison on eight NVIDIA B200 GPUs using the DeepSeek-R1 model, confidential compute retained 96.1% to 98.2% of baseline output-token throughput while adding 1.2% to 4.3% per-token latency overhead across concurrency levels 1 through 16. Nvidia notes that workload characteristics determine how visible CC overhead can be. High request volume can amortize fixed encryption costs by overlapping stalls with other work.
To expose the effects, the post recommends selecting a workload with a long input context, extended output generation, and low concurrency. Long context stresses data movement during prefill, extended generation amplifies small per-token CC overhead during decode, and low concurrency limits the opportunity to hide those effects.
The post directs readers to NVIDIA Trusted Computing documentation, TensorRT LLM release notes, and NVIDIA Confidential Compute news for further information. It is categorized under Trustworthy AI and Cybersecurity and is available in English and Chinese. One caveat sits on the page itself.
The blog carries an AI-generated summary disclosure powered by NVIDIA Nemotron, warning that AI-generated content may summarize information incompletely and urging verification of important information. What to watch next is whether those throughput and latency figures hold up on models and hardware beyond the single configuration Nvidia tested, and whether the CC-aware changes described here land in shipping TensorRT LLM releases.
