DeepSeek has open-sourced DeepInfer, the inference framework that runs its production services, in a move aimed at challenging major AI players and lowering inference costs. The release gives developers a rare look at a frontier laboratory serving stack. According to Bloomberg, the software reveals the methods DeepSeek relies on to hold down inference costs for advanced AI.
Inside the Framework and Hardware Support
DeepInfer is the engine behind DeepSeek-V4.1-Flash and ships under the Apache 2.0 license. The framework supports NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs, according to the software release. Think of an inference framework like the engine of a high-performance sports car, taking the raw power of the underlying silicon chips and turning it into usable speed for users. The codebase was originally built primarily on vLLM, with ideas from SGLang and TensorRT-LLM, as stated in the project documentation.
The framework stacks DeepSeek kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. DeepInfer supports Huawei Ascend 950 chips following a September release in which DeepSeek brought the TileLang kernel programming language to Huawei hardware with the chipmaker help. For hardware requirements, the framework requires Linux, GCC 13.0 or later, Python 3.12, plus CUDA 13 on NVIDIA hardware or CANN 9.2 on Ascend, according to the software requirements.
Performance and Cost at Scale
The practical upshot of this release is a window into how large clusters handle massive traffic. A single cluster serves 155 trillion tokens per day, with revenue at off-peak pricing working out to $38 per GPU-hour, according to deployment documents. The release covers offline inference with the core features needed at scale. Attention uses data parallelism, while mixture-of-experts layers use expert parallelism with an expert-parallel load balancer.
DeepInfer includes optimized kernels, parallelism methods, and a decoding method using draft tokens and confidence estimates. In plain terms, this decoding method named DSpark uses a draft model to propose five tokens at a time, while a confidence head predicts which ones the main model will accept, trimming checked draft tokens by about 30 percent. A second technique runs only the last 128 tokens of each prompt back through the decoder layers, which DeepSeek reports nearly halves total prefill computation. Two features that keep outputs identical from run to run work only on NVIDIA hardware.
What Is Missing and Industry Impact
While the release provides offline inference, it omits parts of a complete inference service. Users must supply a KV cache system, prefill and decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode, according to DeepSeek. The Hangzhou-headquartered company said on X, according to Bloomberg, that sharing the production stack helps the community lower compute costs, reduce energy use, and make artificial intelligence more affordable.
Firms that host or run DeepSeek models in the cloud can apply the methods on their own, and the price of serving a given model can vary widely with the serving software, according to Bloomberg. No outside party has yet benchmarked DeepInfer efficiency gains. Outside developers might read through the code and borrow its techniques for their own systems, shifting competitive dynamics in open-source artificial intelligence inference.
Speculative Decoding and Structured Output Features
DeepInfer implements DSpark, a confidence-scheduled speculative decoding system where a draft model proposes five tokens per step. A confidence head predicts the acceptance rate for each token, allowing the scheduler to verify only the most valuable prefix. According to the deployment Q&A, this process cuts verified draft tokens by about 30 percent, reducing the count from 5.00 to between 3.2 and 4.4. While the accepted length remains nearly unchanged at 3.29 versus 3.20, DeepSeek estimates this optimization yields a 4 percent to 7 percent throughput gain at production load.
Beyond decoding, the framework supports structured outputs using regular expressions, JSON schemas, and EBNF grammars. It also features asynchronous scheduling that overlaps CPU batch scheduling with GPU execution, ensuring the hardware never idles between steps. However, batch-invariant inference and deterministic sampling, which ensure identical outputs from run to run, are available exclusively on NVIDIA hardware. The system also handles cache hits by recomputing only the last 128 tokens of sliding-window attention state, keeping recomputation costs constant regardless of context length.
Sources
- DeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash
- DeepSeek open-sources DeepInfer, the inference framework behind its production services – SiliconANGLE
- DeepSeek open-sources DeepInfer software for running its AI models
Photo: Taylor Vick / Unsplash (Unsplash License)
