The Pulse
DeepSeek Opens DeepInfer, Its Production Inference Framework
DeepSeek has released DeepInfer, the inference framework it says powers its large-scale production services, under the Apache 2.0 license. The first release supports DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, but o
AI.info Team · Written with AI · fact-checked ·
“DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services.”
DeepSeek-AI
DeepSeek published DeepInfer on GitHub on October 20, opening access to software it describes as the inference framework behind its production services. The release is available under the Apache License 2.0. Its first supported model is DeepSeek-V4.1-Flash, and the repository provides instructions for running inference offline.
DeepInfer targets DeepSeek-V4.1-Flash
The framework combines model and sampler runtime code with optimized kernels and tools for distributing work across processors. Its listed capabilities include attention data parallelism and balanced context parallelism, plus expert parallelism and expert load balancing for mixture-of-experts models. DeepSeek also includes DSpark speculative decoding, asynchronous scheduling, deterministic sampling and structured outputs.
Those features address different parts of inference: speculative decoding uses draft tokens to speed generation, while structured outputs can constrain responses to formats such as JSON schemas or grammars. The repository lists kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, alongside custom TileLang kernels. DeepSeek says asynchronous scheduling, which overlaps CPU batch scheduling with device execution, is enabled by default.
Two accelerator platforms, demanding prerequisites
The release names NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs as supported hardware. Building it requires Linux, GCC 13 or later and Python 3.12; NVIDIA users also need CUDA 13, while the Ascend path requires CANN 9.2. The README’s example downloads the DeepSeek-V4.1-Flash-1017 checkpoint into the Hugging Face cache on its first run, or accepts a local model directory.
DeepSeek says the engine grew primarily from vLLM, with additional inspiration from SGLang and TensorRT-LLM. That lineage places the release within an existing ecosystem of inference software, while the repository offers a view into the specific components DeepSeek assembled and optimized for its own model family.
The release is not a complete serving stack
DeepInfer’s limits matter for operators considering a deployment. DeepSeek explicitly says the release does not include a key-value cache system, the components for separating prefill from decode, or load balancing across data-parallel replicas. The project therefore supplies offline inference and many engine features, but its own documentation says additional infrastructure is needed to form a complete inference service.
The repository includes a performance-benchmark section, but its table lists no results in the published README. It says the measurements use four SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to a script for reproducing them. Without reported scores, the release documents its tested hardware setup but does not provide a numerical performance comparison.
DeepSeek’s opening move is to publish production-derived inference code while drawing a boundary around the infrastructure needed to serve requests end to end. Developers can inspect and run the framework under Apache 2.0, but must supply the omitted cache, disaggregation and replica-balancing layers themselves.