# DeepSeek Opens DeepInfer, Its Production Inference Framework

Published 2026-10-20 · AI.info Team · AI.info

> DeepSeek has released DeepInfer, the inference framework it says powers its large-scale production services, under the Apache 2.0 license. The first release supports DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, but omits several components needed for a complete inference service.

> “DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services.”
>
> [DeepSeek-AI](https://github.com/deepseek-ai/DeepInfer)

DeepSeek published DeepInfer on [GitHub](https://ai.info/companies/github) on October 20, opening access to software it describes as the inference framework behind its production services. The release is available under the Apache License 2.0. Its first supported model is DeepSeek-V4.1-Flash, and the repository provides instructions for running inference offline.

## DeepInfer targets DeepSeek-V4.1-Flash

The framework combines model and sampler runtime code with optimized kernels and tools for distributing work across processors. Its listed capabilities include attention data parallelism and balanced context parallelism, plus expert parallelism and expert load balancing for mixture-of-experts models. DeepSeek also includes DSpark speculative decoding, asynchronous scheduling, deterministic sampling and structured outputs.

Those features address different parts of inference: speculative decoding uses draft tokens to speed generation, while structured outputs can constrain responses to formats such as JSON schemas or grammars. The repository lists kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, alongside custom TileLang kernels. Asynchronous scheduling, which overlaps CPU batch scheduling with device execution, is enabled by default.

## Two accelerator platforms, demanding prerequisites

The release names [NVIDIA](https://ai.info/companies/nvidia) SM100 GPUs and Huawei Ascend 950 NPUs as supported hardware. Building it requires Linux, GCC 13 or later and Python 3.12; NVIDIA users also need CUDA 13, while the Ascend path requires CANN 9.2. The README’s example downloads the DeepSeek-V4.1-Flash-1017 checkpoint into the Hugging Face cache on its first run, or accepts a local model directory.

The engine grew primarily from vLLM, with additional inspiration from SGLang and TensorRT-LLM. That lineage places the release within an existing ecosystem of inference software, while the repository offers a view into the specific components DeepSeek assembled and optimized for its own model family.

## The release is not a complete serving stack

DeepInfer’s limits matter for operators considering a deployment. The release does not include a key-value cache system, the components for separating prefill from decode, or load balancing across data-parallel replicas. The project therefore supplies offline inference and many engine features, but its own documentation says additional infrastructure is needed to form a complete inference service.

The repository includes a performance-benchmark section, but its table lists no results in the published README. It says the measurements use four SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to a script for reproducing them. Without reported scores, the release documents its tested hardware setup but does not provide a numerical performance comparison.

DeepSeek’s opening move is to publish production-derived inference code while drawing a boundary around the infrastructure needed to serve requests end to end. Developers can inspect and run the framework under Apache 2.0, but must supply the omitted cache, disaggregation and replica-balancing layers themselves.

## Sources

1. Original source [DeepSeek-AI on GitHub](https://github.com/deepseek-ai/DeepInfer) github.com

## Related stories

- [vLLM 0.33 Drops CUDA 12.9 Builds](https://ai.info/news/vllm-033-drops-cuda-129-builds)
- [DeepSeek Opens Huawei Ascend Programming Tools](https://ai.info/news/deepseek-opens-huawei-ascend-programming-tools)
- [NVIDIA Releases Dory, a 4B Model With Adjustable Inference Depth](https://ai.info/news/nvidia-dory-4b-adjustable-inference-depth)
- [FuriosaAI Raises $620 Million to Expand RNGD Chip Production](https://ai.info/news/furiosaai-620-million-series-d-rngd-production)
- [Epoch AI introduces index for measuring inference capacity](https://ai.info/news/epoch-ai-inference-performance-index)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/deepseek-opens-deepinfer-production-inference-framework
