DeepSeek flips the table again—open-source inference engine.

Just now, DeepSeek shook things up again! Open-source inference engine

• DeepInfer：
  
  https://github.com/deepseek-ai/DeepInfer
• deepseek-recipe：
  
  https://github.com/deepseek-ai/deepseek-recipe

When the news broke, the tech community went crazy.

Many developers exclaim that they are truly deeply impressed by DeepSeek’s open spirit..

This is the model’s inference framework, a core technology for boosting inference performance.

And just like that, it was open‑sourced, fresh and vibrant..

DeepSeek official blog states:

We hope that by sharing internal deployment experience, we can help developers and enterprises optimize the throughput and latency of large‑scale inference, thereby efficiently building private and public cloud deployments, reducing compute costs, cutting energy consumption, and making intelligence accessible to a broader audience at lower prices, achieving true large‑model inclusivity.

What is an inference framework?

When we use AI in everyday life, the most immediate impressions are usually the generation speed and the model’s cost.

But behind the model, when a large number of users send requests simultaneously, which ones should the server handle first, how are computational tasks allocated, and how do multiple chips cooperate?

These matters largely depend on the inference framework to coordinate.

If a trained large model is likened to a master chef, then the inference framework is the kitchen’s scheduling system for the entire restaurant: when tens of thousands of users place orders simultaneously, how should the server divide and handle the requests?

◈DeepInfer

DeepInfer this time publicly releases a set of engineering implementations already used in DeepSeek’s own services.

It integrates high‑performance operators and an optimized inference engine, supporting distributed inference, speculative decoding, SWA Bounded Replay, structured output, and is compatible with NVIDIA and Huawei Ascend platforms.

The technical name may sound a bit convoluted, but the goal is unified:

Make computation more efficient and generation involve less waiting.

Previously, DeepSeek has gradually released a series of single-operator libraries such as FlashMLA, DeepGEMM, and DeepEP.

This time, the publicly released DeepInfer directly assembles these scattered libraries into a complete engineering framework.

Specifically, the open‑source DeepInfer this time has several core components:

1. Core operators and unified parallel scheduling
   
   Package FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang operators into a unified runtime, with built‑in attention data parallelism (Attention DP), context parallelism (CP), MoE expert parallelism (EP), and load balancing.
2. Smart tricks for generation acceleration
   
   Supports DSpark and confidence‑based speculative decoding (quickly guess a few candidate tokens then batch‑verify, default up to 5 guesses per step), while using SWA Bounded Replay (sliding‑window bounded replay) to truncate redundant long‑context computation, reducing work in the pre‑fill stage.
3. Deterministic and structured output
   
   Supports batch‑invariant inference and deterministic sampling, and allows you to directly constrain model output format with regular expressions, JSON Schema, or EBNF grammar, eliminating the hassle of repeatedly cleaning malformed outputs at the application layer.
4. Compatible with both NVIDIA and domestic hardware
   
   The official repository explicitly lists dual‑platform support. It adapts to NVIDIA SM100‑architecture GPUs and also natively supports Huawei Ascend 950 NPU.

According to the official DeepSeek‑V4.1‑Flash benchmark data (based on a single scale‑up domain with 4 SM100 GPUs running Prefill and 64 SM100 GPUs running Decode), in a 128K long‑context scenario:

Prefill throughput can reach up to roughly 29K tokens/s per GPU;

During the Decode phase, at an interaction speed of 80 tokens per user per second, a single card can output throughput of about 2,200 tokens/s per GPU.

◈deepseek-recipe

At the same time, DeepSeek also announced another companion component:deepseek-recipe.

Many companies, after running open‑source models locally or in the cloud, often encounter a pitfall: even though they use the same model weights, the performance they achieve seems a bit “dumber” than the official API.

The root cause often lies in the format conversion before and after the request.

This time we’re publicly releasing deepseek-recipe, which basically does “simultaneous interpretation.”

It can convert API requests from different formats such as OpenAI or Anthropic into a unified conversation representation, strictly assemble prompts according to DeepSeek’s official logic, encode them into tokens for the model, and after the model finishes computing, parse them back into the corresponding API response format.

In the official words,this can help the community achieve the same experience as the official API when deploying privately.

Currently, large‑scale inference services for DeepSeek‑V4.1‑Flash are already running on DeepInfer; the official team has also made it clear that future inference code and subsequent optimizations for new DeepSeek models will continue to be updated in this repository.

From open-source model weights and open-source low-level operators to today handing over the inference framework and API encoding/decoding logic together.

“For cleverness and collective ability.”

This time, DeepSeek has handed the engineering experience that supports its own services over to more developers.