DeepSeek’s Open-Source Is Amazing! vLLM/SGLang Overnight Surpass Performance, Pushing Inference Speed to New Heights

DeepSeek Open Source Is Truly Amazing! vLLM/SGLang Overnight Overtake Performance, Inference Speed Reaches New Heights

Our own inference engineDeepInferhas been out for less than a week, and the open‑source community has already turned in their assignments.

Truly a whale that makes everything flourish; the first wave of open‑source results is now fresh out:

Determined to learn DeepSeek, become DeepSeek, surpass DeepSeek.(just kidding)

Mature inference frameworks like vLLM and SGLang are the first to receive the T0 upgrade package—

Under the disclosed test configurations, performance is already close to, and in some cases even surpasses, DeepInfer.

Not kidding! DeepInfer was originally built on vLLM and its design also draws on SGLang’s ideas.

Now DeepSeek has open-sourced the refined implementation, and both teams are further absorbing and optimizing it.

The guys really know how to makeopen sourcelook spectacular.

There are also some developer personal projects inspired by DeepInfer gradually launching, ranging from hardware adaptation and inference acceleration to deployment optimization, each with its own style.

But the results are surprisingly consistent: All of them are more capable now!

Now that DeepInfer is open‑source, the community has become lively.

The inference race is heating up, it’s showtime～

Open‑Source Community Plays with DeepInfer

First, take a look at vLLM.

This time the results delivered have progressed from “matching DeepInfer” all the way to full rack deployment.

By integrating DeepSeek’s open‑source kernel and heavily optimizing the pre‑fill and decoding stages, vLLM’s performance curve now matches DeepInfer.

Under the same decoding configuration (131K context, 3 draft tokens per step), vLLM measured on GB200 shows a throughput gap of only 6% compared to DeepInfer across batch sizes from 8 to 256.

With pre‑filling, the throughput for new tokens also increased by 35.6%–43.9%.

Then scale the deployment to an NVIDIA GB300 NVL72 rack.

By separating prefill and decode and scaling expert parallelism, vLLM achieves a per‑GPU throughput of up to 954,000 tokens/s in the SemiAnalysis AgentX benchmark, which is 7.9 times the previous single‑node result.

The same DeepSeek, and they really squeezed out more speed…

How did these improvements come about? The team almost left no detail unchecked.

For example, when handling input, sparse attention needs to filter out the content worth focusing on from a batch of candidates, which involves a top‑k selection.

The previous path first converted the BF16 scores output by DeepGEMM to FP32 before filtering.

The new implementation letsDeepSelectprocess BF16 data directly, eliminating that intermediate conversion.

Skipping one computation may seem like a minor tweak, but in the corresponding operator test, this top‑k processing can run about3.7 times faster than before.

The vLLM team also implemented an alternative CPU communication scheme, using recursive doubling and similar techniques to exchange information, reducing the number of synchronization rounds required.

For the small‑batch decoding commonly used by agents, vLLM also provides dedicated support forDSpark draft generation, and executes the specialized decoding path with a full CUDA Graph.

SGLangFollowing the submitted answer sheet, it goes further, covering both the input and output stages.

SGLang tested pre‑filling on four B300 and four GB300 devices, achieving a peak throughput about 4% higher than DeepInfer.

During decoding tests, we also used 64 GB300.

When each card handles 256 requests, DeepInfer takes 31.19 ms per step, while SGLang takes 28.88 ms, a reduction of about 7.4% per step; when each card handles 8 requests, the two are basically on par.

In practice, SGLang first integrates the Mega core (MegaMoE, MegaAttention, Mega-mHC, MegaGate), SWA bounded replay, and Engram optimizations.

After achieving performance comparable to DeepInfer, further iterations introduce additional kernel fusion schemes.

Subsequently, by supporting splittable CUDA graphs, efficient pre‑fill is achieved.

In the final comparison with DeepInfer’s open‑source version, SGLang is more complete, adding key service components such as a KV cache system, a pre‑fill/decoding separation architecture, and a load‑balancing module.

It is worth noting that, to make a more rigorous comparison, SGLang reran DeepInfer on both GB300 and B300.

You wouldn’t know the results without running them; the throughput of some DeepInfer configurations is even higher than the previously announced figures…

Wow! Turns out Teacher D is really honest; the test results are quite genuine!

Inference Acceleration Loop

Developer Eric Zhang has pushed the approach one step further.

Previously he had been developing an experimental building‑block library for constructing large‑model inference engines—Resin.

It builds the parts that multiple models share into a common foundation, such as the scheduler, KV cache manager, etc. Users then add the specific computational implementations required by particular models and hardware, allowing them to assemble an independent engine.

Eric says that in the past week he was inspired by DeepInfer and successfully got many features and parallel mechanisms up and running.

According to test results on DeepSeek-V4.1-Flash, Resin’s performance is about 5%–10% behind DeepInfer, while also offering HTTP service, prefix caching, and metrics features.

Then, he added a Skill to this base.

The task is to teach the AI how to build models based on Resin, and then put Codex on it to work through the night.

According to his disclosure, the Agent has completed the GLM-5.3-Flash engine that supports MTP; under B200, EP4 configuration, input throughput reaches up to 60,000 tokens/s, and output throughput up to 16,000 tokens/s.

And! Resin remains open source.

I’m tearing up, Big Fat Fish, this open‑source boom is just as you wished.

DeepInfer comes from open source and returns to open source. Mature frameworks continue to integrate and optimize, and individual developers can also absorb its designs, passing the torch of open source.

So, Big Fat Fish truly grasps the essence of sustainable open source~

Reference links:
[1]https://vllm.ai/blog/2026-10-23-deepseek-v41-flash-pd-wide-ep
[2]https://gist.github.com/sglang-bot/f03d7d0dd268b0386177eb2ef09f8b3b
[3]https://x.com/BanghuaZ/status/2114083299102691751
[4]https://x.com/vllm_project/status/2112480719952138301
[5]https://x.com/ekzhang1/status/2114516258188153215