DeepSeek open-sources DeepInfer, the inference stack behind its V4.1 Flash model

DeepSeek has open-sourced DeepInfer, the inference framework it says powers its large-scale production services, including the V4.1 Flash model. The code is released under the Apache 2.0 license. The GitHub repository describes the release as offline inference with the core features needed for serving at scale, and it runs on NVIDIA and Huawei Ascend hardware.

DeepSeek's engineering report, as summarized by Latent Space's AINews, claims 155 trillion tokens a day from one cluster, a 96% input-cache hit rate, 400 ms average time to first token and 230 tokens per second of generation, rising to 300 for coding. It also reports $38 of revenue per accelerator-hour at off-peak prices. AINews notes these are first-party claims, not independently audited.

The release is not the full production service. KV-cache management and load balancing are not yet fully released, because they remain coupled to DeepSeek's internal infrastructure. The repository lists the components required for a complete inference service as not included. DeepInfer supports distributed inference, speculative decoding, SWA Bounded Replay and structured output, and separates vision encoding, prefill and decode so each can scale on its own.

The project was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM, which the repository credits. The recommended deployment uses ordinary eight-accelerator machines for prefill and 64 to 128-accelerator supernodes for decode, with wide expert parallelism for mixture-of-experts layers.

The numbers feed a debate about inference profitability. Latent Space's swyx works from DeepSeek's prices of $0.15 and $0.60 per million input and output tokens and assumes all-in costs of $5 to $8 per card-hour. He gets an inference margin above 80%, and an 81-day payback on cluster spend. These figures are his arithmetic, and AINews itself cautions that they are sensitive to workload mix, cache accounting, hardware cost and which expenses are counted.

The heavy caching matters: a 96% hit rate means most input tokens are cheap to serve, so the economics describe a cache-friendly workload rather than inference in general. The vLLM team's reply adds to that caution. It reported decode throughput within 6% of DeepInfer's published results on GB200 under a matched configuration, and published reproduction recipes. That undercuts the idea that DeepSeek's efficiency is out of reach for existing open-source engines.

For operators, the practical value is a reference implementation from a large production deployment under a permissive license. The parts withheld, cache management and load balancing, are the pieces that most affect cost at scale, so the gap between running the engine and matching DeepSeek's service economics remains.

Open questions include whether DeepSeek releases the KV-cache and load-balancing code, and whether independent groups reproduce the throughput and cache figures on their own hardware. DeepSeek says future model inference implementations and optimizations will also appear in the same repository.