The China-based company announced the release on Tuesday, making the framework available through GitHub under the Apache 2.0 licence. Its initial public version supports DeepSeek-V4.1-Flash and includes software for processing requests, coordinating computing resources and generating responses.
The move opens a substantial part of DeepSeek’s production technology to outside developers, although the company has acknowledged that several components necessary for a complete online service remain proprietary or absent from the release.
DeepInfer currently provides offline inference rather than a ready-made replacement for DeepSeek’s hosted application programming interface. Developers seeking to operate a full service must supply additional infrastructure, including a key-value cache system and mechanisms for distributing workloads across computing clusters.
DeepSeek said the framework supports Nvidia SM100 graphics processors and Huawei Ascend 950 neural processing units. That dual compatibility gives operators a choice between two accelerator platforms, although the software’s specified hardware requirements limit the range of machines on which it can run without modification.
The company describes DeepInfer as the framework powering its large-scale production services. Its published features include optimised computing kernels, parallel processing, asynchronous scheduling, structured outputs and mechanisms intended to make generation more predictable.
One technique, speculative decoding, prepares candidate tokens before verification, seeking to accelerate the generation of answers. Another feature overlaps processor scheduling with accelerator execution, reducing the time computing hardware spends waiting for new work.
DeepInfer also supports batch-invariant inference, designed to preserve identical model results despite changes in the composition of processing batches. Deterministic sampling can produce matching outputs from the same input and settings when the documented configuration requirements are met. Both capabilities are currently restricted to Nvidia hardware.
For developers building applications, structured-output controls can constrain responses to formats defined by regular expressions, JSON schemas or formal grammars. Such controls can help systems that require machine-readable answers rather than unrestricted conversational text.
DeepSeek’s technical documentation identifies several specialised components, including FlashMLA, DeepGEMM, DeepEP and TileKernels, which contribute to the framework’s computational performance. It also describes expert parallelism and load-balancing techniques intended to distribute work efficiently across multiple accelerators.
The company has published performance benchmarks covering the stages of processing incoming prompts and decoding generated tokens. Its documentation says the tests used four Nvidia SM100 processors for prompt processing and 64 for decoding, with each configuration operating within a single scale-up domain.
Those figures describe particular test configurations rather than performance guarantees for other deployments. Hardware arrangements, request patterns and the availability of supporting infrastructure can materially affect throughput and response times.
A separate deployment document states that one of DeepSeek’s clusters handles 155 trillion tokens daily. That is a company-provided operating figure, not an independently verified measure of what external users should expect from the publicly available software.
The same document describes a production architecture separating prompt processing, response generation and other tasks across different computing resources. DeepSeek says its key-value cache management and load-balancing systems are closely tied to internal infrastructure and have therefore not been fully released.
Its guidance indicates that a smaller deployment combining prompt-processing and decoding resources is technically possible, while an efficient configuration at its recommended scale may require substantially more hardware. This distinction matters for organisations assessing the cost of running the framework themselves.
Alongside DeepInfer, DeepSeek has made available deepseek-recipe, a component for converting between commonly used language-model API protocols and model token sequences. The company says its processing behaviour aligns with the official DeepSeek API, helping external implementations maintain compatibility.
The DeepInfer repository includes build instructions and examples for offline operation. Its listed prerequisites include Linux, Python 3.12 and GCC version 13 or later, alongside CUDA 13 for Nvidia systems or CANN 9.2 for Ascend systems.
The Apache 2.0 licence permits developers to inspect, modify and redistribute the framework subject to its terms. It does not automatically grant rights over separately distributed model weights or provide the missing infrastructure. Organisations planning deployments must therefore assess software compatibility, hardware capacity and applicable model licensing independently.
DeepSeek credits the open-source vLLM project as a principal foundation for the framework and acknowledges inspiration from SGLang and TensorRT-LLM. The repository also contains evaluation instructions and example scripts for developers examining supported configurations.
Follow Arabian Post
Select Arabian Post as your preferred source on Google and MSN News for trusted business news and Arab politics and updates.
