DeepSeek open-sources inference framework with support for Nvidia and Huawei chips

What happened: DeepSeek released DeepInfer, the inference framework behind its large-scale model services, on October 20. It supports Nvidia GPUs and Huawei Ascend processors and includes distributed inference, speculative decoding and structured output. Key numbers: Released October 20. Why it matters: DeepSeek said the release is intended to help developers and businesses improve throughput and latency while reducing computing costs and energy consumption. The company plans to keep releasing inference code and optimizations for future models through the project. Coverage: 7 outlets in 24h

DeepSeek released DeepInfer, the inference framework behind its large-scale model services, on October 20. It supports Nvidia GPUs and Huawei Ascend processors and includes distributed inference, speculative decoding and structured output.

DeepSeek said the release is intended to help developers and businesses improve throughput and latency while reducing computing costs and energy consumption. The company plans to keep releasing inference code and optimizations for future models through the project.