ByteDance found the reason why DeepSeek is sometimes strong and sometimes weak

What happened: ByteDance Seed researchers found DeepSeek-V4's retrieval accuracy shifts periodically with the input position of information, tied to the model's chunked KV Cache compression. Key numbers: Up to 40.2 percentage point accuracy gap in 128K context tests; 4-Token period for DeepSeek-V4, 2-Token for DeepSeek-V4.1; gaps narrowed to 6.1 points after training. Why it matters: The team says standard benchmarks average results and can hide this phase sensitivity, so models using chunked KV Cache compression should be tested with the same information at different compression phases.

ByteDance Seed researchers found DeepSeek-V4's retrieval accuracy shifts periodically with the input position of information, tied to the model's chunked KV Cache compression.

The team says standard benchmarks average results and can hide this phase sensitivity, so models using chunked KV Cache compression should be tested with the same information at different compression phases.