ByteDance researchers identify cause of inconsistent long-context retrieval in DeepSeek models

What happened: ByteDance Seed researchers identified a mechanism behind inconsistent long-context retrieval in DeepSeek models, tracing it to chunked KV-cache compression and a pattern they call phase sensitivity. Key numbers: Up to 40 percentage points difference in retrieval accuracy across positions. Why it matters: The findings explain how strong average benchmark results can conceal recurring retrieval weaknesses, and the behavior was reproduced in models trained from scratch.

ByteDance Seed researchers identified a mechanism behind inconsistent long-context retrieval in DeepSeek models, tracing it to chunked KV-cache compression and a pattern they call phase sensitivity.

The findings explain how strong average benchmark results can conceal recurring retrieval weaknesses, and the behavior was reproduced in models trained from scratch.