SignalTech/AI
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
High-priority alert based on source significance and keyword relevance.
Context
What happened: A paper analyzes training-inference mismatch in on-policy reinforcement learning for large language models, splitting it into a numerical discrepancy from the inference engine and a policy lag from stale weights, and derives the discrepancy-lag correction (DLC).
Key numbers: DLC keeps KL divergence from on-policy training within twice the KL between two independent
Why it matters: The paper reports that a correction stabilizing training under one mismatch source can fail under the other, and that DLC's resynchronization prevents collapses left by per-token corrections.
What changed
A paper analyzes training-inference mismatch in on-policy reinforcement learning for large language models, splitting it into a numerical discrepancy from the inference engine and a policy lag from stale weights, and derives the discrepancy-lag correction (DLC).
Why it matters
The paper reports that a correction stabilizing training under one mismatch source can fail under the other, and that DLC's resynchronization prevents collapses left by per-token corrections.