SignalTech/AI
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
High-priority alert based on source significance and keyword relevance.
Context
What happened: Researchers propose PR-OPD, a method for training language-model agents that combines a GRPO warm start with privileged representation on-policy self-distillation, where the policy writes a hindsight skill per trajectory and aligns its hidden states to a stop-gradient teacher.
Key numbers: A skill lifts WebShop
Why it matters: The authors argue token probabilities are an insufficient distillation channel, since skills alter hidden states far more than token probabilities. PR-OPD reports best overall results in every tested setting on ALFWorld and WebShop with two backbones.
What changed
Researchers propose PR-OPD, a method for training language-model agents that combines a GRPO warm start with privileged representation on-policy self-distillation, where the policy writes a hindsight skill per trajectory and aligns its hidden states to a stop-gradient teacher.
Why it matters
The authors argue token probabilities are an insufficient distillation channel, since skills alter hidden states far more than token probabilities. PR-OPD reports best overall results in every tested setting on ALFWorld and WebShop with two backbones.