The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

A framework distills multi-turn long-horizon planning capabilities into agent models using on-policy supervision from single or multiple teacher models, spanning both pre-training and post-training phases.

For builders of autonomous agents, this addresses the core bottleneck of generating coherent plans across many steps without reward collapse. The method reduces reliance on expensive human trajectory annotation by leveraging teacher models' own rollouts. Multi-teacher distillation can produce agents with more robust planning policies than any single teacher, which is relevant for systems operating in varied environments.

This shifts agent training workflows from offline imitation learning to online, on-policy distillation, requiring infrastructure for running teacher rollouts and student updates in parallel. Builders already using RLHF pipelines can adapt by adding teacher-generated trajectory buffers. The approach may make specialized planning modules cheaper to train, as high-quality plan generation transfers from teacher to student without reward engineering.