https://video.ut0pia.org/videos/watch/af519812-ab26-44d9-b7ff-1a170a306ea4 Scale on policy self distillation to trajectories with a hundred tool calls and the model collapses into hedging. The tokens it learns to favor fill up with wait, but, and maybe, until, as Ronak Malde puts it, everything just turns into maybe. He calls it the but wait problem, and it happens because the student drifts far enough on a long task that the teacher course corrects at every opportunity, leaving the model parked between two divergent distributions. The algorithm underneath is a good trick. At the frontier there is no smarter model to distill from, so you make the model its own teacher: put privileged information, a hint, in the teacher's prompt, and match the log probs of the student that never saw it. Malde scores post training methods against four properties, an online task distribution, on policy sampling, no parallel rollouts, and a per token reward, and shows SFT, RLHF, and GRPO each buying some at the cost of others. GRPO gets on policy sampling but explodes parallelism and collapses feedback into one sequence level score, which he compares to being handed 87 out of 100 on an essay and told to work out why. Self distillation gets all four, and it optimizes across the entire vocabulary at every token instead of sharpening the one that was sampled, which is why it keeps climbing past where GRPO plateaus while tokens to solve go down rather than up. The failure modes are the useful part: step level KL weighting to handle divergence, and residual guidance for hint leakage, the self distillation analogue of reward hacking, where a hint containing the answer teaches the model to state it and back fill the reasoning afterward. Speaker info: https://x.com/rronak_, https://www.linkedin.com/in/ronak-malde, Timestamps: 0:00 - From Windsurf to Trajectory 1:05 - Benchmarks are saturating and getting expensive 1:58 - The signal we throw away every day 2:51 - Four things a training algorithm should have 3:42 - SFT, and what it got right 4:31 - DPO and RLHF 5:22 - GRPO and the Faustian bargain 6:15 - How GRPO actually works 7:04 - Scored 87 out of 100 and told to figure it out 7:52 - Distillation, then on policy distillation 8:43 - Self distillation: make the model its own teacher 10:21 - Optimizing the whole vocabulary, not the top token 11:11 - Results on short horizon tasks 12:52 - What breaks at 120B and a hundred tool calls 13:41 - The but wait problem 14:31 - Step level divergence weighting 16:11 - Hint leakage, the new reward hacking 17:51 - Residual guidance 19:35 - All four properties, finally 20:25 - What Trajectory is building 21:16 - Q&A: how continual is continual learning 22:08 - Q&A: model and harness improving together Wed, 12 Aug 2026 20:16:51 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://video.ut0pia.org https://video.ut0pia.org/lazy-static/avatars/0287a09a-aae7-4840-9843-b416426e7046.webp https://video.ut0pia.org/videos/watch/af519812-ab26-44d9-b7ff-1a170a306ea4 All rights reserved, unless otherwise specified in the terms specified at https://video.ut0pia.org/about and potential licenses granted by each content's rightholder.