[HN Gopher] Tree Search Distillation for Language Models Using PPO
       ___________________________________________________________________
        
       Tree Search Distillation for Language Models Using PPO
        
       Author : at2005
       Score  : 83 points
       Date   : 2026-03-15 00:51 UTC (22 hours ago)
        
 (HTM) web link (ayushtambde.com)
 (TXT) w3m dump (ayushtambde.com)
        
       | supermdguy wrote:
       | > One might note that MCTS uses more inference compute on a per-
       | sample basis than GRPO: of course it performs better
       | 
       | This part confused me, it sounded like they were only doing the
       | MCTS at train time, and then using GRPO to distill the MCTS
       | policy into the model weights. So wouldn't the model still have
       | the same inference cost?
        
         | at2005 wrote:
         | Ah, I meant that MCTS uses more inference-time compute (over
         | GRPO) to _produce_ a training sample
        
       | natufunu wrote:
       | Great post! I wonder why MCTS is not more popular as a test time
       | compute harness. Did you compare performance of MCTS (without
       | distillation) against other methods (eg best of N) with the same
       | compute budget?
        
       | algo_trader wrote:
       | great write up (and effort!! ;))
       | 
       | what are your thoughts on MCTS for coding?
       | 
       | this can/must be paired with a smart execution harness to
       | optimise roll out and roll back of execution paths and system
       | state.
       | 
       | does this change the calculus for optimal post-training ?
        
       | richardvsu wrote:
       | Why is almost every RL paper done on Qwen-2.5 ? That decreases
       | its credibility.
        
         | yorwba wrote:
         | It makes it easier to compare with other papers. If two
         | different papers apply different methods to different models
         | and get different results, how do you know which method is
         | better?
         | 
         | Once you have identified the best method and want to productize
         | it, it would of course make sense to apply it on top of the
         | best model, but if you're just doing research, you can skip
         | that expensive last step.
        
         | mapontosevenths wrote:
         | > Why is almost every RL paper done on Qwen-2.5 ?
         | 
         | In what way does using this model reduce the authors
         | credibility?
        
       | qumpis wrote:
       | I may never understand what harness means - it's used in so many
       | contexts
        
         | blamestross wrote:
         | Its a thing that isn't part of the "subject", used with the
         | subject, to manipulate the state of the "the subject" to be
         | closer to what we want.
        
       ___________________________________________________________________
       (page generated 2026-03-15 23:01 UTC)