[HN Gopher] Understanding R1-Zero-Like Training: A Critical Pers...
___________________________________________________________________
Understanding R1-Zero-Like Training: A Critical Perspective
Author : pama
Score : 90 points
Date : 2025-03-22 14:35 UTC (8 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| scribu wrote:
| If the base models already have the "reasoning" capability, as
| they claim, then it's not surprising that they were able to get
| to SOTA using a relatively negligible amount of compute for RL
| fine-tuning.
|
| I love this sort of "anti-hype" research. We need more of it.
| mirekrusin wrote:
| So they achived R1-Zero like performance without those long CoT
| that sometimes never end/are impacting inference time with
| fraction of fine tunining resources?
| refulgentis wrote:
| No, they still have "<think>", but it's shorter by removing
| part of a term.
| mirekrusin wrote:
| That's what I mean, those CoT are never ending currently
| until you run out of context.
| refulgentis wrote:
| I'm not sure if you're talking conversationally and I'm
| taking it as a technical query, or you're saying CoT never
| terminate for you and asking for input, or asking what the
| paper implies about CoT, or relaying that you understand
| the papers claim that this method net reduces CoT length.
| mirekrusin wrote:
| Verbose, non terminating CoT is currently common problem
| with open weight models based on R1-zero methods.
|
| Currently it seems that this shift of cost to inference-
| time is a necessary tradeoff that we have to live with
| (for now at least).
|
| It's more of a problem for many people running those
| models locally because they have constrained hardware
| that can't handle those long contexts.
|
| It seems that this paper shows not only that their method
| is cheaper in terms of fine tuning but also significantly
| reduces inference time cost for CoTs.
|
| If what they say gets confirmed it looks to me like quite
| significant contribution?
| drakenot wrote:
| I've seen the same "Superficial Self-Reflection" mentioned in
| their linked blog post[0] as well, where the conclusion doesn't
| naturally follow the output of the thinking tokens. I think
| people are fooled by this, but if you take the time to inspect
| the "chain of thought" tokens they often don't match the final
| output answer.
|
| I don't deny that performance for certain logic tasks goes up
| with these models but I don't fully understand what role the
| thinking tokens take in these cases.
|
| [0] https://oatllm.notion.site/oat-zero
| andai wrote:
| I heard that even just getting the model to print a bunch of
| whitespace ("think for longer") improves the quality of the
| final response, because some kind of processing is still
| happening internally?
| MoonGhost wrote:
| Could it be that model just uses latent space for thinking
| while generating almost garbage? Interesting to check if
| adding repeating something at the end of prompt helps. I.e.
| model uses it for 'thinking'.
| mentalgear wrote:
| Overall the industry needs more review, less hype. I was shocked
| to find out SWE-verified [0] is all but verified.
|
| [0] benchmark used by all major vendors to "showcase" coding
| ability, turns out to be <10% properly solved:
| https://www.youtube.com/watch?v=QnOc_kKKuac
| belter wrote:
| Failure modes are also interesting to show what is happening or
| not really happening. Like the test of asking GenAI to create
| clocks at specific times, or people drawing with the left hand.
| All you get are clocks, at 10 min past two, or people drawing
| with the right hand, since it's 99% of what is in the training
| data.
|
| Like Sabine says, if the LLM models, already read all the Math
| books in the world but are not yet able to do basic math,
| without calling upon a calculator, how much reasoning is really
| emerging?
|
| "The Path to AGI is Coming Into View":
| https://youtu.be/mfbRHhOCgzs?t=219
___________________________________________________________________
(page generated 2025-03-22 23:00 UTC)