[HN Gopher] ReasoningGym: Reasoning Environments for RL with Ver...
___________________________________________________________________
ReasoningGym: Reasoning Environments for RL with Verifiable Rewards
Author : t55
Score : 84 points
Date : 2025-06-02 09:27 UTC (13 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| starzmustdie wrote:
| GitHub: https://github.com/open-thought/reasoning-gym
| phh wrote:
| Cool cool. I'm a bit put off by calling it "reasoning"
| /"thought". These RL targets can be achieved without "thinking"
| model but still cool. Gotta love the brainfuck task.
|
| I personally think that Gemini 2.5 Pro's superiority comes from
| having hundreds or thousands RL tasks (without any proof
| whatsoever, so rather a feeling). So I've been wanting a "RL Zoo"
| for quite a while. I hope this project won't be a one-off and
| will be maintained long term with many external contributions to
| add new targets!
| t55 wrote:
| > I personally think that Gemini 2.5 Pro's superiority comes
| from having hundreds or thousands RL tasks (without any proof
| whatsoever, so rather a feeling).
|
| Given that GDM pioneered RL, that's a reasonable assumption
| flowerthoughts wrote:
| Assuming with GDM, you mean Google-Deep Mind. They pioneered
| RL with deep nets as policy function estimator. The deep nets
| being a result of CNNs and massive improvements in hardware
| parallelization at the time.
|
| RL was established, at the latest, with Q-learning in 1989:
| https://en.wikipedia.org/wiki/Q-learning
| t55 wrote:
| i didn't say they invented everything; in science you
| always stand on the shoulders of giants
|
| i still think my original statement is fair
| lechatonnoir wrote:
| "gdm pioneered rl" is definitely not actually right, but
| it's correct to assert that they were huge players.
|
| people who knew from context that your statement was
| broadly not actually right would know what you mean and
| agree on vibes. people who didn't could reasonably be
| misled, i think.
| olliestanley wrote:
| We definitely plan to maintain the project for as long as there
| is interest in it. If you have ideas for new tasks, we'd always
| welcome contributions!
| phh wrote:
| Thanks for the answer! As a toy project I implemented
| wikiracing with trl. I'll probably try to PR that to your
| gym. (can't say that I managed to improve score with it
| though)
| CuriouslyC wrote:
| Gemini 2.5 Pro's superiority is IMO largely driven by their
| long context support and training methodology. Compare Gemini
| as a beta reader for a 100k token book with GPT4.1 or Claude 4,
| and it becomes quite clear how much more effectively it can
| reason across its context than other comparable models. This
| also makes it much better for architecting new features into a
| system, since you can load a lot of the current system into the
| context and it'll conform to existing styles and architecture
| patterns more closely.
| jacob019 wrote:
| Agreed, 2.5 flash too. I analyze a large json document of
| metrics for pricing decisions. Typically around 200k,
| occtionallly up to 1M, Gemini 2.5 significantly outperforms
| for my task. It isn't 100%, but role playing gets close. I
| suppose that's a form of inference time compute.
| t55 wrote:
| For a 100k token context window; all those models are
| comparable though
|
| gemini 2.5 pro shines for 200k+ tokens
| CuriouslyC wrote:
| I can confirm from first hand experience that even at 100k
| they are most definitely not comparable for the task of
| beta reading.
| throwaway314155 wrote:
| splitting hairs much?
| ninakostoska wrote:
| Cool to see NVIDIA's most recent reasoning model [1] already uses
| Reasoning Gymas a large part of their data mixture
|
| [1] https://arxiv.org/abs/2505.24864
| t55 wrote:
| > prolonged RL training can uncover novel reasoning strategies
| that are inaccessible to base models, even under extensive
| sampling
|
| does this mean that previous RL papers claiming the opposite
| were possibly bottlenecked by small datasets?
| yorwba wrote:
| No, they do not point to any specific examples of novel
| reasoning strategies that were uncovered, nor is their
| sampling that extensive (at most 256 samples vs the 2048 used
| in https://limit-of-rlvr.github.io/ ).
| t55 wrote:
| so you think it's fake news? another example of a paper
| with strong claims without much evidence?
| yorwba wrote:
| I think it's a case of not coming up with alternative
| explanations for the observed evidence and hence not
| designing experiments to distinguish between those
| explanations.
|
| Their results are _consistent_ with novel reasoning
| strategies, but they 're also consistent with more
| reliable execution of reasoning strategies that the base
| model can generate in principle, but rarely succeeds at
| due to a large number of steps. (If you have a model that
| can do each step independently with 99% success rate and
| getting the correct result requires 1000 steps, the
| chance of making it all the way to the end without a
| single error is only about 0.004%.)
| psb217 wrote:
| One challenge with this line of argument is that the base
| model assigns non-zero probability to all possible
| sequences if we ignore truncation due to numerical
| precision. So, in a sense you could say any performance
| improvement is due to shifting probability mass towards
| good reasoning behaviors and away from bad ones that were
| already present in the base model.
|
| I agree with your general point though. Ie, we need more
| thorough empirical investigation of how reasoning
| behavior evolves during RL training starting from the
| base model. And, current RL training results seem more
| like "amplifying existing good behavior" than "inducing
| emergent good behavior".
| yorwba wrote:
| While it's true that the model assigns non-zero
| probabilities to all sequences by design, those
| probabilities can get a lot smaller. E.g. replace that
| 99% per-step success probability with 10% and suddenly
| the overall chance of a correct result is truly
| astronomically small.
|
| For a novel reasoning strategy, I would expect at least a
| few individual tokens where the base model assigns much
| smaller probabilities than the reinforcement-learning
| trained one, as opposed to just being a little smaller
| but spread out over many tokens. (Which would better fit
| a "death by a thousand cuts" scenario.)
| jimmySixDOF wrote:
| RL is proving to be a weird science lately :
|
| >Spurious Rewards: Rethinking Training Signals in RLVR ###
| *TL;DR* We show that you can do RLVR on Qwen2.5-Math models with
| *completely random or incorrect rewards*, and still get massive
| math benchmark gains.
|
| All of the following _spurious rewards_ give 15-20+ points on
| MATH-500 when RLVR training Qwen2.5-Math-7B:
|
| - RLVR + format reward (reward responses with `\boxed{}`):
| *+16.4%* - RLVR + incorrect reward (only incorrect answers
| rewarded): *+24.6%* - RLVR + random reward: *+21.4%* - (as a
| reference) RLVR + ground-truth reward: + 28.8%
|
| How can these spurious rewards possibly work? Can we get similar
| gains on other models with broken rewards?
|
| >Learning to Reason without External Rewards Training large
| language models (LLMs) for complex reasoning via Reinforcement
| Learning with Verifiable Rewards (RLVR) is effective but limited
| by reliance on costly, domain-specific supervision. We explore
| Reinforcement Learning from Internal Feedback (RLIF), a framework
| that enables LLMs to learn from intrinsic signals without
| external rewards or labeled data. We propose Intuitor, an RLIF
| method that uses a model's own confidence, termed self-certainty,
| as its sole reward signal. Intuitor replaces external rewards in
| Group Relative Policy Optimization (GRPO) with self-certainty
| scores, enabling fully unsupervised learning. Experiments
| demonstrate that Intuitor matches GRPO's performance on
| mathematical benchmarks while achieving superior generalization
| to out-of-domain tasks like code generation, without requiring
| gold solutions or test cases. Our findings show that intrinsic
| model signals can drive effective learning across domains,
| offering a scalable alternative to RLVR for autonomous AI systems
| where verifiable rewards are unavailable. [2]
|
| [1] https://rethink-rlvr.notion.site/Spurious-Rewards-
| Rethinking... [2] https://arxiv.org/abs/2505.19590
| t55 wrote:
| yeah, RLVR is still nascent and hence there's lots of noise.
|
| > How can these spurious rewards possibly work? Can we get
| similar gains on other models with broken rewards?
|
| it's because in those cases, RLVR merely elicits the reasoning
| strategies already contained in the model through pre-training
|
| this paper, which uses Reasoning gym, shows that you need to
| train for way longer than those papers you mentioned to
| actually uncover novel reasoning strategies:
| https://arxiv.org/abs/2505.24864
| spmurrayzzz wrote:
| I think the fact that spurious rewards were predominantly only
| effective for Qwen may suggest that it was triggering some
| shift in its language distribution. If you use those models
| long enough you'll see a ton of mandarin that makes its way
| into your outputs, and their logits tend to look more
| "confident" than the ones for english tokens.
|
| So the reward value shifting may act as a sort of unintentional
| regularization technique (similar to adding noise to the
| discriminator input in GAN archs).
| sadboots wrote:
| by the love of god, please stop overfitting on gsm8k
| i5heu wrote:
| It looks like your neural network is overfitted on seeing
| overfitt where is none.
|
| Prejudices is a form of overfitting IMHO
| t55 wrote:
| agree, the RG evals feel like a fresh breeze
| olliestanley wrote:
| Difficult one. GSM8K and MATH evals (both reported in Reasoning
| Gym paper) are common in smaller model RL papers for a reason,
| which is that smaller models can get decent scores on them,
| unlike fresher & harder benchmarks.
|
| Part of the aim of RG is to be used as a difficulty-adjustable
| & non-repeating eval though so if people think it's a good
| benchmark, perhaps it will allow this status quo to shift!
___________________________________________________________________
(page generated 2025-06-02 23:01 UTC)