[HN Gopher] Self-reasoning tokens: teaching models to think ahead
___________________________________________________________________
Self-reasoning tokens: teaching models to think ahead
Author : fesens
Score : 102 points
Date : 2024-04-20 17:54 UTC (5 hours ago)
(HTM) web link (reasoning-tokens.ghost.io)
(TXT) w3m dump (reasoning-tokens.ghost.io)
| wrsh07 wrote:
| Ok so my understanding: you can have the network generate a token
| that can be used as input to future token generation along with
| each output token it generates
|
| These are called reasoning tokens
|
| Initial results with gpt2 are promising
|
| You can generalize this to let the network decide when to
| generate reasoning tokens (I'm unclear on how). There were also
| multiple lines in the loss graph with reasoning tokens that I
| don't quite understand (what's reasoning 1 vs 3? Is it the ratio
| of reasoning tokens? Something else?)
| fesens wrote:
| Reasoning 1 vs. 3 is the number of reasoning tokens between
| each "text" token. The 1 reasoning token is exactly what you
| see in the picture explanation in the article.
|
| The generalization comes from making the network predict a
| <"start reasoning token"> and end the sequence only when it
| predicts a <"end reasoning token">. The training dataset for
| the upcoming experiment contains examples like: """ Q: What is
| 3+2? A: 3 + 2 is equal to <start reasoning> <reasoning> ...
| <reasoning> <end reasoning> 5 """
| pizza wrote:
| I'm just speculating here since I don't know what or where the
| code is but since inference is still autoregressive;
|
| given [a b c] sample [d]
|
| distribution of [d] could be over [reasoning token] | [vocab
| token]
|
| then at next step you have
|
| [a b c d] and each has an embedding vector associated
|
| so when you go to sample [e] it's a function of [a b c d]
| XenophileJKO wrote:
| I have definately and frustratingly seen GPT3.5-Turbo do a bunch
| of anticipation in the outputs.
|
| Basically it will create pre-conditions so that the final output
| aligns to some bias. In my specific case it was the bias to
| provide an answer to a question. This is noticable sometimes in
| chain of thought intermediate outputs. I ended up having to
| create some space between the entangled decisions in the chain of
| thought output.
| PeterisP wrote:
| > In my specific case it was the bias to provide an answer to a
| question
|
| That seems to be a reasonably expected result of the
| "instruction post-training" finetuning with RLHF or otherwise.
| If for some reason you don't want this behavior, you can avoid
| this by using a model version that just has the core language
| modeling without that finetuning, e.g. the llama models have
| such a version available.
| alt0_ wrote:
| > definately
|
| relevant xkcd: https://xkcd.com/2871/
| jacobsimon wrote:
| I've tried similar experiments before by asking the LLM to
| generate "internal" and "external" dialog, which I think is sort
| of the same idea at a higher level---and might be preferable
| because it would allow for easy introspection vs a new set of
| tokens? I'm not enough of an expert to understand whether this
| proposal is intended more for training or inference.
| fesens wrote:
| The main advantage of using a new and constant token for
| reasoning is that, while we would pay the full price during
| training, in the inference phase, we could do most, if not all,
| the "reasoning" in one shot, without having to feed one
| generation token at a time.
| jacobsimon wrote:
| Cool!
| sdenton4 wrote:
| This method is for training. They are using a stop-gradient to
| 'shield' some tokens from contributing to prediction of the
| immediate next token, and thus producing a stream of tokens
| that are only used for longer term prediction.
|
| This is a bit more low level than the usual prompt engineering
| approaches, and to my mind, a bit more promising. There's more
| easily measurable results, and I've seen other context where a
| well placed stop-gradient does wonders...
| earslap wrote:
| For the existing models is beam-search like methods hopeless due
| to combinatorial explosion? Are there no smart ways to improve
| it? Evaluating multiple futures will be slow but if it means that
| the model can give vastly better output, it might be a worthwhile
| trade-off in some cases. I feel like our standard way of sampling
| the output of the LLMs is a bit too simplistic and my hunch is
| that it should be possible to get a lot more out of them even if
| it means losing speed.
| HarHarVeryFunny wrote:
| People are considering that sort of beam-search approach - this
| is what they call "tree of thoughts" - generate a branching
| tree of alternate continuations, then pick the best one based
| on some criteria.
|
| This doesn't seem an ideal approach though, since it amounts to
| generating a bunch of shallow responses and picking the best,
| rather than the preferred thinking more deeply before
| generating. It's not the same as a computer chess program
| considering N-moves ahead where you are guaranteed that one of
| those move sequences really is the best one (as long as you
| don't accidentally prune it out). In contrast, if you generate
| all possible "shallow" N-token responses (bunch of monkeys
| gibbering), there is no guarantee any of those will be the high
| quality response you are hoping for.
|
| Really planning ahead - reasoning deeply before speaking -
| would seem harder to implement though, since it'd involve
| applying a variable number of reasoning steps (maybe looping),
| then determining when to stop. This also seems different from
| the proposed insertion of "reasoning tokens" since those are
| shallow reasoning steps (normal single pass through
| transformer's layers), when it seems what is really needed is
| more depth of reasoning ("more layers"), perhaps coupled with
| some working memory/tokens. Both schemes (more tokens vs more
| depth) are also related to the wish to use a variable amount of
| compute for different tasks/inputs - less compute for simple
| tasks, more for hard ones.
| wantsanagent wrote:
| "The second token, however, duplicates the input of the first one
| and does not receive a gradient "answer" from the very next
| token, only from future tokens; ..."
|
| This formulation doesn't make a lot of sense to me.
|
| I get the motivation here but what you're trying to implement is
| a working memory.
|
| Because transformers have perfect retrospective memory within
| their context window any generation which can be done directly
| from input tokens will be.
|
| At any given point a model might want to write to a working
| memory, but that does not imply that the next non-working-memory-
| step will supply useful information to better write to working
| memory in the future. The model also has to be able to _decide_
| when to compare the work done in working memory to the next
| token.
|
| By allowing the model to both exempt output from gradient updates
| and opt back _in_ to gradient updates, you create a meta-learning
| loop that could be quite flexible.
| exploringBytes wrote:
| First association was to extend the modality of text tokens to
| concept tokens which could be (logical) relationships. Are you
| aware of similar works?
| pizza wrote:
| Not exactly the same game but you might be interested in
| _Mathematical Structure of Syntactic Merge, Marcolli, Chomsky,
| Berwick (2023)_.
|
| When we speak we give a string. When we think we don't have to
| use a string. But we do have to have a functionality to map
| something that has no single ordering to something that has an
| ordering (externalization) - a sentence. And vice versa we have
| a functionality to turn strings into things without a specific
| ordering (internalization) - thoughts.
___________________________________________________________________
(page generated 2024-04-20 23:00 UTC)