[HN Gopher] Training LLMs to Reason in a Continuous Latent Space
___________________________________________________________________
Training LLMs to Reason in a Continuous Latent Space
Author : omarsar
Score : 152 points
Date : 2024-12-10 16:26 UTC (6 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| ilaksh wrote:
| It seems like the latent space could be even more useful if it
| was trained with the transcribed videos.
| mentalgear wrote:
| "We utilize the last hidden state of the LLM as a representation
| of the reasoning state (termed "continuous thought")."
|
| Could someone explain the last hidden state of the LLM ? What it
| shape is and how it is normally used - and why it hasn't been
| used yet to augment the next input? (which seems logical)
| AmazingTurtle wrote:
| Embeddings aka the last hidden state are the mathematical
| representation of an input of the model before a separate model
| (usually the decoder) translates that hidden state to a next
| token (the generative part in generative ai). Normally, the
| this step repeats over and over. This novel approach introduces
| re-using the last hidden state as if it was a token that has
| been generated thus "evolving" the hidden state over each
| iteration.
| tjbai wrote:
| The last hidden state is just the output embedding after N
| residual layers, e.g. input embedding + res1 + res2 + ...
|
| There's typically an "unembedding layer"/"classification head"
| that uses this hidden state to produce a softmax distribution
| over the LLM's vocabulary. In this case, we can think of this
| as "snapping" the hidden state into a single token and feeding
| that token into the next position of the autoregressive LLM.
|
| In this sense, the last hidden state _does_ augment the next
| input. The authors simply propose directly feeding this hidden
| state into the next step rather than reducing it into a single
| token--thus, reasoning in continuous latent space rather than
| discrete token space.
| intalentive wrote:
| Moreover "snapping" the hidden state to a token is akin to
| quantization. It's lossy. By staying in latent space the
| model can "reason" at "full resolution" without
| discretization noise.
| soulofmischief wrote:
| Useful information about conceptual relationships and
| procedure can be captured in the LM head, so there is also
| potential lossiness when short-circuiting it.
| snthpy wrote:
| Sometimes discretization introduces interesting behavior
| though. Compare for example the logistic map and it's
| chaotic regime with the simplicity of the logistic ODE.
| Another example would be quantum mechanics compared to
| classical mechanics and determinism. The Poincare
| Conjecture was only interesting for n=3 due to too much
| connectivity in higher dimensions. Wouldn't it be
| interesting if consciousness only arose in such a
| discretized form, a case of incidental complexity and chaos
| introduced as the result of topological non-triviality from
| quantization?
|
| Don't forget, non-linearity is fundamental to the whole
| process, otherwise you'd just have one large linear
| transformation. Maybe there's a similar role for
| discretization? :shrug:
| psb217 wrote:
| The way the recurrence in this method works -- ie, using last
| LLM hidden state at previous time step as input token for the
| next time step -- isn't directly compatible with how
| recurrence/autoregression is typically handled during LLM
| training. One of the major strengths of transformers is that
| they can be trained for recurrence/autoregression (which have
| sequential dependency) using convolutions (which are
| embarrasingly parallel). The proposed method requires
| introducing some sequential dependencies during training that
| could otherwise be avoided using "causal masking" and
| convolutions to enforce the correct dependencies between time
| steps in a sequence. Introducing these sequential dependencies
| makes training a lot slower.
|
| tldr; the method requires training in a way that loses one of
| the major benefits of transformers, but maybe in some scenarios
| that loss is worth it.
| DalasNoin wrote:
| So the models will no longer be thinking in plain English but
| some embedding space? Seems not like what you want.
| Vampiero wrote:
| Seems exactly like what you want. We don't think in plain
| English, we _rationalize_ our thoughts into English (or
| whatever language comes out) but they must be more fundamental
| than language because language is acquired.
|
| Essentially, English is one of many possible encodings of an
| underlying intuitive, possibly non-symbolic representation.
| ekianjo wrote:
| > We don't think in plain English
|
| That's debatable. Language shapes thoughts much more than you
| might think. Because you learn concepts from language that
| you could not imagine by yourself until you learned/read
| about them, so they are in effect very linked to language.
| Vampiero wrote:
| Fair, but there are many categories of languages.
|
| For example, I can think in formal logic. I've learned to
| do that, and surely my brain takes a step-by-step approach
| to it, but I've also internalized some of it and I don't
| think that my proficiency with English has anything to do
| with it.
|
| I could have learned the same concepts in any other
| language, but the end result would be the same.
|
| And surely there are many thoughts that can't be expressed
| purely with words. For example all that is related to
| qualia. You can think of a color but you can't describe
| what you see in your mind's eye with words, not in a way
| that would let a blind person share the same experience. Or
| try describing "love" without making a similitude. Is love
| a thought? Or a feeling? Is there a meaningful difference
| between the two?
| samiskin wrote:
| Language is definitely a significant part of thinking, but
| when I remember how cold it was outside yesterday to figure
| out if it was colder than today, I'm not bringing words to
| mind. I'm bringing up some other non-discrete information
| that I could never precisely encode into words and then
| factoring that in with the other non-discrete information
| I'm currently taking in through my senses. Its only after
| that processing that I encode it as a lossy "It was colder
| yesterday" statement.
| idiotsecant wrote:
| But the thing you learn is not the word 'purple'. You just
| use the word as the mental scaffolding to build a _concept_
| of purple. The word forms a linkage to a deeper embedding,
| which is further proven by the fact that it 's actually
| slightly different in each mind that has understanding of
| the concept.
|
| This embedded concept is what is doing the work, the word
| was just the seed of the understanding and a method by
| which to convey that understanding to others.
| phkahler wrote:
| I can also think in images and internal visualizations.
| Geometric reasoning is also a thing. Musicians can also
| hear things in their mind - some can write it down, others
| can play it directly, and in my case I'm not good enough to
| get it out of my head!
|
| In all cases though these thoughts are kind of tied to
| representations from the real world. Sort of like other
| languages via different senses. So yeah, how abstract can
| our thoughts actually be?
| intalentive wrote:
| Cognitive scientists called it "mentalese".
| drdeca wrote:
| If you mean "not what we want" for safety reasons, I think I
| agree.
|
| If you don't mean for safety reasons, I'm not sure why.
| miven wrote:
| In section 2 they briefly mention studies such as [1] that
| point out that the token outputs of a chain of thought aren't
| always entirely faithful to the responses of the models
|
| I'm not sure whether it wouldn't be more reliable to let the
| model run on latents and try to train a separate latent-
| reading explainer module that has at least some approximation
| of what we want as an explicit optimization objective.
|
| Assuming it actually is or has the potential to be better
| than CoT, from what I gathered from the paper the current
| results are mostly just more efficient token-wise.
|
| [1] https://arxiv.org/abs/2305.04388
| DalasNoin wrote:
| I was thinking abut safety reasons, but also usability. Seems
| like a pretty big difference to me if you don't understand
| the chain of thought. How faithful cot are is another
| question.
| vouaobrasil wrote:
| > Experiments show that Coconut can effectively augment the LLM
| on several reasoning tasks.
|
| It really seems like we are building a true intelligence, adding
| components to different parts of a "brain" until we have
| something rivalling the human mind. It's exceptionally dangerous
| and it's remarkable how researchers turn a blind eye to any
| possible consequences.
| AmazingTurtle wrote:
| One day researchers will be like "Oh crap what have we done"
| and "Shut it down, shut it down!!!"
| vouaobrasil wrote:
| That is true. Most poeple will just respond to immediate
| physical threats as long as they have the illusory safety net
| of modern society.
| ionwake wrote:
| Just bear in mind while they are yelling "shut it down"
| there will be a bunch of commenters with no idea whats
| happening saying that they are just over reacting
| ionwake wrote:
| agree. Someone should make sure the next ASI develops an
| extension to hide the comments in every AI thread 80% full of
| the brightest minds saying " I tried to build a react app and
| it totally failed doing it the way I wanted ".
| anon291 wrote:
| It's a good, understandable paper. The main issue with chain-of-
| thought (which I think is a solid approach, and one that needs to
| take place) is that we ourselves aren't necessarily 'trained' on
| chain-of-thought. Yes, we do learn mathematical proofs and
| reasoning at some point (usually), but most people settle on
| latent thinking without training, and switch between the two
| modes naturally. My intuition says we're missing something, but
| who knows
| empath75 wrote:
| I wonder what would happen if you just ran this on a continuous
| loop and only intermittently fed in new tokens or queried it for
| token outputs.
| SubiculumCode wrote:
| In LLMs, is there a correlation between layer depth and the
| activations correspondence to the abstract to concrete details
| continuum?
| pizza wrote:
| Yes: for eg BPE, due to how it progressively pushes compound
| tokens of already seen - hence more common - subtokens to the
| 'top' of the vocab), you can train a model to do _regression
| over vocabulary index_ for the next token from the current
| token embedding - using the same single regression model for
| all layer depths. If you plot mse of token index prediction
| versus layer depth then you can see that the mse of the
| prediction decreases steadily per additional layer. This
| appears to be because token index in eg BPE is actually fairly
| smooth and so it seems like the model is capable of localizing
| to the actual correct vocab index as depth increases, so kind
| of like a fuzzy- >discrete refinement as you go deeper in
| layers https://arxiv.org/abs/2408.13442
| fabmilo wrote:
| I like the direction of the research of working in latent space
| but feeding the last layer representation back as a first layer
| embedding feels sketchy to me. Those layers have different
| representation space.
| zxexz wrote:
| Feeding the last layer back as the input embedding has been
| done many times, e.g. Transformer-XL. The models are trained
| like this, it's not like they're taking a pre-trained Llama and
| just feeding it to itself. It's a simple, computationally cheap
| mechanism to add feedback.
| empath75 wrote:
| I read a paper not long ago that showed that deleting,
| duplicating and reordering layers doesn't actually seem to
| matter that much and it feeding back is just a kind of re-
| ordering.
| TeMPOraL wrote:
| So you're saying that feeding the last layer back to the
| first makes the model layer-order independent, or kinda
| _infinitely deep_ , if you squint? :).
| torginus wrote:
| Imo this kind of makes sense - LLMs without a feedback loop
| can learn to have one themselves by encoding information in
| the previously generated tokens.
| imtringued wrote:
| They can't, because that would increase training loss.
| The training loss acts as a gatekeeper for reasoning.
| fabmilo wrote:
| from my understanding that is what they do, see the paper: >
| We use a pre-trained GPT-2 (Radford et al., 2019) as the base
| model for all experiments. I agree the feedback is necessary,
| and the mechanism simple and cheap, but I don't think is
| optimal.
| bick_nyers wrote:
| I wonder if you would want to use an earlier layer as opposed to
| the penultimate layer, I would imagine that the LLM uses that
| layer to "prepare" for the final dimensionality reduction to
| clean the signal such that it scores well on the loss function.
| patcon wrote:
| I think of an LLM model as like a crystallised mathematical
| snapshot of intelligence... like a cell on a microscope slide, a
| _dead_ and mounted form of output from the _living process_ of
| intelligence...
|
| This paper makes me wonder whether, in a very fuzzy sense, we
| could give #LLMs access to some similarly crystallised analog of
| emotion or emotional valence, below the level of language
|
| https://x.com/patcon_/status/1866549080127893613?s=46
| threeseed wrote:
| Intelligence is more than just knowing the probabilistic
| relationship between every word.
| Rhapso wrote:
| "Intelligence" is a continuous process. Without a continuous
| feedback loop, LLMs will never be more than a compression
| algorithm we bullied into being a chatbot.
|
| OpenAi as a mega-organism might be intelligent, but the LLMs
| definitely are not.
|
| The "compressed capture of semantic relationships" is a new
| thing we don't have a word for.
| thrance wrote:
| Funnily enough, there is a mathematical link between data
| compression and AGI [1]. I believe a paper circulated some
| time ago that compared gpt2 to gzip, with interesting
| results.
|
| [1] https://en.wikipedia.org/wiki/AIXI
| 3abiton wrote:
| It's part of the process, given that the "bigger picture"
| remains in context.
| edgyquant wrote:
| Did you really just link to a post from your Twitter saying the
| same thing you did here?
| HeatrayEnjoyer wrote:
| Maybe "stasis" is more appropriate than "dead." Each new
| session is an unfrozen clone of the original mind snapshot.
| ttul wrote:
| I've been looking into using the last hidden layer of an off-the-
| shelf LLM to help my company with a classification task. The last
| hidden layer is obviously super rich in semantic information
| because it has to somehow tell the next layer how to generate the
| next token prediction. That final layer, in some respects, is
| discarding valuable context information that the final hidden
| layer encodes.
|
| I am not surprised at all that Meta was able to generate some
| positive returns by feeding the last hidden layer back into the
| model auto-regressively.
|
| The method of training they describe in the paper is really cool.
| Summarized in Figure 2, they train it with a corpus of step-by-
| step text instructions and then across multiple stages, they
| iteratively replace one of the textual steps with a last-hidden-
| layer embedding and see what the model spits out. The weights are
| then updated through cross-entropy loss as the additional text
| tokens are generated once again.
|
| So they're basically rewinding the output, replacing an
| increasing number of textual steps with hidden state embeddings,
| and playing it forward as the model gradually learns to do all of
| its step-by-step thinking using just the hidden state data.
|
| In a way, this might be how humans learn to think through
| language. Our parents teach us using words and our brain
| gradually replaces the words with thoughts until we can replicate
| the action or solve the problem ourselves without anyone guiding
| us with words.
| ttul wrote:
| Indeed, I would not be surprised if OpenAI one day admits that
| the `o1` model uses the last hidden layer (or some other
| intermediate layer) to feed the "thought process" that you can
| watch as it "thinks" about the answer. I suspect that they may
| take the last hidden layer and feed it back into the front of
| the `o1` model while also feeding a separate, likely much
| smaller LLM that generates the "thought process" as language
| tokens.
|
| In this manner, the model makes use of the rich semantic
| information encoded at the last hidden layer while informing
| the user via an extraction of that hidden layer specifically
| tuned to generate human-legible concepts such as, "I'm
| considering the impact of converting the units from kilograms
| to pounds," or whatever.
| pedrovhb wrote:
| That's certainly possible, but it reminds me a bit of a
| similar thing I've seen in their UI that rhymes in a way that
| makes me think otherwise. In the code interpreter tool, you
| have a little preview of the "steps" it's following as it
| writes code. This turns out to just be the contents of the
| last written/streamed comment line. It's a neat UI idea I
| think - pretty simple and works well. I wouldn't be surprised
| if that's what's going on with o1 too - the thought process
| is structured in some way, and they take the headings or
| section names and just display that.
| impossiblefork wrote:
| I don't think it does, because from this paper this kind of
| backfeeding is apparently quite difficult to train.
|
| I've said it before, but I think it's just something like
| Quiet-STaR, but simplified. They have a bunch of question
| answer pairs, many of which are difficult. They generate a
| lot of tokens from the question (let's say, 3x the length of
| the expected answer), summarise whatever is generated and
| reinforce whenever it generates the right answer.
|
| I don't think o1 is something complicated.
| psb217 wrote:
| "...because it has to somehow tell the next layer how to
| generate the next token prediction." -- This isn't actually
| true in the case of transformers. Features in the final TF
| layer at time t in a sequence do not depend on the features in
| the final TF layer at any other time step. Recurrence in
| transformers is done "depthwise" via "causally masked"
| convolutions. Final layer features at time t can depend on
| penultimate layer features at time t-1, but not on final layer
| features at time t-1.
| danielmarkbruce wrote:
| you are misunderstanding what the person is saying. They are
| saying the final hidden layer outputs a vector which has all
| the information that decides the logits which decide the
| probabilities of _each token in the entire vocabulary_. Ie,
| it is storing a lot of information.
| WiSaGaN wrote:
| This is intriguing. When I learned that a lot of people do not
| have inner monologue, I was fascinated by the fact that people
| can differ on such seemingly fundamental way of being. Maybe
| those who have it just have a "tee" that pipes into words.
| throwawaymaths wrote:
| > using the last hidden layer
|
| iirc this is a well supported task iirc called "classification
| head" instead of "language modeling head" in case anyone else
| wants to do this as a fine-tuning project
___________________________________________________________________
(page generated 2024-12-10 23:00 UTC)