[HN Gopher] AnyGPT: Unified Multimodal LLM with Discrete Sequenc...
___________________________________________________________________
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Author : tkgally
Score : 80 points
Date : 2024-02-21 13:45 UTC (9 hours ago)
(HTM) web link (junzhan2000.github.io)
(TXT) w3m dump (junzhan2000.github.io)
| mdrzn wrote:
| We're clearly heading towards a "Generalist Multimodal Large
| Language Model" that autonomously selects the appropriate
| specialized LLM for any given task, rather than requiring us to
| switch between multiple LLMs. The combination of a mixture of
| experts approach and multimodality appears to be the way forward.
| Very excited for the future.
| ilaksh wrote:
| I don't think it's accurate to call them language models
| anymore actually. Large Multimodal Model?
| phh wrote:
| You're right, that calling them language model is no longer
| accurate. But "Language Model" has a precise signification
| (which is that it tries to create a perfect representation of
| the language). Saying "multimodal model", you're not really
| saying what does it try to represent. "World Model" or "Human
| Model" would be more accurate at this point
| t_mann wrote:
| World model sounds too much of physics than it deserves
| imho. If you want to name it after what's being modeled
| (which it doesn't need to be, it could also be named after
| the methods used eg), why not just call it something like
| general token model?
| throwanem wrote:
| What they model is meaning. I'd call them "semantic
| models".
| Smith42 wrote:
| "Large Observation Model" has a nice ring to it
| ShamelessC wrote:
| I'm fond of Large Sequence Model, which I've seen used a bit
| recently. If your data can be defined as a sequence of unique
| symbols, actions, letters, tokens, patches, whatever - a
| transformer is a great way to model it.
| cs702 wrote:
| I'm partial to Large World Model (LWM).
|
| Large Sequence Model (LSM) is OK too.
| kristjansson wrote:
| As long as the underlying representation is a sequence of
| (embeddings of) tokens , and the training depends on next-
| token prediction, they should still be called language
| models. The 'language' part of 'language models' is about the
| architecture, not the modalit(y/ies).
| CuriouslyC wrote:
| Is it a "language" model though, if you have a multimodal
| autoencoder that converts input into concept tokens, and
| you do sequence extension in concept space?
| kristjansson wrote:
| Unambiguously, yes.
|
| To elaborate a bit, that's effectively what happens in a
| natural-language-only LLM. Text tokens are immediately
| mapped to wide embeddings, which are processed by a
| transformer, producing another embedding, which is
| decoded back to a discrete token. To a first
| approximation, unified multimodal models substitute f:
| text -> [embedding] with g: image -> [embedding,] and act
| on a sequence like [*f(t), *g(i), ...]. So... the
| multimodal model maps text/image parts to wide
| embeddings, which are processed by a transformer
| producing another embedding, which is decoded back to a
| discrete text/image part. Which sounds somewhat similar
| to the former :).
|
| Of course the function g is a bit more complicated, and
| decoding takes more work, but the underlying machinery is
| unchanged. The ability to handle multi-modal
| inputs/outputs uniformly is exactly what's so exciting
| about work like this!
| jasfi wrote:
| Apparently used in GPT-4 and coming in Gemini 1.5.
| kristjansson wrote:
| TFA is the exact opposite of that though? It's a single model
| that acts on a single representation, which is derived from
| (and mapped back to) different input (output) modalities.
| ilaksh wrote:
| I feel like this type of capability or architecture might be the
| future of interactive agents. The quality of the voices leaves a
| little to be desired, but otherwise it seems very powerful.
| thfuran wrote:
| Given how quickly we've been churning through sota
| architectures in the last decade or so, I'd not bet on anything
| we currently have being "the future"
| cs702 wrote:
| The key takeaway for me:
|
| Whether data is continuous or discrete, no matter its modality
| (text, video, music, etc.), we now have an array of proven
| methods for representing it with _discrete_ tokens, enabling us
| to use existing sequence modeling architectures (Transformers,
| linear RNNs).
|
| We live in interesting times!
| justanotherjoe wrote:
| What's with the dummy github page? Anyway. This feels like the
| right step forward. Just like OpenAI, i have near religious faith
| in the transformers architecture. The question is how these
| modalities can work together better.
___________________________________________________________________
(page generated 2024-02-21 23:01 UTC)