[HN Gopher] How to make LLMs go fast
___________________________________________________________________
How to make LLMs go fast
Author : tosh
Score : 158 points
Date : 2023-12-22 12:09 UTC (10 hours ago)
(HTM) web link (vgel.me)
(TXT) w3m dump (vgel.me)
| behnamoh wrote:
| I wonder if the transformer will remain the de facto arch for
| LLMs in a couple years. We already have Mamba, RWKV, etc. which
| are huge improvements over transformer in terms of memory
| requirement and speed. I wonder why there's still so much work
| (and hacks and workarounds) on transformer to make it go faster
| and consume less VRAM whereas the same effort could be spent on
| other architectures that solve some of those problems
| fundamentally.
| __loam wrote:
| The best technology doesn't necessarily win. First to market
| matters, especially when models on the scale of gpt-4 cost
| 100mln to train.
| evilantnie wrote:
| "best" is subjective, but I do think the industry will
| eventually converge onto an architecture that is
| significantly more cost effective than current state of the
| art. Regardless of who is first to market, everyone is
| incentivized to continue down this path with their research
| on improving LLM performance.
| realusername wrote:
| While the first mover advantage exists, I don't think it's a
| sufficient moat in the long term.
|
| OpenAI's moat is the high cost of training as you mentioned
| but this might be obsolete in a few papers down the road.
|
| They themselves realized this by trying to turn it into a
| platform but I don't think it's enough.
| andy99 wrote:
| The speed of transformers is memory bandwidth limited. I
| _think_ that it 's possible to speed them way up with different
| chip architectures, and I know there are lots of people working
| on this e.g. https://etched.ai making asics and
| https://untether.ai making chips will co-located memory and
| compute cells. So unless some architecture really starts
| beating transformers badly on language tasks, I think the speed
| problem is going to disappear as silicon architectures adapt.
| tome wrote:
| Yup, the company I work for has the world's fastest LLM
| appliance, based on a hardware design co-locating memory and
| compute. It requires compilation innovation as well in a sort
| of hardware-software codesign. We don't see transformer
| models as being an impediment in terms of memory requirements
| or speed in the near future.
|
| [I would be happy to say more but I just got a top-level
| comment flag killed, presumably because they though I was
| advertising, so I won't mention the company name.]
| andy99 wrote:
| Are you guys hiring? I'm actually looking for a job in this
| industry.
| tome wrote:
| Yes we are! (I put links in my profile because I'm
| worried if link here or mention the name I'll be flag
| killed again.)
| andy99 wrote:
| Awesome, thanks! Is there someone there you could connect
| me with for an introductory chat? I put my contact info
| in my profile, I completely understand if it's not
| possible. Thanks!
| tome wrote:
| Yes, sure! I've emailed a recruiter to introduce you.
| Thanks for reaching out and showing interest.
| benchess wrote:
| If I understand correctly, Groq chips have 220MB SRAM and
| the next best level is DDR4? How many chips are needed to
| run Llama2-70B at those speeds?
| tome wrote:
| Cool that you know the tech specs of the GroqChip! Yes,
| that's right, 220 MB of SRAM per chip. I think the demo
| where we first broke 200 tokens / sec was running on 1
| GroqRack, so 64 chips. The live public demo that's
| currently running at 275 tokens / sec I think might be
| running on two GroqRacks, so 128 chips. I'm not certain
| of either of these figures so please don't quote me! But
| those are the right ball-park.
| pavelstoev wrote:
| There is still so much performance to gain by applying
| software-based optimizations for superior performance of
| transformer LLMs on GPUs. From ~ 30% GPU utilization to over
| 90% is possible. I wrote a white paper on this, page 5 and 6:
| http://tinyurl.com/uujr3z4f
| beoberha wrote:
| There's hacks and workarounds because billions of dollars have
| been poured into transformers by the biggest tech companies in
| the world.
|
| Your assumption that the same effort could be given to MAMBA or
| RWKV isn't necessarily true. I'm sure there's research arms
| looking to see if they scale to the level OpenAI or Google have
| their transformers at now, but they're still very much in their
| infancy.
|
| This all ignores the risk of them not panning out and the lost
| time not focusing some energy on transformers.
| taneq wrote:
| The current best-in-class performance is from transformer
| models so of course research is focusing on this. You're
| creating a false dichotomy between "work on transformers" and
| "work on something new and fundamentally better." All of the
| current work on optimizing transformer architectures is
| exploring and better understanding the space, and so it IS
| leading us to better architectures. This is the most fertile
| ground for new insights that will outperform the current
| approaches.
|
| New technologies aren't generally just created out of whole
| cloth by some genius, they're build up of layers of incremental
| improvements, each of which is modest in its own right but
| which when taken together are groundbreaking.
| logicchains wrote:
| Theoretically transformers are fundamentally more powerful than
| stuff like Mamba and RWKV because they don't "forget"; the
| newest token can attend perfectly to the oldest token if it
| wants. Mamba and RWKV on the other hand compress the old state,
| based on information available at the time of compression. If
| it turns out the new token wants some information from the old
| token that during the compression was excluded due to the
| compression thinking it wouldn't be needed, the new token has
| no way to access it.
| visarga wrote:
| Yet this all-to-all approach of transformers is bottlenecked
| on compute, while Mamba can take in a context length of a
| million tokens, that might have a positive effect on long
| range tasks.
| maleldil wrote:
| What use is to have infinite context if new tokens can only
| access a lossy compression of past tokens?
|
| There is definitely a use for this kind of model, but this
| also shows why Transformers are still the main architecture
| we use today.
| seo-speedwagon wrote:
| I know almost nothing about this space but "lossy
| compression of millions of past tokens" _feels_ a lot
| more like how actual human memory works than "perfect
| access to a small number of recent tokens".
| thentherewere2 wrote:
| RNNs and LSTMs from the past did this as well (but cannot
| be trained in parallel as each token has to be compressed
| sequentially). Transformers ate their cake.
|
| Newer methods are going back to similar concepts but
| trying to get past previous bottlenecks given what we've
| learned since then about transformers.
| throwup238 wrote:
| _> feels a lot more like how actual human memory works_
|
| Beware of feel-good traps like this.
|
| If you were to map the human connectome to a
| computational neural network down to the ion channel,
| it'd be _at least_ 500 quadrillion parameters*. That 's
| at least 5-6 orders of magnitude beyond what is currently
| possible with SOTA ML which means that even if the human
| brain was 99% devoted to compressing those tokens, that
| 1% that could actually do work with them is still a
| thousand times bigger than GPT4. There be emergent
| dragons.
|
| * This is a fascile argument to begin with since
| biological neuron signals aren't quantized and ion
| channels are far too complex to map to a single static
| parameter
| llamaInSouth wrote:
| Fast is great but uncensored is better
| Der_Einzige wrote:
| They had a whole section on guided decoding which can be used
| to among other thing break censorship/alignment efforts.
| madeofpalk wrote:
| Just run/train it yourself.
| llamaInSouth wrote:
| This one is supposed to be uncensored: https://huggingface.co
| /cognitivecomputations/dolphin-2.6-mix...
| prestonlibby wrote:
| For anyone else who wishes to do so, Fireship has a low-
| barrier video[1] on training the uncensored model in the
| sibling comment (dolphin mistral) for local use.
|
| [1] https://www.youtube.com/watch?v=GyllRd2E6fg
| kristiandupont wrote:
| What do you want to use it for? Out of all the things I use
| LLM's for, there isn't one situation where I care about
| political correctness or whatever.
| throwup238 wrote:
| Court documents. One mention of rape or murder and there's a
| 50:50 chance GPT/Claude/Bard will just shut down, when all I
| want is to extract the entities mentioned in transcripts or
| opinions.
| fuddle wrote:
| Thanks for the blog post, I also enjoyed their previous post on
| making a transformer by hand: https://vgel.me/posts/handmade-
| transformer/
| altruios wrote:
| LLM's generate 'the next token' require a serialization of data,
| and prevents 'too much' parallelization.
|
| I wonder if some sort of diffusion hybrid is being worked on.
|
| Something that approximates a complete text answer, but
| 'increases the resolution' of that text over time.
|
| One key benefit to a 'shot-gun/top-down/all-at-once' approach as
| opposed to a 'guess the next token => add and repeat' is the
| ability for the tokens at the end of the text (a twist in a
| surprise mystery story) have a direct effect on the beginning of
| that story. which (as far as I know) is not possible in current
| LLM's due to their architecture.
|
| how self attention and positional encoding would work in a
| diffusion model... that's the question I have if that could even
| work...
|
| to be clear, I don't mean stable diffusion rendering text as an
| image. I mean diffusion run on raw (random) text: turning that
| into a legible response.
| thatguysaguy wrote:
| People have been trying to get non-autoregressive generation
| working for years. There are some methods which work okay but
| they're always behind AR generation. And yeah they do look very
| diffusion-esque.
|
| One of the issues is that you'd need to be really sold in this
| being a better paradigm before spending the money to pretrain a
| huge non-AR model.
| altruios wrote:
| I figured the raw approach of diffusion-esque was tried
| (without techniques lifted from newer LLMs {chatGPT is only a
| year old}). I am thinking that those attempts were made
| largely before LLM's were popular. Self attention and
| positional encoding are the claimed 'secret sauce': are there
| any models (or attempts) anyone can point to that try to
| combine SA+PE into a diffusion-like (non-autoregressive)
| model.
|
| What are some models that 'work okay'?
| thatguysaguy wrote:
| Self-attention and positional embeddings are ancient at
| this point (i.e., more than 5 years old). Here are a couple
| examples of methods: https://arxiv.org/abs/2112.06749
| https://arxiv.org/abs/2205.12558
___________________________________________________________________
(page generated 2023-12-22 23:00 UTC)