[HN Gopher] The Eleuther AI Mafia
___________________________________________________________________
The Eleuther AI Mafia
Author : swyx
Score : 77 points
Date : 2023-09-03 06:53 UTC (16 hours ago)
(HTM) web link (www.latent.space)
(TXT) w3m dump (www.latent.space)
| ftxbro wrote:
| > Primarily Discord, pseudonymous, GPU-poor volunteer community
| somehow coordinating enough to train >10B, OPT/BLOOM-competitive
| models
|
| what are the discords for Eleuther and RWBY are they public?
| hiddencost wrote:
| Give it a Google? The discord logo is at the bottom of the
| page.
| polygamous_bat wrote:
| Question: I only ever hear how RWKV is threatening the dominance
| of transformers... from the RWKV folks, while everyone
| independent is happily using transformers. Is there any substance
| to the above claim that's been independently verified?
|
| Not an NLP person, so apologies in advance if I've missed any
| obvious papers or references.
| NLPaep wrote:
| RWKV performs poorly in practice.
| capableweb wrote:
| You'd like to expand that a little, suggestively how you know
| this? Or you're expecting rebuttals that amount to "No, it
| doesn't"?
| NLPaep wrote:
| Benchmark results in publications show it being confused in
| chat--like settings and answering questions incorrectly
| allisdust wrote:
| Occam's razor tells us that if it's a great
| architecture/technical breakthrough, it would have taken
| the world by storm by now. Similar to the original
| transformer paper and model. Since 2017, the only
| successful models are variations of transformers. RNNs are
| no where in the picture.
|
| Simply believing a architecture is superior doesn't make it
| so. Nothing converges and performs as good as a model with
| attention in both training and inference. The difference is
| night and day.
| capableweb wrote:
| > Occam's razor tells us that if it's a great
| architecture/technical breakthrough, it would have taken
| the world by storm by now. Similar to the original
| transformer paper and model.
|
| I think you're making the opposite case here without even
| realizing it yourself. The Transformer was proposed in
| 2017, with the softmax-based attention being proposed in
| 2014 but it wasn't until 2023 when GPT-3 et al took the
| world by storm that people started really using the
| Transformer as it is today.
|
| The timeline going from being proposed to being used in
| production models also took multiple steps, with things
| improving with each step. It's a iterative process, not
| "dump -> done".
|
| > Simply believing a architecture is superior doesn't
| make it so
|
| I agree. But also I agree the opposite, that just because
| you believe it's not superior, doesn't make it so. If you
| claim "RWKV performs poorly in practice", I expect you to
| have something to back that up, definitely more than
| "Just because you believe it's superior doesn't make it
| so"
| adw wrote:
| Transformers have been industrially ubiquitous for a lot
| longer than GPT-3, initially in machine translation.
| mistrial9 wrote:
| "a lot longer " quite the statement !
|
| reading here says "the behavior and qualities of these
| large models is poorly understood"
|
| prove me wrong?
|
| ps- I agree that BERT-related models have been "wildly
| popular in NLP for years now"
| ianbutler wrote:
| I've now presented on this model and worked with it. It's
| not phenomenally better than other models but has some
| attractive scaling and speed properties that may or may
| not be worth the trade off relative to its draw backs
| which I detail in my other comment in the GP's thread.
| danieldk wrote:
| This is really a strange take. Encoder-only transformers
| like BERT and RoBERTa have been wildly popular in NLP for
| years now, replacing pretty much every model that came
| prior to it and beating pretty much all traditional NLP
| benchmarks (tagging, parsing, etc.).
| simonster wrote:
| Also BERT was integrated into Google search in 2019 (see
| https://blog.google/products/search/search-language-
| understa...)
| lhl wrote:
| I've been catching up on the papers, and the mechanics of RWKV
| are interesting, but I'm more of a hands on guy and I played a
| bunch with RWKV earlier this year when its quality was
| comparable with the early LLaMA models of similar size and it
| was significantly faster (its CPU inference was comparable to
| the latter's GPU inference speeds 6 months ago), but I feel
| like those advantages have largely dissipated.
|
| With 4-bit quants, and the optimizations for batch=1 with
| llama.cpp, Exllama, MLC on the edge side, and on continuous
| batching w/ vLLM and the like on the high throughput side, I
| think performance (in terms of speed) is not as pressing, at
| least while people are pushing hardest on the capabilities
| front. Meanwhile, the quality of the best Llama 2 (and even the
| last-gen of Llama 1) models have blown by the existing RWKV
| models (although I'm definitely interested in kicking the tires
| on RWKV-5 when it's out). Also, I recently stepped back into
| RWKV world to check up on what was going on, and I was actually
| a bit shocked at how night and day the tooling/ecosystem is
| now. While it's not all roses the open LLM side, there's enough
| critical mass where the evolution has been ... extremely rapid,
| to say the least.
|
| I think over the next few years, obviously there will be
| architectural improvement that succeeds the current crop of
| dense transformer LLMs (one might argue that's already
| happening piece by piece), and I'm happy there are experiments
| like RWKV going on, so I guess we'll just have to see in terms
| of what scales and what doesn't.
| emadm wrote:
| It has good potential and nobody ever thought that RNN's would
| be able to scale as they have led by BlinkDL.
|
| Well we did at Stability AI which is why we provided and scaled
| compute for it.
|
| They have the potential for far better edge inference and also
| benefits longer context windows.
|
| There are other architectures out there too we are supporting,
| I would be surprised if the current transformer LLM paradigm is
| what we use in a few years.
| polygamous_bat wrote:
| With all due respect, I consider you as one of the "RWKV
| folks" I mentioned in the initial comment :) (assuming you
| are Emad Mostaque?)
|
| > They have the potential for far better edge inference and
| also benefits longer context windows. ... I would be
| surprised if the current transformer LLM paradigm is what we
| use in a few years.
|
| What are the studies/evidence that convinced you? Given the
| current landscape, the last claim is quite bold.
| ianbutler wrote:
| Personally, the RNN "mode" runs substantially faster with
| similar quality to other models of the same size. RWKV is
| limited by compute and thus model size at present which makes
| it hard to suggest how it will stack up against models like
| falcon and llama2 in their larger sizes.
|
| There are also trade offs around their specific attention
| mechanism. It's approximate and consists of an exponential
| decay across time steps and some constant that represents the
| magnitude of the attention at that time step. The decay means
| for long running contexts it in theory will loses some of the
| ability to recall nuance you see with perfect attention on
| longer sequences. This isn't a specific problem of RWKV, but
| really any decay based attention.
|
| A perk of the formulation is the attention computation is
| linear in time complexity and only requires one additional
| serial scan at the end, so you get nicer scaling properties
| than quadratic perfect attention for the trade off in lossy
| attention.
|
| The examples in the original paper don't show this to be a
| problem, but they also show much shorter sequences. In the real
| world its a toss up, I'd expect it to be faster but more lossy
| for longer context windows.
|
| TLDR: It has some teeth, but I think saying it's threatening is
| wishful thinking to some degree. Good model, looking forward to
| seeing larger sizes.
| blueblisters wrote:
| How are EleutherAI and Carper funded?
| _delirium wrote:
| A majority of the Carper folks (though not all) are in
| academia. Cambridge, Brown, Georgia Tech, NYU, etc.
| emadm wrote:
| Carper is technically a Stability AI lab and community
| https://twitter.com/carperai
|
| Core are full time Stability AI
| capableweb wrote:
| Eleuther:
|
| > Core Weave, HuggingFace, Google TRC, Nat Friedman, Lambda
| Labs
|
| https://www.eleuther.ai/about
| sverona wrote:
| Now that there are comments here I can express the melancholy of
| not being Stella Biderman, whom I admire from afar.
|
| I wish there were something I could do in AI alignment, I really
| do, but it's beyond me.
| ShamelessC wrote:
| I realize you say "not authoritative", but just to clarify -
| LAION isn't the creation of Emad Mostaque or Stability AI. Emad
| is/was a funder who showed up well after its inception. The same
| applies for his work with Eleuther. Although some members of both
| LAION and Eleuther did go on to become Stability employees. LAION
| was also partially born from _frustrations with_ Eleuther's TPU
| culture and inability to ship anything (at the time, and
| excepting Katherine Crowson's great work) for text-to-image work.
| mkl wrote:
| Tangent: Why do some sites (like this one) disable pinch zooming
| on mobile? It makes graphs and pictures illegible, which is
| incredibly frustrating.
| [deleted]
| d4rkp4ttern wrote:
| Another tangent -- I don't get why people writing technical
| blogs use Substack, where support for math and syntax
| highlighting is nonexistent (math) or minimal (code). Something
| with Hugo plus PaperPad theme would be far better. I suppose
| Substack is popular because of the newsletter features.
| d4rkp4ttern wrote:
| In fact I really like the look of the Mkdocs Material system
| for blogs -- https://squidfunk.github.io/mkdocs-
| material/blog/
|
| And they have integrations with comment systems like giscus.
| nextworddev wrote:
| It's mainly for discovery I believe.
| capableweb wrote:
| Same reason anyone would use a hosted service vs setting it
| up themselves, they don't want to deal with the hassle. For
| many of us, it's trivial, sometimes because we've done it
| before, but for others, it's a huge hill to climb, which
| seems unnecessary if all they want to do is publish some
| content.
| primordialsoup wrote:
| I can do it myself and it's not a huge hill to climb, but I
| don't necessarily have the time to do it. There is also the
| time that needs to be dedicated towards maintenance
| (upgrades etc.).
| d4rkp4ttern wrote:
| This makes sense for people who don't code, but I see
| plenty of developer oriented bloggers who use Substack,
| which is puzzling to me. I also don't know why Substack
| doesn't get with the program and just support MathJax and
| colored syntax highlighting already.
| capableweb wrote:
| > This makes sense for people who don't code, but I see
| plenty of developer oriented bloggers who use Substack,
| which is puzzling to me
|
| No, it makes sense for any person with any skill set.
| Just because you are a developer doesn't mean you want to
| setup everything yourself. We use plenty of tools to make
| our own experience better and more focused, based on what
| we want to do.
|
| If I'm a programmer, and I want to publish blog posts,
| sometimes I don't want to care about the setup that makes
| me able to publish blog posts, I just want to post
| something.
|
| > why Substack doesn't get with the program and just
| support MathJax and colored syntax highlighting already
|
| This, I don't know. Should have been done ages ago.
| AndrewKemendo wrote:
| I haven't been on the Eleuther Discord in a year or so but the
| last time I was there it was bustling and thriving on some really
| hard topics
|
| So while I'm not sure that the RWKV movement will see more
| widespread adoption than the corporate LLM implementation
| movement - I think there's probably still more goodness to come
| out of Eleuther community
| bogwog wrote:
| What a shame that all that discussion is locked behind some app
| and not accessible on the public internet.
| User23 wrote:
| Discord is a cancer. While I don't understand the appeal,
| I've been around long enough to no longer be surprised by
| people sacrificing control of their information for trivial
| short term gain.
| Kiro wrote:
| What do you mean? Making my private content accessible on
| the public internet is the opposite of control.
| pphysch wrote:
| Discord started out as a far better
| ventrilo/teamspeak/Skype for gamers but then larger
| communities started using it as their main "website"
| tbruckner wrote:
| The arrows in their diagram make no sense. Stability funds
| everything. All arrows should be coming out of Stability.
| ftxbro wrote:
| least capitalist hacker news comment. do the arrows mean flow
| of capital? i don't think so. i think it means flow of people
| or cultural influence or leadership
| capableweb wrote:
| This doesn't seem to be true, unless you don't trust what
| Eleuther themselves are saying.
|
| > The work we do would not be possible without the generous
| support of our donors and other sponsors: Core Weave,
| HuggingFace, Google TRC, Nat Friedman, Lambda Labs
|
| https://www.eleuther.ai/about
| emadm wrote:
| We fund Eleuther AI too as it says there (but made sure it
| can run independent when 501(c)3 set up) and provided compute
| for RWKV.
|
| Conjecture was meant to be alignment so can't really support,
| stability scaled resource provision.
| tbruckner wrote:
| Which one of those donors are on their board?
___________________________________________________________________
(page generated 2023-09-03 23:02 UTC)