[HN Gopher] Understanding Large Language Models - A Transformati...
___________________________________________________________________
Understanding Large Language Models - A Transformative Reading List
Author : mariuz
Score : 62 points
Date : 2023-02-11 17:34 UTC (5 hours ago)
(HTM) web link (sebastianraschka.com)
(TXT) w3m dump (sebastianraschka.com)
| mensetmanusman wrote:
| Did anyone predict the unreasonable effectiveness of LLMs?
|
| We seemed to have stumbled upon something quite big as humans...
| loveparade wrote:
| I worked in one of the big labs when the first large models
| came out and I can pretty confidently say that nobody in the
| field predicted this. Sure, there were always people who said
| "let's make models bigger because why not, we have the infra
| and it'll be a good paper" but nobody expected them to become
| _this_ good just by being bigger and using more data. The
| consensus was that they 'd hit a ceiling of what they can do
| much sooner.
| tomr75 wrote:
| aren't humans trained on massive data sets, why not expect
| the same for digital intelligence?
| soraki_soladead wrote:
| Only some model architectures continue to get better as you
| pump in more data. Transformers and their variants have
| this property more so than prior architectures.
| tough wrote:
| Or maybe language just isn't that complex at all?
|
| Same with pixels?
|
| Dunno
| GeorgeTirebiter wrote:
| Perhaps Monica Anderson understood early? https://experimental-
| epistemology.ai/the-red-pill-of-machine...
| corbulo wrote:
| We stumbled upon a mirror of our own cognitive bias.
|
| Another chapter in the era of Narcissus and Echo.
| auggierose wrote:
| Yeah, I was looking for a book about this topic, but there
| doesn't seem to be anything out there except for research
| articles, it is just too new.
|
| When I look at the descriptions of the papers, it sounds
| incredibly complicated, and at the same time incredibly trivial.
| Clearly this whole domain is not well enough understood yet to be
| explained properly. Or maybe there is no interest yet in clear
| and concise explanations.
| CongoDan wrote:
| That's how blockchain was a few years back. Now everyone and
| their uncle has a "how blockchains work" article or video
| corbulo wrote:
| And yet I feel like I know less at this point than I did
| then.
| KHRZ wrote:
| The description of the attention mechanism of GPT architectures
| and a couple of examples can be very brief. Then you have to
| supply your imagination and realize that the model simply found
| a whole bunch of very effective such attention measures by
| itself, which are all computed for every query, and they could
| be anything from the straight forward examples, or some more
| clever abstract types of attention. I think we'll need some
| more descriptions of what the more important attention measures
| it comes up with really are, to understand better.
| qup wrote:
| > Large language models have taken the public attention by storm
| - no pun intended.
|
| As far as I can tell, no pun made, either!
| marginalia_nu wrote:
| "Attention" is the would be pun, I think.
| xwowsersx wrote:
| The pun is in "attention" because GPT uses "attention" to weigh
| each input token and comes up with an attention score between
| whatever token is currently being generated and _all_ the input
| tokens then it 'll take those scores to determine what weight
| each contributes to the output. Something along those lines...
| I'm no GPT expert.
| 0z8 wrote:
| To make a pun in this domain, attention is all you need!
___________________________________________________________________
(page generated 2023-02-11 23:01 UTC)