[HN Gopher] Megatron-Turing NLG 530B, the World's Largest Genera...
___________________________________________________________________
Megatron-Turing NLG 530B, the World's Largest Generative Language
Model
Author : selimonder
Score : 66 points
Date : 2021-10-11 13:13 UTC (9 hours ago)
(HTM) web link (developer.nvidia.com)
(TXT) w3m dump (developer.nvidia.com)
| bane wrote:
| What's really interesting is that these models are using some
| non-trivial portion of all easily accessible human writing -- yet
| humans learn language really well with significantly less input
| data. What's missing in the field to replicate human performance
| in learning?
| dlkf wrote:
| Humans use language to accomplish tasks in their environment -
| establishing relationships, making deals, coaxing others, etc.
| By contrast, all neural language models do is predict the next
| word as a function of the previous word. So far, these language
| models have nothing at all to do with language learning.
| They're only valuable insofar as they advance downstream
| engineering tasks like machine translation.
| visarga wrote:
| Imagine you lived in a black room and all you can see is a
| buffer of text scrolling in front of you. Nobody explains what
| the symbols mean. You don't remember anything medium term, you
| can only access a short snippet of text at a time and form long
| term memories gradually. You just look and predict what will
| come next. Who could become an intelligent person being raised
| in these conditions?
|
| So they are missing 2 years worth of visual, auditory, tactile
| and other modalities (grounding), having direct access to
| change their environment (embodiment) and being part of our
| society or an AI society (social).
| ipsum2 wrote:
| One dimension that isn't being considered is that humans have
| had billions of years of evolution, while neural networks are
| essentially a blank slate.
| xnx wrote:
| This reminds me a little bit of the early 2000's where search
| engines would list the number of indexed pages on their homepage.
| For language models, does large = good? I'm guessing the quality
| of the corpus matters as much.
| ewheeler wrote:
| Maybe? The Scaling Hypothesis[1] suggests that greater
| capabilities of intelligence may emerge from scaling up
| 'scalable architectures' to large sizes. GPT-3 exhibits 'meta-
| learning' capabilities that GPT-2 did not (like learning how to
| sum numbers)--probably just because its a 100x larger version
| of GPT-2.
|
| [1] https://www.gwern.net/Scaling-hypothesis
| thunderbird120 wrote:
| >For language models, does large = good?
|
| The short answer is yes, the long answer is it's complicated.
|
| You could actually think of these models as a type of indexer
| because, at their heart, what they are doing is memorizing the
| training data and storing it in such a way that incomplete
| samples can be used as keys to extract complete samples. The
| magic happens because the models themselves (even the 100+
| billion parameter ones) are nowhere near complex enough to
| actually store all of these possible key value pairs. Instead,
| the model has to compress its representation of the data which
| leads to generalization. Larger models can model more
| complexities which leads to better performance as long as your
| training dataset is sufficiently large and varied.
| bpiche wrote:
| I'm not sure it moves the needle on NLU/classification tasks
| very much, compared to models with many fewer parameters. But
| it does seem to make the NLG better, which is what Microsoft
| seems obsessed with lately.
| captn3m0 wrote:
| Interesting that books3 and The Pile are among the largest corpus
| used for training - both with copyright concerns.
| trash3 wrote:
| How has the previous largest model, gpt3, generateda value? How
| much better is this model at those tasks?
| miket wrote:
| https://en.wikipedia.org/wiki/Wu_Dao
| bpiche wrote:
| Wonder how much compute it would cost to train this thing, if you
| weren't Nvidia..
| posharma wrote:
| This is great. Now, how do we inference these models
| economically? It appears there's some kind of competition to
| train larger and larger models, but the inferencing side of the
| story seems to be neglected?
| dwohnitmok wrote:
| Model inference is actually comparatively very cheap. If you
| have the resources to train a model, you most definitely have
| the resources to run it.
| shock-value wrote:
| Does that hold as the workload scales up? E.g. could this or
| similar models be used as part of a general-purpose search
| engine whereby (at least) one inference is completed per
| unique search? Aside from computation, I know these models
| consume an intense amount of memory -- would that scale
| horizontally easily / economically? Would it need to?
| manquer wrote:
| Not necessarily, you train once , you run inferences billions
| of times maybe. The compute required could be beyond your
| resources.
| buffington wrote:
| When you say "inference", do you mean "interface", or is
| "inference" an ML term I'm not familiar with?
| igorkraw wrote:
| It's a ML term, inference basically means using the
| probability model you learned to draw "inferences" about a
| piece of data. In this context, it means giving the language
| model some context and using some method (either arg max
| sampling or something more sophisticated like beam search) to
| do what amounts to statistical auto implemention on it. As
| you might imagine, doing this with 530 GB of data at speed is
| quite energy intensive, even though there are things you can
| do to compress the model (distillation, pruning,
| compression/discretization) and specialised inference
| hardware.
|
| Technically there is some very specific meaning to inference
| vs. prediction, but it's been heavily overloaded with meaning
| by now
| sanity31415 wrote:
| First you train a model then you use it, "inference" is a
| fancy word for using the model.
| petters wrote:
| Training data has 0.339T tokens, less than the number of training
| parameters. A model like that could store all of the training
| text with 100B+ parameters left for computation.
| visarga wrote:
| But then you try to predict the next token on a completely
| unseen piece of the corpus and fail miserably if all you do is
| store the training data.
| thewarrior wrote:
| Maybe that's what it's doing under the hood.
| knuthsat wrote:
| For some reason this issue with model having insane amounts of
| weights but training data being small is not something that is
| an issue for modern NNs.
| lucidrains wrote:
| https://arxiv.org/abs/2109.02355
| macrolime wrote:
| Will anyone outside of Nvidia be able to access it? GPT-3 at
| least has an API.
| thenightcrawler wrote:
| hoping so!
| cs702 wrote:
| So we now have models with 0.5 trillion parameters, each the
| weight of a connection in a neural network.
|
| Trillion-parameter models are surely within reach in the near
| term -- and that's only within two orders of magnitude of the
| number of synapses in the human brain, which is in the hundreds
| of trillions, give or take. To paraphrase the popular saying, a
| trillion here, a trillion there, and pretty soon you're talking
| really big numbers.
|
| I know the figures are not comparable apples-to-apples, but
| still, I find myself _in awe_ looking at how far we 've come in
| just the last few years, to the point that we're realistically
| contemplating the possibility of seeing dense neural networks
| with hundreds of trillions of parameters used for real-world
| applications in our lifetime.
|
| We sure live in interesting times.
| EvgeniyZh wrote:
| Not all weights are born equal, different paradigms allow more
| parameters while being less parameter-efficient, e.g.
| https://openreview.net/forum?id=TXqemS7XEH
| fspeech wrote:
| A 10 trillion parameter model was mentioned here:
| https://mobile.twitter.com/ethancaballero/status/14458268620...
| ImprobableTruth wrote:
| That's MoE.
| rolisz wrote:
| Mixture of Experts, aka not all 10 trillion parameters are
| used at the same time, just a subset that is an "expert" on
| the "task at hand".
| codeulike wrote:
| Unless we have misunderstood neurons, and microtubules are the
| fundamental computational unit in which case we are out by an
| order of magnitude
| eximius wrote:
| There was a result recently of modeling an organic neuron
| with 1000 digital neurons.
|
| And even if that result was perfect modeling of the neuron,
| that assumes perfect and exhaustive _data readings_ on the
| organic neuron, which is, frankly, unlikely. (Not that I know
| how to estimate how much it 's missing, but I don't think we
| fully understand a single neuron yet.)
| singularity2001 wrote:
| Except that every synapse is not a dumb weight but a highly
| complex system connected to an even more complex system (aka
| neuron) which might each be a (super)computer on its own.
|
| Given how extremely bad we are at computing, there is hope (for
| ai) that the neurons or their circuits are not _that_ powerful
| after all.
| alecst wrote:
| Last I heard (and I believe this could be wrong) my professor
| said that we basically understand how a single neuron works.
| That like basically if we do X input we get Y output, up to
| some accuracy. He used this to discuss the idea behind neural
| networks -- that each neuron is simple enough to model, all
| we need to worry about is the weights and the dynamics of the
| network as a whole.
|
| How much of a simplification is that? And how much does the
| accuracy of such a model matter, in the grand scheme of
| things?
| visarga wrote:
| > All we need to worry about is the weights and the
| dynamics of the network as a whole. How much of a
| simplification is that?
|
| A lot. Parallel optimization is an art form. These models
| are trained on static datasets, they can't intervene in the
| environment to infer causal relations, so they need legs
| and hands.
| manquer wrote:
| I would say quite a bit. Adding even a third body makes it
| impossible to calculate physics with certainty. A complex
| system with any number of individual components is hard to
| understand with certainty and/or calculations can become
| exponentially more complex .
| gisely wrote:
| Your professor lied.
| sydd wrote:
| oh finally something that I learned a lot about :)
|
| Such research is the area of computational neuroscience -
| one thing that such people do is try to model parts of the
| brain (or just a single neuron) with computers.
|
| A Neuron (=nerve cell in the brain) is a very complex
| beast. In rough terms they work like this: They collect
| signals (electrical impulses) via their small appendages
| called dendrites. when the sum of the signals reaches a
| certain threshold a large electrical impulse is generated
| at the cell body that will travel trough its "output"
| appendage (called axon) that connected to another neuron's
| cell body or to its dendrite.
|
| Neurons display a dazzling variety in all these parameters:
|
| - In morphology, e.g. they can look like a pine tree
| http://www.scholarpedia.org/article/Pyramidal_neuron (I
| really recommend scholarpedia, also this article has a nice
| animation on how electrical impulses propagate) or like a
| sea urchin.
|
| - it really matters where the cell gets its impulse from: A
| neuron stimulated near its cell body will be much more
| sensitive to the input than being stimulated far away.
|
| - Their response characteristics are wildly varied too.
| Some give off one large impulse, some a quick burst of
| impulses. Some are preventing others from giving out
| impulses from stimulation (inhibitor neurons)
|
| - This whole mess can be modulated with chemical compounds
| that are released by the body -- some make some neurons
| more sensitive, some less.
|
| - Also we still discover every year some new mechanism that
| modulates how they function.
|
| The issue is that this results in such a complex system
| that a modern PC cant even simulate 1 detailed neuron model
| realtime (these tools are open source, try them out! for
| example https://neuron.yale.edu/ ). Now we know that we're
| simulating things that likely do not matter (e.g. we don't
| need a neuron model that consist of 10.000+ segments), but
| we do not know which parts we need to remove to have a
| faithful simulation. Also we might simply simulate some
| parts wrong because our knowledge of the subject is not
| enough.
|
| But on the upside we've reached some great things already,
| for example we know how our brain calculates from our head
| and eye position the orientation of the things we're
| looking at
| postalrat wrote:
| CPU in kilohertz then megahertz then gigahertz then it stopped.
|
| RAM in kilobytes then megabytes then gigabytes then it stopped.
| manquer wrote:
| Those are material science and physical limitations.
|
| Number of parameters in a neural network is not really
| limited that way, doing useful compute with it is a different
| matter
| savant_penguin wrote:
| Really cool!
|
| I'd love to see a table comparing the results against the other
| gigantic models (I know could Google the other results and merge
| them together but no thanks)
| rustc wrote:
| Has there been any update on the legality of using this kind of
| model? Is it ok to just crawl the web, take any content you want,
| train a model and sell access to the model like
| OpenAI/GPT-3/GitHub Copilot?
| wyldfire wrote:
| For the most part, everything that's not barred by law is
| "legal." Does this use constitute copyright infringement (if it
| were trained on copyrighted material)? IMO no, but it depends
| very much on the use of the model. Copilot is especially
| interesting because instead of being used for simple inference
| the model is being used to author new works that might aspire
| to also be copyrighted. Are those new works derivative works?
| Perhaps. We consider art and science produced by humans to be
| inspired in part by that which they've been exposed to before.
| If the model hasn't been overfitted, it should generalize its
| 'knowledge' sufficiently that it's 'similar' to our
| intelligence. Humans can commit copyright infringement when
| they recall and author content so specifically as to be a
| derived work.
|
| In any case: my opinion matters for naught. The only 'update'
| you'd get that matters is from a court producing a ruling.
| Legal journals might chime in but their opinion isn't binding.
| Theoretically there could be legislation to clarify but that's
| probably a really, really, really long way off.
|
| Certainly some of the training looks to be content that's not
| copyrighted or no longer copyrighted, btw.
___________________________________________________________________
(page generated 2021-10-11 23:02 UTC)