[HN Gopher] ML training compute has been doubling every 6 months...
___________________________________________________________________
ML training compute has been doubling every 6 months since 2010
Author : innerfestival
Score : 84 points
Date : 2022-02-15 16:44 UTC (6 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| egberts1 wrote:
| 10B?
|
| - Is that 2 (as in binary 10)?
|
| - 108 as in a typo, or
|
| - an order of magnitude of 6 thereof?
| danieldk wrote:
| 10 billion. The number of FLOPs has gone from ~1e+14 (6 layer
| MLP MNIST) to ~1e24 (Megatron-turing NLG).
|
| edit: s/parameters/FLOPs
| Tamaybes wrote:
| Not parameters, the amount of FLOPS required to train the
| model.
| danieldk wrote:
| Whoops, thanks! That's what I get for typing too quickly
| between household duties.
| dang wrote:
| HN's software replaces "10 billion" with "10B". Most of the
| time it's clear.
|
| However, I've replaced the submitted title ('ML training
| compute has been growing by a factor of 10B since 2010') with
| what the tweet says now.
| gilbetron wrote:
| Machine learning related, the transcription capability of Zoom
| (which uses some form of machine learning) was amazing me this
| morning. I was watching a recording of a zoom session that was a
| mock interview to help learning a new interview question. In this
| situation, there were two people sitting by each other in an
| echo-y conference room, being recorded by a zoom tablet thing
| microphone. They were talking to each other, both were french but
| speaking english and so had moderately heavy accents, both had
| masks on, and were talking about a coding problem. The accuracy
| of the transcription was blowing my mind. Really the only time it
| stumbled was for very technical terms.
| Datenstrom wrote:
| While this is true there is also a lot of interesting research
| going on that makes model training and adaptation to new tasks
| much more efficient. For example in meta-learning methods like
| Model Agnostic Meta-Learning (MAML)[1] you learn a model that
| learns to initialize weights for another model such that it
| converges very quickly for a new task.
|
| For datasets benchmarking new task adaptation there are Meta-
| Dataset (MD)[2] for Meta-Learning, and the Visual Task Adaptation
| Benchmark (VTAB)[3] for Representation Learning. Recently to
| compare the two approaches VTAB+MD[4] was created. Of course,
| there is also model quantization[5], mixed precision, sparse
| networks[6], brain floats[7].
|
| Well I've dropped a lot there, but there is a lot more outside
| the mainstream. I've been building something on a tight budget
| for a few years so this has been important for success there. We
| are are very focused on being data and compute efficient.
|
| [1]: https://proceedings.mlr.press/v70/finn17a/finn17a.pdf
|
| [2]: https://arxiv.org/abs/1903.03096
|
| [3]: https://arxiv.org/abs/1910.04867
|
| [4]: https://openreview.net/pdf?id=Q0hm0_G1mpH
|
| [5]: https://arxiv.org/abs/2105.08819
|
| [6]: https://arxiv.org/abs/2112.13896
|
| [7]: https://en.wikipedia.org/wiki/Bfloat16_floating-point_format
| ellis0n wrote:
| Growth by 1e+7 per ten years from 1e+14 2011 to 1e+21 2021 and
| should be 1e+28 2030. Probably Doom, host OS and player bot will
| be generated and played by AI from 1e+28 2030.
|
| 2011 1e+14 = 100 000 000 000 000
|
| 2021 1e+21 = 1 000 000 000 000 000 000 000 <- you are here
|
| 2031 1e+28 = 10 000 000 000 000 000 000 000 000 000
| joe_the_user wrote:
| It looks like a useful article showing the trend OpenAI described
| a few years ago has continued.
|
| One thing I'd object to is coining (or maybe just using?) the
| term "Large-Scale Era". This seems like polluting the discussion
| with a meaningless phrase.
|
| The basic message is essentially, deep learning continues to grow
| in scale exponentially, many people consider the dividends still
| worth the costs. No need to "sexy things up" with what no more
| than a marketing term.
| Tamaybes wrote:
| The result about recent compute trends is different from the
| recent trends described by OpenAI. In particular, they find a
| 3.5-month doubling time over the Deep Learning Era, whereas the
| paper finds a 6-month doubling time.
|
| I think the Large-Scale Era does point to a new phenomenon that
| emerged pretty discontinuously, which is that there are now
| 'two lanes' in ML scaling. Prior to 2015, academic and industry
| would train roughly similarly compute intensive models. Since
| then, a small number of industry players frequently train
| models with 10-100x more compute than what the typical
| researcher uses.
| joe_the_user wrote:
| The thing is that the advent of deep learning was a very big
| change in the sense you had a general purpose method appear
| that you could use to throw computing power at many/most
| problems (and tune a bit but still) and get results that
| previously you couldn't get (and when did get results, you
| required domain experts). No doubt we have changes within the
| trajectory of this escalating brute force solutions. But
| relative changes in this paradigm seem fundamentally
| different than the initial advent of the paradigm.
| axg11 wrote:
| For those in the field, the progress of model performance feels
| just as rapid. Language models today are incredibly powerful
| compared to language models in the early 2010s.
|
| I think the advent of retrieval models (retrieval transformers)
| will continue this compute trend in a more efficient manner. They
| allow focusing of the compute onto indexing of the knowledge.
| uses wrote:
| What's a good next step after learning "classical" neural nets,
| i.e. backpropagation ANNs?
|
| I've been working with MNIST with C and CUDA with dynamic
| parallelism for a couple months and it's been extremely
| enlightening but I'm kind of ready to move on.
|
| CNNs maybe?
| nefitty wrote:
| What would you recommend for a complete beginner? I found these
| lectures that have helped me start sketching out the field in
| my mind, but it's still difficult:
| https://www.davidsilver.uk/teaching/
| chickenpotpie wrote:
| https://en.m.wikipedia.org/wiki/Wirth%27s_law
| wirthjason wrote:
| This is one of my favorite laws!
| Victerius wrote:
| Why spend billions of dollars on armies of PhDs when you can
| create a better organic intelligence in 9 months at a fraction of
| the cost?
|
| Why try to beat nature?
|
| I'm being only semi sarcastic here.
| LesZedCB wrote:
| because the kind of work we want the machines to do is
| degrading and monotonous
|
| oscar wilde's _the soul of man under socialism_ probably better
| describes the idea.
| AnEro wrote:
| I wish that's why we all are doing this
| spyder wrote:
| Because it's software and once you created it you can duplicate
| it, customize it with the fraction of the cost and time.
|
| _" Why try to beat nature?"_
|
| If we wouldn't try "beat nature" we would be still in caves or
| would be extinct. Nature is "beating" itself with evolution and
| we are part of that.
| cossatot wrote:
| Plus 20 years of intensive training after that 9 months...
|
| Or maybe your newborn is more capable than mine!
| AnEro wrote:
| Nature's $7 an hour general compute nodes are still too
| expensive to not burn money to try to make an equivalent at the
| cost of electricity and initial purchase.
| throwawaynay wrote:
| AI is already more (cost)efficient at some tasks than humans
| nootropicat wrote:
| 1. Because technology to clone an existing human (not just the
| body) doesn't exist, especially into a more durable substrate
|
| 2. Because ai doesn't have biological scaling limits. There's
| nothing particularly special about human level of intelligence.
| rowanG077 wrote:
| > There's nothing particularly special about human level of
| intelligence.
|
| Big claim since nothing in the known universe has human level
| intelligence besides humans. It's like holding the original
| declaration of independence and saying that there is nothing
| particularly special about this piece of paper.
| nefitty wrote:
| This analogy breaks down easily.
|
| What do you propose is special about human-level
| intelligence? If anything, it is in constant struggle with
| biological and emotional needs. Yes, of course, human
| brains are amazing, the declaration of independence is
| great, but the entire human project is the push for
| progress.
| rowanG077 wrote:
| I can't say what is special about human-level
| intelligence. Most importantly because the scientific
| community(and by extension me) can't define what human-
| level intelligence means. It's just too illusive at this
| point. That something is undefined by humans doesn't mean
| you can simply say it's not special.
| mistrial9 wrote:
| deep, real-time scans of digital media streams, subscription
| behavior, market responses.. things like that, where the inputs
| are digital already
| anonymouse008 wrote:
| I honestly thought you meant waiting for the next iteration
| doubling of ML model performance/training...
|
| which I thought was brilliant, ha!
| ska wrote:
| > Why spend billions of dollars on armies ...
|
| The vast majority of AI research (over all time) has not been
| seriously intended or expected to arrive at a general
| intelligence.
|
| Sure there are some true believers there in some sort of
| handwavy construction argument ("and then a miracle occurs")
| and there have been serious research attempts to understand
| intelligence, of limited scope an resources. But to be clear,
| whether this wave or previous ones, the money washes in and the
| army of PhD's happens when there is a sniff of practical,
| usable algorithmic results from machine learning. It's not the
| same thing.
|
| Parallel application of even relatively stupid algorithms
| targeted at the right tasks will rapidly outproduce any
| reasonable number of organic intelligences of the type you
| describe :)
| jacquesm wrote:
| Because a newborn needs another 18 years or so of nurturing and
| education to become useful, this costs so much that quite a few
| countries see no qualms in saddling these new humans with the
| debt of their education.
| mistrial9 wrote:
| hehe - DeepLearning that is, my supervised models work fine
| without a compute infrastructure the size of a major naval vessel
| thanks!
| itronitron wrote:
| At what point should this start to have a noticeable result
| outside of the profession?
___________________________________________________________________
(page generated 2022-02-15 23:01 UTC)