[HN Gopher] AI training is outpacing Moore's Law
___________________________________________________________________
AI training is outpacing Moore's Law
Author : pseudolus
Score : 60 points
Date : 2021-12-05 12:17 UTC (3 days ago)
(HTM) web link (spectrum.ieee.org)
(TXT) w3m dump (spectrum.ieee.org)
| Jiro wrote:
| Do we even still have Moore's Law? All indications are that it
| died out a while ago.
| wing-_-nuts wrote:
| If you're going strictly on number of transistors, I think
| we're still going up up and away. If you're talking about
| performance, which is what most folks really care about, we
| really are hitting thermal limits especially with consumer
| hardware.
|
| AMD, Nvidia and intel are all hitting the 'heat wall' with
| their hardware just trying to cram more into smaller nodes and
| boost clocks. Nvidia's 4000 series cards are rumored to be 500W
| cards which is well into 'space heater' territory.
| kemiller wrote:
| Transistor density/cost still helps with the inherently-
| parallel ML workloads.
| e12e wrote:
| Really?
|
| http://www.transistorcount.com/
|
| Slope seem a bit more gentle - but certainly not gone?
| hyperpallium2 wrote:
| You're right about literal Moores Law, but its benefits are
| much diminished, of performance, power usage and cost. My
| understanding is the main problem is heat dissipation.
|
| I've long predicted macrochips to solve this, brute force of
| huge quantities of silicon, but it hasn't really happened.
| adverbly wrote:
| If we hit a trillion transistors by 2030 without increasing
| chip size, I'll be shocked.
| armatav wrote:
| Yup, and we don't currently have an architecture that can support
| models 35+ years from now.
|
| Not yet at least.
| brrrrrm wrote:
| The article is a bit lacking in details. Here are some changes in
| ML performance that are happening today and don't require
| increased transistor counts:
|
| 1. newer optimizations algorithms converge faster (Adam, adagrad)
|
| 2. lower-precision numerics can be used (TF32 is 19bit, bf16 is
| popular)
|
| 3. matrix multiplications are implemented in hardware (tensor
| cores, AMX)
|
| 4. neural networks are compiled directly (TVM, halide)
|
| 5. multi-device network topologies are tuned for training (e.g.
| nvlink improvements)
|
| 6. companies employ grad-student descent (hire a bunch of PhDs
| and task them with improving MLPerf results)
| JL-Akrasia wrote:
| The last one is particularly effective.
| wing-_-nuts wrote:
| I didn't really understand #6? Expand?
| hailwren wrote:
| You can overfit your data by coming up w/ a model that is
| very specific to it. Consider an n dimensional data set
| where one variable is a boolean.
|
| A researcher might notice that the True and False cases are
| different enough to warrant their own algorithms. Say they
| now fit a lower dimensional line to the True and False
| cases.
|
| They've just performed gradient descent but the grad
| student was the gradient. It's a common joke in CS/ML
| departments.
|
| (This is an obviously simple example, but the point is that
| hyperparameter tuning and algorithm selection _are a part
| of your algorithm_ and the data you're looking at while
| doing so is part of your training set)
| visarga wrote:
| These large models are going to be the operating system of the
| future. When they are going to run efficiently on edge we are
| going to see an explosion of automation. Being so large they are
| not only more accurate, but also quicker learners. So less
| initial effort to start a project.
|
| Google is teasing us with leaks about Pathways[1], but not
| releasing a paper or demo yet.
|
| [1] https://blog.google/technology/ai/introducing-pathways-
| next-...
|
| I call them multi-multi-models: multi-modality, multi-task,
| multi-language, multi-memory (besides the processing memory it
| can have search engine in the loop and episodic memory).
| Somebody's going to make it in the next couple of years.
| Moru wrote:
| The sad part is that we are all helping to train the futures AI
| but who will own it and make money on it?
| wetpaws wrote:
| You can have ownership in companies that do machine learning.
| It's called "shares". Creating your own business is an option
| too.
| frenchyatwork wrote:
| The complaint was that it's the people who own the
| equipment (or shares in it), and not the people who do the
| work, who are reaping the profit.
|
| When the working class had this problem a couple centuries
| ago, it was solved by building guillotines. I hope we can
| do better this time.
| rowanG077 wrote:
| To start an AI company you generally don't need much. A
| few thousands in GPUs and you are good to go. If AI
| researchers feel they are underpaid there is literally
| nothing stopping them from starting a company themselves.
| gmadsen wrote:
| that is absolutely not true. I'm not sure where you are
| getting your information, but read anything from an angel
| investor. most AI start ups fail because they go into
| huge debt getting correct data and training models, which
| end up bankrupting them in the process with nothing to
| show for it
| wetpaws wrote:
| I don't see any problem with it, stock market is open for
| everyone.
| fartattack wrote:
| Sorry, are you claiming that the French Revolution
| occurred due to bad labor practices?
|
| Or did trust busting and unionization in the late 19th
| century US utilize guillotines?
|
| Either of those would certainly be a.. uh.. let's go with
| "novel" retelling of the history!
|
| Or did you mean Red October? Because I'm pretty sure
| /that/ revolution didn't actually solve any problems for
| the worker in the way things actually panned out
| thehappypm wrote:
| You're conflating two things. Feudalism is like slavery:
| taking credit for someone else's labor. Automation
| removes the need for labor. Automation has almost
| universally led to quality of life improvements, not
| revolution.
| andreyk wrote:
| IMHO (as a AI researcher) 'multi-modality, multi-task, multi-
| language, multi-memory' may take longer than you think. Doing
| multi modal learning is quite hard due to lack of labeled data,
| a lot of tasks either don't offer clean plentiful datasets or
| require RL / human in the loop learning and not just supervised
| or self supervised learning, and on and on. Certainly a lot of
| progress will be made and partial automation is likely to speed
| up (as has happened with Copilot), but full automation seems to
| be a good while away.
|
| Plug - I expand on this a bit in this blog post 'The Inherent
| Limitations of GPT-3' (which was on front page of HN a few
| weeks back and led to a good deal of discussion )
| https://lastweekin.ai/p/the-inherent-limitations-of-gpt-3
| oblio wrote:
| I guess the next step is democratizing ML.
|
| Right now we seem to be into a kind of "mainframe phase", where
| running big models is super expensive.
|
| I wonder what kind of speedup we need for ML to be truly
| accessible for the average developer. 50x? 100x?
| hervature wrote:
| Define "accessible". Very modest hardware (sub $100 and
| certainly sub $1000) can train random forests very quickly.
| This is going to be close to SOTA on basically any
| classification task.
| minimaxir wrote:
| The trick for ML accessibility is minor finetuning the existing
| models that have been trained on these super training models so
| that users can get similar performance, and lately companies
| have been more open about releasing models open source.
| [deleted]
| ska wrote:
| > speedup we need for ML to be truly accessible for the average
| developer. 50x? 100x?
|
| The data scale (and access) is often more important than the
| modelling, and that doesn't have a technological fix, or at
| least not a purely technological fix.
| fumar wrote:
| So Moore's Law is a constraint for average people as hardware
| costs a key factor in performance, is that right?
| ska wrote:
| What I'm saying is the hardware is less of a problem than
| access to data, for most problems that would get you out of
| what you can reasonably do on a consumer accessible (cost)
| machine.
| jsharf wrote:
| Unless we have widely available pre-trained models with
| generic knowledge that is easily transferable. Such a thing
| is obviously difficult, but it isn't impossible if we're
| talking about the future.
| Der_Einzige wrote:
| I love this because it's vindicating Rich Sutton's essay titled
| "The Bitter Lesson".
|
| http://www.incompleteideas.net/IncIdeas/BitterLesson.html
|
| "The biggest lesson that can be read from 70 years of AI research
| is that general methods that leverage computation are ultimately
| the most effective, and by a large margin. The ultimate reason
| for this is Moore's law, or rather its generalization of
| continued exponentially falling cost per unit of computation."
|
| And now we are beating out Moore's law!
| dejj wrote:
| Seems like a step towards singularity.
___________________________________________________________________
(page generated 2021-12-08 23:01 UTC)