[HN Gopher] The importance of exponentially more computing power
___________________________________________________________________
The importance of exponentially more computing power
Author : nkurz
Score : 54 points
Date : 2022-07-29 12:52 UTC (1 days ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| shadowofneptune wrote:
| > In this paper, we assemble direct quantitative evidence of the
| impact that computing power has had on five domains: two
| computing bellwethers (Chess and Go), and three economically
| important applications (weather prediction, protein folding, and
| oil exploration). Computing power explains 49%-94% of the
| performance improvements in these domains.
|
| I'd like for a follow-up paper to look at some different domains,
| since these are AI research and scientific computing,
| respectively. Business computing is more directly relevant to
| economic growth, and with its use dating back to the 1950s it
| might be possible to find data on how it's improved.
| nabla9 wrote:
| It's well studied subject. There are also surveys available.
| Here is slate article: https://slate.com/business/2011/03/the-
| productivity-paradox-...
|
| 1) effect has been surprisingly small. It's often called the
| "productivity paradox" Robert Solow said it in late 80s: "You
| can see the computer age everywhere but in the productivity
| statistics."
|
| 2) The reasons are complex.
|
| One possible reason is that productivity increase -> labor
| share declines -> less demand -> smaller GDP growth.
|
| People always incorrectly assume that unemployment is the
| danger from automation. The real issue is declining labor
| share. People with jobs get smaller share of the work they do,
| because computers (capital asset) is replacing labor.
| marcosdumay wrote:
| I wonder if anybody measured something similar to that number
| on the 18th century.
| jasonhong wrote:
| Economist Robert Solow famously said in 1987 that the computer
| age was everywhere except for productivity statistics.
| Wikipedia has a pretty good article about this Productivity
| Paradox.
|
| https://en.wikipedia.org/wiki/Productivity_paradox#End_of_th...
|
| One hypothesis I've heard that isn't on the Wikipedia page is
| that productivity from IT also requires new kinds of
| organizational structures. If you had many layers of middle
| management, and replaced paper memos with email, it wouldn't
| necessarily improve productivity because that wouldn't harness
| the full potential of computing.
| freemint wrote:
| > we examine the contribution of more computing power to better
| outcomes
|
| No, they pick a set of problems where computational methods are
| known to have a beneficial impact and the plot every progress in
| that field against increased amounts of computing. Since amount
| of computing power used is monotonous and ELO score/Go
| performance/weather prediction success is trending monotonous the
| correlation is pretty high. However computation power is not the
| only thing that rose mostly monotonically during that time. At
| best they derived an upper bound of the contribution of more
| computing power to better outcomes.
|
| For example in Mixed Integer Linear Programming studies were done
| to measure algorithmic vs hardware speedup. "On average, we found
| out that for solving LP/MILP, computer hardware got about 20
| times faster, and the algorithms improved by a factor of about
| nine for LP and around 50 for MILP, which gives a total speed-up
| of about 180 and 1,000 times, respectively."
| https://arxiv.org/abs/2206.09787 This methodology would attribute
| the 1000 times effect to the increase in FLOPs alone.
|
| And just a methodological concern, taking the logarithm of one
| axis, is applying a non-linear transformation, and then doing a
| linear fit results in distorted measure of distance between fit
| and data depending on the data. This effect was not discussed. It
| does only mess with R value so i would not feel comfortable
| applying that R value to derive an attribution.
| jvanderbot wrote:
| To the users of MILP libraries, the difference between
| algorithmic and hardware improvements are not so significant,
| and both can be called "Computing Power" in terms of number of
| application-specific solutions per second.
|
| You're right, and your insight is illuminating into the real
| gains had by the world on the backs of hardware and software
| improvements, but I think you're one level deeper in the
| abstraction than the paper intends to be.
| freemint wrote:
| Then i must have phrased it badly. The fact that all the
| speed up would be attributed to FLOPs in the MILP case with
| that methodology means any improvement in data acquisition,
| management methodology, mathematical modeling, the ability to
| dig deeper or even the increased regulatory cost of boring
| which taken together sorta look linear when logarithmitized
| (for weather prediction or oil exploration (after a dubious
| data selection)) would also be attributed to computational
| improvement.
| naillo wrote:
| I saw a slide somewhere recently that showed that although
| compute per individual CPU/GPU is reaching an upper bound, the
| compute _per dollar_ is still growing exponentially.
|
| Gave me some hope that we'll still be able to do cool things in
| the coming decades even without some giant leap like quantum
| computers etc (although it's not as cool to have the compute in
| the cloud vs on your local machine, but it's something).
| jillesvangurp wrote:
| That threat has been looming over me ever since I had my
| commodore 64 in the nineteen eighties. I'm sure it will hit
| some theoretical limit at some point in the future. But I
| wouldn't go as far as to predict next year for that.
|
| Of course we have been approaching nano scale now for a while
| and my 2012 mac was only able to perform at 70% of the build
| speed of my 2017 model, which in turn is about the same as the
| cheap Linux laptop I picked up a few months ago. GPUs are one
| area where things are still improving rapidly because you
| increase performance by simply having more cores.
|
| My cheap laptop is impressive in the sense that it does what it
| does without thermal throttling or even heating up a lot. Not
| bad for a cheap 700 Euro i5 laptop. My 2017 mac book pro was
| struggling with keeping things cool.
|
| The next leap is going to be an exponentially larger number of
| CPU cores. We've been stuck at 4-16 or so for the last decade
| or so. There's no other reason for this other than legacy
| compilers, languages and CPU architectures. Leveraging
| concurrency is just hard. GPUs kept on increasing number of
| cores and have been doubling fps for the same job much more
| reliably.
|
| All of course amazingly quick compared to my trusty old C64.
| joak wrote:
| Quantum computers are not faster classical computers. They
| solve some problems exponentially faster than classical
| computers. For example we can break RSA keys with a quantum
| computer because we have a quantum algorithm (Shor's algorithm)
| to solve this problem.
|
| We do not have many quantum algorithms that solve interesting
| problems exponentially faster. Finding these algorithms is not
| an easy task and we do not expect all problems to be solvable
| this way.
|
| Disclaimer: I'm a researcher in quantum algorithms.
| marcosdumay wrote:
| Just to add, nobody expects NP-complete or harder problems
| (like the MILP example) to get any speedup form quantum
| computers at all.
| phkahler wrote:
| >> we find that an exponential increase in computing power is
| needed to get linear improvements in these outcomes.
|
| The scaling for weather models and FEA is not linear. To get
| incrementally better requires at least polynomial increases.
| ng55QPSK wrote:
| "weather models" "better" - how do you actually rate weather
| models? Exact prediction or prediction of outliers like
| catastrophic events? Is there more value in knowing tomorrow's
| temperature to be 24C or a thunderstorm happening at 15:00 ?
| pixl97 wrote:
| There is a lot of value in predicting where a hurricane is
| going to hit within 100km in 36 hours. Also things like flood
| prediction so we can drain lakes ahead of time before the
| waters start rising. Giving people a forewarning saves lives.
| palm-tree wrote:
| Surely this is showing correlation, but not necessarily
| causation. There are probably many fields where improvements in X
| have happened, along with greater compute power. How much the
| improvement in X is attributable to greater computing power is
| surely very hard to quantify. Or am I missing something?
| credit_guy wrote:
| I agree with you.
|
| For the oil example, I can think of a very simple explanation
| that has nothing to do with computing power. They state that
| the drilling success rate was 10% in 1940 and 70% in 2010. In
| other words, out of 10 exploratory drills, 9 were dry in 1940
| and only 3 in 2010. And since the companies use computers to
| predict the presence of oil, and the computing power has
| increased, voila, the computing power explains the increased
| success.
|
| But the alternative explanation is that over time oil could be
| found at higher and higher depths. The cost of exploration has
| increased. When it does not cost you much to drill a shallow
| hole (in 1940), you don't mind if you miss 9 times out of 10.
| But if you have to put in millions of dollars to drill to
| thousands of meters depths, then you think twice (or maybe 10
| times) before you drill.
|
| Any prediction system will give you an estimated probability of
| success. If the cost of failure is not high, you may decide to
| drill as soon as p > 10%. But if it's high, you may increase
| the threshold to 90%.
|
| Of course, I'm not saying this is all there is to it. But it's
| likely that this explained part of the increased success.
| Another part is no doubt due to more experience. Another is due
| to algorithmic advances. Yet another to the improvement in the
| seismic sensors and technology used.
| abernard1 wrote:
| > Yet another to the improvement in the seismic sensors and
| technology used.
|
| This is a huge one. Almost all of that geoseismic downhole
| sensing technology uses radioactive emissions. In 1940, it
| was impossible to "see" anything downhole with any
| reliability. Nowadays it's passe.
| sylware wrote:
| where no algebraic, and certainly beautiful solution(s), of some
| set of equations exists, brute force computation to the rescue!
|
| The more computing power, the more accurate the predictions.
| ng55QPSK wrote:
| Just wrong. The more correct input you have and the more
| correct your model of the real world is, the more accurate your
| predictions are.
|
| We already see helpless activities to be better with AI (and a
| lot of computing) then classical optimisation theory (think
| solving PDEs) and failing.
|
| Just more computing power will not help (except people whose
| business model relies on selling you computing power)
| nicoburns wrote:
| More computing power can be a prerequisite to more accurate
| models. E.g weather models modelling things at smaller
| granularity.
| touisteur wrote:
| Let's say that the messy problems that don't have a nice
| analytic solution (a beautiful all encompassing model) are
| more and more leaning on 'very complex systems of equations
| or inequations, linear or non linear' and we can only
| approach or simplify them to heuristics (gradient descent,
| optimization of all sorts) because exact solution is too
| compute-expensive.
|
| A tenfold difference in power might help a bit (being able to
| take 10x more sensors in or data with 10x better resolution
| might already help).
|
| But a 10000x increase is a game changer, even if you waste so
| much of it. Though it all depends how it increases... Some
| processes I know would be immensely better - state of the art
| has already quantified the huge gains, and then spent 15
| years trying to make the thing runnable in real-time - if I
| could run hundreds of millions of complex 400x400 SVDs per
| second or if I could run a maximum likelihood search at the
| same rhythm).
|
| A 10000x increase in perf puts some _exact best answer_ and
| not just approximations or 'best we could compute, sorry' in
| the realm of possibilities.
|
| I'd also say AI itself would not be viable without the huge
| increase in power and memory bandwidth that allows to play
| SGD on such a huge search space.
|
| You need months or 200+ DGX servers (8xA100 each) and a team
| of mlops people babysitting the process, to train a large
| language model today. Isn't that a direct result of computing
| power increase?
|
| I'll take any more computing power, especially if there's a
| 10x or 100x gain on the horizon, if they'll sell it to me.
|
| Yes we need to be clever and not waste the computing power,
| but for many as-yet-not-over-optimized problems any increase
| is an instant win.
| nkurz wrote:
| If the abstract is too dry, here's a press release interview with
| the lead author: https://news.mit.edu/2022/neil-thompson-
| computing-power-inno...
|
| Neil has an impressive educational pedigree that positions him
| well for analyses like this. He's got (among other things) an
| undergraduate degree in physics, separate masters in Comp Sci and
| economics, and a PhD in Public Policy. He's smart and worth
| listening to. I'm excited to see him getting some professional
| traction: http://www.neil-t.com/about-me/
| jarenmf wrote:
| Related to weather, climate science would benefit greatly from
| increased computational power. Current models are forced to make
| assumptions to predict certain processes below the grid
| resolution. Things like cloud processes and transport processes
| are usually parameterized. Similarly, computational fluid
| dynamics are very useful to studying climate and are limited by
| the available computational power.
___________________________________________________________________
(page generated 2022-07-30 23:02 UTC)