[HN Gopher] The Great AI Reckoning
___________________________________________________________________
The Great AI Reckoning
Author : doener
Score : 117 points
Date : 2021-09-30 15:20 UTC (7 hours ago)
(HTM) web link (spectrum.ieee.org)
(TXT) w3m dump (spectrum.ieee.org)
| areoform wrote:
| Something that I'm fascinated with is the question, would other
| models show equivalent results if given the same amount of
| compute?
|
| I looked through the list of solvers for the protein folding
| challenges and there were other deep learning, neural network,
| and classical machine learning approaches on there. Even some
| hybrid ones! But none of the participants had even a fraction of
| the compute power that AlphaFold had behind it. Some of the
| entries were small university teams. Others were powered by the
| computers some professor had in their closet (!). Most of the
| teams were dramatically under-powered as compared to AlphaFold.
| How much did this influence the final result?
|
| What would the other results look like if they'd been on equal
| footing? Would they have been closer?
|
| It's a genuine question.
| dekhn wrote:
| actually, companies can buy the amount of TPU time that
| AlphaFold used. Note that the price breaks from "here is a
| listed price" to "contact sales" at the largest size (TPU
| v3-2048). I assume that during initial training and
| experimentation that DM kept 1 or two TPU v3-2048 busy 24/7 for
| 6 months to a year. That's entirely within the budget of well-
| heeled companies
| vasco wrote:
| Presumably the team started with existing methods on their
| fleet of compute and was forced to develop new ones. While
| development happens your fleet also grows and your ability to
| utilize it expands and you end up not going back to ever try
| methods that are generations behind on your current fleet
| because the overhead of making it work doesn't justify it.
|
| It's like asking if using the google homepage from 10k
| iterations ago would perform better than the current version on
| the present user cohorts. There's just too much invested to
| justify testing things like that when you can use the time to
| improve what is showing promise.
| sdenton4 wrote:
| A couple thoughts...
|
| First, having 'infinite compute' is a way for researchers to be
| sure that compute isn't the thing holding back their method.
| So, DeepMind made a protein folder using all the compute they
| had available; later others managed to greatly reduce the
| amount of compute needed to get equivalent results, by re-
| implementing and innovating on DeepMind's initial write-up.
|
| Second, I think there's a lot of interesting ground to explore
| in hybridizing ML with more classical algorithms. The end-to-
| end deep learning approach gives us models that get the best
| scores on imagenet classification, but are extremely prone to
| domain shift problems. An alternative approach is to take a
| 'classical' algorithm and swap out certain parts of the
| algorithm with a deep network. Then you should often get
| something more explainable: The network is explicitly filling
| in a specific parameter estimation, and the rest of the
| algorithm gives you some quality guarantees based on the
| quality of that parameter estimator. I saw some nice work along
| these lines in sparse coding a couple years ago, for example...
| randcraw wrote:
| No, some DL architectures are unquestionably superior to others
| especially for some tasks. In the first few years of deep CNN
| development, several limits quickly became apparent
| (sensitivity to scale, vanishing gradients, gradient signal
| loss with some activation functions, and many engineering
| constraint tradeoffs like poor RAM use). This was addressed by
| making changes to a simple dense net which begat architectures
| that suffer less from those limits.
|
| Without question, limits exist in every DL architecture; we
| just haven't taken the time to diagnose or quantify all the
| limits yet. Now that attention has shifted to transformers,
| analysis of DNNs' inherent limits is made substantially more
| difficult, given transformers' huge size. That, and their
| multi-million dollar training cost likely will make if
| infeasible to diagnose or cure each design's inherent limits.
| neatze wrote:
| I don't think computational resources will have greater effect,
| because you can see this by simple plot running time x axis and
| model performance y axis.
| WithinReason wrote:
| The advantage of DeepMind from more compute resources is
| cumulative, you can develop better models if you have more
| compute. So the question really is: If academia and other
| research institutions had DeepMind's resources, would their
| output match DeepMind's?
| gradys wrote:
| Probably not. It's not clear how to scale other methods to make
| use of so much compute. They top out sooner, with fewer
| parameters and less compute.
|
| One way to look at why deep learning is having the impact it
| does is that unlike other ML methods, it's actually capable of
| making use of so much compute. It gives us modular ways to add
| more and more parameters and still fit them effectively.
| WanderPanda wrote:
| What always astonishes me that deep learning seems to work on
| human timescales. For other problems (even if they are
| polynomial in complexity) we get into infeasible runtimes
| when we increase the problems complexity by 10x. With deep
| learning the fuzzy, approximative nature seems to help to
| grasp the gist of the 10x problem and somehow allows us to
| reach 95% of the solution in just e.g. 2x the runtime.
| Heuristics might play in the same league, but the development
| time kind of scales with the problem, while in deep learning
| I would put it in the linear or log basket.
| photochemsyn wrote:
| Personally, I'm won't think of it as 'artificial intelligence'
| until an AI system can teach me to speak a human language as well
| as a native human speaker of that language could. Seems a long
| ways off.
|
| All it appears to be now is a collection of very sophisticated
| pattern matching algorithms that build their own opaque internal
| sorting/matching/classification algorithms based on the data
| they're trained on, as I understand it. This is of course
| incredibly useful in some domains, but it's not really
| 'intelligence'.
|
| And, they can't do math very well:
|
| > "For example, Hendrycks and his colleagues trained an AI on
| hundreds of thousands of math problems with step-by-step
| solutions. However, when tested on 12,500 problems from high
| school math competitions, "it only got something like 5 percent
| accuracy," he says. In comparison, a three-time International
| Mathematical Olympiad gold medalist attained 90 percent success
| on such problems."
|
| 1. https://spectrum.ieee.org/ai-failures
| quotemstr wrote:
| > AI system can teach me to speak a human language as well as a
| native human speaker of that language could. Seems a long ways
| off.
|
| And when such a system appears, you'll claim that it's still
| not AI --- just a fancy pattern matching trick --- and say that
| it's not _real_ AI until some other arbitrary benchmark is met.
|
| "AI" is just what machine learning can't quite do yet.
|
| > collection of very sophisticated pattern matching algorithms
|
| What do you think human brains are? Humans are Turing machines
| as well --- all physical computers are. We process inputs,
| match them against internal state, and generate outputs. You
| can't criticize AI on the grounds that it's "pattern matching":
| _everything_ is pattern matching.
|
| You can't look at the output of GPT-3 and tell me that it's
| some kind of dumb regular expression here. You just can't.
|
| https://www.gwern.net/GPT-3
| neatze wrote:
| Except, human brain changes on second by second bases, it is
| highly distributed and concurrent system with unimaginable
| redundancy, furthermore brain capability of
| learning/adaptability from just from one or few
| examples/tries does not compare to any AI models; what aspect
| of GPT-3 you comparing to human brain, specifically ?
| photochemsyn wrote:
| Pattern recognition is certainly a major component of human
| cognition but is hardly the whole story. Solution of complex
| mathematical problems - those are not pattern recognition
| problems. Two quite similar equations may have radically
| different outputs for the same inputs, so that screws up the
| whole pattern recognition approach, doesn't it?
| m_ke wrote:
| The recent advances in deep learning are good enough to keep
| industry busy for the next 20 years. Each year has brought us
| substantial improvements and if you look past the popular
| benchmarks there's no sign of things slowing down.
| tmaly wrote:
| it will be interesting to see if they can start to load models
| like GPT-3 ASICs and realize some serious performance gains.
| antupis wrote:
| I think it is that we are going to hit soon local maximum where
| modern NN architectures with better vertical integration and
| better hardware are going to create lots of value but other
| hand doesn't bring use anywhere near AGI.
| m_ke wrote:
| No sane person believed that deep learning would get us to
| AGI withing the next few decades. Anyone saying so is doing
| it to fool VCs out of their money.
| short_sells_poo wrote:
| I'd like to agree with you but just 1 year ago every AI
| related thread here were chock full of people breathlessly
| predicting AGI within years... and presumably HN is a well
| informed community!
|
| It's often not about expertise or sanity, but AGI captures
| people's imagination to a huge extent and as evolution has
| conditioned us to do, we ascribe certain behaviors and
| desires to systems which only exhibit those by random
| chance.
| m_ke wrote:
| Sorry to break it to you but most of HN is not well
| informed. Look for some threads in your area of expertise
| for proof of that.
|
| Only people selling AGI are marketers and futurists.
| 0xFACEFEED wrote:
| As a layman I wonder when we're going to start bearing the
| fruit of all this effort in DL. In the average person's day to
| day life I mean.
|
| I don't rely on DL driven systems and I'm not even a skeptic. I
| _want_ it to work. But I can 't rely on these systems in the
| same way that I rely on my computer/phone, a light bulb, or a
| refrigerator. Is that ever going to change?
| m_ke wrote:
| Most people unlock their phones using a deep learning model
| based facial recognition system, they talk to their devices
| thanks to deep learning, they translate documents with
| transformers, even google maps uses GNNs for ETA estimates
| and routing (https://arxiv.org/abs/2108.11482).
|
| The cameras on mobile phones got so much better thanks to
| deep learning, snapchat filters, zoom backgrounds, etc all
| use CNNs.
| edgyquant wrote:
| These things work so well and are so ingrained we don't
| even notice them: which is what we should be going for
| potatolicious wrote:
| DL is already bearing fruit everywhere - the key is that the
| places where it works are narrow domains. In general the
| larger the scope of intelligent behavior the less successful
| it has been.
|
| Someone has already mentioned face unlock, but also dictation
| is miles ahead of where it used to be. Similarly text-to-
| speech is _absurdly_ better than it used to be and is
| approaching indistinguishability from human speech in some
| cases (again, narrow domains are more successful!)
|
| Smartwatches are capable of detecting falling motion and
| alerting emergency responders, and are increasingly able to
| detect (some types of) cardiac incidents. Again here the
| theme is intelligent behavior in very narrow domains, rather
| than some kind of general omni-capable intelligence.
|
| The list goes on, but I think there's a problem where so many
| companies have overpromised re: AI in more general
| circumstances. Voice assistants are still pretty primitive
| and unable to understand the vast majority of what users want
| to speak about. Self-driving still isn't here. To some degree
| I think the overpromising and underdelivering re: larger-
| scoped AI has poisoned the well against what _is_ working,
| which is intelligent systems in narrow domains, where they
| are absolutely rocking it.
| 0xFACEFEED wrote:
| > the key is that the places where it works are narrow
| domains
|
| I've observed that not only are the domains narrow, but the
| domains of domains are narrow. In other words the real-
| world applications are mostly limited to pattern
| recognition, reconstruction, and generation.
|
| What I wonder is this. Is DL a dead end?
|
| Are we going to reach a ceiling and only have Face ID,
| Snapchat filters, spam detection, and fall detection to
| show for it? Certainly there'll be creative people that'll
| come up with very clever applications of the technology.
| Maybe we'll even get almost-but-not-really-but-still-
| useful-actually vehicle autonomoy.
|
| I can't imagine a world without the transistor, the
| internet, ink, smart phones, satellites, etc. What I'm
| seeing coming out of DL is _super_ cool but it feels like a
| marginal improvement on what we have now and no more. And
| that 's fine... but a lot of very smart people that I know
| are heavily investing in AI because they're banking on it
| being the new big technological leap.
| potatolicious wrote:
| > What I'm seeing coming out of DL is super cool but it
| feels like a marginal improvement on what we have now and
| no more
|
| "Marginal" here seems to be doing a lot of heavy lifting
| and IMO isn't fair. The ultimate point of technology
| isn't to inspire a Jetsons-like sense of wonder (though
| it is nice when it happens), it's to make life better for
| people generally. The best technology winds up
| disappearing into the background and is unremarked-upon.
|
| Like better voice recognition or text-to-speech. We've
| become accustomed to computers being able to read things
| without sounding like complete robots - and the
| technology has become so successful that it's simply
| become the baseline expectation - nobody says "wow Google
| Assistant sounds so natural" - but if you trotted out a
| pre-DL voice synthesis model it would be immediately
| rejected.
|
| I also wouldn't characterize "ability to automatically
| detect cardiac episodes and summon help" as some kind of
| marginal improvement!
|
| I think there's a bit of confusion here re: a desire for
| DL to be the revolutionary discovery that enables a sci-
| fi expectation of AI (self driving cars! a virtual
| butler!), vs. the reality of DL being a powerful tool
| that enables vast improvements in various narrow domains
| - domains that can be highly consequential to everyday
| life, but ultimately isn't very sci-fi.
|
| Does that make DL a dead-end? For those who practice it
| we aren't close to the limits of what we can do - and
| there are vast, vast use cases that remain to be tackled,
| so no? But for those whose expectations are predicated on
| a sci-fi-inspired expectation, then maybe? It's likely DL
| in and of itself won't lead us to a fully-conversant
| virtual butler, for example.
|
| [edit] And to be fair - the sci-fi-level expectations
| were planted by lots of people in the industry! Lots of
| it was mindless hype by self-described thought leaders
| and various other folks wanting to suck up investment
| money, so it's not fair to blame folks generally for
| having overinflated expectations about ML. There's been a
| vast amount of confusion about the technology in large
| part because companies themselves have vastly overstated
| what it is.
| cowanon22 wrote:
| The truth is that artificial intelligence and machine learning
| today are still aspirational titles - the current tech simple
| does not learn or reason in any way that is similar to human or
| even biological reasoning.
|
| Data mining and probabilistic pattern recognition are much more
| accurate descriptions, but don't sound as exciting.
|
| It's definitely possible that true AI will one day exist, but it
| may be anywhere from 5 to 1000 years away. I suspect the current
| approaches will not resemble the final form when it comes to AI.
| debacle wrote:
| I think the calculus was that you didn't need "true" AI for a
| self-driving car.
|
| That still might be accurate, just maybe not in the near term.
| It may be controversial, but I think that humanity's hubris is
| the biggest barrier towards developing more effective AI.
| HenryKissinger wrote:
| The future is littered with The Onion headlines.
|
| "AI researchers drop silicon, turn to biological computers"
|
| "AI researchers discover AI biological computers not so
| artificial after all"
|
| "AI researchers to allow AI biological computers to grow
| neurons and grey matter"
|
| "AI researchers realize AI biological computers need oxygen and
| organic nutrients, not electricity"
|
| "Shortages of neurology textbooks as AI researchers switching
| careers to neurologists"
|
| "AI researchers turn to live human farms to grow brains"
|
| "AI researchers realize brains in a vat don't work, need
| sensory inputs, to farm human heads instead of just the brain"
|
| "AI researchers realize heads in a vat don't work, need the
| rest of the body"
|
| "'Why spend billions on cutting edge quantum processing units
| when a normal human will do the job?' asks machine learning
| pioneer"
|
| "AI researchers give up, say human brains are better than
| supercomputers for a fraction of the cost"
|
| "AI research community spent $635,920 billion and 537 years,
| has produced a human, says GoogleSoft-Zon report"
|
| The end product of AI research will be Homo Sapiens.
| Veedrac wrote:
| The article "An Inconvenient Truth About AI > AI won't surpass
| human intelligence anytime soon" doesn't even once mention or
| justify the claim in its title.
|
| The article "Deep Learning's Diminishing Returns" was discussed a
| week ago, here: https://news.ycombinator.com/item?id=28646256.
| varelse wrote:
| There are still plenty of targets to hit with unitasking AI like
| we have that could each have their imagenet moment. But there is
| a shortage of domain experts who understand both those targets
| and AI.
|
| That creates opportunity for domain experts who learn AI. Less so
| the other way around because these domains are generally more
| complicated than AI and lack the unending tsunami of online
| courses to learn the details.
| gumby wrote:
| Sadly these days the word for "program" is "AI"
| jasonsync wrote:
| We once utilized an ad agency that relied heavily on "AI" to pre-
| test ad creative.
|
| The agency presented regional branding campaign creative with our
| national flag flying at half mast. The AI predicted success. The
| ad agency stood by the AI.
|
| Certainly would have generated clicks.
|
| But the ad agency lost a customer. Not sure if the AI would have
| predicted that!
| 6gvONxR4sf7o wrote:
| I'd bet people outside the field are going to be disillusioned
| before long as the marketing BS falls apart. Even years ago, I
| was already getting questions from friends that surprised me in
| how detached their perceptions are from the reality of state of
| the art ML. Eventually, they'll realize that no, we're not
| building superhuman AI today or soon.
|
| However, there's a ton of cool shit happening and tons of stuff
| already powering industry $$$. Some of this works because of
| mismatched expectations, like companies who are trying to sell
| magic. They will fail as their customers stop buying their
| products that fail spectacularly 2% of the time, but really can't
| fail 2% of the time.
|
| But there's tons of room in the market for stuff that fails
| spectacularly 2% of the time. For that reason, I don't see a
| "real" reckoning coming. People's expectations are too high,
| sure. But the reality is still Good Enough(tm) for a lot of
| problems and getting better.
| ChefboyOG wrote:
| There certainly are tasks where a 2% failure rate is fine, but
| even more importantly, where we see ML/DL having the biggest
| impact today is in regards to complex system-type problems,
| where there is often a less-than-discrete notion of failure.
|
| Looking at apps we use every day, almost all of them owe some
| core feature to ML/DL. ETA prediction, translation, search,
| spam filtering, speech synthesis, autocomplete, recommendation
| engines, fraud detection--and that's not even touching the
| world of computer vision behind nearly every popular photo app.
|
| A key understanding gap in the general public's knowledge of ML
| is that people think AI === Skynet, and they've therefore been
| lied to about the field's progress and impact, when in reality,
| they probably interface with a dozen pieces of technology that
| are built on top of recent breakthroughs in ML/DL.
| Animats wrote:
| _But there 's tons of room in the market for stuff that fails
| spectacularly 2% of the time._
|
| That's the key insight. It's why the commercial successes of AI
| have been primarily in advertising, marketing, and investing.
| worik wrote:
| Investing?
|
| Is there any evidence that any AI is better than index buy
| and hold?
|
| I do not think there is.
| tomComb wrote:
| The only thing consistently better than that is insider
| info, which is very common and quite reliable.
| ssivark wrote:
| That's an interesting insight. What are the long term
| societal implications of enabling/inducing (Jevons paradox)
| activities which are not "important enough" to need 100%
| correctness?
| Jensson wrote:
| > commercial successes of AI have been primarily in
| advertising, marketing, and investing.
|
| Which is why people say that it has few useful applications.
| People don't care if those areas become more effective.
| Another important application is mass surveillance, which
| people would also argue isn't a good thing.
| credit_guy wrote:
| Can I propose another insight? AI sometimes works and works
| spectacularly and does not fail in 2% of the cases, or even
| in 0.02%. Just look at iPhone photos. But that AI application
| require deep domain knowledge + deep AI (or, well, ML,
| whatever) knowledge. You can't simply throw sklearn at an
| arbitrary problem and expect to get great results.
| 6gvONxR4sf7o wrote:
| IPhone photos do some weird shit a small portion of the
| time too. I've had mostly great experiences as well as the
| rare unrealistically colored outcomes or misplaced bokeh.
| ashtonkem wrote:
| This appears to be the pattern of AI development overall: booms
| and busts of hype and disillusionment that leave behind useful
| tools that most don't consciously consider to be remnants of a
| previous AI bubble.
| Animats wrote:
| This round got a lot further than the last round.
|
| I went through Stanford CS in the mid 1980s, just as it was
| becoming clear that expert systems didn't really do much.
| There had been Stanford faculty running around claiming
| "strong AI real soon now" to Congress. Big-time denial as the
| field collapsed. Almost all the 1980s AI startups went bust.
| The "AI winter" followed. Today, expert systems are almost
| forgotten.
|
| This time around, there are large, successful industries
| using machine learning. Tens of thousands of people
| understand it. It's still not "strong AI", but it's useful
| and profitable. So work will continue.
|
| We're still at least one big idea short of "strong AI". A
| place to look for it is in "common sense", narrowly defined
| as "not doing something really bad in the next 30 seconds".
| This just requires lower mammal level AI, not human level.
| Something with the motion planning and survival capabilities
| of a squirrel, for example, would be a big advance.
|
| (I once had a conversation with Rod Brooks about this. He'd
| been building six-legged robot insects, and was giving a talk
| about how his group was making the jump to human level AI,
| with a project called "Cog". I asked why such a big jump? Why
| not try for a robot mouse, which might be within reach. He
| said "I don't want to go down in history as the man who
| created the world's best robot mouse". Cog was a flop, and
| Brooks goes down in history as the the creator of the robot
| vacuum cleaner.)
| dreamcompiler wrote:
| > There had been Stanford faculty running around claiming
| "strong AI real soon now" to Congress.
|
| That would be Ed Feigenbaum, the man who (IMHO) almost
| single-handedly brought on the AI winter of the 80s.
| Because of him we had to call all the AI research we did in
| the 90s something other than "AI" lest it get shut down
| instantly.
| Animats wrote:
| Um, yes. His book. "The Fifth Generation - Artificial
| Intelligence and Japan's Computer Challenge to the World"
| lays out his position. For a short version, here's his
| testimony before a congressional committee, asking for
| funding.[1] "The era of reasoning machines is inevitable.
| It is the "manifest destiny" of computing."
|
| He was taken seriously at the time. Chief Scientist of
| the USAF at one point. Turing Award.
|
| [1] https://stacks.stanford.edu/file/druid:mv321kw4621/mv
| 321kw46...
| listenallyall wrote:
| > The era of reasoning machines is inevitable. It is the
| "manifest destiny" of computing."
|
| Isn't this pretty much the current opinion of the
| majority of the thousands of AI researchers and
| programmers today? Maybe this guy was early to the party
| but his vision seems in alignment with today's
| practitioners.
| m463 wrote:
| that's the right answer. New technologies are always
| overestimated in the short term, but underestimated in the
| long term. The long-term usually is not a generalized
| solution, but is still a true solution.
|
| speech recognition hype... now most phone trees use them.
|
| electric car hype... now california is full of them.
|
| self driving cars... now many new cars have driver assist.
| mdoms wrote:
| The same will happen with deep learning networks because if
| we're honest no serious researcher would really consider this
| "AI" either. It's just a big self-correcting linear algebra
| machine. Researchers will keep using that "AI" label for as
| long as it gets them research grants or selling products.
| bopbeepboop wrote:
| Why do you think DNNs are unrelated to "AI"?
|
| They develop an internal language[*] through experience in
| which they model the world, and make decisions based upon
| that language; what more do you want from them?
|
| [*] You can interpret that "linear algebra machine" as the
| Euclidean VM modeling some latent type theory; Curry-Howard
| + algebra-geometry correspondences connect systems of
| difference equations to type theories.
| WanderPanda wrote:
| I wouldn't call it a "linear algebra machine" if the
| central point (of NNs) is that they are highly non-linear
| (the deeper the more non-linear, roughly)
| flir wrote:
| When it works, it's no longer AI.
|
| I _know_ I 'm paraphrasing someone smarter than me there.
|
| (Edit: Larry Tesler maybe)
| Jensson wrote:
| People believe that achieving X means the agent will be
| intelligent and can thus do Y, Z, A, B, C etc.
|
| But then someone builds an agent that can do X, but it
| can't do anything else. So people don't view it as
| intelligent, since it doesn't do any of all the other
| things people associated with being able to do X.
|
| So it has nothing to do with being used to computers
| solving new problems, it is that we now realize that
| solving those problems doesn't really require the level of
| intelligence we thought it would. Of course solving new
| problems is great, but it isn't nearly as exciting as being
| able to create intelligent agents.
|
| Edit: And it isn't just hobbyists making this mistake, you
| see big AI researchers often make it as well and they have
| for a long time. You will find tons of articles of AI
| researchers saying something along the lines of "Now we
| solved X, therefore solving A, B, C, D, E, F is just around
| the corner!", but they basically never deliver on that.
| Intelligence was harder than they thought.
| 6gvONxR4sf7o wrote:
| Exactly. People make the case that something along the
| lines of AGI is necessary and sufficient for some task,
| then someone comes along and solve it with something
| completely unlike AGI, proving that it's not a necessary
| condition. Or in more straightforward but less precise
| terms, solving this is AI and then it isn't.
| Blammar wrote:
| Don't you think, though, that each boom and bust cycle leaves
| us closer to real accomplishments?
|
| We now have protein folders and superhuman Go players --
| that's new.
|
| I agree that ML ("AI") is currently at the alchemy stage.
| And, guess what? A neural network isn't even Turing complete!
| [citation needed - correct me if I am wrong.] So ML can only
| compute SOME functions.
|
| AGI, when it comes, and believe me, it will, will have ML as
| part of its structure, but only a small part.
| randcraw wrote:
| Of course that depends on what you mean by "real
| accomplishments".
|
| It seems to me that that deep nets have effectively
| maximized the potential of using gradient pursuit to model
| patterns. But if you remove gradients from your data, or
| shrink your data down to tens of samples, or shift the
| problem to logic, or need to use functions that aren't
| convex or differentiable, deep nets run smack into a wall.
|
| Luckily human perception makes extensive use of gradients,
| as does most search, so problems in those arenas have been
| unsurprisingly amenable to solution using deep nets
| (vision, speech, game play, etc). But many of the problems
| that remain untouched by DL, like human cognition, are NOT
| driven by gradients. Will deep nets eventually fill that
| void? I doubt it. You can convert only so many problems
| with big data into gradients to pursue them efficiently
| with DL before that transformation trick runs out of steam.
|
| Personally I think deep net language modeling is one of
| those areas, and soon we'll encounter the limit of their
| generalizable contextual phrase association. Then because
| deep nets are so difficult to selectively revise or extend
| the specifics that they have learned, the vanguard of ever
| more complex deep nets (transformers) will eventually sink
| beneath their own weight, taking the last best hope for DL-
| based general AI with them.
| nicolapede wrote:
| > But many of the problems that remain untouched by DL,
| like human cognition, are NOT driven by gradients.
|
| This seems to me quite a deep insight. But how would you
| formally define a process that is gradient-based?
| 6gvONxR4sf7o wrote:
| I mean, each cycle does leave us with real accomplishments.
| If the question is whether it leaves us closer to AGI, then
| it's an open question. Like, when AGI happens, it will
| certainly trace its roots back to things happening in each
| period, but its roots will also go back to Gauss and Newton
| and co, so nobody knows whether it gets us closer in the
| way you probably mean.
| ashtonkem wrote:
| I have no idea if each boom brings us closer. Scientific
| discovery is not exactly a linear process; we can't observe
| before the fact if we're getting closer or are on a dead
| end path, and that's even assuming we can ever get there at
| all.
|
| But it's hard to deny that each boom doesn't give us
| something useful. Neural Nets might not exactly make an
| AGI, but they do have uses.
| Dr_Birdbrain wrote:
| ML is Turing complete in the sense that every computable
| function can be approximated to arbitrary precision by a
| 3-layer neural network. Classic result from the 90s, the
| paper in question (iirc) is titled something like "neural
| networks are universal approximators" Turing machines also
| can only approximate to arbitrary precision, so the
| computation models are equivalent.
| narrator wrote:
| I think there are these technological problems, many that
| overlap with AI, that a lot of politically powerful people
| want to come to fruition to enable the World Economic Forum
| lauded "Fourth Industrial Revolution" [1] and they are
| willing to invest insane amounts of money in them without
| expecting much of a return, thus creating really big p/e
| ratios. Examples include:
|
| 1. Self-driving cars. The ultimate being no steering wheel
| and carefully software controlled and permissioned as to
| where you are allowed to go.
|
| 2. AI powered Body monitoring devices (e.g Fitbit) that can
| be used to biologically monitor millions of consumers body
| states and functions.
|
| 3. Any new kind of surveillance technology. Miracle mass
| surveillance is something that's really interesting to this
| market.
|
| 4. Large centralized social media networks that aggressively
| moderate content with AI.
|
| 5. Various kinds of transhumanist biotech that design all
| sorts of biotech stuff with AI. What the heck is Calico up to
| anyway?
|
| 6. Blockchain stuff. AI enabled or otherwise.
|
| They are also interested in environment tech like any kind of
| alternative energy, no matter how speculative or impractical
| and fake meat, but I digress.
|
| So I would guess the bubble is the WEF crowd and friends with
| absurd amounts of money trying to make their future, for good
| or ill, a reality and investing without much regard for the
| economics of the project.
|
| [1]https://www.weforum.org/agenda/2016/01/what-is-the-fourth-
| in...
| sli wrote:
| I worked for a startup for a while that, late in its life,
| decided it needed to use AI in their product. Then that turned
| into needing ML in the product. Then it turned out it was just
| the owners trying to market the product as AI and ML powered
| when half of the product was the frontend used to configure the
| decision engine. The manually-configured decision engine was
| being sold as AI and ML, with the terms used interchangeably. I
| was actually a little surprised when it didn't work out, but
| not _that_ surprised.
| rscho wrote:
| The main issue is that people have unrealistic expectations in
| domains that are personally relevant, medicine being an iconic
| example. So, there there might still be a 'reckoning' of sorts,
| even if Walmart applies AI successfully behind the scenes to
| power non-critical user-facing stuff.
| 6gvONxR4sf7o wrote:
| Oh for sure. I guess I could have worded it better (not just
| scare quoted "real"), but it seems like a public expectations
| reckoning will happen, but not a reckoning that really
| affects practitioners who aren't 'script-kiddies' for lack of
| a better term.
|
| (side note, does anyone know a less insulting way to refer to
| the group people call 'script-kiddies' that still gets the
| same point across?)
| CityOfThrowaway wrote:
| To your side note: "technicians" is a reasonable label
| here. They know how to use a technique, but aren't
| themselves capable of creating one.
| popcorncowboy wrote:
| True Scotsmen. (I mean your practitioners.. everyone else
| is of course, not a true scotsman).
| 573e91b4e282b2 wrote:
| I don't believe that there is a polite way to say "These
| people have no idea what they're actually doing, they're
| applying other people's logic blindly, and if anything goes
| wrong they're stuck."
|
| If that's not what you meant to communicate, you should
| probably explain more what you mean by 'script kiddie'.
| m0zg wrote:
| Precisely. AI outside the beaten-to-death academic problems and
| datasets is a vast and completely un-plowed field with enormous
| potential. And unlike last time, there are some sub-fields of
| this that do actually work this time (computer vision, signal
| processing, speech recognition/synthesis, some NLP tasks, etc)
| and beat older baselines, often by a mile. There is a lot of
| froth, but there's quite a bit of substance as well, and the
| field is moving faster than any other I have ever experienced.
| Jugurtha wrote:
| > _Some of this works because of mismatched expectations, like
| companies who are trying to sell magic._
|
| We were asked to look into a company that had an, I kid you
| not, "AI-powered problem solving platform".
|
| The suggestions it spit out in their demo for agriculture were
| things like "What if the earth was upside down", and I read it
| like "Maaaaan, what if, like, just imagine like, the earth
| wasn't like below, but it was like, above... Duuuuude. Just
| imagine.". i.e: you could get these recommendations with a few
| dollars worth of haschish.
| snidane wrote:
| Human level AI in 2021 is still "tree climbing with one's eyes
| on the moon" (Hubert Dreyfus).
| worik wrote:
| I am very disappointed that I do not have a self driving car.
|
| Looks like I never will get one. Too hard, it is.
| monkeydust wrote:
| There's so much BS around use if AI/ML in the enterprise space
| right now.
|
| It's totally self perpetuating.
|
| My sales team keep asking me to add AI to X product. Doesn't
| matter if it's not required, or even makes sense they ask because
| the competition is doing it and they get asked by our
| clients/prospects in an expecting tone, if we offer some AI on
| our products.
|
| There are places we do offer it where it genuinely adds value but
| this 'sprinkle AI on everything for the sake of it' needs to die.
| fzingle wrote:
| > My sales team keep asking me to add AI to X product. Doesn't
| matter if it's not required, or even makes sense they ask
| because the competition is doing it and they get asked by our
| clients/prospects in an expecting tone, if we offer some AI on
| our products.
|
| It could be worse, they might be asking you to put it on a
| blockchain :P
| [deleted]
| version_five wrote:
| Currently, the cost of ML R&D has a floor based on what
| advertising companies are willing to pay people to work in the
| space. This actually has a huge upside, as most of the tools that
| have made ML so accessible (pytorch, tensorflow, all the research
| advances) are coming out of these companies. But it has the
| downside that if I want to get someone to work on my ML problem I
| have to compete with what google and fb can pay.
|
| A consequence, I guess, is that there are lots of unexplored /
| underexplored problems waiting to be tackled, and there are tools
| around that can make it happen. If there is a reckoning in the
| advertising space, there will be lots of other applications to
| focus on.
| airstrike wrote:
| Well, one issue is that Google and FB can finance said research
| because they make so much money off ads to begin with, and by
| doing so essentially provide enough funding to keep the field
| going
|
| So this reckoning in the advertising space would only be a net
| positive for society if others came in to fill that funding gap
| and threw enough money at researchers to keep the field afloat
| in a similar fashion
| dr_dshiv wrote:
| So long as researchers are willing to reify AI as a thing, the
| collective delusion will continue. I'm specifically looking at
| researchers focused on the ethics of AI; they set up AI as a
| thing more than other researchers. AI is not a thing. It's a
| field of research. Know thyself.
| gibsonf1 wrote:
| Right now Machine Learning with "neural" networks is all A and no
| I.
| sabhiram wrote:
| We get better and better at building pattern matchers that
| carefully oscillate between overfit and under-parametrized.
|
| Funny how we spend so much time and effort to mimic something
| that grows free by the billions :)
| woopwoop wrote:
| All of their arguments for why deep learning is reaching its
| limits seem to me to be arguments for why its future is bright.
| The fact that we have no even halfway plausible theory of why
| deep models generalize well, or why training them via sgd works,
| or even what the effect of depth is in a neural network, is
| exciting, not discouraging. I do not believe these are impossible
| problems to solve (well maybe the generalization one, but even
| there surely more can be said than the current state of knowledge
| which is basically a shrug). And even partial solutions should
| yield big practical steps forward.
| m_ke wrote:
| some good insights on depth: https://arxiv.org/abs/2106.09647
| carom wrote:
| Am I the only one who just sees links to other articles? Checked
| on desktop and mobile.
| pineconewarrior wrote:
| Same, even with content blockers off!
| lifekaizen wrote:
| Seems to assume carbon emissions from electricity will stay
| constant, yet that is likely to fall which reduces the impact in
| one dimension (cost remains).[0][1]
|
| [0]Diminishing Returns https://spectrum.ieee.org/deep-learning-
| computational-cost
|
| [1] renewables and non-renewables in $.05~.15/kWh range
| https://www.forbes.com/sites/dominicdudley/2019/05/29/renewa...
| hwers wrote:
| I used to be a huge sceptic of this cycle of deep learning. But
| not anymore. GANs are incredible and there's a huge amount of low
| hanging fruit left to pluck. I definitely don't believe AI will
| repeat it's history (history doesn't _have_ to cycle).
| bilater wrote:
| Can you elaborate on some of the use cases of GANs? I'm
| particularly interested in generative content but the output
| has been pretty meh so far.
| hwers wrote:
| Yeah I was totally anti hype when everyone was excited about
| style transfer and stuff like that. I found most of the
| VQGAN+CLIP way overhyped (sure it's abstract and funky but
| you get bored of that look after 10 images - it's like that
| deep dream filter from 5 years ago, shouldn't we have
| progressed more by now) but lately CLIP guided diffusion has
| blown me away. I literally thought it was all deception and
| cherry picking until I tried it myself[1] and realized the
| amazing outputs from https://twitter.com/RiversHaveWings are
| real.
|
| [1] https://colab.research.google.com/drive/1QBsaDAZv8np29FPb
| vjf...
| supperburg wrote:
| Imagine every program that fits within 10TB. It's a finite number
| of programs. Inside there is GTP-3, 2 and 1. Every GAN. Tesla
| FSD. It's everything. What this entire industry is really about
| at its core is finding programs in that set that behave in
| intelligent ways. That's all. Recently we've had success in
| letting the computer dig through programs automatically until it
| finds a good one. So overnight we went from testing maybe
| hundreds per year to billions per month or something like that.
| So obviously things have been getting weird.
|
| Imagine how many we've dug up so far out of the total set. It's
| an infinitesimal percentage. We haven't even scratched the
| surface.
|
| Imagine the set. What's inside? Is there something inside that we
| will regret?
| hyperion2010 wrote:
| In order to search that space you have to have a specification
| language for the behavior of the target program. Things like
| copilot/GPT-3 currently take natural language as input, but
| that cannot be used for anything that needs to be verifiable or
| have correctness. Maybe start by having the nets generate
| implementations of the IETF RFCs in a variety of languages
| given the text of the RFC as input? Not easy.
| bmc7505 wrote:
| While there are those in the AI community who believe scaling
| laws will unearth such programs, they are deluded. There are
| problems which scale faster than our computational resources
| allow, even if hardware scaling continues unabated for
| millennia to come.
|
| The space of all programs in 10TB is far too large to count,
| even if we could harness galactic computation. Even within a
| much smaller search space, there are valid programs which
| cannot be found by gradient descent. Let BT(n) be the number of
| distinct binary trees less than or equal to height n. This
| number scales according to the following recurrence relation:
| BT(n+1)=(BT(n)+2)2-1
|
| Consider the space of all binary trees of height 20 - there
| fewer atoms in the visible universe. And this is just laying
| out bits on a hard drive. There are other functions (e.g. Busy
| beaver and friends) which scale even faster. The space of valid
| programs in 10TB is too large to enumerate, never mind
| evaluate.
|
| In case anyone here is interested in learning more about
| program synthesis, there is a new workshop at NeurIPS 2021
| which explores some of these topics. You can check it out here:
| https://aiplans.github.io/
| supperburg wrote:
| Yeah
| Veedrac wrote:
| Like most anti-AGI arguments, this one disproves humans, and
| is therefore wrong.
| twofornone wrote:
| You're not wrong, but the numbers are quite different if you
| have a strong search heuristic. And that is where neural nets
| excel. See AlphaGo and derivatives. So this isn't just some
| random gradient descent exploring the full space; a properly
| designed and trained neural net will effectively cordon off
| vast swaths of the search space, and suddenly the problem is
| probably many orders of magnitude more tractable.
|
| That's really why deep learning shines in technical
| applications. Solution search heuristics were until recently
| solely within the domain of biological neural networks; now
| we have created technology which is capable of extracting
| superior heuristics over the course of learning. And it's
| already paying off in industrial science, despite the cries
| of naysayers, though the applications are still in infancy.
| bmc7505 wrote:
| There is a vast chasm of computational complexity between
| Chess, Go, and protein folding, to program induction.
| Unlike problems where the configuration space grows
| exponentially, the space of valid programs is at least
| super-exponential and depending on the language family,
| (e.g. context free, context sensitive, recursively
| enumerable), can often be undecidable. Furthermore, many
| language induction problems do not have optimal
| substructure or overlapping subproblems, two important
| prerequisites for reinforcement learning to work. In these
| settings, gradient-based heuristics will only get you so
| far.
|
| If you are interested in learning more about the limits of
| gradient descent, you should look into the TerpreT problem
| [1]. There are surprisingly tiny Boolean circuits which can
| be found using constraint solving, but we have not yet been
| able to learn despite the success of reinforcement learning
| in other domains. I'm not saying that program induction is
| impossible, but it is extremely hard even for relative
| "simple" languages like source code.
|
| [1]: https://arxiv.org/pdf/1608.04428.pdf
| twofornone wrote:
| So what do you think is different about human learning
| that allows us to learn to find solutions in such large
| spaces?
| bhntr3 wrote:
| Thank you. I just finished my master's thesis in program
| synthesis/induction. You're explaining this better than I
| could.
| supperburg wrote:
| And don't forget that we haven't even really begun to
| search the areas that resemble aspects of human neural
| anatomy. The ultimate heuristic.
| metalliqaz wrote:
| >Is there something inside that we will regret?
|
| Unambiguously yes. 10TB is a lot of space. That's large enough
| for a program that does nothing but show 8 hours of HD footage
| of your kids being tortured and eaten.
|
| It's also more than enough to define a program that would
| reliably precipitate a global thermonuclear war if connected to
| the Internet.
| jeremysalwen wrote:
| That's a very interesting way of thinking about things, but I
| think unfortunately the search techniques we are using
| (gradient descent with certain datasets/self-supervision tasks)
| limit us to exploring a very small subset of that 10TB of
| possible programs. A search strategy that could actually find
| the optimal 10TB program to accomplish some task would actually
| be a superintelligence beyond anything we have ever thought of
| creating using present day AI techniques.
| shawnz wrote:
| Maybe one of the lesser search strategies will find a better
| search strategy? Isn't that how humans came around after all?
| visarga wrote:
| Evolution used an entire planet and billions of years to
| come up with us.
| bobthechef wrote:
| > find the optimal 10TB program to accomplish some task would
| actually be a superintelligence
|
| I am having a difficult time understanding what the operative
| meaning of "intelligence" here is. "AI" doesn't transcend the
| Turing machine. Intelligence doesn't mean more computer
| cycles per unit time either and compute cycles don't
| transcend the TM. What makes intelligence intelligence is
| what it can in principle do; speed is irrelevant. There is no
| essential difference between AI and non-AI.
| FredPret wrote:
| But we _want_ our search strategy to limit us to the
| infinitesimal fraction of programs that does useful work.
|
| 99.99% of possible bit arrangements in that 10TB will not do
| anything good, or useful, or even anything at all.
| margalabargala wrote:
| The number possible bit arrangements in 10TB is 2 raised to
| the power of 80 trillion.
|
| This comes out to 3.1 x 10^24082399653118
|
| To emphasize the size of that search space- let us measure
| the diameter of the observable universe in planck lengths,
| the shortest possible length. We would need over 24
| trillion digits to write the number of universes we would
| need for the quantity of planck lengths to equal the number
| of possible programs.
|
| That 99.99% number is missing several billion additional
| 9s.
| soVeryTired wrote:
| Meh. Imagine the set of 10k x 10k pixel RGB images. The mona
| lisa is in there, as are images from your worst nightmares.
|
| But almost all of the images in that set look like grey fuzz.
| On average there's nothing interesting there.
| [deleted]
| undershirt wrote:
| https://en.wikipedia.org/wiki/The_Library_of_Babel
| phyalow wrote:
| Sorry I dont follow at all.
|
| Are you stating that you have a corpus of 10TB of software
| source and you are frankenstiening it with interesting results?
| I find that hard to believe.. Surely its like Monkeys and
| typewriters, 10TB in that context wouldnt be nearly enough.
| mrbungie wrote:
| Parent is not talking about a 10TB corpus of software but the
| corpus/set of all possible software weighing 10TB (or less I
| guess, specially if you take the lottery ticket hypothesis
| into consideration).
|
| A sibling post shows this, it presents the concept very well:
| https://en.wikipedia.org/wiki/The_Library_of_Babel
___________________________________________________________________
(page generated 2021-09-30 23:01 UTC)