[HN Gopher] Deep reinforcement learning is a waste of time (2019)
___________________________________________________________________
Deep reinforcement learning is a waste of time (2019)
Author : headalgorithm
Score : 75 points
Date : 2021-07-10 15:56 UTC (7 hours ago)
(HTM) web link (www.jtoy.net)
(TXT) w3m dump (www.jtoy.net)
| thunderbird120 wrote:
| There has been a lot of discussion recently in certain AI
| research communities recently about whether or not the entire
| idea of reinforcement learning is even necessary. Self supervised
| sequence models seem to represent the best path forward for
| general purpose problem solving agents because we can essentially
| just keep improving them by increasing parameter counts. There
| was a previous HN discussion about the idea here
| https://news.ycombinator.com/item?id=27659526
| thomasahle wrote:
| Those models require a _lot_ of training data to exist. And I
| have never seen supervised sequence model achieve super human
| performance at anything, the way AlphaGo did.
| MathYouF wrote:
| Makes me wonder if anyone has looked into using genetic
| algorithms combined with RL where the genetics determine the
| reward function.
|
| This seems to be how humans have evolved. Ultimately, all living
| animals are here based on only one reward function, the ability
| to have had an uninterrupted chain of reproduction. Our nervous
| system provides stimuli and our brains chemicals provide positive
| or negative rewards (pain or pleasure) that optimize us taking
| actions that result in having an uninterrupted chain of
| reproduction (it's why sex feels good and putting your hand on a
| stove feels bad).
|
| Presumably, both the reward function within our brain, as well as
| the signal it interprets (nervous system) evolved to find a more
| optimal combination of inputs and reward scalars for each input
| to maximize for this singular goal (reproduction).
|
| Maybe we need to frame RL goals in much more simple terms, and
| allow genetic algorithms to evolve their own inputs and reward
| functions on their own.
|
| RL is one of my weakest fields of knowledge in the AI field, so
| I'm sure some of this has been tried before, I'm curious how much
| and what the results have been.
| javitury wrote:
| > anyone has looked into using genetic algorithms combined with
| RL where the genetics determine the reward function
|
| There is an introductory guide to rust, deep learning and
| genetic algorithms that I really like, it's called "Learning to
| fly" [^0]
|
| [0]: https://pwy.io/en/posts/learning-to-fly-pt1/
| kris-s wrote:
| > and allow genetic algorithms to evolve their own inputs and
| reward functions on their own
|
| I've been playing with genetic algorithms for years now as a
| hobby and this type approach was a dead end for me, the GA
| entities would just "game the system" as it were and would
| min/max in surprising ways.
|
| My latest genetic algorithm creation https://littlefish.fish
| has performed far better at pattern recognition than I
| expected. I really think they've got massive potential.
| polishdude20 wrote:
| I think the stove and sex examples are on the right track but
| these qualities are also what every animal experiences. Well...
| judging by the face of a dog when he's humping your leg, I'm
| sure it feels good for him.
|
| Anyways, I think there's another ingredient that's missing that
| we humans uniquely have. I think that ingredient is the fear of
| death. The knowledge that of all our intellect and powers as a
| human, we will inevitably die. Its better summed up by terror
| management theory I believe.
| MathYouF wrote:
| Dogs have achieved an unbroken chain of survival dating back
| as far as your ancestors have, so they've achieved the same
| survival goals as you.
|
| They've managed to do so without our intellectual abilities,
| which goes to prove that our goals of making RL algorithms
| become "smart" is malformed since high level complex and
| abstract reasoning skills apparently aren't a necessarily
| prerequisite to survival, at least in our earth environment.
| cma wrote:
| Worker ants end the chain of survival, but are still
| necessary, so chain of surival among individuals is a
| flawed metric.
| MathYouF wrote:
| Indeed, the genetics of every living thing are not
| evidence of its fitness, but once it has reproduced, it
| is.
| lumost wrote:
| The problem with this view is that the outside world has such
| an immensely vast amount of data to it that the problem becomes
| uncomputable.
|
| It took ~3 billion years of real evolution to reach Humans.
| This evolution occurred on a planet scale, including naturally
| formed barriers which rose and fell, changes to climatic
| conditions, and even a few stellar events to shake up
| evolution. There isn't even a compelling reason to think that
| Humans couldn't have arisen anytime within the last ~300
| million years. Implying that the probabilities of intelligent
| life emerging are low, or that the conditions are poorly
| understood and rare.
|
| Effectively each attempt to learn an agent which has to
| interact with the real world runs into these problems. The
| solution is to make the reward more complex and the simulated
| environment more realistic - both actions which increase the
| computational costs of the problem faster than the improvements
| arrive.
| MathYouF wrote:
| Well the failure of multilayer perceptrons to converge into
| useful models was because of limited compute scale, which
| eventually was solved with the advent of powerful GPU's and
| CUDA popularising using their parallel computationa meant for
| graphics rendering for the linear operations used during back
| propogation.
|
| Maybe the problem isn't that the RL algorithm is wrong, but
| that it just doesn't work without a 3 billion year, atom
| resolution, planet scale computer.
| Barrin92 wrote:
| The purpose of any scientific field is to generate knowledge,
| i.e. to actually understand the conceptual underpinnings of
| something like intelligence.
|
| This idea that all that's necessarily for _engineering_
| intelligence is throw some chemicals into a bucket and turn the
| heat on is bad. By that logic you can just write a universe
| simulator, wait a million years and maybe you solve AI as a
| side challenge. if AI is just evolution and genetics and
| genetics is just physics just solve that and we 're good to go.
|
| It's like if someone tried to build a bridge and he just
| clobbers things together until it stands up and then prays that
| it doesn't fall down. That's not how engineering works
| obviously, but that's the attitude we have towards AI.
|
| What AI needs at this point is the very opposite. An actual
| theory of intelligence at a high level because we haven't
| really made progress on that front in decades.
| esac wrote:
| >genetic algorithms combined with RL where the genetics
| determine the reward function.
|
| I have been working on this problem for years (2+ as
| researcher, 2 as PhD student).
|
| The main issue is that evolution is both massively parallel and
| had plenty of runtime to get to human level intelligence.
|
| The person that pushes this evolution/evolved reward point is
| Andrew G. Barto and his students/collaborators over the years.
|
| Satinder Singh in particular is actively working on gradient
| based algorithms to find rewards (e.g.
| https://arxiv.org/abs/2102.06741)
|
| > Maybe we need to frame RL goals in much more simple terms,
| and allow genetic algorithms to evolve their own inputs and
| reward functions on their own.
|
| I was checking HN while the current iteration of this (gradient
| based, genetic was my master thesis) algorithm, the main
| complexity is figuring out:
|
| 1) What are the sub-goal e.g. grasping things 2) How to solve
| those goals e.g. motor control 3) How to do something useful,
| e.g. surviving
|
| Balancing those three processes is the current hurdle.
|
| For more info my email is delvermm at mila.quebec
| whimsicalism wrote:
| > The main issue is that evolution is both massively parallel
| and had plenty of runtime to get to human level intelligence.
|
| How many entities are we talking about for substantial
| evolution? I know that there have been 100 billion "humans"
| (not that it's so clear-cut) alive, so guessing this is on
| the order of ~trillions of entities to simulate some
| evolution for (but maybe I'm really underestimating the early
| tail of tons and tons of microorganisms and small short-lived
| life that got us to this point).
|
| Is the bandwidth of evolution that much larger than what we
| could possible simulate with computation, especially for a
| much simpler world/task than "generally survive"?
| lelandbatey wrote:
| Given that a single teaspoon of soil probably has about a
| billion bacterial organisms in it, I suspect you're a
| couple orders of magnitude short.
| goatlover wrote:
| Also, evolution isn't trying to get to human level
| intelligence. It's just one out of millions of adaptations
| that work, it's recent, and it's rare. Change Earths
| parameters a little over the past several million years, and
| maybe we don't evolve.
| hervature wrote:
| To me, this entire piece reads like goal-post moving and lacks
| understanding of the field and was incredibly premature.
|
| > I am confused as to why they continue to spend so much time on
| building Deep RL systems that beat games with clearly defined
| rules and point systems. I think everyone gets it now, with
| enough time, money, and computers almost brute forcing every
| single possible action, Deep RL can beat almost any game.
|
| Right, before AlphaGo, superhuman Go was decades away. Then it
| became obvious and therefore easy. He brings up a laundry folding
| robot almost as if discrete games can be trivially applied to
| robotics. If you think about it for one second, Shogi, the game
| that AlphaZero solves with the largest action space has 10^5
| actions. Consider a robot with 3 joints that can move in any
| dimension motors. Discretizing the continuous action space into
| 10 buckets yields an action space of 10^9. Something AlphaZero
| will never solve. In that sense, the entire analogy is a straw
| man and everyone in the field knows. This doesn't even touch the
| fact that 10 buckets would be useless for fine motor control.
|
| Since they published this, there have been great advancements in
| learning action embeddings for robotic tasks. With a couple of
| expert examples, an algorithm can learn to open a door
| effectively. That's not that far from folding laundry.
|
| Personally, my entire PhD thesis attempts to solves the exact
| deficiencies they raise. Simply put, the work goes on. Instead of
| saying that the field is useless, try to understand and
| contribute to it. At the very least, give technical reasons that
| it is a dead end path.
| lysozyme wrote:
| That's a really interesting way of looking at the difficulty of
| the problem that's being solved. I'm curious, how do you arrive
| at the number 10^9?
| hervature wrote:
| You have 3 joints with 3 dimensions. So 9 variables you can
| control. If each variable has 10 choices, and you need to
| make a choice for each, that is 10^9 possible combinations.
| whimsicalism wrote:
| "3 joints" -> What are the 3 dimensions that joints move
| along? I naively would have thought one or two (ie. how
| much spin on one axis and how much spin on the other axis)
| just by naively looking at my elbow for a second.
|
| For an arm of fixed length, polar coordinates have 2 dof.
| phunehehe0 wrote:
| I presume it's 3 dimensions for each joint, so 10^3 * 10^3 *
| 10^3
| YeGoblynQueenne wrote:
| >> With a couple of expert examples, an algorithm can learn to
| open a door effectively.
|
| "A door". How about "any door"? I'm curious- do you have a link
| to a demonstration or something similar?
| bruce343434 wrote:
| Note to author: TFA has a couple of spelling mistakes
| roenxi wrote:
| There is that lovely quote floating around that change is slower
| than people expect in the short term but faster in the long term.
|
| It is far too early to write of plain ol' deep reinforcement
| learning as a failure. It hasn't yet been 5 years since AlphaGo
| really shocked the unwary, and that was rather cutting edge at in
| 2016.
| beefman wrote:
| I think short- and long-term predictions are wrong in different
| ways. Short-term predictions often fail to come true, leading
| to the diagnosis of slowness. Long-term predictions often fail
| to even come false, because the world has changed so much the
| prediction is no longer applicable. In these cases, change
| _was_ faster than expected, but not in a way that settles the
| original point (often pertaining to progress or some other
| value judgement).
| QuesnayJr wrote:
| I first saw the quote in something by Robert Cringely. (The
| Internet claims that Cringely attributed it to Amara, but I
| couldn't find the original source. Maybe "Accidental Empires"?)
| AlexCoventry wrote:
| I saw a similar quote from Asimov or Heinlein or the like, in
| the introduction to a science-fiction anthology in the
| eighties.
| stevesimmons wrote:
| Maybe quoting Roy Amara, a 1960s Stanford Computer
| scientist. He said "we overestimate the impact of
| technology in the short-term and underestimate the effect
| in the long run" [0].
|
| I dimly remember similar sentiments in Arthur C. Clarke's
| Profiles of the Future (1662) [1], where he talked about
| "Hazards of Prophecy", where predictions suffered from
| either "failure of imagination" (predicted too little
| change) or "failure of nerve" (could/should have foreseen
| far reaching change, but chickened out and wrote down a
| watered-down version).
|
| As ACC said it "The failure of nerve seems to be the more
| common; it occurs when even given all the relevant facts
| the would-be prophet cannot see that they point to an
| inescapable conclusion."
|
| [0] https://fs.blog/2019/05/gates-law/
|
| [1] https://fabiusmaximus.com/2017/12/26/arthur-c-clarke-
| about-p...
| api wrote:
| Reward function design and overfitting or "cheating" (optimizing
| to some incidental thing instead of the real problem) is a major
| reason genetic and evolutionary AI approaches never took off. I
| don't think anyone ever figured out how to make reward function
| design easier or make any kind of unsupervised evolutionary
| learning work.
|
| I played with EC a lot in college and these systems were almost
| comically good at the "cheating" part of overfitting. I watched
| evolving programs do things like learn the scheduling behavior of
| the OS kernel (because the reward function was threaded) or the
| disk timing differences resulting from where different parts of
| the sample set were stored on the drive. They could guess the
| answer by inferring load time.
| toisanji wrote:
| how do you think the article stands with a paper just recently
| publishing "Reward is enough" by Richard Sutton (father of
| Reinforcement Learning)
| dannyz wrote:
| I find these kind of articles just perplexing. Research is
| incremental, tiny steps pushing the boundaries of knowledge.
| DeepMind has done things that were thought to be decades away
| using deep reinforcement learning. These research advancements
| may or may not end up being important for AGI in the future, but
| that's just what research is.
| freeone3000 wrote:
| It's refuting the premise that supervised RL becomes less
| supervised because you put the feedback in a handcrafted
| function and use a neural network. Deep RL in its current state
| should be grouped with Supervised RL, in other words (which is
| why I personally think that imitation learning is a great way
| forward, in contrast with the author). The issue is the amount
| of interactive tweaking and lack of a natural reward function
| that prevents DeepRL from being unsupervised.
| rich_sasha wrote:
| AlphaZero is not supervised, in the sense that it learned
| from known correct actions (earlier versions of AlphGo did
| learn from online games). So although it needed human
| supervision, sure, it didn't need us to provide correct
| answers.
|
| The authors point does somewhat stand that you don't have the
| problem of reward engineering in board games so they are a
| dead end from that point of view - they skirt around the core
| problem instead of tackling it.
| shadowlight wrote:
| I don't agree with this article but it is not perplexing at
| all. Dead ends exist. The universe is highly, highly limited
| and everything eventually has a dead end. The question is, are
| we there yet?
|
| For certain things yes, for other things no. But to assume
| there is never a dead end and that everything can be overcome
| through incremental development and research is patently a
| false assumption. There are many examples of dead ends within
| research and development.
|
| Thus in short his proposal is likely wrong, but it is not a
| perplexing proposal. Nor is his proposal guaranteed to be wrong
| and there is a possibility he may be right. For example Elon
| predicted self driving will be a finished problem in a year.
| Guess what?
|
| I actually sort of dislike this whole "perplexing" attitude
| that some people have. It's like yeah his opinion seems wrong
| or his opinion is not the norm, but there's no need to treat it
| as if it's "perplexing." It's like you observing animal
| behavior in a lab and your so "perplexed" on how someone can
| have a differing opinion.
|
| People can have differing opinions and sometimes these opinions
| can be right and overturn an existing paradigm.
|
| Instead of saying you find someone perplexing or strange, just
| say you disagree. It's more civil and it respects the underdogs
| of the past who fought against overwhelming odds to change
| entire schools of thought and bring our knowledge closer to
| answering the ultimate question.
|
| So perplexing how some people are so rude nowadays. See what I
| did there?
| shadowlight wrote:
| I actually find this technique used a lot on HN. They
| disagree with someone but they want to insult them without
| violating HN rules so they treat the person as if they're
| some kind of lab experiment and observing how they're
| behavior is so "strange" or "perplexing". The admins likely
| fail to see just how insulting these kinds of comments are.
|
| Perplexing is when someone jumps off a cliff while detonating
| a stick of dynamite. Someone with a differing opinion is NOT
| perplexing.
|
| "I find it so perplexing that someone would think that...
| despite that... " and so on.
|
| Really people should call it out. It's rude and manipulative.
| inglor_cz wrote:
| Thank you for formulating precisely what I felt, but could
| not describe in detail.
|
| Calling someone else's opinion basically outside any
| rational Overton window is usually meant as a veiled
| insult.
| abeppu wrote:
| I wish the author had picked a title like "Single-task
| reinforcement learning is a waste" or "Reward function
| engineering is a waste".
|
| At the very end of his essay he mentions some directions:
| homeostasis, Friston's free-energy principle, and predictive
| processing. I agree that all of these are very interesting. A few
| steps less ambitious is recent work on reinforcement learning to
| reach desired outcomes without specifying reward functions. All
| of these seem to require more focus on learning the model of
| dynamics in the relevant domain (what happens when I take action
| A from state S?) and less on value / policy learning.
|
| Ok great, but why can't that happen in the context of "deep
| reinforcement learning"? For complex environments with partial
| observability, don't we probably want something like states
| represented in an embedding? As we improve our model, is there
| anything broken about the approach of optimizing a differentiable
| function?
|
| "There are likely better approaches to deep RL, and they include
| ..." seems like a better framing than "deep RL is a waste of
| time."
| sdenton4 wrote:
| The new hotness this month just says 'screw it; treat RL as a
| supervised sequence problem with causal transformers.'
|
| https://arxiv.org/abs/2106.01345
|
| 'We consider the following shift in paradigm: instead of training
| a policy through conventional RL algorithms like temporal
| difference (TD) learning [6], we will train transformer models on
| collected experience using a sequence modeling objective. This
| will allow us to bypass the need for bootstrapping for long term
| credit assignment - thereby avoiding one of the "deadly triad"
| [6] known to destabilize RL. It also avoids the need for
| discounting future rewards, as typically done in TD learning,
| which can induce undesirable short-sighted behaviors.
| Additionally, we can make use of existing transformer frameworks
| widely used in language and vision that are easy to scale,
| utilizing a large body of work studying stable training of
| transformer models.'
| mountainriver wrote:
| Transformer all the things
| sillysaurusx wrote:
| Yuck. I hate how expensive transformers are. You can see it
| clearly in transformers vs gans; GANs can generate a frame in
| about 20ms, whereas it takes seconds or more to make a frame
| with transformers. It's not even clear that it's necessarily
| better quality.
|
| That said, I doubt we'll be able to make an RL GAN, so maybe
| this is the best way. Though now I wonder how well an RL GAN
| might work...
| singhblom wrote:
| Aren't actor critic algorithms very close to GANs already?
| You have a generator/actor/policy that produces data and a
| discriminator/critic/q that says if the data is good or
| bad. The critic trains on the data generated by the actor
| and some extra info given by the user (rewards or example
| data) and the actor learns from the signal given by the
| critic.
| sdenton4 wrote:
| Yeah, I work on on-device audio, so definitely agree on the
| expense problem. In audio we've now got a few different
| approaches that work really well for sequence modeling, and
| we're constantly finding cool ways to make inference run
| faster.
|
| Reframing reinforcement learning as sequence modeling
| /should/ make it possible to reuse (m)any of the approaches
| we use for audio, including GANs. Generative audio is
| nicely analogous to RL problems : There's complex state and
| interesting predictive distributions, which shift subtly
| over time, and you need to combine short-term good behavior
| (good individual samples) in a reasonable way to get good
| long-term behavior (matching melspectra).
| unixpickle wrote:
| I think you are conflating "Transformers" and
| "autoregressive models". Transformers are a general purpose
| architecture for transforming sequences into other
| sequences with self-attention. AR models / GANs are
| frameworks for generative modeling. The model architecture
| is almost entirely orthogonal to the generative framework.
|
| You can use transformers as part of GANs [1], and you can
| even use them as discriminative models for images [2].
|
| [1] https://arxiv.org/abs/2102.07074 [2]
| https://arxiv.org/abs/2010.11929
| PaulHoule wrote:
| Deep-learned convolutional nets work wonders for visual
| recognition. Visual recognition via "the old AI" looked
| impossible by 1980, but today it looks easy.
|
| (I see the visual segmentation models for self-driving cars from
| the "autonomous systems lab" in the next building over and
| think... It would be so easy to make something that honks for
| cyclists.)
|
| Text analysis, reinforcement learning, etc. seem to be areas
| where deep learning might very well reach a plateau. In the case
| of images the meaning is not changed by a random "hot pixel", but
| changing one letter in a sentence can reverse the meaning of a
| sentence, changing one piece position in a chess game is the
| difference between a win or loss. These "binary" situations
| aren't a good match with the assumptions of continuity,
| differentiablity and such that neural networks depend on.
| throwawaygh wrote:
| _> Text analysis, reinforcement learning, etc. seem to be areas
| where deep learning might very well reach a plateau... changing
| one piece position in a chess game is the difference between a
| win or loss. These "binary" situations aren't a good match
| with the assumptions of continuity, differentiablity and such
| that neural networks depend on._
|
| Deep learning can't do board games like chess and go?!
|
| Wait... are you using _pretty darn subtle_ dry sarcasm to argue
| that deep learning _won 't_ reach a plateau?
| PaulHoule wrote:
| I think that markov chain monte carlo is pretty cool even
| with lightweight playouts. (pick a random move)
|
| The neural net by itself is a "half-baked" chess or go
| player, it needs the MCMC to be a strong player. (MCMC plus
| lightweight playouts can beat me at chess if not at go.)
|
| Same with text-analysis, code generation and such. If you can
| build a hybrid system where the neural net comes up with
| half-baked answers that can be corrected by a system which is
| capable of comprehending things like "well-formed" and
| "valid" then you could be cooking with gas.
|
| What I am seeing though is that people aren't "beginning with
| the end in mind" the way the Wright Brothers did with flying,
| rather they are throwing stuff at the wall and seeing what
| sticks.
| soVeryTired wrote:
| I think you mean Monte Carlo Tree Search rather (MCTS)
| rather than MCMC?
| xvilka wrote:
| At this point it's clear that "AI" was overhyped (as usual), and
| we are heading into one more "winter". AGI is still probably
| centuries away.
| inglor_cz wrote:
| "probably centuries away"
|
| Making technological predictions on such a timescale is folly.
|
| 1821 is two centuries away - the minimum necessary to use a
| plural form of the noun "century". For a person of 1821, the
| technologies that we rely upon routinely would be completely
| unknown. What would they recognize? Not cars, not computers,
| not electrical appliances and light, not pretty much anything
| in our households save furniture.
|
| Of all the tech that you can meet in an American street of
| 2021, the only thing that would be somewhat familiar to people
| of 1821 would be guns.
| aikinai wrote:
| Is that clear? I see all sorts of new and useful
| implementations of AI coming out everyday.
| jstx1 wrote:
| You're implying that it's either AGI or nothing and I can't
| agree. There are plenty of applications and use cases for ML
| out there. Should that be called "AI"? I don't care too much
| personally. Is it overhyped? Sure. But there is enough useful
| stuff there that I don't expect another winter or AGI any time
| soon.
| throwawaygh wrote:
| I think AI is over-hyped now in exactly the same way that the
| dot com bubble over-estimated the impact of the internet at
| exactly the same time that most people were under-estimating
| the impact of the internet.
|
| "In 2020 everyone will laugh at the idiots investing in the
| dot coms, and also three of the five largest companies in the
| world will be internet companies founded after 1990" would've
| seemed like a contradiction. Both sides were wrong in the
| late 90s.
|
| The techno-hippies talking about AGI are insane, and so were
| the techno-hippies who imagined the internet would connect
| the world and solve all our problems.
|
| The MBAs are selling bullshit as always and a lot of it'll go
| bust.
|
| But there's a lot of value in the past decade of advancements
| in ML and a lot more to come, with a lot of chaff in the
| wheat.
|
| The future will be shaped by ML more than the average person
| thinks, _and_ those changes will be more prosaic than the
| hypsters would have us imagine, _and_ those changes will be
| huge in ways that people kind of see now but take on an
| unexpected shape.
|
| I think in 2050 we'll be nowhere close to fully autonomous
| robotics, but also the combined forces of the USA military
| will in aggregate have a robotic land/air/sea fighting force
| that's 1) larger than most of the world's militaries and 2)
| capable of toppling a nation-state like Iraq or Afghanistan
| completely autonomously (ie, without any remote control).
|
| I think in 2050 we won't have robotaxis, but every warehouse
| and port in the developed world will be a nearly lights-out
| operation.
|
| I think in 2050 we'll still have wait staff and baristas but
| most non-sitdown food prep establishments will have at most
| one employee.
|
| I think in 2050 chatbots will still be useless for replacing
| callcenter work but also video games will have incredibly
| immersive social environments that are at least as
| stimulating as real social interactions.
|
| And so on.
| Grimm1 wrote:
| I'm not sure how you reach that conclusion when we keep doing
| crazy things that haven't ever been possible before with AI
| literally like every 6 months right now.
| vsskanth wrote:
| Has there been any work on "growth" based RL models ? Like how a
| human baby starts off with a small brain and little capability to
| move around, but slowly starts to roll over, gets head control,
| crawl and eventually walk, grasp and develop fine motor control
| as their brains and physical abilities grow together.
|
| The current method looks to me like starting from scratch with a
| fully capable human with a huge untrained brain instead of
| progressively expanding their actuators and control plane
| mountainriver wrote:
| Yeah there is a field that does incremental learning and
| "lifelong learning" which employ these tactics. It's successful
| on some domains
| Digitalis33 wrote:
| I prefer to listen to people who actually put AI/ML models into
| production, instead of random programmer / AI dilettante types.
| soVeryTired wrote:
| But this is part of the author's point: where are all the
| production reinforcement learning models?
| cesaref wrote:
| You'll only hear one side of the coin. I would also want to
| hear from people who understand AI/ML and have decided _not_ to
| use it in an application domain. That 's probably the most
| valuable info, knowing when not to use a tool.
| [deleted]
| wizzwizz4 wrote:
| I understand ML reasonably well, and have decided to use
| classical AI instead, because I want real-time performance on
| low-end hardware (and I also want to be able to predict and
| extend the operating parameters). It's a lot harder, though,
| because I'll have to understand the problems at a fundamental
| (mathematical) level better than I currently do, and all the
| time I can hear a little voice saying "a neural network could
| do this in half an hour of work and a week of training".
| phonebucket wrote:
| I have to disagree with a good chunk of this article.
|
| The article completely misses the main advantage (to me) of
| reinforcement learning:
|
| Reinforcement learning allows you to optimise on non-
| differentiable outcomes.
|
| I can't differentiate real life, but I want to optimise a process
| within real life. This feels tantalisingly close to AGI. If I can
| figure out a reward function, I can use reinforcement learning.
|
| Yes, this requires a reward function to be defined. Yes, this is
| a challenge to AGI. But to say that the big labs are not aware
| that this is a challenge to AGI is unfair. DeepMind is actively
| investigating open ended learning:
| https://deepmind.com/research/publications/open-ended-learni....
|
| Just because the labs haven't tackled all the questions doesn't
| mean that they're not busy tackling difficult questions.
| soVeryTired wrote:
| Can you clarify what you mean by 'non-differentiable outcomes'?
| ArnoVW wrote:
| My guess: a function that you can evaluate but that you can
| not differentiate. Differentiate being "know the slope".
|
| Differentiable functions are great because you can run
| gradient descent on them in a very optimized way. Example: if
| your objective is to have a very high value, search in the
| direction that has a mounting slope.
|
| Though maybe I'm missing something 'cos it seems to me you
| can run gradient descend on non-differentiable functions. It
| just requires more evaluations.
| dang wrote:
| Discussed a bit at the time:
|
| _Deep Reinforcement Learning is a waste of time_ -
| https://news.ycombinator.com/item?id=21916775 - Dec 2019 (3
| comments)
| max_ wrote:
| AI people don't have a formal/scientific structure of
| intelligence.
|
| Noam Chomsky thinks that deep learning is more of engineering
| than science [0]
|
| He says what we need is something like a Turing machine but for
| intelligence.
|
| [0]: https://youtu.be/ndwIZPBs8Y4
| sdenton4 wrote:
| Why is Chomsky worth listening to on ML/AI?
|
| It's birds and airplanes: airplanes don't have feathers, but
| they solve different problems. Aerodynamics is a science,
| actual airplanes are engineered. But aerodynamics as a science
| evolved more slowly than actual flying airplanes; that's where
| we're at with ML.
| vsskanth wrote:
| As of 2021, has anything improved ? I'm not sure if this was
| before or after alphazero
| sdenton4 wrote:
| The article is about the failure of reinforcement learning to
| make it out of games and into any kind of real world task, so I
| think alpha go doesn't really change the argument.
| nuclearnice3 wrote:
| Maybe AlphaFold does?
| https://deepmind.com/blog/article/alphafold-a-solution-
| to-a-...
|
| Notably it's still an entry in a common task framework
| contest, not a piece of software for the lab, as far as i
| know.
| xiphias2 wrote:
| AlphaFold is from DeepMind, but it uses modern neural
| networks, but not reinforcement learning. DeepMind is not
| just simply wasting money, they are doing important AI
| research in other areas as well.
| throwawaygh wrote:
| One thing that's changed is:
|
| _> So now we have the top machine learning research
| institutes, DeepMind and OpenAI, still spending the majority of
| their time and resources on Deep RL_
|
| DeepMind has diversified at least some since 2019, and I'm
| fairly confident that OpenAI is spending more resources on huge
| transformer models than on RL these days.
|
| Which is really the only thing that has changed, since even in
| 2019 there are at least a dozen world-class institutions doing
| AI/ML research aimed at addressing issues raised in this blog
| post (and others).
|
| The blog post is accurate about OpenAI/Deepmind c. 2019, but is
| wrong about the overall composition of research effort in the
| field c. 2019. Outside of two small and very new labs, most ML
| research wasn't focused on RL, and most RL research wasn't
| focused on DRL as a silver bullet.
|
| Sort of of the west coast SV version of only paying attention
| to work out of MIT and Stanford and therefore missing most of
| the interesting things happening in the world.
| xiphias2 wrote:
| Most of the improvements in real world have been by improving
| data representation for perception (improvements in
| transformers, self supervised learning), so so far the article
| seems right.
| timkam wrote:
| I think when looking at least at large scale applications in
| the context of games (and not just super-expensive showcases),
| like in the Stockfish chess engine, we see that it's not
| primarily about depth, it's about architecture design.
| Reference: start here
| https://stockfishchess.org/blog/2021/stockfish-14/ and go down
| the rabbit hole...
|
| In the broadly useful domain of recommender systems (which
| typically make use of some type of RL-like feedback loop, but
| can be implemented using simple clustering approaches), at
| least in 2019, neural network-based approaches didn't seem to
| fair too well, either: https://arxiv.org/pdf/1907.06902.pdf
| (arXiv pre-print, but this is an award-winning paper).
|
| Since then, it seems that researchers are moving away from
| getting deeper and deeper (the low-hanging fruit), and try to
| be more creative instead: new architectures, combining symbolic
| (logic-based) and sub-symbolic (ML-based) AI, etc.
___________________________________________________________________
(page generated 2021-07-10 23:01 UTC)