[HN Gopher] ConvNetJS Deep Q Learning Demo (2013)
___________________________________________________________________
ConvNetJS Deep Q Learning Demo (2013)
Author : eigenvalue
Score : 33 points
Date : 2023-11-24 22:22 UTC (1 days ago)
(HTM) web link (cs.stanford.edu)
(TXT) w3m dump (cs.stanford.edu)
| eigenvalue wrote:
| This is related to the Q* stuff that is making the rounds now. I
| was quite impressed with this when I first saw it years ago. The
| little 2D robot is able to learn sophisticated looking behaviors
| with just reward based reinforcement. If you find this
| interesting, you should also check out Karpathy's ScriptBots
| project from back when he was a college student:
|
| https://youtu.be/RjweUYtpNq4?si=IGUwl7cZ47V2DSGB
| aliljet wrote:
| This is incredibly interesting. Why wasn't this seen as fruitful
| over the last decade?
| smuser wrote:
| My understanding (not an expert) is a lot of problem domains
| have very sparse / infrequent rewards - imagine if the only
| reward you gave a minecraft agent was when it mined a diamond,
| it would take a _lot_ of gameplay for it to randomly do that
| and get a reward. So researchers spend time tuning the reward
| space (oh you mined some dirt, here 's a tiny reward. Oh you
| mined rock, a greater reward, etc) but it's kind of akin to
| hand crafted feature detection from the pre-neural network
| days. The Q* mystery is did OpenAI 'solve' reward modelling the
| same way neural networks solved feature detection.
| pyinstallwoes wrote:
| Reward for a successful prediction against a goal, then the
| nuance is defining a goal?
| throwaway4aday wrote:
| Sounds like the process of tuning the reward space is a type
| of labelling and ranking problem. If I'm not mistaken, those
| are two things that GPT-4 is pretty good at. You wouldn't
| even necessarily pre-label every possible action since GPT-4
| could do it in real time.
| eigenvalue wrote:
| It was. Around the time this came out, something like half of
| the new ML papers were about reinforcement learning. The
| problem is that it's incredibly slow and inefficient compared
| to any learning where you have access to a gradient and can use
| that to choose more targeted weight updates.
|
| But there are certain applications where it's the only good way
| of doing it (for example in games, where you don't have access
| to a gradient over the space of how good a certain move is
| given the current game state, and it's relatively quick and and
| efficient to simulate the evolution of the game).
| rnimmer wrote:
| neuroevolution strategies are another approach to games, for
| what it's worth (since you said 'only').
| Jagerbizzle wrote:
| For those of us like me who are unfamiliar, can you
| recommend any useful reading on the topic?
| tnecniv wrote:
| Well Q learning has been incredibly fruitful over the last
| decade. If you want to know why it wasn't fruitful before the
| last decade, the answer is that it was. Data driven techniques
| went in and out of vogue for this kind of stuff. The difference
| this time is how much data we have due to the internet and
| smart phones and now we have computers capable of drinking from
| the data fire hose
| mortallywounded wrote:
| When I was in grad school and implemented Q-learning for the
| first time, was the moment when machine learning "clicked" for
| me. It went from being a "oh this algorithm is pretty cool, but
| it doesn't learn anything" to "oh my god, the program figured
| things out on its own."
|
| I've always had a soft spot for q-learning ever since :)
|
| edit:
|
| In grad school, I implemented q-learning to learn strategies for
| playing games (like tic-tac-toe, towers of hanoi, etc).
| jwilber wrote:
| For a very high-level, visual introduction, see:
|
| https://mlu-explain.github.io/reinforcement-learning/
| ffwd wrote:
| I don't know much about RL but I was wondering has anyone tried
| the opposite? Like have a fixed set of actions, and a
| fixed/ranged movement speed, then a punishment every time it
| doesn't reach the goal? Or does this not work?
| eigenvalue wrote:
| That's just a negative reward, so basically the same thing. You
| train most efficiently with a mix of positive and negative
| reinforcement, just like with children.
| ffwd wrote:
| Thanks, I can't write a full reply now (need to think) but
| for some reason my intuition was, let's say you have a
| constant punishment signal, and a timer, and if it doesn't
| solve whatever problem by the time the timer goes down, then
| it has to find the optimal action set, and if it reaches a
| goal earlier than the timer then it has to weigh that
| solution stronger? Like at least how I see with organisms
| it's about finding the optimal use of the limbs in order to
| solve an ongoing problem/goal state, and if you accumulate
| specific actions (instead of one network that optimizes one
| "space"), and different types of goal states, then it has to
| find the optimal set of actions to reach the different goals,
| it just seemed more efficient. But this is off the cuff a bit
| right now.
| PodgieTar wrote:
| I implemented Q Learning in University. Only very basic, sort of
| like the Mario style Q-Learning. It's quite intuitive, as a
| concept.
|
| You do need to know the expected outputs - positive rewards and
| negative.
___________________________________________________________________
(page generated 2023-11-25 23:00 UTC)