[HN Gopher] RT-2 AI model translates vision and language into ro...
       ___________________________________________________________________
        
       RT-2 AI model translates vision and language into robotic actions
        
       Author : BhattMayurJ
       Score  : 151 points
       Date   : 2023-07-28 10:45 UTC (12 hours ago)
        
 (HTM) web link (blog.google)
 (TXT) w3m dump (blog.google)
        
       | alphabetting wrote:
       | google demo'd this to an NYT podcast in the final 20 minutes
       | here: https://www.nytimes.com/2023/07/28/podcasts/elons-x-
       | machina-...
       | 
       | There's a cool part where they ask the robot to pick up a lion
       | from a group of toy figures it hasn't seen before. After it does
       | it correctly the NYT reporter asks it to pick up the extinct
       | animal and the robot picks up the dinosaur toy.
        
         | xnx wrote:
         | Corresponding article where they also describe RT-2 and their
         | visit: https://archive.is/DoeyT
        
       | aliljet wrote:
       | Robotics is such a lovely world. For the hobbyists, I'm really
       | curious about how to get the home-brew robotics training lab
       | working at home? The last time I reviewd this, you'd spend
       | thousands of dollars just to get a reasonable robotic arm. Right
       | now, if work is relegated to the rich research departments of
       | mega corporations, this certainly doesn't seem more interesting
       | than a corporate press release..
        
         | NalNezumi wrote:
         | Depends on what part of robotics you want to explore. You can
         | get a quadruped with an arm for around 400-1000$ if you don't
         | mind it being small, and with a limited set of sensors.
         | 
         | It becomes expensive when you want an robot arm with high
         | dexterity and torque-sensors, which is often a requirement for
         | some tasks, but position-based control can be done on cheaper
         | models.
        
           | nojvek wrote:
           | Are you taking about the unitree quadrupeds?
           | 
           | Quadruped with an arm for <$1000 would be a steal. Got any
           | recommendations?
        
             | NalNezumi wrote:
             | With emphasis on "small" I got one called XGO mini 2 (and
             | 1) that I sponsored on Kickstarter, which had it at half
             | the price (400$ ish) but it's around $800 here[1].
             | 
             | It runs on Raspberry Pi which is a big +, as version 1 was
             | running on custom hardware. People had to plug a Jetson
             | Nano or other device to run their code (such as this [2]).
             | It comes with their own App, connected via bluetooth, for
             | simple commands too.
             | 
             | I (and most kickstart backers) funded/bought it for either
             | hobby or research purpose (some backers were from labs) so
             | if you're looking for *practical* application this one
             | might be a bit too weak.
             | 
             | But version 1&2 did come with a processing power that could
             | run YOLO detection so there's enough power to test things
             | out and learn.
             | 
             | [1] https://www.robotshop.com/products/xgo-
             | mini-2-quadruped-robo... [2]
             | https://www.youtube.com/watch?v=xqfwCwu4-Rc
        
           | euroderf wrote:
           | I thought by now there'd be kits like plastic airplane
           | models. Injection-mold a bunch of parts, connected by sprues,
           | and throw in the odd bits of (metal) hardware where they are
           | critical to the whole.
        
         | joshvm wrote:
         | Someone else mentioned sims - Mujoco is pretty common and
         | you'll want to learn ROS anyway. Also robotics doesn't mean
         | just arms, you can play with navigation and perception
         | algorithms with a single camera.
         | 
         | Good off-the-shelf arms are low thousands last time I looked.
         | You can DIY for $1-2k https://www.anninrobotics.com/robot-kits
        
         | thanatropism wrote:
         | Is there any reason why anthropomorphic is needed for an
         | enjoyable hobby? Get an Arduino, a couple of servos, make a
         | claw that grips on things.
        
         | neatze wrote:
         | This is false to very large degree, you have simulations for
         | robotics, and you can buy/build your self cheaper versions.
        
           | mrits wrote:
           | It is also likely even after you purchase a physical arm
           | you'd prefer to use the simulation as the feedback loop is
           | both testable and much faster.
        
           | amelius wrote:
           | What are some good open source simulators suitable for
           | robotics development?
           | 
           | (preferably a physics simulator library, not an entire
           | framework)
        
             | tnecniv wrote:
             | PyBullet (I guess Bullet more broadly) sees a lot of use
        
         | _visgean wrote:
         | > work is relegated to the rich research departments of mega
         | corporations
         | 
         | idk there is plenty of universities that have robotics labs..
         | Probably the best value is to apply for masters etc if you are
         | interested in that..
        
         | mhb wrote:
         | Impressive dog:
         | 
         | https://www.youtube.com/watch?v=6zPvT0ig1VM
        
       | taneq wrote:
       | The Portal-themed animation really sells it as a step in the
       | right direction. ;)
       | 
       | Seriously though, I'm excited to see where this goes. AI is
       | progressing so fast now that newcomers loudly proclaim "AI is
       | dead" and "going nowhere" when they haven't seen SotA beaten for
       | a whole month, because they don't remember the times when a small
       | improvement in a decade was big news.
        
         | ChatGTP wrote:
         | You know where it's going dude...to the military and law
         | enforcement. It's completely naive to think otherwise. Advanced
         | robotics might be used to unpack your dishwasher, or save
         | kittens stuck in trees, but we know where the $$$ lies...
        
           | taneq wrote:
           | Depends on who sparks the hard takeoff. There's a whole bunch
           | of interesting times coming up and I don't think our human
           | power structures are going to have any control over it. The
           | next five years are going to be less about status quo and
           | more about... good parenting?
        
       | itissid wrote:
       | It would be interesting to see how it solve AI Planning Problems
       | like BlocksWorld[1]. I've read that in the past with these
       | things[2][3] when multiple goals needed to be met at once and
       | there was interaction between them, it just falls over itself.
       | Being able to generate coherent plans and execute them, as I
       | understand, an important aspect of generating Action Sequences
       | given a State and thus for planning in robotics. How are these
       | overcome in RT2?
       | 
       | P.S.: I also was told that the key here is that in automated
       | planning you can't have a human in the loop doing the _actual_
       | learning. If you are going to prompt _engineer_ or get a human in
       | the loop to a degree that you effectively fool yourself that the
       | robot is solving a problem then its not planning.
       | 
       | [1] https://en.wikipedia.org/wiki/Sussman_anomaly [2]
       | https://chat.openai.com/share/16a8a0e9-7422-41da-a192-6393cc...
       | [3] https://twitter.com/rao2z/status/1599462959788744704?s=20
        
         | Xeophon wrote:
         | Related work is the planning paper by Valmeekan et al [1]. The
         | gist is that LLMs are incapable of planning, which is due to
         | their autoregressive nature. METAs Head of AI Yann Lecun also
         | talks about this topic in a talk [2]. As RT2 is based on a
         | similar architecture, I think the results will be similar.
         | 
         | [1] https://arxiv.org/abs/2305.15771 [2]
         | https://youtu.be/x10964w00zk
        
           | itissid wrote:
           | That arxiv link to the paper is also the same guy I mentioned
           | in my twitter reference. Subbarao :)
        
       | peterleiser wrote:
       | I find the lack of videos... curious.
        
         | rl_agent wrote:
         | Hi! We have many videos on our RT-2 website: https://robotics-
         | transformer2.github.io/
        
       | bilsbie wrote:
       | Any startups working in this area? I'd love to get involved!
        
       | Jeff_Brown wrote:
       | If it was tethered to the floor, I would trust a good LLM in 2023
       | with folding laundry, but very little else. No dishes. No letting
       | kids or pets bear it. No plumbing. I wouldn't even let it wander
       | around cleaning stuff, because I'd expect it to knock over
       | things.
        
         | HereBePandas wrote:
         | Sure, but this time in 2022, you probably wouldn't have let it
         | do that much (or even had this thought).
         | 
         | Progress!
        
         | formulathree wrote:
         | [dead]
        
         | falcor84 wrote:
         | I personally am less concerned about knocking things over,
         | which is a thing we already trust vacuum robots with; rather
         | I'd be much more concerned about the robots accidentally (or
         | intentionally?) "folding" my kids or pets.
        
           | JimtheCoder wrote:
           | I would like to see one try to fold my cat.
           | 
           | I would then have one cat and one messed up LLM robot.
        
       | formulathree wrote:
       | [dead]
        
       | GaggiX wrote:
       | This is a step towards what I think AGI would be: a model trained
       | on visual and language data from the Internet used as a prior for
       | an action model (finetuned perhaps with reinforcement learning)
       | that would be able to learn how to use the frozen prior to make
       | useful actions; I would separate the prior from the action model
       | so there is no possibility of catastrophic forgetting (this is
       | dealt by co-fine-tuning the model in the RT-2 paper), and because
       | a more advanced robot would need to control more actions quickly,
       | so it would be expensive to run the entire prior just to do basic
       | movements in real time.
       | 
       | Edit: also, the model should be smart about how to use its
       | "context window", when the robot is taught how to do a task and
       | it needs to do that task, it must retain the knowledge.
        
         | valine wrote:
         | That seems like a real step in the direction of AGI but not
         | anywhere near the full solution. The current context windows
         | are far too small to replicate human intelligence. I can't
         | quite put my finger on it but it feels like we are missing a
         | sort of bridge between learning done in-context and offline
         | training. A true AGI would be able to learn in-context and then
         | quickly apply those lessons to the base model. If in context
         | learning is analogous to a person's short term memory, we need
         | a mechanism to move short term memory to long term memory.
         | 
         | In the near term I expect we will see much more general
         | robotics that know how to do lots of tasks and can follow basic
         | instructions, but lack the ability to develop complex new
         | skills over time. Robots doing dishes and laundry will soon be
         | feasible, just don't expect unbounded self improvement.
        
           | GaggiX wrote:
           | I agree that it would be much better if a model could move
           | the knowledge to the weights, but I don't think it's
           | necessary. Models like RNN have practically infinite lossy
           | "context window", and I think it would be much more
           | reasonable to find an RNN architecture that would scale like
           | a transformer than what you propose, while still achieving
           | our goal. An alternative analogy you can think of is that the
           | frozen weights in the model are our DNA, while the state or
           | "context window" of the model is our state.
        
       | itissid wrote:
       | An immediately useful application of this is in roombas that can
       | not just clean the floor but effectively avoid/move obstacles.
       | All the vaccums i have gotten generally get stuck or suck in
       | paper/usb cables and get stuck in corners. There are two planning
       | aspects here
       | 
       | 1. Choosing a workflow(i.e. a series of general steps that can
       | achieve a goal) 2. Generating low level policy actions given a
       | state(i.e. sensor data).
       | 
       | Like lets say the Workflow is to clean the room. Now there is a
       | big chair in the way and a USB wire there too, the robot could
       | just decide to move around the chair but it could pick up the
       | wire and place it on the table and clean the area. Generating
       | policy actions this way seems to be very much in line with simple
       | VQA based things proposed in the paper, or so it would seem...
        
         | valine wrote:
         | I don't think its even necessary to pick up the wire. The
         | newest vacuums use cameras to try to avoid common obstacles,
         | but the image classification is so rudimentary they get stuck
         | anyway. Having the common sense to stay away from the end of
         | the cable is all the robot needs to not get stuck.
        
       | ChatGTP wrote:
       | Why does Google actually build stuff like this? What is their
       | actual end game and how does it relate to what they actually do
       | as a business?
       | 
       | They're an advertising company with a mission to "organize the
       | worlds information", pivoting to robots? Do they need robots to
       | organize the worlds information?
       | 
       | No idea how big of a breakthrough this is, but it is absolutely
       | undeniable that Googles PR team is going wild on anything AI
       | since OpenAI provided them with their first real existential
       | crisis / code red.
        
         | falcor84 wrote:
         | > Do they need robots to organize the worlds information?
         | 
         | Well, my bookshelves are a mess and I'd be ok with getting a
         | robot to help organize my information, if I had some assurances
         | that it won't kill me, or at the very least a "Don't be evil"
         | clause.
        
         | alphabetting wrote:
         | google runs on ad money but they're also in the business of
         | acquiring elite ML talent and publishing sota research is one
         | of the main drivers in accomplishing that.
        
           | ChatGTP wrote:
           | ML talent for? There search is absolute trash, so maybe they
           | could work to use ML to solve that?
           | 
           | I'm so sick of Google I just paid for Kagi.
        
       | darkclouds wrote:
       | Sounds cool, any youtube evidence?
        
         | peterleiser wrote:
         | The post links to: https://robotics-transformer2.github.io/
         | 
         | There under "demo" are a few videos at 2x and 4x speed. It's
         | slow. None of the videos include audio of the verbal commands
         | or the latency between commands and action.
        
           | [deleted]
        
       | [deleted]
        
       | [deleted]
        
       | NalNezumi wrote:
       | This seem like an cool upgrade from RT1, judging from the result.
       | It seems to now also output the delta of the end-effector pose,
       | which was previously handled by a different motion planner.
       | 
       | It does seem like this work (and a lot of robot learning works)
       | are still stuck on position/velocity control and not impedance
       | control. Which is essentially output _where_ to go, either
       | closed-loop with a controller or open-loop with a motion planner.
       | This seem to dramatically lower the data requirement but it feel
       | like a fundamental limit to what task we can accomplish.
       | 
       | The reason robot manipulation _is hard_ is because we need to
       | take in to the account not just what 's happening in the world
       | but also how our interaction alters it and how we need to react
       | to that.
       | 
       | Robot Learning right now is either Reinforcement Learning or
       | Imitation Learning (often latter) and I'm not sure _how_ one
       | would collect data that capture this.
       | 
       | Edit: I'm surprised Google still do robotics work, I thought
       | Everyday Robotics was shut down [1]
       | 
       | [1] https://www.therobotreport.com/alphabet-closes-everyday-
       | robo...
        
         | [deleted]
        
         | blovescoffee wrote:
         | They collect data for RL/IL in simulation which _can_
         | generalize to the real world. Also, being Google, they have the
         | resources to collect data by brute force i.e. scientists
         | manually collecting that data. The paper says - one main source
         | of the data is internet scale vision/llm data. The second
         | source is 6k trial runs.
         | 
         | A principal idea behind this work is that you can collect data
         | in one domain to avoid collecting data in another. Training an
         | LLM can train the 'reasoning' portion of the robot so that it
         | can perform real-world skills with less training.
        
           | NalNezumi wrote:
           | They use the RT1 data which is (if I remember correctly) 11
           | month of data using like 6 research engineers sitting
           | everyday, remote controlling the robot. They might even use
           | ROSIE data which is data-augmentation using Stable-diffusion
           | inpainting.
           | 
           | And for that they can do mobile manipulation, which have been
           | possible for a long while (as long as we know ground-truth of
           | object location, and have a some-what consistent map.
           | Although the latter is solved with SLAM). it is way, way more
           | versatile (if their claims are correct) so this is cool
           | result.
           | 
           | as for sim2real, that's a whole different set of issue....
           | Simulator have come a long way, but from last time I talked
           | with team-lead of Mujoco sim, it seems like completely
           | realistic ones (at a reasonable compute time) is still far
           | away. Deform-able objects are hard to simulate
           | 
           | My point is just that the "reasoning" you mention have a
           | limit;[1] there's no amount of reasoning with language that
           | can zero-shot a robot to do gymnastics or ride a bide. I'd be
           | more than happy to be proven wrong on this point though, then
           | my line of research is not dead-end =)
           | 
           | [1] https://youtu.be/x10964w00zk
        
             | blovescoffee wrote:
             | Sure, I wouldn't say reasoning can guide a robot to do
             | gymnastics zero-shot either. But it is a substantiated
             | approach to generalizing existing capabilities - just like
             | in humans. You or a robot can't reason your way to bike-
             | riding but if you know how to bake cookies or brush your
             | teeth, you can perform much better zero-shot on related
             | tasks like baking a chicken or brushing a mirror.
        
         | lucidrains wrote:
         | thank you for sharing your thoughts! is there anyway I can get
         | in touch with you, through email or some other means?
        
         | empath-nirvana wrote:
         | Seems to me that the way to architect this is to have multiple
         | quasi independent embedded controllers at different levels. For
         | example, you might have 3 independent finger controllers,
         | managed by a hand controller, and so on, on up to the highest
         | level LLM that drives everything. So you have an LLM that just
         | says, pick up the green can, issues whatever structured data
         | needs to go to the arm controller and on down the line, going
         | to down to more real-time and less high level processing as you
         | go.
         | 
         | As a human, I don't understand the detailed micro-second by
         | microsecond movements my fingers have to do to pick something
         | up, let alone how I'm touch-typing this sentence. It just sort
         | of "happens" when I want it to happen. I don't think you need
         | to design a robotic AI that understands how every part of it's
         | mechanics work. The fingers don't need to know how the feet
         | work, for example. There should be semi-autonomous
         | "intelligence" embedded throughout the system, with only
         | necessary feedback being fed back up.
        
           | RecycledEle wrote:
           | I agree. We need feedback loops on each joint that keep doing
           | what they are doing until a higher feedback loop decides to
           | change things.
           | 
           | For example, as I enter this text, most of my fingers are
           | holding the phone and do not need to be told to keep holding
           | it.
           | 
           | The low level controllers should also be able to respond to
           | simple things.
           | 
           | It's like when I get hurt, my hand yanks back before my brain
           | realizes what has happened. Or when I almost trip and my legs
           | correct before I realized it happened.
           | 
           | Low level tasks should be at a low level and not bother the
           | higher professors.
        
             | RecycledEle wrote:
             | LLMs have terrible spatial awareness. This probably comes
             | from ONLY being trained on text.
             | 
             | I wonder if it would help a LLM running a robot to have a
             | separate controller calculating it's position and what is
             | around it, and feed that into the LLM constantly.
             | 
             | It would be like when a video game has a radar display or
             | map to let you know where you are in relation to other
             | things.
        
               | moffkalast wrote:
               | I actually tried that a while back, giving 3.5-turbo a
               | multishot prompt that consisted of distance readings for
               | ahead, left, right and back in an array, as extracted
               | from lidar data, then giving it movement instructions. It
               | performed rather terribly.
               | 
               | You've very much correct that their spatial awareness is
               | terrible. Something as simple as drive forward, then
               | back, turn left, etc. works just fine and they can
               | generally translate it to a specified message format
               | reasonably reliably, but give them something more complex
               | to execute, like drive a robot in a square pattern (an
               | example answer would be go forward, turn right, go
               | forward, turn right, etc.) they start to generate
               | nonsense.
               | 
               | I also tested it with the 30B WizardLM at the time which
               | performed almost as well in terms of message format but
               | had even worse awareness.
               | 
               | Part of the problem is that the training data contains
               | next to no examples that would teach it how 3D space
               | works. I considered making a dataset of driving a robot
               | around with human movement commands and then logging the
               | aggregated sensor data and commands for fine tuning so
               | the prompt format would be pre-learned, but I'm not
               | entirely sure how much it would help.
        
               | StackOverlord wrote:
               | LLMs have a firm grip on common sense. It's because it
               | allows them to deal with the utterly unexpected they are
               | deemed useful in robotics. Not to perform delicate
               | movements, but stop doing so when police enters the room.
        
           | NalNezumi wrote:
           | That is indeed what a lot of Machine Learning turned Robotics
           | researchers/enthusiasts are banking on. A counter argument to
           | that is what Dhruv Batra responded to the question "lol why
           | not use LLM for everything" [1].
           | 
           | >As a human, I don't understand the detailed micro-second by
           | microsecond movements my fingers have to do to pick something
           | up, let alone how I'm touch-typing this sentence. It just
           | sort of "happens" when I want it to happen. I don't think you
           | need to design a robotic AI that understands how every part
           | of it's mechanics work.
           | 
           | This is true for us ofc, and is encompassed in what is called
           | the "Moravec's paradox" [2]. You don't understand it because
           | it's unconscious process, and It's harder to reverse-engineer
           | an unconscious process (motor movement) than conscious ones
           | (calculating math, playing game, writing text, reading).
           | 
           | But the thing is that in the real world we do need to take in
           | to account everything, including noise and time-delay.
           | Evolution gave rise to complex language in the last 100k
           | years compared to _millions_ of year for motor movement. I do
           | agree that there must be some  "hierarchical" structure for
           | complex motion, but we currently don't know _where and how_
           | that hierarchy is. Boston dynamics uses Model Predictive
           | Control for complex movement, which means that at least
           | _some_ model of the world is required, for dexterous motion.
           | Now if that model is part of LLM or not is a hard guess.
           | 
           | But if we don't know this it's hard to say what kind of data
           | we need to collect to train a LLM-model applied to embodiment
           | (robotics).
           | 
           | Researchers in the past have already made the mistaken
           | assumption of "oh cognition and language is the hard part of
           | intelligence. Perception and motion is easy" [3] and then
           | their work amounted to nothing because turns out the latter
           | was way, wayyyyyy harder.
           | 
           | There's an implicit bias in us that think Language, puzzles
           | and logic are harder [2] and therefore models that accomplish
           | this can just be rammed in to the "easier" issues.
           | 
           | Edit: I too would like "LLM models will solve these" attitude
           | because otherwise the research I'm doing right now is a dead-
           | end, but the more I try (with my limited compute) the less
           | I'm sure
           | 
           | [1] https://imgur.com/eWsH5ui originally
           | https://twitter.com/DhruvBatraDB/status/1641871357020614656
           | 
           | [2] https://en.wikipedia.org/wiki/Moravec%27s_paradox
           | 
           | [3] https://youtu.be/x10964w00zk?list=PLSQhB89mdG7PsZsDz2_5hZ
           | L8C...
        
             | whinenot wrote:
             | >It's harder to reverse-engineer an unconscious process
             | 
             | Aside from some basic life support systems, don't almost
             | all movements start with conscious effort? Whether you are
             | deliberate about the exercise or not, you practice and
             | practice until you develop 'muscle memory' where it becomes
             | unconscious: walking, dribbling a basketball, holding a G
             | chord on a guitar, etc.
        
           | StackOverlord wrote:
           | This is in fact what happens in the human body.
           | 
           | For example, when you reach out to pick up a green can, your
           | brain makes the decision to do the task but it's your spinal
           | cord and peripheral nerves that carry out the detailed work -
           | orienting the hand, managing grasp strength, controlling the
           | arm movements etc. This process is mostly unconscious - you
           | don't need to actively think about how to tense each muscle
           | in the same way that an embedded controller wouldn't need to
           | understanding the working of the entire robotic system to
           | carry out its specific task.
           | 
           | Much like the model suggested, the human body communicates
           | feedback across layers -- this process is crucial to
           | maintaining balance, coordination and effectively reacting to
           | the environment. For instance, if your fingers touch a hot
           | stove, the sensory receptors in your skin will immediately
           | send a signal to your spinal cord and a reflex action will
           | make you pull your hand back even before you consciously
           | perceive that the stove is hot.
        
           | jimbokun wrote:
           | Maybe the octopus is a more tractable model for robotic
           | control. I understand that octopus neurons are not as
           | concentrated in a central brain, but spread throughout its
           | limbs that are autonomous compared to humans.
        
       | random5245 wrote:
       | So we gonna have Robot like in girl front line in near future ?
        
       | martythemaniak wrote:
       | I'm actually pretty bullish on humanoid robots like the Tesla bot
       | - combination of LLMs, cheap batteries/motors/controllers from
       | cars and vision research should be able to come together in
       | useful and cheap ways in a few years, say 2030.
       | 
       | $35K for a robot that can putter around the house doing basic
       | stuff is just not that high of a bar. With a 10 year life span,
       | that's $3.5k/year, or $10/day. Doing 1 hour of useful minimum
       | wage work around the house is just not that high of a bar - doing
       | laundry, cleaning, tidying up, weeding, wiping surfaces down,
       | taking out the garbage etc. If it can do some combination of
       | those, it would make sense for basically every household. And it
       | doesn't need to be able to do the crazy parkour of Boston
       | Dynamics to achieve this. Our world is generally designed to be
       | operable by all sorts of people - disabled, old etc. Crazy
       | athleticism isn't required to do useful work.
        
       | amelius wrote:
       | How long until we can tell it to take a LEGO instruction booklet
       | and assemble the model?
        
       | FrustratedMonky wrote:
       | One more step towards the 'Terminator' scenario. And everyone
       | said it would be decades away.
       | 
       | Edit. So many downvotes -- here have a /s (sarcasm)
        
         | falcor84 wrote:
         | They did, in 1984 and 1991. It's now been decades. It's even
         | almost two decades since people felt it was now ok to call a
         | robotics company Cyberdyne[0].
         | 
         | Decades have passed, and with pretty much no actual surprises
         | on the way, we're steadily marching towards creating robots
         | with the power to destroy humanity. We haven't made much
         | progress on time travel though, so whatever happens, we
         | probably won't have a chance at a do-over.
         | 
         | [0] https://en.wikipedia.org/wiki/Cyberdyne_Inc.
        
           | FrustratedMonky wrote:
           | Skynet was supposed to become self-aware on August 29, 1997
           | 
           | Since it didn't happen, maybe a Terminator Genisys type reset
           | of the timeline has occurred. So we have a few more decades
           | buffer again, to maybe 2050?
        
       | beefman wrote:
       | Better URL: https://www.deepmind.com/blog/rt-2-new-model-
       | translates-visi...
        
       | Vox_Leone wrote:
       | I never see mention of another type of integration that seems to
       | me to be necessary for the improvement of AI and for any attempt
       | to establish general artificial intelligence: integration with
       | the senses. Chains of thought are born in the senses and are
       | constantly updated in the background with information from them.
       | 
       | To fully match the human model, AGI based on neural networks will
       | need to dynamically receive information from sensors of all kinds
       | [to be able to sample at least five major categories of physical
       | stimuli, like us]. If we want to reach the level of the neural
       | networks that we carry in our heads, we have to learn more about
       | the role of other brain structures, such as glial cells, which
       | are known to influence the activation and moderation of synapses
       | in the human brain [which correspond to the 'weights' in
       | artificial neural networks], and then apply them to our
       | technological endeavors.
       | 
       | Without it two things will happen
       | 
       | 1. We will never have strong AI 2. Strong AI that works virtually
       | only, aware of the world and material reality, but denied access
       | to them by its creators.
       | 
       | Imagine that it is possible to take a snapshot of a human mental
       | state [an analogue of .h5 weights] and then run it on hardware as
       | we do with our neural networks. It would certainly be a nightmare
       | for a mind endowed with "qualia" to find itself enclosed in a
       | metal box [brain in a vat] with no access to the senses. There
       | are clinical cases analogous to this condition, for example in
       | states where the patient is aware of everything around him but is
       | unable to respond.
       | 
       | These issues highlight the ever-present ethical boundaries of IA
       | research. Without integration with real-time sensors AI will
       | always be incomplete and/or inhumane.
        
       | falcor84 wrote:
       | The linked post from DeepMind has more detail -
       | https://www.deepmind.com/blog/rt-2-new-model-translates-visi...
       | 
       | EDIT: And here's the paper pdf - https://robotics-
       | transformer2.github.io/assets/rt2.pdf
        
       ___________________________________________________________________
       (page generated 2023-07-28 23:02 UTC)