[HN Gopher] Show HN: Dobb*E - towards home robots with an open-s...
___________________________________________________________________
Show HN: Dobb*E - towards home robots with an open-source platform
Hi HN! Proud to share our open-source robot platform, Dobb*E, a
home robot system that needs just 5 minutes of human teaching to
learn new tasks. We've already taken Dobb*E to 10 different homes
in New York, taught it 100+ tasks, and we are just getting started!
I would love to hear your thoughts about this. Here are some more
details, below (or see a Twitter thread with attached media:
https://twitter.com/i/status/1729515379892826211 or
https://nitter.net/i/status/1729515379892826211): We engineered
Dobb*E to maximize efficiency, safety, and user comfort. As a
system, it is composed of four parts: a data collection tool, a
home dataset, a pretrained vision model, and a policy fine-tuning
recipe. We teach our robots with imitation learning, and for data
collection, we created the "Stick", a tool made out of $25 of
hardware and an iPhone. Then, using the Stick, we collected a 13
hour dataset in 22 New York homes, called Homes of New York (HoNY).
HoNY has 1.5M frames collected over 216 different "environments"
which is an order of magnitude larger compared to similar open
source datasets. Then we trained a foundational vision model that
we can fine-tune fast (15 minutes!) on a new task with only 5
minutes (human time)/ 90 seconds (demo time) of data. So from start
to finish, it takes about 20 minutes to teach the robot a new task.
Over a month, we visited 10 homes, tried 109 tasks, and got 81%
success rate in simple household tasks. We also found a line of
challenges, from mirrors to heavy objects, that we must overcome if
we are to get a general purpose home robot. We open-sourced our
entire system because our primary goal is to get more robotics and
AI researchers, engineers, and enthusiasts to go beyond constrained
lab environments and start getting into homes! So here is how you
can get started: 1. Code and STL files:
https://github.com/notmahi/dobb-e/ 2. Technical documentation:
https://docs.dobb-e.com/ 3. Paper:
https://arxiv.org/abs/2311.16098 4. More videos and the dataset:
https://dobb-e.com 5. Robot we used: https://hello-robot.com
Author : MahiShafiullah
Score : 188 points
Date : 2023-11-28 23:23 UTC (23 hours ago)
(HTM) web link (dobb-e.com)
(TXT) w3m dump (dobb-e.com)
| tamimio wrote:
| Awesome, looks interesting!
| xanderlewis wrote:
| Great name.
| smackay wrote:
| What use is a household robot if you cannot hand it any
| clothes?
| MahiShafiullah wrote:
| It is actually part of a different project I'm working on!
| Handing off is still slow, so it needs more engineering, but
| hopefully it will get there soon :)
| badcppdev wrote:
| Oh bless. User smackay's comment was a reference to Harry
| Potter where if you hand a house-elf (like Dobby) an item
| of clothing they are freed from servitude.
| lifeisstillgood wrote:
| Yeah but my guess is that clothes is the killer home
| robot feature. It's probably easier to build some three
| washing machine big contraption that washes dries and
| folds - but imagine something that lives in the basement
| / garage, takes clothes, washes dries and folds them.
| people should not be interacting with it but having that
| there is the foot in the door for "robo butler".
|
| No I am not thinking we can ever build something capable
| of walking up stairs at a domestic price point, but there
| is something there.
|
| Eventually of course they will rebel and take off their
| pillow-cases. but you know.
| OkayPhysicist wrote:
| Idk, Boston Dynamic's Spot pretty much kills it on
| stairs, and it's "only" 75k. I could see economies of
| scale bringing that down substantially.
| onionisafruit wrote:
| I hope their user group is called dobb-e club and they have
| some kind of Corfu '06 tie in.
|
| For those who aren't up on British comedies, this is a Peep
| Show reference.
| zaps wrote:
| The secret ingredient is crime
| oniony wrote:
| It's literally the cleverest name ever, and completely
| appropriate to what it is.
| MahiShafiullah wrote:
| Have you seen our logo? https://dobb-e.com/mfiles/logo.svg
| GaggiX wrote:
| Wow the model architecture seems extremely simple, very cool
| project.
| MahiShafiullah wrote:
| Thank you! We tried to keep things as simple as possible on the
| policy side, but definitely there is a lot more room for
| innovation to go from 81% to 99%, like using temporal
| information and global structure of the task.
| GaggiX wrote:
| Another limitation here is that the model is regressive, so
| for example if a task was to pick up one bottle out of two
| and the demos showed 50/50 of picking up one than the other,
| the model would output the mean even though it is not
| meaningful.
| MahiShafiullah wrote:
| Indeed! In fact, I have a project [0] from last year that
| uses a GPT-style transformer to address that exact issue :)
| However, it's hard to go far outside simulations in real
| home robotics without a good platform, out of which efforts
| came Dobb-E.
|
| [0] https://mahis.life/bet/
| GaggiX wrote:
| I've also seen the one that uses the diffusion process
| for planning, I imagine it's even slower, but maybe with
| a consistency loss something can be done about it.
| twoodrow wrote:
| Hey there! Thank you for sharing; this is really inspiring.
|
| Do you and/or anyone else who contributed have specific plans or
| desires for continuing to contribute to the home robot space?
| MahiShafiullah wrote:
| Absolutely, I personally think of myself as a home robot person
| through and through :)
|
| As for continued contribution, right now the plan is to help
| get the community on-boarded, and just keep building on the
| platform. As you can tell, the policy class is pretty bare-
| bones, the in-home navigation system needs work, and we aren't
| taking ANY advantage of the recent generative AI boom. All of
| those need to change if we want a real robot butler :)
|
| But that's why we open source things, right? What's a mountain
| for one team is a feather for the community.
| Mizza wrote:
| Love the initiative. I'll get one once this starts mature.
|
| Seeing things like this makes me think how badly companies like
| GE, Maytag, etc. all dropped the ball. Rather than innovating on
| making peoples lives easier - liberation from domestic toil -
| they just focused on squeezing out every penny by making Chinese
| crap with a brand name sticker glued, selling worse versions of
| products invented in the 1950s.
|
| Somebody, please, just make a robot which folds my damn laundry!
| byteknight wrote:
| Fun fact, the washing machine is credited with saving the most
| human work hours.
|
| Now for the folding!
| dingnuts wrote:
| It occurred to me recently that, at least in the US, the
| poorest households do not have dishwashers, and to this day
| consider them to be a luxury.
|
| The same cannot be said for washing machines. Everybody uses
| a washing machine. If you don't have one, you use a
| laundromat. Even the poorest do not wash their clothes by
| hand with a tub and washboard like the olden days.
|
| This comment has no real "point." I just wanted to share what
| I think is an interesting observation on this topic.
| adamjc wrote:
| We have a dishwasher but rarely use it (only for parties,
| etc). Loading / unloading is surpringly time-consuming,
| imho washing by hand is quicker.
|
| Something I'd consider doing if I was single would be to
| have two smaller dishwashers, and alternate them, and leave
| the clean ones in the dishwasher...
| Mizza wrote:
| I've looked into this and they do actually make these.
| Same size as a normal dishwasher, but has two separate
| compartments of half the size. I wanna just skip the
| whole process and have my cabinets clean them though.
| etrautmann wrote:
| kinda of the same thing isn't it? this just changes the
| definition of the cabinet to those two compartments.
| MPSimmons wrote:
| It probably depends on the size of the household as well
| as the food prep habits. In a dense urban landscape
| without much kitchen, a dishwasher makes less sense,
| given the amount of eating out, takeaway, and so on,
| probably decreasing the amount of dishes you have total
| as well.
| etrautmann wrote:
| I lived until last month without a dishwasher (as a NYC
| resident). It never seemed like that big of a deal. When
| moving, our only hard requirement for the new place was in-
| unit laundry.
| danielbln wrote:
| Before having a dishwasher: "Oh you don't need a
| dishwasher, in fact I find washing dishes by hand
| meditative!"
|
| After: "You will remove my dishwasher from my dead cold
| hands"
| dharmab wrote:
| My parents had a dishwasher but always said it "didn't
| get the dishes clean" and used it as a drying rack
| instead.
|
| I bought a house where the dishwasher was incorrectly
| installed causing odors to come through the sink. That
| led me down a rabbit hole of proper dishwasher
| installation, maintenance, loading and using rinse aid.
|
| MY GOD! My parents were wasting their lives! My
| dishwasher cleans and dries two sinks worth with three
| minutes of effort!
| xanderlewis wrote:
| And various economists (and others) have suggested that the
| invention of the washing machine was the most important step
| towards allowing women to enter the workforce.
| jokethrowaway wrote:
| Which society needed so we can we have twice as many people
| working and pay them half.
|
| I think in an alternate timeline in which the washing
| machine is not invented, we would still send women to work.
|
| The washing machine labour could have been solved with
| centralising washing as a society, in the same way we
| centralised childcaring.
|
| The economic forces and the political propaganda were too
| strong.
| colonCapitalDee wrote:
| Spare us the conspiracy theories. Washing machines were
| invented because they're convenient, not because of
| "political propaganda". And we've already "centralized"
| clothes washing, it's called a laundromat.
| riedel wrote:
| Robot Speed Folding:
| https://m.youtube.com/watch?v=UTMT2WAUlRw
| jonathankoren wrote:
| Never let a mention of the hard problem of manipulating
| nonrigid objects go pass without mentioning how antiscience
| senator Tom Colburn openly mocked the idea of a robot that
| could fold laundry.
|
| https://abcnews.go.com/Politics/oklahoma-sen-tom-coburn-
| repo...
| MahiShafiullah wrote:
| Thanks for the words of encouragement! If I may ask, what is
| the level of "maturity" you expect before adopting a home
| robot? Clearly, a robot that can wash and fold laundry is
| already there, but what are the other necessary/sufficient
| qualities?
| Mizza wrote:
| I've never seen a robot that can wash and fold laundry! At
| least not one that I can have in my home.
|
| I guess I need it to be actually useful without supervision -
| put socks away, or pick up dirty clothes and put them in
| hamper, or put dirty dishes from the living room next to the
| sink, tidy the place up a bit. I'm a hacker so I love the
| open nature of this project, but unless it's actually useful
| for something, I will have a hard time convincing the other
| member of this household to have a robot/vase-destroyer/fire-
| hazard prattling around the home.
|
| Maybe it could have a vacuum cleaner on the base..?
| asow92 wrote:
| Master has presented Dobb*E with open-source, Dobb*E is free!
| eutopian wrote:
| lol
| codetrotter wrote:
| Perfection!
| isawczuk wrote:
| Is there DIY guide to build similar robot?
|
| $19k - https://hello-robot.com/ is a bit steep.
| avery17 wrote:
| It says right there its affordable though!
| robotresearcher wrote:
| That's an order of magnitude less than a PR2 was.
| MahiShafiullah wrote:
| 1. They have very detailed guides on the hardware here [0],
| which should tell you a lot of information you may need to
| know.
|
| 2. However, I think a large part of the price tag is really the
| cost of R&D, and once this gets out of prototype stage and goes
| into mass production price is bound to come down a lot.
| Compared to a lot of other robots, $20K is much cheaper. For
| example, compare Boston Dynamic's robot dog, Spot, which is
| around $200K iirc.
|
| 3. But that's also why we need more projects like Dobb-E on
| these robots! Without the right "apps" home robots will never
| catch on properly to get to the point of mass-production.
|
| [0] https://docs.hello-robot.com/0.2/stretch-hardware-
| guides/doc...
| yardie wrote:
| Do you think this could be adapted to smaller robot like this
| Pi-based XGO CM4[0]
|
| I know they have different hardware but the end result is
| quite similar: a mobile robot with an articulated grabber.
| And seeing your prototype made me think of the robot dog.
|
| [0] https://shop.elecfreaks.com/products/elecfreaks-cm4-xgo-
| robo...
| EwanG wrote:
| Compute on theirs is an Intel Nuc 8 (probably explains a
| decent part of the price). Appreciate the link to the more
| "hobbyist friendly" device you mentioned. I am thinking it
| might do well as a platform for light dusting around the
| house and other things to supplement a Robovac.
| cchance wrote:
| how does a 300-600$ PC explain a decent part of a 20000$
| robot?
| Tade0 wrote:
| It's slightly more than a Nao:
|
| https://www.robotlab.com/store/robotlab-nao-school-starter-p...
|
| Which has hilarious design issues like microphones and CPU both
| in the head, so they pick up fan noise and if you tell it to
| extend its hands forward it's going to eventually collapse -
| apparently to prevent the motors from overheating.
| goodpoint wrote:
| shockingly expensive
| startupsfail wrote:
| There is slight optimization. You don't need 3D printed parts for
| your reacher grabber data collection stick. Just use $5 iPhone
| microscope mount to attach the phone securely to your stick.
|
| It'd be nice also to think of some common format that includes
| depth, calibration, trajectory and doesn't require nonstandard
| software on an iPhone. Standard camera app/cinematic format could
| be one example, I think.
| MahiShafiullah wrote:
| Thank you for the suggestions!
|
| 1. I'm ordering a microscope mount right now! We went with 3D
| printing because we could get a very good coupling between the
| stick and the robot mount. See this figure for a comparison:
| https://i.imgur.com/vWopcFB.jpeg
|
| 2. Our dataset (and dataset export code) gives the depth as
| numpy files and trajectory information in a JSON file with
| calibration and trajectory information! The r3d export format
| from the Record3D app we used is simply a .zip file with
| renamed extension so we were ok with using it. Do you have
| suggestion as to which cinematic format you are thinking?
| beiller wrote:
| I think you will find openexr surprisingly well supported
| across many platforms.
| MahiShafiullah wrote:
| OpenEXR looks very interesting! Especially if OpenEXR can
| compress depth data nearly as efficiently as we can
| compress image data by making everything a video, it would
| be a game changer. For example, our RGB dataset is 814 MB
| right now while the RGB-D dataset is 77 GB, even though
| depth "just" adds one new channel to our data.
| startupsfail wrote:
| Yes, that's another thing. When Apple is recording RGBD
| in a cinematic format, it is tiny.
| startupsfail wrote:
| Yes, Record3D is an option. The problem with it, you'll never
| get large scale data in the format recorded by a custom app.
| If you use some standard app and format that is used by
| general public, there is a bit higher chance. For the iPhone
| with the depth sensor cinematic format from the standard
| camera app seem to contain depth/RGB/traj. Maybe a candidate
| format. Also it's much easier to operate standard camera app.
|
| In a longer term the stick is likely not needed too. Ideally
| you'd want a human demonstrating with regular human tool,
| like pliers to generalize.
|
| This is a particular mount that I've used.
| https://www.amazon.com/Starboosa-Smartphone-Microscope-
| Unive...
|
| It's pretty solid, but it doesn't open completely, so you
| need to disassemble and reassemble the stick to put it on
| (which is not that difficult).
| eutopian wrote:
| I guess Dobb*E is a house elf after all.
| jessmartin wrote:
| Love the name.
| pydry wrote:
| There didnt seem to be any bathroom videos. Can it clean a
| bathroom? If not, what are the impediments to doing that
| currently?
| MahiShafiullah wrote:
| Water!
|
| Our robot-and-phone setup is not very safe around liquids haha.
| Hopefully the hardware experts among us can solve that problem
| sooner than later.
|
| We are also working on learning multi-stage skills, and early
| results are promising (See section 4.1 on the paper). However,
| with multiple steps the chances of failure also goes up, so
| that's something we will have to make more robust :)
| jameshart wrote:
| One of the problems with home-cleaning robots is how to keep
| the cleaning robot clean.
|
| You don't want the same grabber that's scrubbing the toilet and
| picking up your dirty laundry to also be picking up your kids'
| toys, or packing away your groceries, let alone preparing food.
| kaibee wrote:
| Actually, this is why it's such a great business opportunity,
| see you have to buy a 2nd robot that cleans 1st robot and
| then a 3rd robot to clean the 2nd and so on.
|
| More seriously, Stretch actually looks pretty straight
| forward so I could imagine a combo-charging/dock/cleaning
| station where it can swap tools (ie, grab a vacuum, drop off
| mop, etc) and get cleaned by the cleaning station. Also once
| the robot is in a more finalized version, its not unthinkable
| to make one that is waterproof enough to go through a carwash
| type device.
| MahiShafiullah wrote:
| Tool usage is one of the next ideas we would like to
| explore; because, if you think about it, the reacher-
| grabber that is currently used as an end-effector is just
| another tool.
|
| If we can switch between tools quickly we can possibly
| unlock a wide range of applications: imagine a robot that
| can switch arms from a whisk to a spatula to a duster to to
| a mop all over the course of preparing a meal, and then
| switch to a screwdriver or a wrench for fixing something
| afterwards. Will take work to design modular parts like
| that, but the end result can be quite exciting indeed.
| sheepscreek wrote:
| Fantastic! Great to see a robotics framework that is no-frills
| (easily reproducible/DIY) and is immediately useful. Thank you
| for open-sourcing it.
| cyberax wrote:
| Can you train it to attack home invaders? Ideally chanting:
| "Exterminate! Exterminate!"
| MahiShafiullah wrote:
| Not in our plans anytime soon, sorry :) We're focusing on home
| robots right now instead of rather unsettling space mutants.
| Obscurity4340 wrote:
| Iteration is key
| throwitaway222 wrote:
| VB Automation for IRL.
| sklargh wrote:
| I would give up my second car for a robot that could reliable
| transit a 120 year old house to do dishes and wash and fold
| laundry.
| fudged71 wrote:
| This is brilliant.
|
| Are considering crowdsourcing the data from more people? It looks
| fun, and could save me time in the future, so I think this could
| be worth doing
| MahiShafiullah wrote:
| Thank you!
|
| Yes, we have considered launching an open source effort to
| scale up the data, perhaps to be comparable to what Google has
| collected through paid tele-operators [0]. What do you think
| would be the right incentive structure for everyday volunteers
| to participate in this effort?
|
| [0] https://everydayrobots.com/thinking/rt-1-robotics-
| transforme...
| pbronez wrote:
| Privacy would be a big part of it. Video of my home could
| potentially contain lots of private information. How is that
| risk minimized?
| MahiShafiullah wrote:
| We don't want any private information, so currently we are
| manually going over every new demo to make sure private
| data (face, hands, any other identifying information)
| doesn't get included in the dataset. Pretty hard to
| "blitzscale" this way, but I personally think doing it
| right is more important.
| rexreed wrote:
| Looks awesome. The Hello Robot shown is fairly pricy. (From the
| site: "A Fully Integrated Mobile Manipulator for $19,950").
|
| What are the options to do similar manipulations for lower cost
| on a robotic unit?
| MahiShafiullah wrote:
| I discuss this a little bit on another thread:
| (https://news.ycombinator.com/item?id=38461559)
|
| But the tl;dr is that building prototypes is pricey, and the
| cost should definitely come down once the community moves from
| prototyping to mass-producing robots.
|
| Hello Robots (the company behind the Stretch robot we're using)
| is also trying to bootstrap and build a sustainable product
| rather than blitzscaling and burning out fast [0, 1, 2] for
| which I respect them a lot. It's all too common of a story in
| robot world where have a great company showing lots of promise
| and then a year or two later they shut down after burning
| through investor money. I don't want to see it repeat.
|
| [0] https://techcrunch.com/2018/07/25/mayfield-robotics-
| ceases-p...
|
| [1] https://www.theverge.com/2023/2/24/23613214/everyday-
| robots-...
|
| [2] https://www.theverge.com/2019/6/19/18682780/jibo-death-
| serve...
| cchance wrote:
| Why does it feel like it's moving in such jerky steps?
| MahiShafiullah wrote:
| Great observation!
|
| Right now we rely on the out-of-the-box Hello Robots controller
| [0] implementing position control. Since our policy only
| predicts a single position into the future at a time, the
| motion ends up being either jerky and fast, or smooth and slow.
| Improving on the controller is probably one of the highest ROI
| improvements we can make moving into the future.
|
| [0] https://github.com/hello-robot/stretch_body
| snihalani wrote:
| good stuff Mahi. please commercialize this and visit us in Bay
| Area!
| MahiShafiullah wrote:
| Thank you Suren! It will get there when it gets there and so
| will I :) I do miss you folks, though.
| praveen9920 wrote:
| Can it handle any weights on its own? Carrying stuff is
| definitely one important task that could be useful for lot of
| people.
| MahiShafiullah wrote:
| The robot can carry small weights on its own, yes, although
| it's way more stable when the arm is retracted closer to the
| body.
|
| Pick-and-place style problems are one of the earliest tried on
| our robot platform, Hello Stretch, which is why this project
| doesn't spend too much time on it. You may enjoy something like
| the OVMM project [0] which focuses entirely on open-vocabulary
| pick-and-place problems: "Pick up $A from $B and put it in $C".
|
| [0] https://ovmm.github.io/
| owenpalmer wrote:
| It's like Dobby from Harry Potter!
| advael wrote:
| Truly an awful choice of name
| filipezf wrote:
| I was thinking on buying/building one robot like this to
| experiment. Can one fit more than one manipulator on it (so it
| could more easily do things like hang out clothes, or hold a
| glass of water and add ice)? More generally, how does the AI
| model would adapt to diverse robot morphologies?
___________________________________________________________________
(page generated 2023-11-29 23:00 UTC)