https://dobb-e.com/

[logo]DOBB*E
 
Videos
 
Hardware
 
Dataset
 
Models
 
Paper
 
Code
Contact us
Start using Dobb*E

Dobb*E

An open-source, general framework for learning household robotic
manipulation

109 tasks

10 NYC homes

81% success rate

20 minutes to learn a new task

Nur Muhammad "Mahi" Shafiullah*, Anant Rai*, Haritheja Etukuru,
Yiqian Liu, Ishan Misra, Soumith Chintala, Lerrel Pinto

Dobb*E in homes Read the paper GitHub repo
Expand abstract

Throughout history, we have successfully integrated various machines
into our homes. Dishwashers, laundry machines, stand mixers, and
robot vacuums are just a few recent examples. However, these machines
excel at performing only a single task effectively. The concept of a
"generalist machine" in homes - a domestic assistant that can adapt
and learn from our needs, all while remaining cost-effective - has
long been a goal in robotics that has been steadily pursued for
decades. In this work, we initiate a large-scale effort towards this
goal by introducing Dobb*E, an affordable yet versatile
general-purpose system for learning robotic manipulation within
household settings. Dobb*E can learn a new task with only five
minutes of a user showing it how to do it, thanks to a demonstration
collection tool ("The Stick") we built out of cheap parts and
iPhones. We use the Stick to collect 13 hours of data in 22 homes of
New York City, and train Home Pretrained Representations (HPR). Then,
in a novel home environment, with five minutes of demonstrations and
fifteen minutes of adapting the HPR model, we show that Dobb*E can
reliably solve the task on the Stretch, a mobile robot readily
available on the market. Across roughly 30 days of experimentation in
homes of New York City and surrounding areas, we test our system in
10 homes, with a total of 109 tasks in different environments, and
finally achieve a success rate of 81%. Beyond success percentages,
our experiments reveal a plethora of unique challenges absent or
ignored in lab robotics. These range from effects of strong shadows
to variable demonstration quality by non-expert users. With the hope
of accelerating research on home robots, and eventually seeing robot
butlers in every home, we open-source Dobb*E software stack and
models, our data, and our hardware designs.

Dobb*E, in a nutshell:

Videos

Dobb*E in action

In 10 homes of New York City, Dobb*E attempted 109 tasks. Here are
sample rollouts from each of the tasks.

[Home 1 ]
[Opening horizontal blinds          ]
Shuffle

Hardware

The Stick

We believe one of the largest roadblocks to safe and scalable
progress in home robotics, especially in imitation learning based
approaches, is the lack of a cheap, ergonomic, and easy way to
collect demonstrations for robots.

To address this, we built the Stick, a demonstration collection tool
we built out of a $25 Reacher-grabber stick, some 3D printed parts,
and an iPhone.

Get the reacher-grabber
Download the 3D files
Build and usage guide
Dataset creation code

Dataset

Homes of New York (HoNY)

22 homes
216 environments
5620 trajectories
13 hours
1.5 million frames

Homes of New York (HoNY) is a dataset containing 13 hours of
interactions at 22 different homes of New York City collected with
the Stick. The dataset contains RGB and depth videos at 30 fps, as
well as full action annotations for 6D pose of the gripper as well as
the gripper's opening angle normalized between (0, 1).

RGB + actions dataset (814 MB)
RGB-D + actions dataset (77 GB)

Model

Home Pretrained Representations (HPR)

HPR

Home Pretrained Representation (HPR) is a model pre-trained on the 
HoNY dataset that we used to initialize a robot policy to perform a
new task in a novel enviroment. HPR is a ResNet-34 model trained on
the HoNY dataset using the MoCo-v3 self-supervised learning
objective.

During deployment,we used HPR to initialize a policy, the trunk of
which was simply our pretrained ResNet-34 model followed by two
linear layers on top.

 Get the model at Huggingface

Or if you are using  Pytorch Image Models (TIMM), you can simply
start using it in a couple of lines:

import timm

model = timm.create_model(
    "hf-hub:notmahi/dobb-e",
    pretrained=True
)

import timm

model = timm.create_model("hf-hub:notmahi/dobb-e", pretrained=True)

Paper

On Bringing Robots Home

Paper
[arxiv] Read the paper (Arxiv)
Read the paper (PDF)
Citation (bibtex)

@misc{shafiullah2023dobbe,
    title={On Bringing Robots Home}, 
    author={Nur Muhammad Mahi Shafiullah and Anant Rai and Haritheja Etukuru and Yiqian Liu and Ishan Misra and Soumith Chintala and Lerrel Pinto},
    year={2023},
    eprint={2311.16098},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Code

Get the code on github.

GitHub repo
Documentation
Questions? Contact us.

Dobb*E by Mahi Shafiullah. The source code is licensed MIT. The
website content is licensed CC ANS 4.0.