https://danijar.com/project/director/ [face] Danijar Hafner Artificial Intelligence Researcher Research Mentoring Resources Deep Hierarchical Planning from Pixels Danijar Hafner, Kuang-Huei Lee, Ian Fischer, Pieter Abbeel Preprint Paper Twitter Introducing Director While humans can easily break down complex long-horizon tasks into managemable subgoals, artificial agents often struggle with tasks that require more than a few hundred decisions despite enormous computational budgets. Research on hierarchical reinforcement learning aims to overcome this limitation but has proven to be challenging, current methods rely on manually specified goal spaces or subtasks, and no general solution exists. We introduce Director, a practical method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. The manager policy maximizes task and exploration rewards by selecting latent goals and the worker policy learns to achieve the goals. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director outperforms exploration methods on tasks with very sparse rewards, including 3D maze traversal with a quadruped robot from an egocentric camera and proprioception, without access to the global position or top-down view used by prior work. Director also learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels. We summarize our main contributions as follows: 1. We describe a practical, general, and interpretable algorithm for learning hierarchical behaviors from pixels, which we call Director. 2. We identify two critical design choices, namely discretizing the possible latent goals through a goal autoencoder and the max-cosine goal reward. 3. We introduce two sparse reward benchmarks that underscore the limitations of current RL agents and find that Director solves these challenging tasks. 4. We demonstrate that Director successfully learns across a wide range of traditional RL domains, including Atari, Control Suite, DMLab, and Crafter. 5. We visualize the internal latent goals that Director uses to break down various tasks, providing insights into its decision making. How Director Works Director learns a world model from pixels that enables efficient planning in a compact latent space. From imagined rollouts in representation space, it learns a manager and a worker which communicate via internal goals. [method] This is how Director works: * The world model provides continuous 1024-dimensional representations and allows planning in this space. * The goal autoencoder compresses the world model representations into compact discrete codes. * The manager selects codes in this discrete space every 8 steps. The goal autoencoder decodes these codes into feature space goals. * The worker learns to reach these goals via the max-cosine similarity that considers both angle and magnitude between current state and goal. The manager maximizes task reward plus exploration bonus and the worker only learns from the feature distance between the goal and current state. Both policies learn from shared imagind rollouts. All model components are optimized concurrently and throughout learning on end-to-end reinforcement learning tasks. Egocentric Ant Maze Benchmark Prior work often resorted to custom evaluation protocols, such as assuming diverse practice goals during training, providing top-down XY coordinates to the agent, and requiring dense ground-truth distance rewards. In this paper, we train end-to-end on reinforcement learning tasks with very sparse rewards directly from raw sensory inputs, without assuming access to semantic goal spaces. [prior] In this benchmark, a quadruped robot is controlled through joint torques to navigate to a fixed location in a 3D maze, given only first-person camera and proprioceptive inputs. The only reward is given at time steps where the agent touches the reward object. Director solves all four tasks by breaking them down into manageable subgoals that the worker can reach. [mazes] Visual Pin Pad benchmark The agent controls the black square to move in four directions. Each environment has a different number of pads that can be activated by walking to and stepping on them. A single sparse reward is given when the agent activates all pads in the correct sequence. The history of previously activated pads is shown at the bottom of the screen. Unlike prior algorithms, Director succeeds across difficulty levels. [pinpads] Standard Benchmarks To evaluate the generality of Director, we train on a wide range of standard benchmarks, including continuous control from pixels, Atari games, DMLab mazes, and Crafter. We find that Director learns successfully across all these domains, despite the worker learning only from the intrinsic goal reward. When additionally giving task reward to the worker, Director fully closes the gap to Dreamer, demonstrating that Director is a generally applicable RL algorithm. [standard] Goal Visualizations While Director uses latent feature vectors as goals, the world model allows us to decode them into images for human inspection. Each video shows: * Left: The environment inputs of an episode as seen by the agent * Right: The visualized goals that Director chooses internally We find that Director discovers diverse strategies for breaking down tasks into internal subgials, such as leveraging robot poses, salient landmarks in the environment, and inventory and score displays on the screen. The goals generally stay ahead of the worker, efficiently directing it often without giving it enough time to fully reach the previous goal. Visual Pin Pad Six There is only one sparse reward after activating all pads in the right order. The manager directs the worker via the history display at the bottom of the image and sometimes via the black player square. Atari Pong Because the game is reactive and requires no long-term reasoning, the manager learns to communicate the task to the worker by requesting a higher score via the score display at the top of the screen. Egocentric Ant Maze M The quadruped robot has to explore the maze through low-level locomotion and find the target zone to receive a sparse reward. The colored walls allow the agent to localize itself in the maze and can serve as meaningful subgoals. Egocentric Ant Maze XL This is the most challenging task in our benchmark, requiring the quadruped to find the sparse goal in the largest maze. The manager proposes multiple intermediat subgoals for this tasks, including red, pink, yellow, and gray walls. Cartpole Swingup Early during the episode, the manager requests a sideways angle to help the worker swing up the pole, after which the goal remains an upright position with small left and right movements to correct imbalances. Acrobot Swingup The upright reward is difficult to discover in this task. But once found, the manager can easily request the rewarding upright pose. From then on, the worker can follow its dense feature space reward to learn swing up the two link pole. Walker Walk The manager abstracts away the detail of leg movement, directing the worker through a forward-leaning pose with both feed above the ground and a shifting floor pattern. The latent goal also likely contains velocity information. The worker fills in the leg movement to pass through the goals. Humanoid Walk For the humanoid task, the manager uses an upright pose to direct the worker to stand up, following by a shifting floor pattern for carefully walking forward without falling. Director is the first hierarchical agent to solve this challenging control task end-to-end without demonstrations. Crafter The manager directs the worker via the item display to collect wood and create a pickaxe. It then sends the worker to a cave to collect stone and iron. As it gets dark, the manager tells the worker to find a small cave or island to hide from mosters. DMLab Goals Small The manager requests the teleport animation that occurs when collecting the reward object. Because there is no locomotion challenge in this environment, the worker can navigate to the goal object on its own, without fine-grained goals. Without Goal Autoencoder Director uses a goal autoencoder to aid manager learning by providing a discretized action space and allowing only goals that are similar to states from the replay buffer. As an ablation, we here show the selected goals if we remove the goal autoencoder so that the manager chooses goal vectors directly in the continuous feature space of the world model. Each video shows: * Left: The environment inputs of an episode as seen by the agent * Right: The visualized goals that Director chooses internally As you can see, the goals are completely uninterpretable and cause the agent to fail in many, but not all, of the tested environments. Visual Pin Pad Five Crafter DMLab Goals Small Walker Walk