https://github.com/apple/ml-depth-pro Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Security Find and fix vulnerabilities + Codespaces Instant dev environments + GitHub Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} apple / ml-depth-pro Public * Notifications You must be signed in to change notification settings * Fork 12 * Star 481 Depth Pro: Sharp Monocular Metric Depth in Less Than a Second. License View license 481 stars 12 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 2 * Pull requests 2 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Security * Insights apple/ml-depth-pro This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Last Last Name Name commit commit message date Latest commit History 2 Commits data data src/depth_pro src/depth_pro ACKNOWLEDGEMENTS.md ACKNOWLEDGEMENTS.md CODE_OF_CONDUCT.md CODE_OF_CONDUCT.md CONTRIBUTING.md CONTRIBUTING.md LICENSE LICENSE README.md README.md get_pretrained_models.sh get_pretrained_models.sh pyproject.toml pyproject.toml View all files Repository files navigation * README * Code of conduct * License Depth Pro: Sharp Monocular Metric Depth in Less Than a Second This software project accompanies the research paper: Depth Pro: Sharp Monocular Metric Depth in Less Than a Second, Aleksei Bochkovskii, Amael Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun. [depth-pro-] We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. The model in this repository is a reference implementation, which has been re-trained. Its performance is close to the model reported in the paper but does not match it exactly. Getting Started We recommend setting up a virtual environment. Using e.g. miniconda, the depth_pro package can be installed via: conda create -n depth-pro -y python=3.9 conda activate depth-pro pip install -e . To download pretrained checkpoints follow the code snippet below: source get_pretrained_models.sh # Files will be downloaded to `checkpoints` directory. Running from commandline We provide a helper script to directly run the model on a single image: # Run prediction on a single image: depth-pro-run -i ./data/example.jpg # Run `depth-pro-run -h` for available options. Running from python from PIL import Image import depth_pro # Load model and preprocessing transform model, transform = depth_pro.create_model_and_transforms() model.eval() # Load and preprocess an image. image, _, f_px = depth_pro.load_rgb(image_path) image = transform(image) # Run inference. prediction = model.infer(image, f_px=f_px) depth = prediction["depth"] # Depth in [m]. focallength_px = prediction["focallength_px"] # Focal length in pixels. Evaluation (boundary metrics) Our boundary metrics can be found under eval/boundary_metrics.py and used as follows: # for a depth-based dataset boundary_f1 = SI_boundary_F1(predicted_depth, target_depth) # for a mask-based dataset (image matting / segmentation) boundary_recall = SI_boundary_Recall(predicted_depth, target_mask) Citation If you find our work useful, please cite the following paper: @article{Bochkovskii2024:arxiv, author = {Aleksei Bochkovskii and Ama\"{e}l Delaunoy and Hugo Germain and Marcel Santos and Yichao Zhou and Stephan R. Richter and Vladlen Koltun} title = {Depth Pro: Sharp Monocular Metric Depth in Less Than a Second}, journal = {arXiv}, year = {2024}, url = {https://arxiv.org/abs/2410.02073}, } License This sample code is released under the LICENSE terms. The model weights are released under the LICENSE terms. Acknowledgements Our codebase is built using multiple opensource contributions, please see Acknowledgements for more details. Please check the paper for a complete list of references and datasets used in this work. About Depth Pro: Sharp Monocular Metric Depth in Less Than a Second. Resources Readme License View license Code of conduct Code of conduct Activity Custom properties Stars 481 stars Watchers 15 watching Forks 12 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 99.5% * Shell 0.5% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.