https://github.com/facebookresearch/segment-anything-2 Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + GitHub Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} facebookresearch / segment-anything-2 Public * Notifications You must be signed in to change notification settings * Fork 159 * Star 4k The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. License Apache-2.0, BSD-3-Clause licenses found Licenses found Apache-2.0 LICENSE BSD-3-Clause LICENSE_cctorch 4k stars 159 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 30 * Pull requests 4 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights facebookresearch/segment-anything-2 This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 11 Commits assets assets checkpoints checkpoints notebooks notebooks sam2 sam2 sam2_configs sam2_configs sav_dataset sav_dataset tools tools .clang-format .clang-format .gitignore .gitignore CODE_OF_CONDUCT.md CODE_OF_CONDUCT.md CONTRIBUTING.md CONTRIBUTING.md LICENSE LICENSE LICENSE_cctorch LICENSE_cctorch README.md README.md pyproject.toml pyproject.toml setup.py setup.py View all files Repository files navigation * README * Code of conduct * Apache-2.0 license * BSD-3-Clause license * Security SAM 2: Segment Anything in Images and Videos AI at Meta, FAIR Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Radle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, Christoph Feichtenhofer [Paper] [Project] [Demo] [Dataset] [Blog] [BibTeX] SAM 2 architecture Segment Anything Model 2 (SAM 2) is a foundation model towards solving promptable visual segmentation in images and videos. We extend SAM to video by considering images as a video with a single frame. The model design is a simple transformer architecture with streaming memory for real-time video processing. We build a model-in-the-loop data engine, which improves model and data via user interaction, to collect our SA-V dataset, the largest video segmentation dataset to date. SAM 2 trained on our data provides strong performance across a wide range of tasks and visual domains. SA-V dataset Installation Please install SAM 2 on a GPU machine using: git clone git@github.com:facebookresearch/segment-anything-2.git cd segment-anything-2; pip install -e . To use the SAM 2 predictor and run the example notebooks, jupyter and matplotlib are required and can be installed by: pip install -e ".[demo]" Getting Started Download Checkpoints First, we need to download a model checkpoint. All the model checkpoints can be downloaded by running: cd checkpoints ./download_ckpts.sh or individually from: * sam2_hiera_tiny.pt * sam2_hiera_small.pt * sam2_hiera_base_plus.pt * sam2_hiera_large.pt Then SAM 2 can be used in a few lines as follows for image and video prediction. Image prediction SAM 2 has all the capabilities of SAM on static images, and we provide image prediction APIs that closely resemble SAM for image use cases. The SAM2ImagePredictor class has an easy interface for image prompting. import torch from sam2.build_sam import build_sam2 from sam2.sam2_image_predictor import SAM2ImagePredictor checkpoint = "./checkpoints/sam2_hiera_large.pt" model_cfg = "sam2_hiera_l.yaml" predictor = SAM2ImagePredictor(build_sam2(model_cfg, checkpoint)) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image() masks, _, _ = predictor.predict() Please refer to the examples in image_predictor_example.ipynb for static image use cases. SAM 2 also supports automatic mask generation on images just like SAM. Please see automatic_mask_generator_example.ipynb for automatic mask generation in images. Video prediction For promptable segmentation and tracking in videos, we provide a video predictor with APIs for example to add prompts and propagate masklets throughout a video. SAM 2 supports video inference on multiple objects and uses an inference state to keep track of the interactions in each video. import torch from sam2.build_sam import build_sam2_video_predictor checkpoint = "./checkpoints/sam2_hiera_large.pt" model_cfg = "sam2_hiera_l.yaml" predictor = build_sam2_video_predictor(model_cfg, checkpoint) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): state = predictor.init_state() # add new prompts and instantly get the output on the same frame frame_idx, object_ids, masks = predictor.add_new_points(state, ): # propagate the prompts to get masklets throughout the video for frame_idx, object_ids, masks in predictor.propagate_in_video(state): ... Please refer to the examples in video_predictor_example.ipynb for details on how to add prompts, make refinements, and track multiple objects in videos. Model Description Model Size Speed (FPS) SA-V test MOSE val LVOS v2 (M) (J&F) (J&F) (J&F) sam2_hiera_tiny 38.9 47.2 75.0 70.9 75.3 sam2_hiera_small 46 43.3 (53.0 74.9 71.5 76.4 compiled*) sam2_hiera_base_plus 80.8 34.8 (43.8 74.7 72.8 75.8 compiled*) sam2_hiera_large 224.4 24.2 (30.2 76.0 74.6 79.8 compiled*) * Compile the model by setting compile_image_encoder: True in the config. Segment Anything Video Dataset See sav_dataset/README.md for details. License The models are licensed under the Apache 2.0 license. Please refer to our research paper for more details on the models. Contributing See contributing and the code of conduct. Contributors The SAM 2 project was made possible with the help of many contributors (alphabetical): Karen Bergan, Daniel Bolya, Alex Bosenberg, Kai Brown, Vispi Cassod, Christopher Chedeau, Ida Cheng, Luc Dahlin, Shoubhik Debnath, Rene Martinez Doehner, Grant Gardner, Sahir Gomez, Rishi Godugu, Baishan Guo, Caleb Ho, Andrew Huang, Somya Jain, Bob Kamma, Amanda Kallet, Jake Kinney, Alexander Kirillov, Shiva Koduvayur, Devansh Kukreja, Robert Kuo, Aohan Lin, Parth Malani, Jitendra Malik, Mallika Malhotra, Miguel Martin, Alexander Miller, Sasha Mitts, William Ngan, George Orlin, Joelle Pineau, Kate Saenko, Rodrick Shepard, Azita Shokrpour, David Soofian, Jonathan Torres, Jenny Truong, Sagar Vaze, Meng Wang, Claudette Ward, Pengchuan Zhang. Third-party code: we use a GPU-based connected component algorithm adapted from cc_torch (with its license in LICENSE_cctorch) as an optional post-processing step for the mask predictions. Citing SAM 2 If you use SAM 2 or the SA-V dataset in your research, please use the following BibTeX entry. @article{ravi2024sam2, title={SAM 2: Segment Anything in Images and Videos}, author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph}, journal={arXiv preprint}, year={2024} } About The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. Resources Readme License Apache-2.0, BSD-3-Clause licenses found Licenses found Apache-2.0 LICENSE BSD-3-Clause LICENSE_cctorch Code of conduct Code of conduct Security policy Security policy Activity Custom properties Stars 4k stars Watchers 37 watching Forks 159 forks Report repository Releases No releases published Packages 0 No packages published Contributors 3 * @CharlesCNorton CharlesCNorton * @ronghanghu ronghanghu Ronghang Hu * @haithamkhedr haithamkhedr Haitham Khedr Languages * Jupyter Notebook 99.1% * Other 0.9% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.