https://github.com/YavorGIvanov/sam.cpp Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} YavorGIvanov / sam.cpp Public * Notifications * Fork 19 * Star 691 License MIT license 691 stars 19 forks Activity Star Notifications * Code * Issues 2 * Pull requests 1 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights YavorGIvanov/sam.cpp This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 4 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/Y] Use Git or checkout with SVN using the web URL. [gh repo clone YavorG] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @ggerganov @YavorGIvanov ggerganov and YavorGIvanov sam : segment on move ... 3dbd1f6 Sep 6, 2023 sam : segment on move 3dbd1f6 Git stats * 31 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time examples sam : segment on move September 6, 2023 15:48 ggml @ 69bf842 Update README example output August 28, 2023 15:53 .gitignore add model checkpoint link to README.md September 5, 2023 21:24 .gitmodules Add first version of GUI example using ImGUI + SDL September 4, 2023 18:49 CMakeLists.txt cmake : fix build on Mac OS September 5, 2023 17:54 LICENSE Add MIT LICENSE September 6, 2023 14:10 README.md Update README.md with threads note September 6, 2023 15:47 convert-pth-to-ggml.py Add output dir to pth conversion script August 28, 2023 15:40 img.jpg Initial commit August 23, 2023 17:27 sam.cpp [FIX] Integer Division by zero on Windows build September 6, 2023 14:12 sam.h Add option to drop new image and continue segmenting September 5, 2023 15:40 View code SAM.cpp Quick start Downloading and converting the model checkpoints Example output on M2 Ultra References Next steps README.md SAM.cpp Inference of Meta's Segment Anything Model in pure C/C++ demo-0.mp4 Quick start git clone --recursive https://github.com/YavorGIvanov/sam.cpp cd sam.cpp Note: you need to download the model checkpoint below (sam_vit_b_01ec64.pth) first from here and place it in the checkpoints folder # Convert PTH model to ggml. Requires python3, torch and numpy python convert-pth-to-ggml.py checkpoints/sam_vit_b_01ec64.pth . 1 # You need CMake and SDL2 SDL2 - Used for GUI windows & input [libsdl](https://www.libsdl.org) [Ubuntu] $ sudo apt install libsdl2-dev [Mac OS with brew] $ brew install sdl2 [MSYS2] $ pacman -S git cmake make mingw-w64-x86_64-dlfcn mingw-w64-x86_64-gcc mingw-w64-x86_64-SDL2 # Build sam.cpp. mkdir build && cd build cmake .. && make -j4 # run inference ./bin/sam -t 16 -i ../img.jpg -m ../checkpoints/ggml-model-f16.bin Note: The optimal threads parameter ("-t") value should be manually selected based on the specific machine running the inference. Downloading and converting the model checkpoints You can download a model checkpoint and convert it to ggml format using the script convert-pth-to-ggml.py: # Convert PTH model to ggml python convert-pth-to-ggml.py sam_vit_b_01ec64.pth . 1 Example output on M2 Ultra $ > make -j sam && time ./bin/sam -t 8 -i img.jpg [ 28%] Built target common [ 71%] Built target ggml [100%] Built target sam main: seed = 1693224265 main: loaded image 'img.jpg' (680 x 453) sam_image_preprocess: scale = 0.664062 main: preprocessed image (1024 x 1024) sam_model_load: loading model from 'models/sam-vit-b/ggml-model-f16.bin' - please wait ... sam_model_load: n_enc_state = 768 sam_model_load: n_enc_layer = 12 sam_model_load: n_enc_head = 12 sam_model_load: n_enc_out_chans = 256 sam_model_load: n_pt_embd = 4 sam_model_load: ftype = 1 sam_model_load: qntvr = 0 operator(): ggml ctx size = 202.32 MB sam_model_load: ...................................... done sam_model_load: model size = 185.05 MB / num tensors = 304 embd_img dims: 64 64 256 1 f32 First & Last 10 elements: -0.05117 -0.06408 -0.07154 -0.06991 -0.07212 -0.07690 -0.07508 -0.07281 -0.07383 -0.06779 0.01589 0.01775 0.02250 0.01675 0.01766 0.01661 0.01811 0.02051 0.02103 0.03382 sum: 12736.272313 Skipping mask 0 with iou 0.705935 below threshold 0.880000 Skipping mask 1 with iou 0.762136 below threshold 0.880000 Mask 2: iou = 0.947081, stability_score = 0.955437, bbox (371, 436), (144, 168) main: load time = 51.28 ms main: total time = 2047.49 ms real 0m2.068s user 0m16.343s sys 0m0.214s Input point is (414.375, 162.796875) (currently hardcoded) Input image: llamas Output mask (mask_out_2.png in build folder): mask_glasses References * ggml * ggml SAM example * SAM * SAM demo Next steps * [*] Reduce memory usage by utilizing the new ggml-alloc * [*] Remove redundant graph nodes * [ ] Make inference faster * [*] Fix the difference in output masks compared to the PyTorch implementation * [*] Filter masks based on stability score * [*] Add support for point user input * [ ] Support F16 for heavy F32 ops * [ ] Test quantization * [*] Support bigger model checkpoints * [ ] GPU support * [ ] Add support for mask and box input About No description, website, or topics provided. Resources Readme License MIT license Activity Stars 691 stars Watchers 9 watching Forks 19 forks Report repository Releases No releases published Packages 0 No packages published Contributors 5 * @YavorGIvanov * @ggerganov * @iboB * @flaviopontes * @danieloi Languages * C++ 95.2% * Python 3.9% * CMake 0.9% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.