https://github.com/iterative/dvc Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} iterative / dvc Public * Notifications * Fork 1.1k * Star 12.2k Data Version Control | Git for Data & Models | ML Experiments Management dvc.org License Apache-2.0 license 12.2k stars 1.1k forks Activity Star Notifications * Code * Issues 573 * Pull requests 7 * Discussions * Actions * Projects 1 * Wiki * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights iterative/dvc This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 7 branches 505 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/i] Use Git or checkout with SVN using the web URL. [gh repo clone iterat] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @dberenbaum @efiop dberenbaum and efiop fix for DVCFileSystem().ls() ... 4a2c4f7 Oct 31, 2023 fix for DVCFileSystem().ls() 4a2c4f7 Git stats * 9,049 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .dvc make it a dvc repo (#9670) June 29, 2023 08:50 .github ci: build: split publishing and building into separate jobs (#9970) September 23, 2023 18:05 dvc fix for DVCFileSystem().ls() October 31, 2023 23:38 tests fix for DVCFileSystem().ls() October 31, 2023 23:38 .dvcignore chore: remove bin dir and simplify tests.__main__ (#8507) November 1, 2022 13:05 .flake8 [pre-commit.ci] pre-commit autoupdate (#9756) August 2, 2023 18:08 .git-blame-ignore-revs add .git-blame-ignore-revs February 4, 2023 10:02 .git_archival.txt deps: remove setuptools_scm_git_archive June 30, 2022 17:00 .gitattributes dvc: fix installation from archives December 11, 2021 05:49 .gitignore dvc.stage: log time duration on stage collection walk in verbose mode April 27, 2022 18:04 .mailmap add .mailmap August 14, 2019 01:11 .pre-commit-config.yaml [pre-commit.ci] pre-commit autoupdate (#9976) September 26, 2023 08:55 .pre-commit-hooks.yaml pre-commit: don't run dvc in parallel (#3912) May 29, 2020 21:20 .zenodo.json metadata: add Zenodo, tidy badges and keywords (#3373) February 21, 2020 04:01 CODE_OF_CONDUCT.md update pre-commit; add some more hooks (#8492) October 31, 2022 21:04 CONTRIBUTING.md Update URLs by replacing /contributing with /contributing/code (#2657 ) October 24, 2019 02:12 LICENSE add licence year + company (#6717) October 1, 2021 09:54 README.rst readme: point to dvc.org for packages July 4, 2023 18:47 pyproject.toml deps: bump dvc-data to >=2.20.0 October 31, 2023 04:51 View code [ ] Quick start How DVC works Visual Studio Code Extension Installation Snapcraft (Linux) Chocolatey (Windows) Brew (mac OS) Anaconda (Any platform) PyPI (Python) Package (Platform-specific) Ubuntu / Debian (deb) Fedora / CentOS (rpm) Contributing Community and Support Copyright Citation README.rst DVC logo Website * Docs * Blog * Tutorial * Related Technologies * How DVC works * VS Code Extension * Installation * Contributing * Community and Support GHA Tests Python Version Codecov VS Code Extension DOI PyPI PyPI Downloads deb|pkg|rpm|exe Homebrew Conda-forge Chocolatey Snapcraft Data Version Control or DVC is a command line tool and VS Code Extension to help you develop reproducible machine learning projects: 1. Version your data and models. Store them in your cloud storage but keep their version info in your Git repo. 2. Iterate fast with lightweight pipelines. When you make changes, only run the steps impacted by those changes. 3. Track experiments in your local Git repo (no servers needed). 4. Compare any data, code, parameters, model, or performance plots. 5. Share experiments and automatically reproduce anyone's experiment. Quick start Please read our Command Reference for a complete list. A common CLI workflow includes: Task Terminal Track data $ git add train.py params.yaml $ dvc add images/ $ dvc stage add -n featurize -d images/ -o features Connect code and / python featurize.py data $ dvc stage add -n train -d features/ -d train.py -o model.p -M metrics.json python train.py Make changes and $ dvc exp run -n exp-baseline experiment $ vi train.py $ dvc exp run -n exp-code-change Compare and $ dvc exp show select $ dvc exp apply exp-baseline experiments $ git add . Share code $ git commit -m 'The baseline model' $ git push Share data and ML $ dvc remote add myremote -d s3://mybucket/ models image_cnn $ dvc push How DVC works We encourage you to read our Get Started docs to better understand what DVC does and how it can fit your scenarios. The closest analogies to describe the main DVC features are these: 1. Git for data: Store and share data artifacts (like Git-LFS but without a server) and models, connecting them with a Git repository. Data management meets GitOps! 2. Makefiles for ML: Describes how data or model artifacts are built from other data and code in a standard format. Now you can version your data pipelines with Git. 3. Local experiment tracking: Turn your machine into an ML experiment management platform, and collaborate with others using existing Git hosting (Github, Gitlab, etc.). Git is employed as usual to store and version code (including DVC meta-files as placeholders for data). DVC stores data and model files seamlessly in a cache outside of Git, while preserving almost the same user experience as if they were in the repo. To share and back up the data cache, DVC supports multiple remote storage platforms - any cloud (S3, Azure, Google Cloud, etc.) or on-premise network storage (via SSH, for example). how_dvc_works DVC pipelines (computational graphs) connect code and data together. They specify all steps required to produce a model: input dependencies including code, data, commands to run; and output information to be saved. Last but not least, DVC Experiment Versioning lets you prepare and run a large number of experiments. Their results can be filtered and compared based on hyperparameters and metrics, and visualized with multiple plots. Visual Studio Code Extension VS Code Extension To use DVC as a GUI right from your VS Code IDE, install the DVC Extension from the Marketplace. It currently features experiment tracking and data management, and more features (data pipeline support, etc.) are coming soon! DVC Extension for VS Code Note: You'll have to install core DVC on your system separately (as detailed below). The Extension will guide you if needed. Installation There are several ways to install DVC: in VS Code; using snap, choco, brew, conda, pip; or with an OS-specific package. Full instructions are available here. Snapcraft (Linux) Snapcraft snap install dvc --classic This corresponds to the latest tagged release. Add --beta for the latest tagged release candidate, or --edge for the latest main version. Chocolatey (Windows) Chocolatey choco install dvc Brew (mac OS) Homebrew brew install dvc Anaconda (Any platform) Conda-forge conda install -c conda-forge mamba # installs much faster than conda mamba install -c conda-forge dvc Depending on the remote storage type you plan to use to keep and share your data, you might need to install optional dependencies: dvc-s3, dvc-azure, dvc-gdrive, dvc-gs, dvc-oss, dvc-ssh. PyPI (Python) PyPI pip install dvc Depending on the remote storage type you plan to use to keep and share your data, you might need to specify one of the optional dependencies: s3, gs, azure, oss, ssh. Or all to include them all. The command should look like this: pip install 'dvc[s3]' (in this case AWS S3 dependencies such as boto3 will be installed automatically). To install the development version, run: pip install git+git://github.com/iterative/dvc Package (Platform-specific) deb|pkg|rpm|exe Self-contained packages for Linux, Windows, and Mac are available. The latest version of the packages can be found on the GitHub releases page. Ubuntu / Debian (deb) sudo wget https://dvc.org/deb/dvc.list -O /etc/apt/sources.list.d/dvc.list wget -qO - https://dvc.org/deb/iterative.asc | sudo apt-key add - sudo apt update sudo apt install dvc Fedora / CentOS (rpm) sudo wget https://dvc.org/rpm/dvc.repo -O /etc/yum.repos.d/dvc.repo sudo rpm --import https://dvc.org/rpm/iterative.asc sudo yum update sudo yum install dvc Contributing Code Climate Contributions are welcome! Please see our Contributing Guide for more details. Thanks to all our contributors! Contributors Community and Support * Twitter * Forum * Discord Chat * Email * Mailing List Copyright This project is distributed under the Apache license version 2.0 (see the LICENSE file in the project root). By submitting a pull request to this project, you agree to license your contribution under the Apache license version 2.0 to this project. Citation DOI Iterative, DVC: Data Version Control - Git for Data & Models (2020) DOI:10.5281/zenodo.012345. Barrak, A., Eghan, E.E. and Adams, B. On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects , in Proceedings of the 28th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2021. Hawaii, USA. About Data Version Control | Git for Data & Models | ML Experiments Management dvc.org Topics python git data-science machine-learning ai collaboration developer-tools reproducibility data-version-control hacktoberfest Resources Readme License Apache-2.0 license Code of conduct Code of conduct Activity Stars 12.2k stars Watchers 134 watching Forks 1.1k forks Report repository Releases 491 3.28.0 Latest Oct 31, 2023 + 490 releases Packages 0 No packages published Used by 10.2k * @BoutainaELYAZIJI * @ravikumarsinha234 * @Rajnish-Ranjan * @Vampaxx * @SonSid * @Sachinsen1295 * @karthika-selvaraj * @raftaarrashedin + 10,173 Contributors 268 * @efiop * @skshetry * @pmrowla * @dmpetrov * @dependabot[bot] * @pared * @casperdcl * @daavoo * @isidentical * @Suor * @karajan1001 + 257 contributors Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.