https://github.com/facebookexperimental/hermit Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} facebookexperimental / hermit Public * Notifications * Fork 29 * Star 907 * Hermit launches linux x86_64 programs in a special, hermetically isolated sandbox to control their execution. Hermit translates normal, nondeterministic behavior, into deterministic, repeatable behavior. This can be used for various applications, including replay-debugging, reproducible artifacts, chaos mode concurrency testing and bug analysis. License View license 907 stars 29 forks Branches Tags Activity Star Notifications * Code * Issues 20 * Pull requests 2 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights facebookexperimental/hermit This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 241 Commits .github .github common common detcore-model detcore-model detcore detcore docs docs examples examples flaky-tests flaky-tests hermit-cli hermit-cli hermit-verify hermit-verify scripts scripts tests tests .gitignore .gitignore CODE_OF_CONDUCT.md CODE_OF_CONDUCT.md CONTRIBUTING.md CONTRIBUTING.md Cargo.toml Cargo.toml LICENSE LICENSE README.md README.md rust-toolchain.toml rust-toolchain.toml rustfmt.toml rustfmt.toml View all files Repository files navigation * README * Code of conduct * License * Security Hermit: A reproducible container Hermit forces deterministic execution of arbitrary programs and acts like a reproducible container. That is, it hermetically isolates the program from sources of non-determinism such as time, thread interleavings, random number generation, etc. Guaranteed determinism is a powerful tool and it serves as a basis for a number of applications, including concurrency stress testing, record/replay, reproducible builds, and automatic diagnosis of concurrency bugs, and more. Hermit cannot isolate the guest program from sources of non-determinism such as file system changes or external network responses. Instead, in order to provide complete determinism, the user should provide a fixed file system base image (e.g., with Docker) and disable external networking. Warning Hermit is no longer under active development within Meta and is in maintenance mode. There is a long tail of unsupported system calls that may cause your program to fail while running under Hermit. Unfortunately, we (the team behind this project) don't have the resources to triage issues, fix major bugs, or add features at this point in time. If you are interested in this project and want to make contributions, please submit a pull request. We will prioritize merging these over anything else. How it works Hermit sits between the guest process and the OS intercepting system calls made by the guest (using Reverie). In some cases, it can completely replace the functionality of the kernel and suppress the original system call. In other cases, it forwards the system call to the kernel and sanitizes the response such that it is made deterministic. As a concrete example, lets say we have a program that reads random bytes from /dev/urandom. Hermit will see that the guest opened this file (a known source of non-determinism) and intercept subsequent reads to this file. Instead of letting the OS fill a buffer with random bytes, Hermit uses a deterministic pseudorandom number generator with a fixed seed to fill in the buffer. The contents of the buffer are then guaranteed to be the same upon every execution of the program. The most complex source of non-determinism is in the thread scheduler. The way threads are scheduled by the kernel depends on many external factors, including the number of physical CPUs or other threads running on the system that require CPU time. To ensure that the threads of the guest process (and all of its child processes) are scheduled in a repeatable way, we first make sure that all thread executions are serialized so that there is effectively only one CPU. Then, we deterministically pick which thread is allowed to run next. In order to only allow a thread to run for a fixed number of instructions, we use the CPU's Performance Monitoring Unit (PMU) to stop execution after a fixed number of retired conditional branches (RCBs). Read below about how to build Hermit, and you can get an idea of what it does from running examples in the ./examples folder. Building and running Hermit Hermit is built with the standard Rust cargo tool. cargo build This builds the whole cargo workspace. The actual binary is located in target directory (target/debug/hermit). Then, once you've built Hermit, all you need to run your program deterministically is: hermit run After that you can try running it in a concurrency stress testing (chaos) mode, or varying other parameters of the configuration such as the speed at which virtual time passes inside the container, or the random number generation seed: hermit run --chaos --sched-seed=3 You can use hermit as a replay-debugger as well, either recording a non-deterministic execution (real time, real randomness, etc), or repeatedly running a controlled, deterministic one (virtual time, pseudo-randomness, etc). hermit record start hermit replay Example programs See the the examples folder for example programs and instructions on how to run them. These showcase different sources of nondeterminism, and how hermit eliminates or controls them. In order to explore more advanced examples, you can look at some of the integration tests built from ./tests/ or ./flaky-tests/. For example, using the commands below you can run a racey example multiple times to see its nondeterminism. Then run it under hermit to watch that nondeterminism disappear. Then run it under hermit --chaos to bring that nondeterminism back, but in a controlled way that can be reproduced based on the input seed. cargo build for ((i=0; i<20; i++)); do ./target/debug/hello_race; done for ((i=0; i<20; i++)); do hermit run ./target/debug/hello_race; done for ((i=0; i<20; i++)); do hermit run --chaos --seed-from=SystemRandom ./target/debug/hello_race; done The state of CI and testing At Meta, this repository is built using buck. We have over 700 integration tests that run under this setup. But as of this initial release (2022-11-21), we have not ported these tests to an external build system yet. A few unit tests run under cargo test, but the integration tests are more complicated because they combine various run modes with each of the test binaries (which are built from tests/, flaky-tests/, and the rr test suite too). We plan to get the internal Buck configuration files building externally with buck or bazel. Applications Hermit translates normal, non-deterministic Linux behavior, into deterministic, repeatable behavior. This can be used for various applications, including: record/replay debugging, simple reproducibility, "chaos mode" to expose concurrency bugs in a controlled and repeatable way. Generally, Hermit makes implicit inputs into explicit ones, and so enables searching over possible executions by explicitly varying these inputs and study the changes in outcomes. This can be used for either searching for bugs or trying to narrow down their causes. Diagnosing concurrency bugs One experimental application that Hermit has built-in support for is diagnosing concurrency bugs, in terms of identifying the stack traces of racing critical operations which, if their order is flipped in the schedule, cause the program to crash (also called an "order violation" bug). Hermit can detect these even if the racing instructions are in different processes and written in different programming languages. You can kick off analyze with any program invocation, and tell it to search for failing (and passing) executions, and then diagnose the difference between them. hermit analyze --search -- License Hermit is licensed under a BSD-3 clause license, included in the LICENSE file in this directory. Support Hermit currently supports x86_64 Linux. Aarch64 support is a work in progress. About Hermit launches linux x86_64 programs in a special, hermetically isolated sandbox to control their execution. Hermit translates normal, nondeterministic behavior, into deterministic, repeatable behavior. This can be used for various applications, including replay-debugging, reproducible artifacts, chaos mode concurrency testing and bug analysis. Resources Readme License View license Code of conduct Code of conduct Security policy Security policy Activity Custom properties Stars 907 stars Watchers 12 watching Forks 29 forks Report repository Releases No releases published Packages 0 No packages published Contributors 23 * @rrnewton * @VladimirMakaev * @zertosh * @jasonwhite * @facebook-github-bot * @stepancheg * @sergik * @diliop * @capickett * @krallin * @cooperlees * @mjhostet * @c-ryan747 * @AndreasBackx + 9 contributors Languages * Rust 92.2% * C 3.7% * Shell 2.4% * Starlark 0.8% * Python 0.6% * C++ 0.3% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.