https://engineering.fb.com/2022/11/16/culture/meta-code-review-time-improving/ fbpx Skip to content Engineering at Meta Search this site [ ] # * Open Source + Open Source + Meta Open Source * Platforms + Android + iOS + Web * Infrastructure Systems + Core Data + Data Infrastructure + DevInfra + Production Engineering + Security + Research Publications * Physical Infrastructure + Connectivity + Data Center Engineering + Networking & Traffic + Research Publications * Video Engineering & AR/VR + Video Engineering + Virtual Reality + Research Publications * Artificial Intelligence + ML Applications + AI Research + Research Publications * Watch Videos [ ] POSTED ON NOVEMBER 16, 2022 TO Culture, DevInfra, Open Source Move faster, wait less: Improving code review time at Meta [Eng-Blog-Self-Serve-Hero-Images-DEBUGGING-203-] By Patrick Riggs * Code reviews are one of the most important parts of the software development process * At Meta we've recognized the need to make code reviews as fast as possible without sacrificing quality * We're sharing several tools and steps we've taken at Meta to reduce the time waiting for code reviews When done well, code reviews can catch bugs, teach best practices, and ensure high code quality. At Meta we call an individual set of changes made to the codebase a "diff." While we like to move fast at Meta, every diff must be reviewed, without exception. But, as the Code Review team, we also understand that when reviews take longer, people get less done. We've studied several metrics to learn more about code review bottlenecks that lead to unhappy developers and used that knowledge to build features that help speed up the code review process without sacrificing review quality. We've found a correlation between slow diff review times (P75) and engineer dissatisfaction. Our tools to surface diffs to the right reviewers at key moments in the code review lifecycle have significantly improved the diff review experience. What makes a diff review feel slow? To answer this question we started by looking at our data. We track a metric that we call "Time In Review," which is a measure of how long a diff is waiting on review across all of its individual review cycles. We only account for the time when the diff is waiting on reviewer action. [Code-Reviews-at-Meta-image2]Time In Review is calculated as the sum of the time spent in blue sections. What we discovered surprised us. When we looked at the data in early 2021, our median (P50) hours in review for a diff was only a few hours, which we felt was pretty good. However, looking at P75 (i.e., the slowest 25 percent of reviews) we saw diff review time increase by as much as a day. We analyzed the correlation between Time In Review and user satisfaction (as measured by a company-wide survey). The results were clear: The longer someone's slowest 25 percent of diffs take to review, the less satisfied they were by their code review process. We now had our north star metric: P75 Time In Review. Driving down Time In Review would not only make people more satisfied with their code review process, it would also increase the productivity of every engineer at Meta. Driving down Time to Review for our diffs means our engineers are spending significantly less time on reviews - making them more productive and more satisfied with the overall review process. Balancing speed with quality However, simply optimizing for the speed of review could lead to negative side effects, like encouraging rubber-stamp reviewing. We needed a guardrail metric to protect against negative unintended consequences. We settled on "Eyeball Time" - the total amount of time reviewers spent looking at a diff. An increase in rubber-stamping would lead to a decrease in Eyeball Time. Now we have established our goal metric, Time In Review, and our guardrail metric, Eyeball Time. What comes next? Build, experiment, and iterate Nearly every product team at Meta uses experimental and data-driven processes to release and iterate on features. However, this process is still very new to internal tools teams like ours. There are a number of challenges (sample size, randomization, network effect) that we've had to overcome that product teams do not have. We address these challenges with new data foundations for running network experiments and using techniques to reduce variance and increase sample size. This extra effort is worth it -- by laying the foundation of an experiment, we can later prove the impact and the effectiveness of the features we're building. [Code-Reviews-at-Meta-image3]The experimental process: The selection of goal and guardrail metrics is driven by the hypothesis we hold for the feature. We built the foundations to easily choose different experiment units to randomize treatment, including randomization by user clusters. Next reviewable diff The inspiration for this feature came from an unlikely place -- video streaming services. It's easy to binge watch shows on certain streaming services because of how seamless the transition is from one episode to another. What if we could do that for code reviews? By queueing up diffs we could encourage a diff review flow state, allowing reviewers to make the most of their time and mental energy. [Code-Reviews-at-Meta-image4] And so Next Reviewable Diff was born. We use machine learning to identify a diff that the current reviewer is highly likely to want to review. Then we surface that diff to the reviewer after they finish their current code review. We make it easy to cycle through possible next diffs and quickly remove themselves as a reviewer if a diff is not relevant to them. After its launch, we found that this feature resulted in a 17 percent overall increase in review actions per day (such as accepting a diff, commenting, etc.) and that engineers that use this flow perform 44 percent more review actions than the average reviewer! Improving reviewer recommendations The choice of reviewers that an author selects for a diff is very important. Diff authors want reviewers who are going to review their code well, quickly, and who are experts for the code their diff touches. Historically, Meta's reviewer recommender looked at a limited set of data to make recommendations, leading to problems with new files and staleness as engineers changed teams. We built a new reviewer recommendation system, incorporating work hours awareness and file ownership information. This allows reviewers that are available to review a diff and are more likely to be great reviewers to be prioritized. We rewrote the model that powers these recommendations to support backtesting and automatic retraining too. [Code-Reviews-at-Meta-image1] The result? A 1.5 percent increase in diffs reviewed within 24 hours and an increase in top three recommendation accuracy (how often the actual reviewer is one of the top three suggested) from below 60 percent to nearly 75 percent. As an added bonus, the new model was also 14 times faster (P90 latency)! Stale Diff Nudgebot We know that a small proportion of stale diffs can make engineers unhappy, even if their diffs are reviewed quickly otherwise. Slow reviews have other effects too -- the code itself becomes stale, authors have to context switch, and overall productivity drops. To directly address this, we built Nudgebot, which was inspired by research done at Microsoft. For diffs that were taking an extra long time to review, Nudgebot determines the subset of reviewers that are most likely to review the diff. Then it sends them a chat ping with the appropriate context for the diff along with a set of quick actions that allow recipients to jump right into reviewing. Our experiment with Nudgebot had great results. The average Time In Review for all diffs dropped 7 percent (adjusted to exclude weekends) and the proportion of diffs that waited longer than three days for review dropped 12 percent! The success of this feature was individually published as well. [Code-Reviews-at-Meta-image5]This is what a chat notification about a set of stale diffs looks like to a reviewer, while showing one of the potential interactions of "Remind Me Later." What comes next? Our current and future work is focused on questions like: * What is the right set of people to be reviewing a given diff? * How can we make it easier for reviewers to have the information they need to give a high quality review? * How can we leverage AI and machine learning to improve the code review process? We're continually pursuing answers to these questions, and we're looking forward to finding more ways to streamline developer processes in the future! Are you interested in building the future of developer productivity? Join us! Acknowledgements We'd like to thank the following people for their help and contributions to this post: Louise Huang, Seth Rogers, and James Saindon. Read More in Culture View All Engineering manager Mark Harman wins 2019 IEEE Harlan D. Mills award on Code.fb.com, Facebook's engineering blog FEB 13, 2019 Engineering manager Mark Harman wins 2019 IEEE Harlan D. Mills award The team behind Infer: Facebook researchers Cristiano Calcagno, Dino Distefano, and Peter W. O'Hearn and Hongseok Yang of KAIST win Most Influential POPL Paper Award at ACM SIGPLAN 2019. FEB 8, 2019 Team behind Infer wins POPL 2019 Most Influential Paper Award Leon Bottou and Olivier Bousquet DEC 6, 2018 Leon Bottou and Olivier Bousquet win Test of Time award [YoE] NOV 16, 2018 Facebook London and U.K. Year of Engineering campaign focus on future of engineering [AI-logo1] JUL 17, 2018 Facebook AI Research Expands With New Academic Collaborations [14518967_1092940280802422_7370702438770147328_n] JUN 30, 2017 Accelerating growth through Facebook's Rotational Engineering Program Related Posts --------------------------------------------------------------------- [Eng-Blog-Self-Serve-Hero-Images-DEBUGGING-201-Orange] Oct 24, 2022 From zero to 10 million lines of Kotlin [Eng-Blog-Self-Serve-Hero-Images-DATACENTERS-503-TealeLight] Sep 12, 2022 MemLab: An open source framework for finding JavaScript memory leaks [FixFast_HERO_FINAL] Feb 17, 2021 Faster, more efficient systems for finding and fixing regressions Related Positions --------------------------------------------------------------------- * Data Engineering Manager - Enterprise Finance Products MENLO PARK, US * Data Engineering Manager - Enterprise Finance Products SEATTLE, US * Data Engineering Manager - Enterprise Finance Products NEW YORK, US * Data Engineering Manager - Enterprise Finance Products REMOTE, US * Engineering Manager, Security Infrastructure MENLO PARK, US See All Jobs Available Positions --------------------------------------------------------------------- * Data Engineering Manager - Enterprise Finance Products MENLO PARK, US * Data Engineering Manager - Enterprise Finance Products SEATTLE, US * Data Engineering Manager - Enterprise Finance Products NEW YORK, US * Data Engineering Manager - Enterprise Finance Products REMOTE, US * Engineering Manager, Security Infrastructure MENLO PARK, US See All Jobs Stay Connected * footer-fb-engineering Engineering at Meta Like * footer-fbopensource Meta Open Source Follow * footer-research Meta Research Like * footer-developers Meta for Developers Like * footer-rss RSS Subscribe Open Source Meta believes in building community through open source technology. Explore our latest projects in Artificial Intelligence, Data Infrastructure, Development Tools, Front End, Languages, Platforms, Security, Virtual Reality, and more. * android ANDROID * ios iOS * web WEB * backend BACKEND * hardware HARDWARE Learn More Meta Engineering at Meta is a technical news resource for engineers interested in how we solve large-scale technical challenges at Meta. * Home * Company Info * Careers (c) 2022 Meta * Terms * Privacy * Cookies * Help To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy I Agree