https://engineering.fb.com/2023/08/07/developer-tools/fixit-2-linter-meta/ Skip to content Engineering at Meta Search this site [ ] # * Open Source + Open Source + Meta Open Source * Platforms + Android + iOS + Web * Infrastructure Systems + Core Infra + Data Infrastructure + DevInfra + Production Engineering + Security + Research Publications * Physical Infrastructure + Connectivity + Data Center Engineering + Networking & Traffic + Research Publications * Video Engineering & AR/VR + Video Engineering + Virtual Reality + Research Publications * Artificial Intelligence + ML Applications + AI Research + Research Publications * Watch Videos [ ] POSTED ON AUGUST 7, 2023 TO DevInfra, Open Source, Production Engineering Fixit 2: Meta's next-generation auto-fixing linter [Eng-Blog-Self-Serve-Hero-Images-DEBUGGING-203-Teale-1] By Amethyst Reese * Fixit is dead! Long live Fixit 2 - the latest version of our open-source auto-fixing linter. * Fixit 2 allows developers to efficiently build custom lint rules and perform auto-fixes for their codebases. * Fixit 2 is available today on PyPI. Python is one of the most popular languages in use at Meta. Meta's production engineers (PEs) are specialized software engineers (SWEs) who focus on reliability, efficiency, and scalability. They work on various projects, including debugging production services, rewriting inefficient libraries, orchestrating project deployments at scale, or capacity planning and scheduling. And Python is often one of the first tools that PEs reach for, as it offers rapid development, easy to read syntax, and a massive array of open source libraries. Meta's Python Language Foundation team -- a hybrid team of both PEs and traditional SWEs -- helps own and maintain the infrastructure and tooling for Python at Meta. The team supports engineers, data scientists, researchers, and anyone else at Meta using Python to get their work done. One of the ways we accomplish this is building tools that enable Python developers to write better, and more reliable code more efficiently. This includes tools like automatic formatting and import sorting that eliminate tedium, or linters that guide engineers toward maintainable code with fewer bugs. This year, we have been building a new linter, Fixit 2, designed from the ground up to make developers more efficient and capable, both in open source projects and the diverse landscape of our internal monorepo. At Meta, we are using Fixit 2 with a few early adopters, and plan to roll it out to the rest of our monorepo soon. But any developer can use it to perform auto-fixing more efficiently and make faster improvements to their own codebases. Why a new linter? (why not X?) There are a variety of excellent linters in the Python ecosystem, many of which have a large community of third-party plugins providing a diverse array of lint rules. We have used Flake8 internally at Meta since 2016, and it has been very successful in helping developers reduce bugs and keep a clean codebase. The popular flake8-bugbear plugin was even created by Lukasz Langa (author of Black, PSF developer-in-residence, and release manager for Python 3.8 and 3.9) while working at Meta (then Facebook), as a home for more opinionated lint rules that we could both use internally and share with the rest of the Python developer community. We also have a large number of internal plugins built by various teams, and Flake8 allows them to write and enable custom lint rules directly in the codebase without getting sign-off from a central gatekeeper, and without waiting for a new deployment of Flake8 to roll out. But while Flake8 has long been a cornerstone of our linting solution, it also has some rough edges. Writing new lint rules requires building entire plugins (each claiming a portion of the "namespace" for error codes) and encourages developers to build complicated plugins covering multiple classes of errors. When those lint errors are found, Flake8 can only point to a line and column number where it occurred, but has no way of suggesting changes to the developer looking at a list of lint results, leaving them in a state of trial and error to find changes that make the linter happy. Also, Flake8 uses the stdlib ast module, making it unable to parse future syntax features and forcing developers to wait for tools to upgrade before they can use the shiny new hotness. There are alternatives to Flake8 of course, but many of them suffer from one or more drawbacks: * A lack of support for "local" in-repo plugins or custom lint rules. * Limited or no support for hierarchical configuration for different projects within a monorepo. * No option for auto-fixes when errors are found. * Slow performance on large codebases. While some of those features aren't critical, the most important for developer efficiency is offering auto-fixes - automatic suggested changes that would satisfy the lint rule. This takes the guesswork out of using a linter, and allows users to quickly review and accept those changes when possible, eliminating the need to re-run the linter until the code is finally clean. Combining these auto-fixes with in-repo, custom lint rules provides a level of tailored code quality improvements that is hard to beat. Unfortunately, even Fixit, the auto-fixing linter that we built for Instagram and open sourced, did not support local lint rules or hierarchical configuration - core requirements for our monorepo that is home to thousands of projects, many of which are themselves open source projects with their own distinct needs for linting and CI. We received many requests from developers to support Fixit in our monorepo, but there were enough hurdles that we were only able to support a small set of security lint rules, reducing the direct benefits to our Python codebase. Meet Fixit 2 After discussions with other teams, especially in the rapidly growing AI/ML space, we considered our options and decided upon a partial rewrite of Fixit. We intentionally designed the new version with an open source-first mindset, while incorporating the needs and requirements of our own monorepos and open source projects from day one. The framework and linting engine would be rebuilt from the ground up while leaving the core design of lint rules largely untouched. The new system provides a hierarchical configuration based on the TOML format; support for local, in-repo lint rules similar to Flake8; and a much improved CLI and API for integration with other tools and automation. Fixit itself builds on top of another Instagram open source project, LibCST, a concrete syntax tree for Python with a tree and node API following the patterns of the ast module in the standard library. The "concrete" part of CST means that LibCST includes every part of the source file in the resulting tree after parsing, including whitespace, comments, and formatting elements that are ignored by the ast module. This is what allows Fixit (and other tools we built, like usort) to safely modify source files, without using regular expressions or the risk of producing broken syntax, and provides the foundation for Fixit to offer auto-fixes suggested by the lint rules themselves. Writing a new lint rule can be done with less than a dozen lines of code, and test cases are defined inline. You can even place it right next to the code that it will be linting: # teambread/rules/hollywood.py import fixit import libcst class HollywoodName(fixit.LintRule): VALID = [...] # no lint errors here INVALID = [...] # bad code samples here def visit_SimpleString(self, node: libcst.SimpleString): if node.value in ('"Paul"', "'Paul'"): self.report(node, "It's underbaked!") Suggesting auto-fixes for the user is as easy as including a new CST node when reporting an error: def visit_SimpleString(self, node: libcst.SimpleString): if node.value in ('"Paul"', "'Paul'"): new_node = libcst.SimpleString('"Mary"') self.report(node, new_node) Enabling this new rule within the project's codebase can be done with a simple config change: # teambread/sourdough/fixit.toml [tool.fixit] enable = [".rules.hollywood"] Now we can run our linter against our project: # teambread/sourdough/baker.py name = "Paul" print(f"hello {name}!") $ fixit lint --diff sourdough/baker.py sourdough/baker.py@7:11 HollywoodName: It's underbaked! (has autofix) --- a/baker.py +++ b/baker.py @@ -6,3 +6,3 @@ def main(): - name = "Paul" + name = "Mary" print(f"hello {name}") [?] 1 file checked, 1 file with errors, 1 auto-fix available [?] [1] The `lint` command only shows errors and suggested changes. If we use the `fix` command, we can apply those suggested changes back to the codebase: $ fixit fix --automatic sourdough/baker.py sourdough/baker.py@7:11 HollywoodName: It's underbaked! (has autofix) [?] 1 file checked, 1 file with errors, 1 auto-fix available, 1 fix applied [?] Now that our auto-fixes have been applied, we can confirm that the project is now clean and lint-free: $ fixit lint sourdough/baker.py 1 file clean When running Fixit 2 with auto-fixing lint rules, any code that triggers the lint rule is an opportunity to get an automatic replacement, improving the codebase with less effort from the developer. Applied more broadly, Fixit 2 can even be used as a tool to enact sweeping codemods against a large codebase, while leaving a lint rule in place to handle any matching code in the future. Try Fixit 2 Fixit 2 is available today on PyPI. You can install and test Fixit 2 with pip install fixit. We have a roadmap with plans for future improvements and features, and a rich set of documentation and user guides to help you get started with Fixit 2 in your own projects or repositories. We hope it proves useful in your projects, and we look forward to hearing your feedback! Share this: * Click to share on Facebook (Opens in new window) * Click to share on X (Opens in new window) * Click to share on LinkedIn (Opens in new window) * Click to share on Hacker News (Opens in new window) * Click to email a link to a friend (Opens in new window) * TAGS: Instagram Read More in DevInfra View All [5-Things-about-Buck2] OCT 23, 2023 5 Things you didn't know about Buck2 [Eng-Blog-Self-Serve-Hero-Images-ANALYTICS-402-Orange] OCT 5, 2023 Meta contributes new features to Python 3.12 [GettyImages-1388687270] SEP 7, 2023 Arcadia: An end-to-end AI system performance simulator Meta Tech Podcast: What's it like to code at Meta? SEP 5, 2023 What is it like to write code at Meta? [Scheduling-Jupyter-Notesbooks-HERO] AUG 29, 2023 Scheduling Jupyter Notebooks at Meta [Eng-Blog-Self-Serve-Hero-Images-DEBUGGING-203-Orange] AUG 15, 2023 Introducing Immortal Objects for Python Related Posts --------------------------------------------------------------------- [Dev-Workflow-1848x1200-Full] Jun 27, 2023 Meta developer tools: Working at scale [Buck2-Hero] Apr 06, 2023 Build faster with Buck2: Our open source build system [Self-Serve-Hero] Oct 31, 2022 Improving Instagram notification management with machine learning and causal inference Related Positions --------------------------------------------------------------------- * Software Engineer, Infrastructure LONDON, UK * Software Engineer, Machine Learning - Monetization SUNNYVALE, US * Software Engineer, Machine Learning - Monetization BELLEVUE, US * Software Engineer, Machine Learning - Monetization MENLO PARK, US * Software Engineer, Machine Learning - Monetization SEATTLE, US See All Jobs Available Positions --------------------------------------------------------------------- * Software Engineer, Infrastructure LONDON, UK * Software Engineer, Machine Learning - Monetization SUNNYVALE, US * Software Engineer, Machine Learning - Monetization BELLEVUE, US * Software Engineer, Machine Learning - Monetization MENLO PARK, US * Software Engineer, Machine Learning - Monetization SEATTLE, US See All Jobs Stay Connected * footer-fb-engineering Engineering at Meta Like * footer-fbopensource Meta Open Source Follow * footer-research Meta Research Like * footer-developers Meta for Developers Like * footer-rss RSS Subscribe Open Source Meta believes in building community through open source technology. Explore our latest projects in Artificial Intelligence, Data Infrastructure, Development Tools, Front End, Languages, Platforms, Security, Virtual Reality, and more. * android ANDROID * ios iOS * web WEB * backend BACKEND * hardware HARDWARE Learn More Meta Engineering at Meta is a technical news resource for engineers interested in how we solve large-scale technical challenges at Meta. * Home * Company Info * Careers (c) 2023 Meta * Terms * Privacy * Cookies * Help To help personalize content, tailor and measure ads and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookie Policy Accept *