Fixit 2: Meta’s next-generation auto-fixing linter

  • Fixit is useless! Lengthy dwell Fixit 2 – the newest model of our open-source auto-fixing linter.
  • Fixit 2 permits builders to effectively construct customized lint guidelines and carry out auto-fixes for his or her codebases.
  • Fixit 2 is accessible at the moment on PyPI.

Python is likely one of the hottest languages in use at Meta. Meta’s manufacturing engineers (PEs) are specialised software program engineers (SWEs) who deal with reliability, effectivity, and scalability. They work on numerous tasks, together with debugging manufacturing companies, rewriting inefficient libraries, orchestrating challenge deployments at scale, or capability planning and scheduling. And Python is commonly one of many first instruments that PEs attain for, because it affords speedy improvement, straightforward to learn syntax, and an enormous array of open supply libraries.

Meta’s Python Language Basis workforce — a hybrid workforce of each PEs and conventional SWEs — helps personal and keep the infrastructure and tooling for Python at Meta. The workforce helps engineers, information scientists, researchers, and anybody else at Meta utilizing Python to get their work carried out.

One of many methods we accomplish that is constructing instruments that allow Python builders to jot down higher, and extra dependable code extra effectively. This consists of instruments like automatic formatting and import sorting that remove tedium, or linters that information engineers towards maintainable code with fewer bugs.

This 12 months, we’ve got been constructing a brand new linter, Fixit 2, designed from the bottom as much as make builders extra environment friendly and succesful, each in open supply tasks and the various panorama of our inside monorepo. At Meta, we’re utilizing Fixit 2 with a couple of early adopters, and plan to roll it out to the remainder of our monorepo quickly. However any developer can use it to carry out auto-fixing extra effectively and make sooner enhancements to their very own codebases.

Why a brand new linter? (why not X?)

There are a number of wonderful linters within the Python ecosystem, lots of which have a big neighborhood of third-party plugins offering a various array of lint guidelines. We have now used Flake8 internally at Meta since 2016, and it has been very profitable in serving to builders scale back bugs and hold a clear codebase. The favored flake8-bugbear plugin was even created by Łukasz Langa (writer of Black, PSF developer-in-residence, and launch supervisor for Python 3.8 and three.9) whereas working at Meta (then Fb), as a house for extra opinionated lint guidelines that we might each use internally and share with the remainder of the Python developer neighborhood. 

We even have a lot of inside plugins constructed by numerous groups, and Flake8 permits them to jot down and allow customized lint guidelines immediately within the codebase with out getting sign-off from a central gatekeeper, and with out ready for a brand new deployment of Flake8 to roll out.

However whereas Flake8 has lengthy been a cornerstone of our linting answer, it additionally has some tough edges. Writing new lint guidelines requires constructing total plugins (every claiming a portion of the “namespace” for error codes) and encourages builders to construct sophisticated plugins protecting a number of courses of errors. When these lint errors are discovered, Flake8 can solely level to a line and column quantity the place it occurred, however has no means of suggesting modifications to the developer an inventory of lint outcomes, leaving them in a state of trial and error to search out modifications that make the linter completely satisfied. Additionally, Flake8 makes use of the stdlib ast module, making it unable to parse future syntax options and forcing builders to attend for instruments to improve earlier than they’ll use the shiny new hotness.

There are alternate options to Flake8 after all, however lots of them endure from a number of drawbacks: 

  • A scarcity of help for “native” in-repo plugins or customized lint guidelines. 
  • Restricted or no help for hierarchical configuration for various tasks inside a monorepo.
  • No possibility for auto-fixes when errors are discovered.
  • Gradual efficiency on giant codebases.

Whereas a few of these options aren’t vital, crucial for developer effectivity is providing auto-fixes – automated advised modifications that might fulfill the lint rule. This takes the guesswork out of utilizing a linter, and permits customers to rapidly assessment and settle for these modifications when attainable, eliminating the necessity to re-run the linter till the code is lastly clear. Combining these auto-fixes with in-repo, customized lint guidelines gives a degree of tailor-made code high quality enhancements that’s exhausting to beat.

Sadly, even Fixit, the auto-fixing linter that we constructed for Instagram and open sourced, didn’t help native lint guidelines or hierarchical configuration – core necessities for our monorepo that’s dwelling to 1000’s of tasks, lots of that are themselves open supply tasks with their very own distinct wants for linting and CI. We obtained many requests from builders to help Fixit in our monorepo, however there have been sufficient hurdles that we have been solely in a position to help a small set of safety lint guidelines, lowering the direct advantages to our Python codebase. 

Meet Fixit 2

After discussions with different groups, particularly within the quickly rising AI/ML area, we thought of our choices and determined upon a partial rewrite of Fixit. We deliberately designed the brand new model with an open source-first mindset, whereas incorporating the wants and necessities of our personal monorepos and open supply tasks from day one.

The framework and linting engine can be rebuilt from the bottom up whereas leaving the core design of lint guidelines largely untouched. The brand new system gives a hierarchical configuration based mostly on the TOML format; help for native, in-repo lint guidelines just like Flake8; and a a lot improved CLI and API for integration with different instruments and automation. 

Fixit itself builds on high of one other Instagram open supply challenge, LibCST, a concrete syntax tree for Python with a tree and node API following the patterns of the ast module in the usual library. The “concrete” a part of CST signifies that LibCST consists of each a part of the supply file within the ensuing tree after parsing, together with whitespace, feedback, and formatting parts which might be ignored by the ast module. That is what permits Fixit (and different instruments we constructed, like µsort) to soundly modify supply information, with out utilizing common expressions or the danger of manufacturing damaged syntax, and gives the inspiration for Fixit to supply auto-fixes advised by the lint guidelines themselves.

Writing a brand new lint rule could be carried out with lower than a dozen strains of code, and check circumstances are outlined inline. You’ll be able to even place it proper subsequent to the code that it is going to be linting:

# teambread/guidelines/
import fixit
import libcst
class HollywoodName(fixit.LintRule):
    VALID = [...] # no lint errors right here
    INVALID = [...] # unhealthy code samples right here
    def visit_SimpleString(self, node: libcst.SimpleString):
        if node.worth in ('"Paul"', "'Paul'"):
  , "It is underbaked!")

Suggesting auto-fixes for the person is as straightforward as together with a brand new CST node when reporting an error:

def visit_SimpleString(self, node: libcst.SimpleString):
    if node.worth in ('"Paul"', "'Paul'"):
        new_node = libcst.SimpleString('"Mary"'), new_node)

Enabling this new rule inside the challenge’s codebase could be carried out with a easy config change:

# teambread/sourdough/fixit.toml
allow = [".rules.hollywood"]

Now we are able to run our linter towards our challenge:

# teambread/sourdough/
title = "Paul"
print(f"hiya title!")
$ fixit lint --diff sourdough/
sourdough/ HollywoodName: It is underbaked! (has autofix)
--- a/
+++ b/
@@ -6,3 +6,3 @@
def most important():
-    title = "Paul"
+    title = "Mary"
    print(f"hiya title")
🛠️  1 file checked, 1 file with errors, 1 auto-fix accessible 🛠️

The `lint` command solely reveals errors and advised modifications. If we use the `repair` command, we are able to apply these advised modifications again to the codebase:

$ fixit repair --automatic sourdough/
sourdough/ HollywoodName: It is underbaked! (has autofix)
🛠️  1 file checked, 1 file with errors, 1 auto-fix accessible, 1 repair utilized 🛠️

Now that our auto-fixes have been utilized, we are able to affirm that the challenge is now clear and lint-free:

$ fixit lint sourdough/
🧼 1 file clear 🧼

When working Fixit 2 with auto-fixing lint guidelines, any code that triggers the lint rule is a chance to get an automated alternative, bettering the codebase with much less effort from the developer. Utilized extra broadly, Fixit 2 may even be used as a instrument to enact sweeping codemods towards a big codebase, whereas leaving a lint rule in place to deal with any matching code sooner or later.

Attempt Fixit 2 

Fixit 2 is offered at the moment on PyPI. You’ll be able to set up and check Fixit 2 with pip set up fixit

We have now a roadmap with plans for future enhancements and options, and a wealthy set of documentation and user guides that will help you get began with Fixit 2 in your individual tasks or repositories. We hope it proves helpful in your tasks, and we look ahead to hearing your feedback!