[HN Gopher] Pyre: A performant type-checker for Python 3
___________________________________________________________________
Pyre: A performant type-checker for Python 3
Author : tosh
Score : 99 points
Date : 2021-05-10 16:14 UTC (6 hours ago)
(HTM) web link (pyre-check.org)
(TXT) w3m dump (pyre-check.org)
| azhenley wrote:
| Big discussion with over 500 comments from 3 years ago:
| https://news.ycombinator.com/item?id=17048446
| flakiness wrote:
| So three years passed and static checkers have gotten even more
| traction. What's the best option for Python so far for small
| projects today?
|
| I tried mypy [1] before but it was a bit cumbersome to keep it
| checking my code. At work I use pytype [2]. It's good enough
| but that's only because someone else made a build system
| integration for me. Pyre's scan-all-the-files approach seems
| easier to get started. Is there any catch? Which one are you
| using and what's your impression? [1]
| https://mypy.readthedocs.io/en/stable/ [2]
| https://github.com/google/pytype
| 6gvONxR4sf7o wrote:
| Whichever option you take, stick it into pre-commit. A good
| pre-commit-config.yaml is quality-of-life changing.
| jchonphoenix wrote:
| Mypy and Pyre appear to be the only two that work really well
| and can solve type-checking issues at scale.
| hiq wrote:
| As someone who hasn't started using types in Python seriously
| yet, it's a bit discouraging to see that there are already so
| many competing type checkers with different approaches and no
| clear winner. I understand that the problem is a bit harder
| than `{go, cargo} fmt`, but I feel we'd be better off with
| one type checker to rule them all.
|
| Or does it mean that we should check types using all type-
| checkers available to have the most compliant code possible?
| ben509 wrote:
| No, multiple checkers will drive you crazy because one type
| checker will support one feature while another will report
| errors for it.
| Mehdi2277 wrote:
| In practice they are quite similar in errors reported.
| There are differences that pop up but most of the time
| either it is bug to report or one has implemented a certain
| pep faster than another. A few rare things are extra type
| checking support entirely like pyright supports recursive
| types but mypy does not (or pyre has best tensor related
| support but those are future peps). Mypy's biggest unique
| thing is plugin support and some very dynamic libraries use
| plugins for better coverage. The differences should become
| peps if important enough and then eventually be supported
| by all 4.
|
| I think for most people any of the 4 big ones is fine. Most
| of the common type errors will look identical regardless of
| which you pick. Configuring multiple in CI is pretty
| straightforward if you want to go full. My current codebase
| has mypy + pyright configured. Pyright was a mix of fast
| response time on github issues + I use vscode and was the
| one that added type checks to CI.
|
| If you want a deep dive pycon is this week I think and
| there's a day for type checking related talks including one
| talk comparing the 4. I think type checking day is
| thursday.
| flakiness wrote:
| Thanks for the tip! Posting link here for the reference:
| https://us.pycon.org/2021/summits/typing/
|
| Hope they post the recordings somewhere later.
| pl-94 wrote:
| I personally use neovim with coc-pyright plugin. It
| integrates remarkably well mypy. It has saved me so many
| hours of development, I just can't dev without it nowadays.
| tomwojcik wrote:
| I'd advise to give mypy yet another try but disable in config
| file error codes that are painful and # noqa or # type:
| ignore lines that can't be understood well by mypy.
| nickm12 wrote:
| I'm not sure what you mean by "cannot be understood well".
| Either the type annotation is correct or not. If there is a
| genuine bug in mypy, the recommended convention is to use
| "type: ignore" (ideally with a reference to a bug tracking
| id). If you are genuinely subverting the typesystem, then
| use a cast(). If the code is too complex to type correctly
| or the type cannot be expressed in the typesystem (e.g.
| recursive types) then use type Any.
| zem wrote:
| have you had any problems using the open source version of
| pytype? we've tried to make it easy to just drop in to your
| setup.cfg and automatically analyse an entire project; if
| there are things that don't work we'd love bug reports for
| them!
| flakiness wrote:
| No. I didn't expect it to be non-blaze/bazel tool friendly,
| but sounds like it is (aims to be at lest)?
|
| It does seems to support the entire-directory parsing as
| well. That's nice! Let me give it a try next time. Thanks
| for the tip!
|
| As a reminder to myself, here is the link to the doc:
| https://google.github.io/pytype/
| MapleWalnut wrote:
| I've used mypy and pyright. Mypy generally "just works" for
| all the Python idioms I use, while pyright is less reliable.
|
| Pyright's VSCode integration, Pylance, provides a great auto
| complete experience. So I use Pyright for autocomplete in
| VSCode and mypy for type checking.
| davepeck wrote:
| If you haven't used pyright lately for checking itself, I'd
| recommend giving it a shot. These days, I find it's far
| faster and more reliable than mypy for the projects I work
| on. I'm curious if others have had similar experiences.
|
| (The one downside is that pyright doesn't support mypy
| plugins; the mypy Django plugin is theoretically handy for
| capturing some of Django's dynamism, particular in the ORM.
| "Theoretically" because I find it's quite buggy in
| practice, causing mypy to generate spurious error
| messages.)
| throwaway894345 wrote:
| Last I checked, mypy struggled to represent recursive
| types. Think `JSON = Union[None, bool, str, float, int,
| List['JSON'], Dict[str, 'JSON']`. That's a bummer because
| recursive data types are darn handy.
|
| Worse, importing third party packages often fails silently
| and when you can get error messages they tend to be
| completely inactionable (I recall one error message which
| linked to a web page that had lots of details and
| workarounds for fixing _other_ problems, but none of which
| solved the error itself).
|
| Figuring out how to distribute my own type annotations was
| similarly painful. IIRC, you have to drop a specially-named
| file into a particular directory and this is all
| undocumented save for a dense PEP and the error messages
| are unsurprisingly terrible. All of these things are
| actionable, but the progress seems slow (these have been
| among my top grievances since the project debuted, so the
| maintainers and I have different priorities, clearly).
|
| The problems for which I'm less optimistic tend to revolve
| around shoehorning typing into existing Python syntax--
| e.g., to get a callback that takes kwargs you have to
| define a protocol with a `__call__` method that takes
| kwargs because you can't express it with `typing.Callable`.
| Similarly where a language with first-class support for
| types might have `type Foo<T>`, Python makes you write `T =
| TypeVar("T"); class Foo(Generic[T])` or something like
| that, and it gets more confusing when you only want one of
| the methods to be generic and I can never remember whether
| that `T` takes on a single type across all uses or which
| scope I need to define it in, etc. This is largely an
| ergonomic nightmare and I don't have lots of optimism for
| this stuff to improve unless the Python community really
| comes to embrace typing as the default way to use Python
| (but I suspect most people who care a lot about this kind
| of stuff will leave for Go or other languages where these
| things just work out of the box).
| codethief wrote:
| > Similarly where a language with first-class support for
| types might have `type Foo<T>`, Python makes you write `T
| = TypeVar("T"); class Foo(Generic[T])` or something like
| that, and it gets more confusing when you only want one
| of the methods to be generic and I can never remember
| whether that `T` takes on a single type across all uses
| or which scope I need to define it in, etc.
|
| I don't know about you but I find PEP 484 very clear
| here: https://www.python.org/dev/peps/pep-0484/#scoping-
| rules-for-...
| throwaway894345 wrote:
| Fair enough. Maybe I didn't come across that at the time
| or perhaps it's been since revised. Even still, it's a
| poor substitution for the more familiar / intuitive
| `class Foo[T]:` / `def foo[T](...)` style.
| codethief wrote:
| Hmm it might take some getting used to but I've actually
| found it quite straight-forward to comprehend and reason
| about.
|
| The nice thing is that, since TypeVars are ordinary
| variables, you can re-use them / import them in multiple
| modules and avoid a lot of boilerplate code.
| throwaway894345 wrote:
| > The nice thing is that, since TypeVars are ordinary
| variables, you can re-use them / import them in multiple
| modules and avoid a lot of boilerplate code.
|
| Maybe I'm missing something, but the boilerplate only
| exists _because_ Python makes you define them as ordinary
| variables in the first place. So while you can have a
| `foo.py` file like this: T =
| TypeVar("T")
|
| And a `bar.py` file like this: from foo
| import T class Bar(Generic[T]):
| ...
|
| So are you saying that this is less boilerplate than a
| `bar.py` that just defines its own `T` TypeVar? Because
| that seems like a pretty comparable amount of
| boilerplate. Or are you saying that it's less boilerplate
| than languages that don't treat TypeVars as ordinary
| variables, e.g., Rust? Because in Rust our `bar.rs` file
| would look like this: `struct Bar<T> {...}` (and the
| scoping rules are patently obvious, to boot).
| codethief wrote:
| No, in this particular example, there's no real
| advantage. What I meant were situations where the TypeVar
| gets a little more complicated, compare the additional
| parameters on https://docs.python.org/3/library/typing.ht
| ml#typing.TypeVar . Then importing TypeVars starts to pay
| off because you only have to define your bounds etc.
| once.
| throwaway894345 wrote:
| Ah, that makes sense. Thanks for clarifying.
| staticassertion wrote:
| The PEP is clear, it's just extremely unintuitive that,
| within the scope of declaration, a type T can be many
| different types.
| codethief wrote:
| > The problems for which I'm less optimistic tend to
| revolve around shoehorning typing into existing Python
| syntax--e.g., to get a callback that takes kwargs you
| have to define a protocol with a `__call__` method that
| takes kwargs because you can't express it with
| `typing.Callable`
|
| You might be interested in the discussion over here then
| -> https://github.com/python/typing/issues/769#issuecomme
| nt-741...
| staticassertion wrote:
| I have yet to successfully build a project with the
| strict mypy settings, without resorting to either turning
| some off or having to use an explicit Any. And a lot of
| the non-default settings are absolutely critical if you
| care about correctness.
|
| Lack of recursive types has been a major deal breaker for
| me. If you have a class Foo that can construct a class
| Bar, and a class Bar that can construct a class Foo, you
| can't express that in mypy without Any-Generics.
|
| To me, at this point, mypy is barely a linter, it's more
| of a "this helps my IDE autocomplete faster", and
| definitely not a type system.
| Groxx wrote:
| This pretty much mirrors my mypy experience too. The only
| project I've ever successfully used it on was a single-
| file CLI tool that only used the stdlib.
|
| And by "successfully" I mean "it ran at all". Every other
| thing I've thrown at it has caused it to crash. I quite
| like the concept, but implementation has been rather
| abysmal as far as I've seen. Has it improved in ~ the
| past year?
| staticassertion wrote:
| Not really, no.
| connorbrinton wrote:
| I also struggle with Mypy's strict mode, but I think
| recursive types are different from classes that can
| construct each other. Mypy doesn't have any problem with
| the following code from __future__
| import annotations class X:
| def __init__(self, value: int) -> None:
| self.value = value def to_y(self) -> Y:
| return Y(self.value) class Y:
| def __init__(self, value: int) -> None:
| self.value = value def to_x(self) -> X:
| return X(self.value) x =
| X(10).to_y().to_x() print(x.value)
| staticassertion wrote:
| Sorry, yes, that works. I should have been a little more
| explicit. This breaks down when Y is generic over X and X
| is generic over Y. IDK, I've had tons of problems with
| mutually recursive TypeVars, though my example was _not_
| of that - sorry.
| tusharsadhwani wrote:
| This is all true. So true infact that I wrote a huge
| article documenting this for people who are new to it:
| https://tshr.me/mypy
|
| I'm planning to migrate this to my own blog soon.
| mplanchard wrote:
| Seconding this setup for both VSCode and emacs
| dunefox wrote:
| As with package managers, there seem to be half a dozen
| different, competing type checkers for Python now...
| short_sells_poo wrote:
| It's starting to look like the huge embarrassment that is
| python packaging and environment management. I work with the
| language every day, and I still have no idea what am I supposed
| to use between conda, pip, setuptools/setup.py, pyproject,
| meta.yml, poetry, pipenv, etc...
|
| Conda is so slow that I sometimes wonder if we are being
| trolled by some cruel God of programming. Pip is faster, but
| version resolution is iffy.
|
| Hell, even just assigning versions to python packages is
| nothing short of ridiculous. Do you use version.txt in the root
| folder and set it manually? Do you have it set from SCM? Which
| of the half a dozen packages do you use to have it set from
| SCM? setuptools_scm? Versioneer?
|
| There are a set of tools in the python ecosystem that have
| basically no equal in any other language and these tools and
| the surrounding mindshare make python irreplaceable in the near
| term. The language itself is easy to learn and powerful enough
| to be able to do data analysis with ease. Good python code is
| easy on the eyes, which I personally consider an important
| aspect.
|
| Outside of these tools, core parts of the ecosystem are
| basically an XKCD joke.
|
| I tried switching to Julia as I find both the language and the
| ecosystem are vastly superior in their foundations.
| Unfortunately the maturity is not there yet, and neither is the
| mindshare. If I had to bet my career on adopting the language
| in a business setting, I'd not be prepared to do so. Which is a
| shame, because the situation turns into a Catch-22.
| lacker wrote:
| I feel like the problem is that pip is clearly the most
| popular, but philosophically the pip developers do not want
| to build a solution to all packaging and environment
| management into pip.
|
| If I were pip dictator, I would try to make pip the one tool
| to handle all python packaging and environment management. In
| particular, that means pip would handle the management of
| different Python versions, different Python environments,
| native dependencies, running tests, making builds perfectly
| reproducible, releasing new versions of libraries, and
| creating new projects.
|
| "Creating new projects" seems like it is not a big deal, but
| in practice I think that if there were simply a "pip new"
| command that set up a new project using the best practices
| advocated by the pip team, it would go a long way toward
| standardizing the ecosystem here.
| alexgrover wrote:
| Starting to? The Python ecosystem has been pretty behind
| other languages for more like 10 years now. I'm still forced
| to use it for work but it's becoming more and more likely
| that "no python" will be a hard requirement for my next job
| search...
|
| What tools do you find irreplaceable?
| Demiurge wrote:
| There are actually clear answers to all your questions.
|
| All the tools you listed do different things, except maybe
| poetry and pipenv, so you can pick whichever one you like,
| you're not _supposed_ to do anything. You can have choice,
| illusion of free will, etc...
|
| As to module version, there is a standard on how to define it
| in __version__:
| https://www.python.org/dev/peps/pep-0008/#module-level-
| dunde...
| whimsicalism wrote:
| > There are actually clear answers to all your questions.
|
| Strongly disagree. Conda, pipenv, pyenv, venv, poetry are
| all trying to solve the same problem (although conda tries
| to solve some other problems too).
|
| Choice is not always good, this is why we have standards.
|
| I would recommend pyenv. I understand why people are
| attracted to poetry, but pyenv arguably offers all of the
| same benefits that poetry has as well, and has better
| adoption.
| Demiurge wrote:
| ... is there some problem with having multiple choices of
| libraries?
| whimsicalism wrote:
| I think it'd be nice if they settled on a standard as to what
| is a correctly typed program.
|
| Mypy and Pyre are pretty similar, but Pytype has very
| different standards.
| Demiurge wrote:
| If type analysis tools show different issues, fixing one
| should not break the other, unless the other is wrong? They
| both are trying to model the same language/runtime.
| fouronnes3 wrote:
| Python type checking has entered a serious xkcd 927 phase that
| will get worse before it gets better.
| short_sells_poo wrote:
| I'd posit you are being optimistic there. Just look at the
| python packaging story. It had 20 years to get better and
| it's still only getting worse.
| IshKebab wrote:
| Yeah and even worse they disagree about type errors!
|
| We really need a modern Python alternative. I don't know of any
| that don't give up the REPL / single file script features which
| are pretty huge advantages of Python to be honest.
| iudqnolq wrote:
| It's radically different, but the language where I've had the
| most pleasant repl-driven experience is Elixir. Because of
| how it's error handling works it's even sane to debug with a
| repl into live production in some cases.
| Sohcahtoa82 wrote:
| Node.js has a REPL, and there's nothing stopping you from
| putting your entire project into a single file.
| pjmlp wrote:
| All ML derived languages, Lisp based languages have REPL, you
| can put everything on one file if you feel like it, and they
| compile to native code.
|
| Then there is Julia as well.
| onerandompotato wrote:
| Wouldn't Julia be that language? Its fast, supports static
| typing, and is built from the ground up for machine learning/
| data science. It also has a built in REPL.
| j1elo wrote:
| Unrelated funny little fact for lovers of scifi novels: Pyre
| (PyrE) was the name of an extremely rare element and main plot
| device for the novel _Tiger! Tiger!_ (aka. " _The Stars My
| Destination_ "), by Alfred Bester:
|
| https://www.amazon.com/dp/B01MRJVUPC/
|
| Maybe not the best science fiction book of all times, but it left
| a mark on my memory... and curious things like this name have
| stayed there since when I was a kid avidly reading everything
| that fell on his hands :-)
|
| -- EDIT: Link to the book
| jmugan wrote:
| PyCharm works well for me. Probably not as fancy as all this
| other stuff, but the type hints make the code easier to
| understand, and they help PyCharm help me by enabling it to find
| more errors and by enabling enhanced autocompletion.
| nerdponx wrote:
| PyCharm having type checking built in is excellent, but it is
| definitely the lowest quality out of the major 3 (Mypy,
| Pyright, Pyre) with respect to correctness.
| vorticalbox wrote:
| I've always found its type checking to be perfectly fine, but
| hearing its not the best I might look into the others.
| nerdponx wrote:
| It tends to be overly-lenient about certain return types
| and tends to miss some more advanced type inferences.
|
| It's a great tool when editing code. But you still should
| run Mypy or another checker for more-correct analysis.
| kbumsik wrote:
| What type checker does PyCharm use? In-house?
| whimsicalism wrote:
| Yes, in house
| a3w wrote:
| A Tale of two types systems -> now: a Tale of three types
| systems? To make it a book title, The three type systems problem?
| Demiurge wrote:
| How are three type systems? The type annotations are part of
| the Python 3. You can use multiple libraries to analyze the
| types. There are also multiple static type analysis tools for
| C, and for other languages that have type definitions as part
| of the language. How is this an issue, what am I missing
| something?
| erjiang wrote:
| When you write types in Python, the interpretation of your
| type annotations are up to the type checker. Your annotations
| that you write for e.g. Pyre may not type check in mypy. The
| end result is that your type annotations are not "Python"
| type annotations, but "Pyre"/"mypy"/etc. type annotations.
|
| In C, you ultimately care about what the compiler says. And
| this has also led to dialect-specific C code that works fine
| in one compiler but doesn't compile or runs incorrectly in
| another compiler.
| Mehdi2277 wrote:
| Not that much as they all follow pep 484 + a few other
| peps. The number of type features unique to each one are
| generally bugs/recent peps they haven't implemented. The
| biggest one is mypy plugin ecosystem although the hope
| there is to either do more peps that plugins become no
| longer needed and the type system is strong enough to cover
| more things or make a plugin pep.
|
| If you are using the most recent pep features possible than
| yeah you might have an issue. That's similar to clang/gcc
| both taking time to implement new c++ standards and not
| being compatible there. If you only use pep 484 which
| covers most basics well then you should be good for any
| checker.
| endgame wrote:
| $ python -c 'import this' | sed -n 15p There should
| be one-- and preferably only one --obvious way to do it.
|
| Sadly, the python type-checking situation appears to be
| going the way of the python packaging situation.
| skrtskrt wrote:
| Been using coc-pyright with coc.nvim and I am a fan.
|
| Issues seem to be from libraries with crappy, incomplete, or
| incorrect typing/stubs, not from the checker itself.
|
| Every day I tire more and more of using Python for larger
| projects for these reasons
| brobinson wrote:
| We unfortunately had to switch away from this as it took too long
| for pyre to support Python 3.8 after it was released. I've been
| using mypy since then, but I had no complaints about pyre when I
| used it.
| globular-toast wrote:
| Anyone considering using type hints should listen to this talk:
| https://www.infoq.com/presentations/dynamic-static-typing/
|
| I personally think it's mostly a waste of time to do type hinting
| for the purpose of catching errors in a strongly typed language
| like Python. Type errors just aren't practically a problem in
| dynamic languages. Doing types for performance is a great reason
| to do it, though.
| mumblemumble wrote:
| I agree with you that the benefits of static typing are
| sometimes greatly exaggerated. But I also think that using type
| hints and type checkers in Python isn't quite the same thing as
| static typing.
|
| For starters, type checking can't actually guarantee you won't
| have type errors at run time. Because, unlike in a statically
| typed language, in Python, anyone can always choose to just not
| use type hinting. Whenever that happens, as far as the type
| checker is concerned, anything goes.
|
| But type hints are very useful as hints. They help with editor
| tooling, which can make it easier to navigate an unfamiliar
| codebase. They provide extra information that makes the code
| easier to read. And they give me an opt-in form of type linting
| that allows me to set up regions of code where I don't have to
| take quite so much personal responsibility for ensuring that
| arguments are compatible with parameters. In short, it's not a
| correctness prover; it's an energy saver.
| flowerlad wrote:
| > _I agree with you that the benefits of static typing are
| sometimes greatly exaggerated._
|
| For large, multi-developer projects, it is impossible to
| exaggerate the benefits of static typing.
| nickm12 wrote:
| I've been on teams at multiple companies that have used type
| annotations for Python. Developers have generally loved
| adopting type annotations because it (1) makes the code easier
| to understand (2) makes the code easier to refactor (3)
| improves integration with IDEs and (4) catches real bugs.
|
| I disagree that type errors "aren't practically a problem".
| NoneType errors and AttributeErrors turn up all the time in
| Python.
| globular-toast wrote:
| Clearly didn't watch the talk then. He found that about 1-2%
| of errors in real world projects were type-related errors.
| bobbylarrybobby wrote:
| It's not just about type errors. (Good) autocomplete and
| documentation are only available when types are either
| explicitly labeled or correctly inferred. It's important that I
| be able to 1. type in just the first few letters of a method
| name and press <tab> to complete it, and 2. hover over a
| method's name and get docs for the method. Both of these are
| only reliable if the type of the caller is known.
| globular-toast wrote:
| But we're commenting on a type checker which implies this is
| being done to catch errors.
| mvzvm wrote:
| It's a problem the moment it has to interact with the outside
| world, especially DBs/serialization/... really any kind of I/O.
| globular-toast wrote:
| So Python can't do IO? Django doesn't exist? What are you
| talking about exactly?
___________________________________________________________________
(page generated 2021-05-10 23:01 UTC)