[HN Gopher] Writing and linting Python at scale
___________________________________________________________________
Writing and linting Python at scale
Author : el_duderino
Score : 105 points
Date : 2023-11-22 13:22 UTC (9 hours ago)
(HTM) web link (engineering.fb.com)
(TXT) w3m dump (engineering.fb.com)
| ayhanfuat wrote:
| (It's a podcast, not an article)
| gostsamo wrote:
| The description links to the tech article.
|
| https://engineering.fb.com/2023/08/07/developer-tools/fixit-...
| jpc0 wrote:
| https://engineering.fb.com/2023/08/07/developer-
| tools/fixit-...
|
| Just a test, for some reason your link isn't clickable for
| me.
| chefandy wrote:
| Looks like they unintentionally (or intentionally for some
| reason I can't parse ATM?) triggered code formatting. If
| you have a line preceeded by two or more blank lines that
| is indented with two or more spaces then it'll <pre>
| formatting. Like This
| gostsamo wrote:
| unintentionally, sorry for that one.
| dang wrote:
| Changed to that from
| https://engineering.fb.com/2023/11/21/production-
| engineering.... Thanks!
|
| (I fixed the link in your comment btw. More at
| https://news.ycombinator.com/formatdoc)
| infocollector wrote:
| They are talking about this project :
| https://pypi.org/project/fixit/2.0.0a1/#description
|
| https://libcst.readthedocs.io/en/latest/ powers Fixit 2. I would
| love to hear if there are any production users here for this new
| linter, and how is that working out for them?
| lagt_t wrote:
| But but but HN told me that python is only good for small scale
| and prototypes, the engineers at meta are wrong
| DarkNova6 wrote:
| The engineers at meta worte a language + runtime on top of PHP
| to make it work "at scale".
|
| If Facebook wants to make something work they have enough
| resources to throw at the problem to solve it. Regardless of
| whether it makes sense or not.
| jerf wrote:
| I mean, as far as I'm concerned, everything Facebook has done
| has done nothing but reconfirm how unwise it is to build
| large-scale infrastructure on dynamic scripting languages.
| They have the resources to move heaven and earth to do what
| amounts to turning their dynamic scripting language back into
| static languages in everything but name, and they have the
| hole they've dug themselves into that justifies it. I have
| neither.
|
| I don't think that Facebook necessarily made a mistake at the
| time, though. Static languages have come a long way. I can
| prototype in them very nearly as quickly as I can in dynamic
| languages now, with the crossover point for where the static
| language is simply straight-up an advantage being roughly one
| to two weeks. Circa 2004 I would not even remotely make that
| claim. By the time static languages reached that state they
| were already in deep.
|
| But I would consider it a mistake for most cases to start
| right now, in 2023, with dynamic scripting languages as a
| base layer.
| crabbone wrote:
| > dynamic scripting languages.
|
| Why keep repeating this nonsense? "Dynamic" or "scripting"
| aren't features of languages. When anyone says something
| like this, it's like talking about square chicken... (i.e.
| a category error). Obviously, you had some idea in your
| mind, and you wanted to communicate it somehow, but your
| readers will not know what it was unless you make an effort
| to analyze what you want to say and make sure it doesn't
| have internal contradictions and that readers can within
| reason understand what you are trying to say.
|
| You seem to be complaining about something. My guess is
| that your problem isn't even with the language. Your
| problem is that some tools for working with your kind of
| programs are missing or aren't effective.
| jerf wrote:
| Because it a common term used for a category of languages
| that everybody understands.
|
| If you have a problem with it, you can take it up with
| the aforementioned everybody. Personally I think it gets
| perilously close to the error that if you argue
| definitions enough you can change reality. It doesn't
| matter what labels we slap on clearly related languages
| Python/Perl/Javascript/Lua/PHP/Javascript, I consider it
| a mistake to build a large system with them in 2023
| because of their deficiencies, and given the extreme
| efforts being exerted trying to fix those very same
| deficiencies with things like gradual typing and
| strongly-typed languages like TypeScript that compile
| down into them, which includes the effort Facebook has
| exerted, clearly it is not a terribly heterodox opinion.
| lijok wrote:
| > I consider it a mistake to build a large system with
| them in 2023
|
| A mistake in what sense? What are the risks that you see
| with using these languages for large scale systems, and
| how do those risks balance around all the other risks
| that come with not using one of these languages and using
| a different one?
| jerf wrote:
| I think I pretty much laid out the bulk of the argument.
| Facebook is moving heaven and earth to mitigate their
| choice of a dynamic language, even if it was the right
| choice at the time. You can go look at everything they've
| done, and everything Typescript has done, and all the
| gradual typing initiatives. I left plenty of threads to
| pull there.
| myvoiceismypass wrote:
| Dynamic Typing is absolutely a feature of languages.
| Static Typing is, as well.
|
| I was not confused by using these terms. I cannot imagine
| many people on HN were either.
| dukeyukey wrote:
| > "Dynamic" or "scripting" aren't features of languages
|
| Surely dynamic typing is a language feature? I can't
| imagine what else someone would refer to with "dynamic".
| OrderlyTiamat wrote:
| Could you enlighten me as to how a type system is not
| part of a language?
|
| For example, in your own words, what is the main
| difference between typescript and javascript?
| tyingq wrote:
| Out of the box php also scaled pretty damn far for them
| before they had to do anything significant. Hundreds of
| millions of users as early as 2009, which seems to be before
| the original Hiphop existed.
| sweettea wrote:
| https://discuss.python.org/t/a-fast-free-threading-python/27...
|
| In which Meta promises 3 engineer-years for enhancing python.
| pjmlp wrote:
| Like PHP, where FB has spent many man years writing AOT
| compilers, an new VM, and a strong typed variant from PHP?
| corethree wrote:
| Python not only has types but it's type system is superior to
| typeScript.
|
| Get this python has sum types and exhaustive pattern matching
| exactly like rust or haskell.
|
| The only problem with python are the libraries are sometimes
| written with tricks that make static typing ineffective. Other
| than that it is really really good at scale. Better API then
| typescript imo which is really it's main competitor.
|
| edit: (rate limiter is preventing from replying to everyone...
| I will respond to everyone.)
| seabrookmx wrote:
| Yes but the if you want to do ahead-of-time type checking you
| need to run a tool like mypy, and none of those tools are as
| comprehensive or performant as Typescript. Also the ecosystem
| of libraries with type annotations is much smaller
| (Typescript has the first mover advantage, after all).
| corethree wrote:
| I literally said it supports exhaustive pattern matching
| which adds a level of safety and flexibility superior to
| that of type script.
|
| Many IDEs have real time type checking that highlights the
| errors so don't even have to run the external checker.
|
| Even if you don't use IDEs running the type checker is
| measured in seconds. Not far off from linters that most
| people will also use for TS.
|
| What you say about the libraries is true though. But you
| can always place a type "shell" around those libraries such
| that your code is type safe.
|
| The other main problem with the libraries in python is that
| a lot of people who use python are data scientists who
| haven't figured out why types are so great. Those guys are
| the main ones holding the libraries back.
| chlorion wrote:
| >I literally said it supports exhaustive pattern matching
| which adds a level of safety and flexibility superior to
| that of type script.
|
| I don't have an opinion on which language is superior
| here, I've never written typescript or even javascript
| before, but I think saying that python's type
| system/checkers is superior because of this one feature
| is not correct.
|
| I'm also skeptical of the claim that typescript doesn't
| support exhaustive branching. This very well could be
| true but it seems hard to believe.
|
| I'm a big fan of exhaustive branching, but I think there
| are other things that are as or more important.
| insanitybit wrote:
| You can't even represent `Json` in Python's type system
| because it would require a recursive type.
|
| edit: I _think_ this is actually a Python type annotation
| limitation but it 's possible that's mypy, although I think
| in 99% of cases those are fine to conflate (I have used other
| Python type systems).
| dazzlefruit wrote:
| In VSCode (which uses PyRight/PyLance) and Python 3.10:
| JSONObject = None | str | int | bool | list["JSONObject"] |
| dict[str, "JSONObject"] # this type checks a:
| JSONObject = {"a": [1, 2, "7", True, {"false": None}]}
| # this doesn't type check b: JSONObject = {"a": [1,
| 2, "7", True, {"false": object()}]}
| insanitybit wrote:
| Cool, guess they finally fixed this. Must've been in the
| last ~1 year, give or take. Of course, it relies on
| quoting your types, which is... a matter of taste, I
| suppose.
| dazzlefruit wrote:
| Quoting types that are defined later is a wart but it's
| not very bad. VSCode's UI will happily handle it as if
| the quotes weren't there.
| npage97 wrote:
| You can also use from __future__ import annotations so
| the quotes become unnecessary.
| https://peps.python.org/pep-0563/
| insanitybit wrote:
| Does that work with recursive types? I have had mixed
| results with `from __future__ import annotations`
| personally, but I haven't written much Python in ~a year
| or so.
| corethree wrote:
| You can. from typing import Dict, List
| JSON = Dict[str, 'JSON'] | List['JSON'] | float | int | str
| | None
|
| I would say the only thing ugly here is the string 'JSON'
| that is used for recursion. From a static safety
| perspective with an external type checker the static safety
| effect is identical.
| crabbone wrote:
| Surprise: all languages have types.
|
| Superior to TypeScript is neither a high bar, nor is this any
| kind of objective metric. I don't know why sum types are a
| blessing, also I don't know why pattern matching makes
| anything better.
|
| I can name a lot of problems with Python, and I'm sure that
| libraries isn't the _only_ one. For example, for no reason,
| Python has multiple unrelated mechanisms to manage program
| state (object, clojures, context managers). Or, here 's
| another one: it has multiple public API for dealing with
| filesystem (i.e. os.path and pathlib), both of which are all
| sorts of bad, but just the fact that it has two, where one
| would do is bad.
|
| It's not possible to make static typing ineffective. Static
| typing is a given, it happens in the virtue of you having a
| program. You may argue that it doesn't exist if unobserved in
| the same way how a tree that falls in the forest doesn't
| really fall if unnoticed. But, really, it's there, and it's
| there in every language. You cannot make it ineffective. It's
| like making centimeters ineffective -- I would struggle to
| imagine what that may possibly mean.
| corethree wrote:
| >I don't know why sum types are a blessing, also I don't
| know why pattern matching makes anything better.
|
| That's because you're inexperienced and haven't used them
| before. Try haskell or rust. This level of type safety
| actually reduces logical branching errors. And the key word
| is <exhaustive> pattern matching.
|
| Googling isn't going to give you the insight here imo you
| need the experience (probably a couple months). If you
| don't plan on getting it I suggest you ask a haskell
| developer or rust developer about why exhaustive pattern
| matching is such a great feature.
|
| > Python has multiple unrelated mechanisms to manage
| program state
|
| This is like a slightly bad for people who have ocd towards
| library organization. Two ways to do the same thing exists
| everywhere. Do you use looping or tco recursion? Why do
| some languages support both? Doesn't matter that much. This
| doesn't make a language horrible just makes it a bit
| bloated. Much worse is stuff like javascripts undefined
| value. Also TS has tons and tons of libraries that do the
| same thing. Why isn't that setting off your "bad" red flag
| instincts? Is it because they aren't in the std? So
| redundant commands outside of the std are ok but within the
| std.. bad bad bad? Have you actually hit a real problem
| related with this or is it just something that feels bad
| because of ocd?
|
| >It's not possible to make static typing ineffective.
|
| Categorically false. Python does have patterns and tricks
| which static type checkers can't catch. You struggle with
| meaning here because you failed to comprehend what I wrote
| and you're now arguing against a misinterpretation of my
| statements. Reread that part again, you definitely
| misunderstood.
| dazzlefruit wrote:
| Why is pathlib bad?
|
| Edit: I'm asking because pathlib is as good as a Python lib
| could be for me. Path manipulations are extremely clear and
| always safe. What more do you need?
| Izkata wrote:
| I'm guessing verbosity? It also reads like they don't
| know why pathlib exists and assume they were created at
| the same time.
|
| os.path came first, often works by poking the filesystem
| directly even when it doesn't seem like it needs to
| (vague memory, not completely certain), and I believe has
| os-specific quirks (so code won't necessarily work
| without changes).
|
| pathlib was created later as a pure-python implementation
| that allows for manipulation paths without touching the
| filsystem except when you explicitly tell it to. Because
| it's also pure python I don't think it has any os-
| specific quirks either, but I haven't explored it in
| depth. Code should work across operating systems without
| changes.
|
| I think I also remember at one point people talking about
| completely replacing os.path with pathlib, or at least
| gutting os.path to the essentials that wouldn't work as
| part of pathlib.
| bafe wrote:
| You seem to confuse "real"state management solutions
| (closures, objects) with syntactic sugar like context
| managers. Any class implementing __enter__ and __exit__ can
| be used as a context manager. The protocol doesn't impose
| any semantics to it. As for the presence of closures
| additionally to objects: this is a natural consequence of
| having nested functions and lexical scoping. However, it is
| quite uncommon to use closures to manage state in Python.
| Timon3 wrote:
| > Python not only has types but it's type system is superior
| to typeScript.
|
| I strongly disagree. It might be better at some things, but
| it's much worse at others. Many functions can't be accurately
| typed (try, for example, to make a well-typed function that
| concatenates two arbitrary fixed-size tuples), and as far as
| I know generic type transformations can't be implemented
| (random example "this type takes a Dict[str, Any] and returns
| an object with each key turned into a function").
| dazzlefruit wrote:
| For the tuple example: from typing import
| TypeVar T, U, V, W = TypeVar('T'),
| TypeVar('U'), TypeVar('V'), TypeVar('W') def
| concatenate(a: tuple[T, U], b: tuple[V, W]) -> tuple[T, U,
| V, W]: return a + b
|
| For the generic type transformation example, I'm not sure
| what you mean: from typing import Any,
| Callable Transformer = Callable[[dict[str,
| Any]], dict[Callable, Any]]
|
| This seems to match your question but it's really weird.
| Timon3 wrote:
| Your tuple example _only_ works if both tuples have two
| elements. I specifically mentioned arbitrary fixed-size
| tuples (as in, tuples with an arbitrary non-variable
| length).
|
| Your generic type transformation example also doesn't
| come close to what Typescript does. The resulting dict
| will not have known keys based on the keys of the input
| dict. In Typescript I can write a function that takes an
| object with known keys, and it returns an object with
| those same keys having their values mapped to a different
| type, _with the keys still known_. I threw together a
| quick example - just look at the type of resultA /resultB
| by hovering over the variables[0].
|
| But both are great examples - they are probably the
| closest you can get in Python, and they are so far
| removed from the thing I want to represent that they are
| completely useless.
|
| [0]: https://www.typescriptlang.org/play?#code/MYewdgzgLg
| BAhjAvDA...
| Izkata wrote:
| Tuples are already fixed size by nature, so adding a
| redundant "fixed-size" in that description was confusing.
| I also thought you meant a predefined size like always
| 2-tuples or always 3-tuples.
| Timon3 wrote:
| > Tuples are already fixed size by nature, so adding a
| redundant "fixed-size" in that description was confusing.
|
| No, you can type variable-length tuples in Python. A
| variable int tuple, for example, can be typed as
| Tuple[int, ...].
|
| You can't concatenate two variable-length tuples, which
| makes sense - where would the cutoff be? But you should
| absolutely be able to concatenate two fixed-size tuples,
| and it's very limiting that you can't.
| Izkata wrote:
| > A variable int tuple, for example, can be typed as
| Tuple[int, ...].
|
| That's a type that matches tuples of any length, not a
| variable-length tuple. The size of a tuple can't be
| changed. A variable-length tuple doesn't even really make
| sense, what you'd want there is a list.
|
| > You can't concatenate two variable-length tuples, which
| makes sense - where would the cutoff be? But you should
| absolutely be able to concatenate two fixed-size tuples,
| and it's very limiting that you can't.
|
| This whole statement doesn't make sense. I'm assuming
| you're still talking about type definitions and not
| actually tuples.
| Timon3 wrote:
| > That's a type that matches tuples of any length, not a
| variable-length tuple.
|
| A tuple with a type that matches variable lengths of
| tuples is a variable-length tuple for that piece of code.
| You're free to show me some official definitions that
| proves this wording false, but until then it's useless
| nitpicking. Though you should probably take that up with
| Guido, who also calls them variable-length tuples:
| https://github.com/python/typing/issues/30
|
| > This whole statement doesn't make sense. I'm assuming
| you're still talking about type definitions and not
| actually tuples.
|
| The statement makes perfect sense, thank you. If you have
| trouble understanding my messages without me repeating
| the whole definition every time, maybe just skip them.
| dazzlefruit wrote:
| You mean that the tuples' size is statically known at the
| calling site, while your message could be interpreted as
| the size being statically known in the callee.
|
| I think this is clearer. The statement "arbitrary fixed-
| size tuples" sounded a bit like "an immutable mutable
| variable". It doesn't really say what's arbitrary about
| the tuples and in what context the size is fixed.
| Timon3 wrote:
| Considering the opposite is called a "variable-length
| tuple", and I wanted to express that I'm talking about
| arbitrary tuples with non-variable length, what wording
| would have made this clearer?
| dazzlefruit wrote:
| What's missing is in which context the tuple's length is
| variable, and in which context it is fixed. You can have
| a tuple size fixed everywhere (because the callee sets
| it) or a tuple size fixed at each call (and propagated to
| the callee statically).
|
| "Arbitrary" doesn't really help because it could refer to
| the elements' values, to their types, or to the tuple's
| length. Also "arbitrary" and "variable-length" sound like
| synonyms to me.
|
| Guido might use some expressions in the context of Python
| steering discussions but that doesn't make them less
| obscure for the rest of us who read C++ docs every day
| instead.
| Timon3 wrote:
| > What's missing is in which context the tuple's length
| is variable, and in which context it is fixed.
|
| Simple example: a function has a parameter whose type is
| "variable-length tuple of int". You can pass any tuple in
| that is known to have 0..n elements, all of type int.
| What would you have me call that, other than the name
| I've seen used in discussions on this feature?
|
| > "Arbitrary" doesn't really help because it could refer
| to the elements' values, to their types, or to the
| tuple's length.
|
| Read it as (arbitary (fixed-size tuples)). It was meant
| to forgo answers describing functions with known tuple
| sizes.
| dazzlefruit wrote:
| > Simple example: a function has a parameter whose type
| is "variable-length tuple of int". You can pass any tuple
| in that is known to have 0..n elements, all of type int.
|
| And n is fixed at the calling site, right? I wonder if
| something like "TypeVar, but for a list of type
| arguments" could solve your problem.
|
| What's funny is that this is already kind of implemented
| in `typing.Concatenate`, but only for function parameters
| [1], not for type hint parameters.
|
| Anyway, I would have written "a well-typed function that
| concatenates two arbitrary tuples whose size is
| statically known at the call site". Can't really remove
| "at the call site" or "statically known" without being
| ambiguous.
|
| Edit: just found out about `TypeVarTuple`. So really
| we're only missing `concatenate`.
|
| [1] https://docs.python.org/3/library/typing.html#typing.
| Concate...
| Timon3 wrote:
| > And n is fixed at the calling site, right? I wonder if
| something like "TypeVar, but for a list of type
| arguments" could solve your problem.
|
| Yep, and TypeVarTuple should - all the syntax etc. is in
| place, there is an Unpack operator for TypeVarTuples,
| allowing you to e.g. append or prepend individual types
| to a TypeVarTuple. But you can't unpack more than one
| TypeVarTuple in an expression, it's specifically
| disallowed - so I can't properly type my function.
| youerbt wrote:
| I'd call that function: polymorphic over tuple length.
| dazzlefruit wrote:
| "Arbitrary fixed-size tuples" probably don't have a
| widely accepted meaning :) I read it as "size known when
| compiling the function".
|
| The second example is cool. But I can't find a good
| practical use case for either example.
|
| If you have a collection that's both heterogeneous and
| whose [size/key set] is statically known, when would it
| make sense to apply such generic transformations to them?
| This sounds like you have tuples or dataclasses where all
| elements have different meanings (since their types are
| fixed and different) _and_ you want to treat them like a
| generic collection _and_ you need the type checker to
| infer the result type.
|
| The main use of tuples or dataclasses or `NamedTuples` is
| to pass or return values to/from functions without
| dealing with long lists of arguments. The elements aren't
| in the same category, it doesn't make sense to process
| them as one big collection, they mean different things.
|
| (Also I think you made a mistake in your previous
| message, you wrote "an object with each key turned into a
| function" but it's the values that change types here.)
| Timon3 wrote:
| > "Arbitrary fixed-size tuples" probably don't have a
| widely accepted meaning :) I read it as "size known when
| compiling the function".
|
| That is exactly what I'm talking about - the size of the
| tuples is known statically.
|
| > The second example is cool. But I can't find a good
| practical use case for either example.
|
| There are many interesting use cases in libraries,
| especially for some of the more esoteric features.
| Everything can be used for additional type safety.
|
| > If you have a collection that's both heterogeneous and
| whose [size/key set] is statically known, when would it
| make sense to apply such generic transformations to them?
| This sounds like you have tuples or dataclasses where all
| elements have different meanings (since their types are
| fixed and different) _and_ you want to treat them like a
| generic collection _and_ you need the type checker to
| infer the result type.
|
| Simple example - I have functions that return a Rust-like
| Result type, and I want to transform that into a
| different tuple-based format using a decorator. The
| transformation itself is static, but I can't write one
| function that handles it all, because Pythons type system
| is simply not developed enough. Something that would be
| incredibly easy in Typescript.
|
| > The main use of tuples or dataclasses or `NamedTuples`
| is to pass or return values to/from functions without
| dealing with long lists of arguments. The elements aren't
| in the same category, it doesn't make sense to process
| them as one big collection, they mean different things.
|
| But I have a use case for exactly this feature. Why
| should the language limit me? Why should I implement x
| functions that take different tuple lengths, with me
| having to choose the correct one for each use case, when
| I could write one function that does all?
|
| > (Also I think you made a mistake in your previous
| message, you wrote "an object with each key turned into a
| function" but it's the values that change types here.)
|
| Sure, though I could also literally write a Map that
| turns all object keys into functions.
| dazzlefruit wrote:
| > But I have a use case for exactly this feature. Why
| should the language limit me? Why should I implement x
| functions that take different tuple lengths, with me
| having to choose the correct one for each use case, when
| I could write one function that does all?
|
| If the PSF ran a poll for the most-wanted type checking
| features, I don't think this would come close to first.
| This sounds very niche. The people working on typing
| seemed very busy in the last few versions.
|
| > Simple example - I have functions that return a Rust-
| like Result type, and I want to transform that into a
| different tuple-based format using a decorator. The
| transformation itself is static, but I can't write one
| function that handles it all, because Pythons type system
| is simply not developed enough. Something that would be
| incredibly easy in Typescript.
|
| Returning errors as values isn't really how you're
| supposed to use Python though. And why is there a second
| tuple-based format that does the same thing?
|
| It also smells slightly off that the rest of the code
| takes an object that's exactly similar to the first
| function's result, but with a transformation uniformly
| applied over the values. Shouldn't the first part's
| output and the second part's inputs both be clearly
| declared independently of each other? And then wouldn't
| it be an extremely niche case that both types are
| identical except for one transformation applied to all
| values? Is it worth the language complexity and a
| dedicated function?
| Timon3 wrote:
| > If the PSF ran a poll for the most-wanted type checking
| features, I don't think this would come close to first.
| This sounds very niche. The people working on typing
| seemed very busy in the last few versions.
|
| Sure, but while it's not possible to type basic functions
| like ones that concatenate tuples, I can say that Pythons
| typing system is not superior to Typescripts.
|
| > Returning errors as values isn't really how you're
| supposed to use Python though.
|
| Okay, so how am I supposed to handle non-exceptional
| errors? Because using exceptions for that kind of thing
| absolutely isn't good practice.
|
| > And why is there a second tuple-based format that does
| the same thing?
|
| Legacy. Typescript allows me to do refactoring of things
| like these step by step and very easily. Python doesn't,
| because it's inflexible.
|
| > It also smells slightly off that the rest of the code
| takes an object that's exactly similar to the first
| function's result, but with a transformation uniformly
| applied over the values. Shouldn't the first part's
| output and the second part's inputs both be clearly
| declared independently of each other? And then wouldn't
| it be an extremely niche case that both types are
| identical except for one transformation applied to all
| values? Is it worth the language complexity and a
| dedicated function?
|
| Are we taking apart my code now or what? I can't stop the
| world and focus only on refactoring things to be neat and
| tidy for months on end. But I _can_ improve individual
| parts, bit by bit - and in a language with a better
| typing system, I can do so way easier.
| dazzlefruit wrote:
| > Sure, but while it's not possible to type basic
| functions like ones that concatenate tuples, I can say
| that Pythons typing system is not superior to
| Typescripts.
|
| That's not really fair. The uses that make sense when
| considering Python's convention ("Pythonic" code,
| nebulous but usually well-understood) are supported.
|
| I think what confused me about these examples is that
| they imply multiple values that have completely different
| meanings, but all get processed as equals anyway. That
| was before you talked about refactoring old code though.
|
| > Okay, so how am I supposed to handle non-exceptional
| errors? Because using exceptions for that kind of thing
| absolutely isn't good practice.
|
| If an operation on a homogeneous collection can say
| "nope" for some values and process others, the values
| would be typed "T | None". If the data isn't really a
| collection but a structured mapping, in general,
| attributes would be made optional on a case-by-case
| basis. If all attributes happen to be optional but the
| mapping itself is non-optional, that sounds more like an
| accident of this specific case than something we should
| complicate a language over. If this happens over a whole
| codebase, I guess I feel for you. Maybe that's when it
| makes sense to give up a bit of static typing and treat
| these values a bit more like data and a bit less like
| separate arguments, no matter what kind of complicated
| typing the language can do.
|
| > Legacy. Typescript allows me to do refactoring of
| things like these step by step and very easily. Python
| doesn't, because it's inflexible.
|
| > Are we taking apart my code now or what?
|
| Honestly yes, this example seems so unusual that it
| doesn't make sense to debate it without knowing
| concretely what's happening in your code that this needs
| to be supported.
|
| I guess that this specific example would get easier. But
| I would hardly call one type system superior over that.
|
| Edit: my final opinion on this is "this is something
| that's technically possible if you follow language and
| conventions to the letter, but with experience you see
| that it's a bad idea that won't fit well with the
| language and you should change the design to avoid it".
| It happens in all languages IMO.
| Timon3 wrote:
| > That's not really fair. The uses that make sense when
| considering Python's convention ("Pythonic" code,
| nebulous but usually well-understood) are supported.
|
| You're focussing on a very specific part of what I wrote.
| Just because my specific example consists of a Result
| type being transformed into a Tuple, it doesn't mean that
| the basic use case of "concatenate two tuple types" is so
| far out there.
|
| > f an operation on a homogeneous collection can say
| "nope" for some values and process others, the values
| would be typed "T | None". If the data isn't really a
| collection but a structured mapping, in general,
| attributes would be made optional on a case-by-case
| basis. If all attributes happen to be optional but the
| mapping itself is non-optional, that sounds more like an
| accident of this specific case than something we should
| complicate a language over. If this happens over a whole
| codebase, I guess I feel for you. Maybe that's when it
| makes sense to give up a bit of static typing and treat
| these values a bit more like data and a bit less like
| separate arguments, no matter what kind of complicated
| typing the language can do.
|
| But the static typing is extremely helpful, it prevents
| many kinds of errors. Not being able to use it for these
| kinds of things makes Python a worse language, no matter
| how you cut it (in the sense that it would be a better
| language if you could).
|
| > Honestly yes, this example seems so unusual that it
| doesn't make sense to debate it without knowing
| concretely what's happening in your code that this needs
| to be supported.
|
| Again, my specific example doesn't matter. The use case
| is "function takes in two tuple types and returns a
| concatenated version". That's something a type system
| should be able to handle.
|
| > I guess that this specific example would get easier.
| But I would hardly call one type system superior over
| that.
|
| On what metric besides expressiveness would you rate type
| systems?
| chlorion wrote:
| The tuple thing requires variadic generics from my
| understanding.
|
| I don't thing variadic generics support is supported in
| most statically typed languages. The only one I can think
| of right now that supports this is C++.
| Timon3 wrote:
| Typescript supports it too (quick example[0]) :) and
| Python actually as well, but currently you can't unpack
| two TypeVarTuples in the same type expression:
| https://peps.python.org/pep-0646/
|
| [0] https://www.typescriptlang.org/play?#code/C4TwDgpgBAg
| lC8UDaA...
| chlorion wrote:
| Wow this is very cool thanks for sharing!
| Timon3 wrote:
| You're welcome! That's why I'm very excited about
| Typescript, the system is very powerful :)
| corethree wrote:
| > I specifically mentioned arbitrary fixed-size tuples
| (as in, tuples with an arbitrary non-variable length).
|
| This is wrong. Again, Arbitrary fixed-size tuples are
| equivalent to structs with an arbitrary amount of
| properties. Languages shouldn't do this, it destroys the
| nature of what a TUPLE is which is essentially just a
| struct with no names.
|
| The concept you are going for is isomorphically
| encapsulated by ANOTHER type: List[Any]
|
| You should be using the above type to encode what you
| want conceptually.
|
| That being said if javascript has variadic tuples then
| it's not a very good type system imo. It encodes
| redundant concepts. Why have a tuple with Variadic
| arguments when I have Arrays that do the exact same
| thing?
| Timon3 wrote:
| > This is wrong. Again, Arbitrary fixed-size tuples are
| equivalent to structs with an arbitrary amount of
| properties. Languages shouldn't do this, it destroys the
| nature of what a TUPLE is which is essentially just a
| struct with no names.
|
| Okay, that might be your personal feelings on the topic.
| But do you understand the concept of "generic functions"?
| Sometimes you have to apply generic transforms to data.
| Being able to correctly express your transformations in a
| type system isn't "wrong", it's useful.
|
| > List[Any]
|
| Sorry, but I really think you don't understand what I'm
| talking about. If I write a function that handles tuples
| of arbitrary length and that function returns a
| transformed version of that tuple, I keep the information
| about individual tuple elements. This is thrown away in a
| list.
|
| > That being said if javascript has variadic tuples then
| it's not a very good type system imo. It encodes
| redundant concepts. Why have a tuple with Variadic
| arguments when I have Arrays that do the exact same
| thing?
|
| Arrays don't do the same thing, so they are not redundant
| concepts. Tuples have elements in specified positions
| with specified types. Arrays have one type (possibly a
| union type) over many elements.
| dragonwriter wrote:
| Python can _almost_ do this with Variadic Generics from
| Python 3.11; its missing an exception to the single-
| unpacking rule (which exists to prevent ambiguity) to allow
| unambiguous cases. Then you would have:
| from typing import TypeVarTuple Ts =
| TypeVarTuple("Ts") Us = TypeVarTuple("Us")
| def tconcat( t1: tuple[*Ts], t2:
| tuple[*Us] ) -> tuple[*Ts, *Us]: ...
| Timon3 wrote:
| That's the conclusion I also arrived at. The type system
| is slowly getting there, but many operations like this
| one are still not possible.
|
| It's a shame, because it makes features like decorators
| significantly harder to use with static typing.
| corethree wrote:
| I haven't seen a type system that allows variadic types
| for Tuples. This would be equivalent to creating a struct
| with variadic amount of properties.
|
| The definition for tuples here is similar to a struct.
|
| They are one in the same except structs have names for
| each property while tuples don't. That is literally the
| main concept of a tuple, just a struct with no names for
| properties.
|
| The type system for python is already "there", it is in
| fact superior to many other type systems from other
| popular languages.
| Timon3 wrote:
| That's confusing, considering Python has them with
| TypeVarTuples?
| corethree wrote:
| No this is actually wrong. It works but it doesn't
| capture the meaning of the nature of a tuple.
|
| A Tuple is Typed as something as a Fixed size. That's
| right, it's like this at the Type level. The entire
| concept of a tuple is a Product type or essentially like
| a struct but with no names for each parameter.
| Tuple[int, str, float] #correct
|
| Doing what you're doing here is equivalent to creating a
| Struct with variadic properties.
|
| If you want some container that holds an arbitrary amount
| of things that is a List
|
| The correct type for what you want is actually this:
| List[Any]
| Timon3 wrote:
| It absolutely isn't wrong, and it's strange that you have
| some aversion against it. Think of Tuple[int, ...] as a
| union of Tuples of all lengths containing only ints. It's
| a specific language feature, and it's not "wrong" in any
| way.
|
| And I have no idea why you claim that a list is
| "correct".
| corethree wrote:
| def concat_tuples(tuple1: Tuple[int, str, float], tuple2:
| Tuple[str, str, str]) -> Tuple[int, str, float, str, str,
| str]: return tuple1 + tuple2
|
| Your second function you just broke out of the type system
| with Any. Give me a more exact, are you saying Any is a
| constrained type variable? The only possibility here is
| this: T = TypeVar('T') def
| transform(x: Dict[str, T]) -> Dict[str, Callable[[], T]]:
| return {key: lambda : value for key, value in x.items()}
|
| If you want that lambda to do something else you have to
| constrain T. Constraining T will give me more options in
| the definition: from numbers import Number
| def transform2(x: Dict[str, Number]) -> Dict[str,
| Callable[[Number], Number]]: return {key: lambda
| y: value + y for key, value in x.items()} def
| transform3(x: Dict[str, Number]) -> Dict[str, Callable[[],
| Number]]: return {key: lambda: value + value for
| key, value in x.items()}
|
| etc...
|
| The thing is because these type checkers are external,
| anyone can add arbitrary features to them and extend it.
| Someone can make it to the level of proof checking...
| eliminating the need for testing in general. Of course the
| syntax has to support it too.
| Timon3 wrote:
| > concat_tuples
|
| I specifically talked about arbitrary tuples. Your
| function only handles specific tuples.
|
| > Your second function you just broke out of the type
| system with Any. Give me a more exact, are you saying Any
| is a constrained type variable?
|
| I'm not sure what you're trying to say here. Yes, you can
| implement the function, I never claimed otherwise. I'm
| talking about the _type system_.
|
| > The thing is because these type checkers are external,
| anyone can add arbitrary features to them and extend it.
|
| The type system is a core feature of Python. Until it
| supports what I want, I can't really use what I want with
| Python.
| timeon wrote:
| > sum types
|
| Even PHP have those now. TypeScript is lagging here.
| Hopefully, they will include it too one day.
| crabbone wrote:
| I've been writing in Python for over ten years, in different
| roles, for wildly different projects (research, infra, Web,
| testing, education).
|
| I'm yet to find anything Python was good for. On engineering
| merits alone Python isn't best for anything, nor is it best for
| combinations of things. It's silly to think that any tool that
| works with Python does so because Python was the best language
| for the job, and they only needed that tool to make it even
| better.
|
| Most stuff written around Python is yet another layer of band
| aids on top of a huge ball of band aids that's already there.
|
| So, you may wonder, with all those band aids, didn't they all
| make it better? And, in a sense, yes, that's what they are for.
| The band aids improve the experience of the user end. But this
| isn't how I use the word "better". When I use "better" I mean
| the quality of execution, not the satisfaction it gives to the
| user.
| devjab wrote:
| I'm not really a fan of Python as such, but after a few
| decades in the industry, I'm beginning to think that being
| good at being bandaid is "better". I can't think of a single
| tech where we don't have a bunch of duct tape (as we refer to
| it), not so much because we want to but because that's just
| how things end up in the imperfect world of organisations. I
| value the techs that fit into this reality more than the ones
| which don't, but you're right, Python isn't really great for
| any technology based merits, it's good because the world is a
| messy place where being productive with your band aid is
| often more valuable than using the "better" programming
| language.
| Izkata wrote:
| Old proverb that I think applies here: A jack of all trades
| is a master of none, but oftentimes better than a master of
| one.
| corethree wrote:
| I've been writing python for a longer amount of time.
|
| What makes python popular is it's the ease of use and
| debugging. It may not have been type safe but every time it
| crashed it's extremely easy to find out where and why it
| crashed. This is not the case languages like JavaScript or
| for even something like golang which is ironically type safe.
| I like to emphasize this here. It is more important for a
| language to be extremely clear about the origin and nature of
| a runtime error then it is to be type and statically safe.
| Look at C++. It has powerful static typing but is marred by
| seg faults and memory leaks which are hidden errors that are
| extremely hard to tease out.
|
| That being said, modern python now has types. The bandaids on
| top of python from sheer coincidence turns it into a powerful
| scalable language for the web rather then an ugly patchwork.
|
| The main problem with python right now would be the
| performance and the library ecosystem. Performance is of
| course mainly the gil and libraries are often written with
| tricks that make static typing impossible. It's just old tech
| debt getting in the way not the modern language itself.
| Linters should prevent anyone starting new code using those
| patterns again.
|
| Other then that, pythons type system + checkers supports even
| sum types with exhaustive pattern matching added with extreme
| ease of debugging runtime errors makes python one of the best
| languages for big projects. That level of safety isn't even
| offered by go or typescript. The only alternatives are rust
| or haskell for sum types.
|
| I highly disagree with you. Python is not without old tech
| debt warts holding it back, but overall it is a superior
| language.
| atoav wrote:
| This very subjective, but I think python is good at syntax.
| If I had to teach non-programmers programming it would be
| easily my number one choice. The included libraries are also
| pretty good.
| yoyohello13 wrote:
| Python is always the second-best language for the job. Which
| makes it a great language to know.
| diarrhea wrote:
| Exactly. It's not about asking if Python is the best tool.
| It's about looking at the alternatives. Python comes out
| ahead of all alternatives, save for 1-3 others. And those
| others might not be feasible for a host of other reasons.
| That's how you end up with Python. And it's fine!
| dehrmann wrote:
| Disclosure: Meta employee, but these views are my own, and not
| all the experiences are from Meta.
|
| Type annotation have made Python much more scalable in terms of
| engineers and codebase size.
|
| It still has other scale problems, especially if you actually
| need threads. One project I worked on managed Python worker
| tasks, and we resorted to subprocesses (within subprocesses!)
| because what we thought was IO-bound became CPU-bound, and
| workers started timing out on RPC calls. I also worked on a
| Python API service that scaled beautifully horizontally, but we
| had to manage extra logic for spinning up one worker per CPU.
|
| At some point, you actually start caring about performance, but
| you're more likely to hit other issues before you care about
| the extra hardware cost.
| wdroz wrote:
| I'm happy with Ruff[0], it's very fast.
|
| [0] -- https://github.com/astral-sh/ruff
| ltbarcly3 wrote:
| Unfortunately ruff is very inconsistent and has lots of
| differences from the flake8 plugins it tries to emulate. Lots
| of rules are confused by irrelevant context so that it can miss
| lots of things it should find when the equivalent flake8 plugin
| still find them. It's automatic fixing of issues will happily
| introduce other issues that it doesn't find until the next run.
| I've tried pretty hard to use it and gave up, it's just not
| remotely ready.
| charliermarsh wrote:
| If you notice differences vs. the originating plugins, would
| definitely appreciate it if you could file an issue! We tend
| to be very responsive especially when it comes to matters of
| correctness.
|
| Candidly, for Flake8 plugins, my experience is that Ruff
| tends to be more consistent, more robust in its inference
| and, at this point, more extensively tested than the original
| implementations -- both via our own testing and via the
| significant number of projects and companies that now use
| Ruff in production. (As compared to Pylint, though, we catch
| fewer issues, since Pylint does some type inference across
| files, which Ruff doesn't support yet.)
|
| Ruff is also designed such that we will iteratively lint-and-
| fix until there are no more fixable issues, so if you've seen
| the linter introduce _new_ fixable diagnostics, that would be
| a bug too.
|
| Regardless, thanks for giving it a try :)
| jamwil wrote:
| This isn't really accurate in my experience. I've been using
| it on several projects large and small for some time now and
| can only recall one time autofix introduced an issue of any
| sort and that was many versions ago. It will also handle
| cascading fixes just fine.
| drcongo wrote:
| We've gone all in on Ruff for several months now, across many
| projects, and not noticed _any_ of these issues.
| misnome wrote:
| > It's automatic fixing of issues will happily introduce
| other issues that it doesn't find until the next run.
|
| I think a few of the "bad" auto fixes have been recently
| disabled by default, they now separate fixes into "safe" and
| "unsafe" (for example: it used to always change `x == True`
| to `x is True`, which broken common numpy selection patterns)
| _whiteCaps_ wrote:
| There's a lot of neat Python research coming from Facebook.
|
| libcst (which Fixit uses) is super cool - I use it in
| https://gitlab.com/harford/logzy
| kstrauser wrote:
| Why do you use that instead of flake8, which also finds that?
|
| (Not meant to be a pointed question; genuinely curious.)
| driggs wrote:
| It looks like the interesting feature of their tool Fixit 2 is
| that its lint rules know how to auto-apply themselves. I'm fine
| with an auto code formatter, an auto import organizer, but not
| sure how much I trust a linter to auto-apply "fixes".
| n00j wrote:
| In the C#/.NET world this is the standard and it works very
| well. It definitely increases productivity to not have to
| double check each and every instance of violating a lint error,
| as you can just have the fix applied to an entire project
| without having to worry to much.
| crabbone wrote:
| > increases productivity to not have to double check each and
| every instance of violating a lint error,
|
| At what cosmic speed should you be pumping out code for this
| to be a concern?
|
| Also, in C#/.NET, where programmers predominantly use MSVS,
| which is an atrocious editor with MSBuild, which is an
| atrocious build system, both hampering productivity...
|
| Also, plenty of linter errors are actual errors that need
| non-trivial fixing. So, I struggle to understand how that may
| be automatically fixed (at least in the context of Python). A
| trivial example: misspelled variable name -- how would the
| linter know if that's a typo, or that the programmer intended
| to declare this variable, but forgot to do that?
| insanitybit wrote:
| > Also, plenty of linter errors are actual errors that need
| non-trivial fixing.
|
| These sound like they're not lints at all.
| HdS84 wrote:
| It's very useful for multiple scenarios: 1. Learning new
| syntax. E.g. the new switch expressions are pretty neat and
| better than the old switch statements. Changing and
| learning how to write them is super easy thanks to this
| feature.
|
| 2. A code base with less then good practices is vastly
| easier to adjust to better standards than in python. I've
| done both and dotnet is mostly applying suggestions and
| then looking for more advanced problems which where not
| caught by automatic analysis. It's not perfect, but it
| reduces useless churn.
|
| This also holds true for rider and vs with resharper.
| neonsunset wrote:
| And VS Code too with the analyzers that come with the
| sdk! (or numerous others that are available as extensions
| e.g. Roslynator)
| joshuamorton wrote:
| As someone who worked on a similar tool
| (https://github.com/ssbr/refex/tree/main/refex/fix/fixers, I
| did a bunch of the work to prep this for open-sourcing, though
| I think all my contributions are hidden behind the "Google-
| internal" anonymization), having auto-applied or auto-appliable
| fixers like this is super useful.
|
| They can be auto-applied by post-commit (e.g. a generic `git
| fixcommit` style command that runs _all_ the relevant lint
| tools and fixes them in the working copy, letting you review
| before push), or applied during code review (automatic comments
| with a "click here to apply fix" interface), both of which are
| nice.
|
| Plus the same underlying tooling can be used to write more
| complex one-off fixes that may be used for migrations or
| cleanups.
| Izkata wrote:
| "black" for python has already done this for quite a while.
|
| Even before that, some flake8 linting rules could automatically
| apply the fix, but not all of then.
| IshKebab wrote:
| You don't have to trust it - just commit your changes then run
| the linter and inspect the diff.
| theonething wrote:
| Ruby's Rubocop linter has had this for a long time and it works
| great. I thought this would be the case for most of the
| mainstream languages.
| llwu wrote:
| Adding to what other commenters said, ESLint also works like
| this.
|
| However in an IDE setting it's not exactly "auto"; you have to
| click the light bulb and accept the fix (idk about VSCode, but
| in Neovim you can even get a preview of the diff [1]). This is
| what I'm working on a Fixit PR for right now.
|
| [1] https://github.com/aznhe21/actions-preview.nvim
| dang wrote:
| Related and a bit recent:
|
| _Fixit 2: Meta's auto-fixing linter for Python_ -
| https://news.ycombinator.com/item?id=37036262 - Aug 2023 (11
| comments)
___________________________________________________________________
(page generated 2023-11-22 23:02 UTC)