[HN Gopher] I Want a New Duck
___________________________________________________________________
I Want a New Duck
Author : BerislavLopac
Score : 70 points
Date : 2021-03-17 08:53 UTC (1 days ago)
(HTM) web link (glyph.twistedmatrix.com)
(TXT) w3m dump (glyph.twistedmatrix.com)
| neolog wrote:
| I think this isn't quite what I want, because implicit structural
| typing has a single shared namespace of methods. If the Ducky
| protocol says "implementers should have .quack()" and I define
| FringeScientist().quack(), Mypy will think it's Ducky but it's
| just a naming coincidence. I want to say which protocols I'm
| implementing. Mypy can't tell whether my Foo.quack() is supposed
| to be an implementation of FringeScientist.quack() or
| Ducky.quack().
| morelisp wrote:
| In practice this happens an order of magnitude less than "I
| want to handle two things that match the same protocol that I
| didn't write and can't extend" and "I need to pass something
| that would work fine to a poorly-typed library" combined.
|
| That's before we even get into the philosophical issue of, who
| are you, the lowly library implementor, to say those aren't
| something appropriate to abstract over in my application?
| erik_seaberg wrote:
| How can two objects present exactly the same API without
| either one knowingly emulating the other? Examples always
| seem to use one method taking no args, which don't seem
| realistic.
| XorNot wrote:
| What's missing in this example is stub typing of the output
| values.
|
| Scientist.quack really shouldn't be returning plain
| strings, and neither should ducky.quack if the data in them
| is contextually different.
|
| Things should only become basic types at the last possible
| moment.
| neolog wrote:
| Which language ecosystems operate like that in practice?
| SatvikBeri wrote:
| A common example I ran into is machine learning methods and
| data transformations such as scaling have have `fit` and
| `transform` methods, and I'd like to write code that can
| take multiple such objects and compose them in a pipeline,
| along with some custom logic. Structural typing makes that
| easy to typecheck, nominal typing makes it difficult
| because those two classes don't have a parent class that
| contains those methods.
|
| Extending this, if the class has a `partial_fit` method,
| sometimes I want to use that, and fallback to `fit` when it
| doesn't.
| nemothekid wrote:
| One thing I did come across once before is when two APIs do
| something similar such as parsing and provide the same
| name. A couple of years ago I remember I was working with
| an API in Go that the function Scan which was
| `fn(interface{}) -> Error`, which was used to decode SQL
| values. I pulled in another library which provided that
| function but in a separate context - text parsing.
|
| The interfaces matched, the compiler accepted it, but the
| application blew up at runtime.
| morelisp wrote:
| > Examples always seem to use one method taking no args,
| which don't seem realistic.
|
| This is one of the reasons it is so rare to actually have
| confusable types in a real problem.
| orblivion wrote:
| Don't Golang interfaces work this way as well? (Not that I'm a
| fan).
| coolreader18 wrote:
| Well, that's duck typing - looks like a duck, quacks like a
| duck, it's a Duck[y]. And while it's obviously hard to
| extrapolate simple examples like `duck_war` to real-world use-
| cases, I think signatures being different between methods of
| the same name on different protocols is generally enough to
| differentiate between them.
| morelisp wrote:
| It's so bizarre to me - though I guess in-line with me finding a
| lot of decisions around Python 3 bizarre - that after years of
| Python being one of duck typing's biggest defenders and success
| stories, mypy went with nominal typing by default.
|
| Both Go _and_ TypeScript were _right there_ showing you how
| effective it was.
| phailhaus wrote:
| Yep. An absolute travesty, and makes working with mypy a pretty
| huge pain, even though the advantages of static typing still
| make it worth it. But man, when I switch back to Typescript, I
| remember how nice it could have been.
| musingsole wrote:
| Mypy will only ever make it into a small fragment of Python
| code. Its existence is more likely an example of the those
| insistent on type checking crossing the picket line and
| bringing their last holdout features to the language than it is
| an acceptance that duck typing is not the way.
|
| For the record, duck typing is the way. Mypy is a curiosity to
| me; I'm happy those who need it are getting that itch
| scratched; I'm happier it's boxed away from what I work with.
| bitwize wrote:
| No, no, no. Strong static typing is an _unmitigated_ win. A
| stitch in time saves nine: with static typing you can check
| for type constraint violations at compile time that would
| otherwise throw runtime errors, eliminating a large class of
| bugs before your program even runs.
|
| What's more, with static typing your IDE can provide you with
| more helpful guidance by immediately showing you the valid
| method names associated with an object, type-checking their
| parameters as you use them, and suggesting possible
| parameters of the required types from the variables currently
| in scope.
|
| With dynamic/duck typing, you give all that up -- for no
| clear advantage.
|
| Use static typing in a project of significant size. Always.
| throwaway894345 wrote:
| > Mypy will only ever make it into a small fragment of Python
| code.
|
| I don't think this has to be the case. If it got serious
| investment and the Python community was more open to making
| ergonomic syntax improvements so type annotations could be
| natural, we could see improvement. Similarly, I think it
| would need to figure out a saner solution to finding/loading
| type annotations, because the current set up and error
| messaging are immensely painful. I also don't know if Mypy
| will ever be sufficiently performant so long as it's written
| in Python (I think people really underestimate the difference
| between instant feedback and a delay of several seconds).
| Having good editor support would also drive adoption.
|
| If Mypy were able to improve on all of these distinct problem
| areas, I think more people would opt into type annotations
| naturally, but it certainly feels like Mypy is being treated
| as a thing to pacify the people who whine about static typing
| (of course I don't think that's the real intention, only how
| it comes across).
| sseagull wrote:
| I thought like you at one point, but then jumped into a
| moderately-sized project that I was unfamiliar with. It took
| months to get an intuition for what the data was that was
| being passed around.
|
| Inside a function you need modify, you are passed a foo. What
| can you do with it? Take the length? Add a number to it? You
| don't know unless you know the type, and then the only place
| that info exists is in your head. Why not write it down in a
| way that can be checked and therefore never get out of date,
| like comments or some weird reincarnation of Hungarian
| notation?
|
| As a bonus: Editors will tell you when you are passing
| incorrect types to functions before you even run your tests.
| No more strings accidentally treated as lists!
|
| I am now absolutely convinced that types are important, even
| in python.
| musingsole wrote:
| I remain unconvinced that the utility in the scenarios you
| describe outweighs the overhead in scenarios where it is
| irrelevant.
| morelisp wrote:
| Please look at languages with good structural typing
| (e.g. Go - not so powerful but lots of syntactic
| convenience - and TypeScript - lots of power but you pay
| in compilation time, albeit not worse than mypy).
|
| If mypy is doomed to remain marginal it is because it
| picked a poor type system for the language it tries to
| support, not because stricter typing generally is the
| wrong choice for that language.
| sseagull wrote:
| To each their own. I find almost all functions I write
| are really designed with one type in mind anyway.
|
| One difference may be if you primarily work in your own
| projects or work with other people's projects. And how
| skilled the other developers are. In my own projects I
| instinctively know the types, but not in other projects,
| and having the types documented allows me to make changes
| much more quickly.
| war1025 wrote:
| My biggest pain as far as Python3 typing / mypy goes has been
| interfacing with major libraries and trying to add types in a way
| that isn't just me tossing `Any` everywhere.
|
| If memory serves, Twisted hasn't been too bad. SqlAlchemy is
| pretty well a nightmare though. I think I saw that the latest
| release has type stubs. That will be nice. PyQt is also less than
| ideal...
| joobus wrote:
| > With Mypy, you get all the benefits of high-level dynamic
| typing for rapid experimentation, and all the benefits of
| rigorous type checking...
|
| Except that most typed languages are far faster than python.
| Retrofitting existing python projects with types makes sense for
| reliability, but starting a new project with python and types
| makes less sense, imo.
| gnulinux wrote:
| I start new projects with python and mypy. If you're convinced
| that there are _some_ problems that better be solved in python,
| you can easily see that once you have a project in python, mypy
| just adds more safety to it. So there is nothing wrong with
| python+mypy for new projects.
| jhayward wrote:
| Using that logic, it never made sense to start any project with
| Python, correct?
|
| People use Python for its ease and speed of development, and
| extensive ecosystem. Things that need to be fast can have the
| appropriate effort expended to execute as fast as necessary.
| memco wrote:
| > People use Python for its ease and speed of development,
| and extensive ecosystem.
|
| Of these, I think only the ecosystem stands out anymore. For
| work purposes I wouldn't be able to move away from Python
| anytime soon, but if I could pick any language for new code
| it wouldn't be Python. If I had to pick the closest
| competitor, Julia seems like a great Python-like language in
| terms of speed and ease of development with much better
| typing and performance, but with a less extensive ecosystem.
| Nim might also fit that bill. If ecosystem was important, Go
| probably fills a similar space without being quite as steep
| of a learning curve as some of the other languages.
| mixmastamyk wrote:
| Cython is an option when a (potential) 100x speedup is
| required. Choosing the right algorithm up front, thru the rapid
| experimentation Python gives, can sometimes make it faster than
| a real world implementation in a static language, that didn't
| get enough time to experiment.
| glyph wrote:
| The post isn't about performance, and is aimed at people using
| Python for whatever reason, so, sure, retrofit away.
|
| That said, if you care about the performance improvements that
| typing can give you with Mypy, you might want to look here:
|
| https://github.com/mypyc/mypyc-benchmark-results/blob/master...
|
| It won't be going toe to toe with Rust any time soon, but a 4x
| to 17x speedup is nothing to sneeze at.
| throwaway894345 wrote:
| That's cool if there's any indication that this could
| actually be used in production. Namely, what is the
| compatibility with the ecosystem. The history of Python could
| be aptly characterized as a series of big problems -> tools
| promising miraculous solutions -> tools utterly failing to
| deliver on those solutions because they failed to consider
| some important part of the ecosystem. We need more than a
| benchmark graph to suggest that this is a panacea to Python's
| longstanding performance problems.
| slaymaker1907 wrote:
| There is a very similar tool called Cython that is used by
| scikitlearn. Coincidentally Cython also solves a lot of
| python 2/3 problems since it compiles for both.
| throwaway894345 wrote:
| Also, Mypy has a thousand paper cuts compared to other type
| systems. Pretty sure we still can't model JSON in Mypy (e.g.,
| JSON = Union[List['JSON'], Dict[str, 'JSON'], None, bool, int,
| float, str] or whatever) because Mypy doesn't support recursive
| types. Moreover, since it's largely shoehorned into existing
| Python syntax, it's very awkward. E.g., I'm still not sure what
| the scoping rules are for TypeVar, describing a callback that
| takes kwargs involves defining a protocol class with a
| `__call__()` method that takes your function's kwargs, etc.
| Further, getting the type definitions to load properly is
| incredibly painful, the error message directs you to a page
| with a few things you might try to debug further, but never has
| my issue been among them. Also, like everything in the Python
| world, Mypy is slow, and you really want your type checks to be
| instant.
|
| Most of these things can be improved upon, but progress feels
| slow and there are so many other serious issues in the Python
| ecosystem (performance, package management, etc) that I've
| given up (after 15 years of daily use). It makes me sad, but Go
| and Rust seem to offer better tradeoffs for the things that I
| care about.
| scudd wrote:
| I can't help but feel like if you want static typing, you're
| better off recognizing that python is not the right tool for the
| job.
|
| Admittedly I've never done serious work with mypy (or
| typescript), so I'm approaching the value proposition of dynamic
| typing at face value rather than experience. However, it seems
| like the primary benefit of these languages was ease and
| flexibility, ableit at the cost of structure. Or said
| differently, adding mypy feels like trying to get out of a trade-
| off decision.
|
| This situation reminds me of a talk Bryan Cantrill gave on
| platform core values, and as examples he gave his interpretation
| of the platform core values of languages like C, Awk, and Scala:
| https://youtu.be/2wZ1pCpJUIM?t=349
|
| For me, platform core values that stick out for python would be
| Approachability, Simplicity, and Velocity. I understand the
| posited value mypy brings to the table, but it feels in
| contention with the original core values that made python
| appealing to begin with.
| seanmclo wrote:
| At this point, another big benefit of Python is its huge number
| of libraries. This is really attractive to lots of people who
| _also_ want static type checking.
|
| Oftentimes, the benefits of using Python in a code base that
| would benefit from static typing outweigh the costs. Especially
| when tools like MyPy exist, which aren't perfect but help
| tremendously.
| nonameiguess wrote:
| This can't be emphasized enough. I'm presently pushing mypy
| and type hints hard in a Python project (passing static
| analysis and type checks is required to get through the CI
| pipeline) and it's just a compromise. Honestly, I don't want
| a dynamic duck-typed language at all, but it is what it is.
| Right now, the single most important factor determining
| whether we have any future at all is speed to market, not the
| increased reliability you get from type safety. And Python
| just has libraries for basically everything. It's enabled us
| to spin up brand new services that do exactly what we need to
| do from scratch in a matter of hours sometimes.
|
| With competing priorities like that, this is the best we can
| do right now. The other factor is the rest of the team is
| already familiar with Python. They're mostly infrastructure
| developers. They're more likely to do everything in Bash if
| given the choice. I'm not going to teach them Scala or
| Haskell or Rust in the next three weeks before we need to
| deliver something. But I can at least teach them Python's
| optional type hints and mypy.
| mixmastamyk wrote:
| Projects tend to change over their lifetime. What might have
| been yesterday's freewheeling experimental prototype is
| tomorrow's boring mission-critical geriatric legacy. Sometimes
| this is (or should be) known in advance, but often not.
| konschubert wrote:
| Sometimes you know in advance, but you still want to optimize
| for starting and not for maintaining.
| freeone3000 wrote:
| What's the equivalent for numpy? Tensorflow? Pytorch? The
| python library support is simply unmatched: regardless of what
| the base language has as features, the proper choice for many
| projects is python.
| scudd wrote:
| I think that consideration of libraries is a good point, I
| didn't initially consider, and especially relevant for
| Python.
|
| I think the one connection I'd make back to my original point
| is that perhaps python becoming the defacto interface for
| certain libraries is still a reflection of its core values.
|
| This is especially evident with libraries like Tensorflow,
| which have interfaces for a breadth of languages, and which
| the core is implemented in C++. The reason people tend to
| reach for python to call into Tensorflow is still the core
| platform values of ease of use, rapid prototyping (imo).
| SatvikBeri wrote:
| Python is one of my least favorite languages, but it is a
| language I absolutely have to use for data work - it would be
| impractical to rewrite everything in Julia.
|
| Type annotations help a lot. They're not perfect, but for long-
| running jobs it's a huge help to catch something when PyCharm
| highlights it instead of 30 minutes into the job.
| dleslie wrote:
| It's a wee bit amusing to see the major icons of duck typing,
| python and javascript, slowly converge on the truth: static
| typing is a powerful and useful tool for mature projects, that
| should not be dismissed.
|
| Looks like mypy has discovered interfaces. Lovely.
| war1025 wrote:
| The difference between "Interfaces" as I have seen them used
| most of the time, and "Protocols" as mypy uses them, is that
| usually an Interface is generally part of the inheritance
| hierarchy.
|
| A Protocol is just the same structural subtyping (duck typing)
| Python has always preferred.
| thechao wrote:
| These protocols match the old Indiana `concepts` pretty
| precisely; they appear to be more like Haskell type-classes
| than interfaces -- they're retroactive, not inherited.
| mtone wrote:
| In Typescript classes are nominally typed (during runtime)
| while interfaces are structural, both can be combined. Seems
| similar to types and protocol in the article.
|
| So are we seeing a move towards a mix of nominal objects
| (great for core models, probably performance) and structural
| types (great for functional style and general purpose tools)
| where we can opt for one or the other as appropriate? Or is
| anything still nominal just a necessary legacy?
|
| In C# I now often find myself longing for functions that
| accept anything with an int Id {get;} property, a
| discriminated union, or similar structural typing concerns.
| Spivak wrote:
| This is one of those things that I wish type systems could just
| do for us. The pattern of every single class in Java being an an
| interface with an InterfaceImpl is exhausting.
|
| Having to fight the type system just for testing seems silly.
| Just have every class's public bits define an interface and the
| class's name in code always refer to the interface.
|
| Then testers can just implement ClassName with no fanfare and you
| stop having to write "test-friendly" classes.
| Animats wrote:
| I could never get why Python added _unchecked_ static typing.
| That seems so useless. If the compiler checks it, it can use it
| to optimize. Huge win for built-in types, like integer and float.
| CPython has to box everything. PyPy has to try to guess and use
| just-in-time recompilation when it guesses wrong.
| neolog wrote:
| For now, it improves documentation and static typechecking. But
| Mypyc is an annotated-python-to-C compiler that's in the works.
| mfgs wrote:
| Type hints are super handy when using smart IDEs like PyCharm,
| which will list or highlight (most of) your type errors. This
| alone can greatly improve confidence in your program,
| especially for more complex cases.
|
| Getting that cheaply while still having types unchecked is a
| nice compromise between improved functionality and not changing
| the language majorly.
| davesque wrote:
| Python didn't add static typing. It added arbitrary annotations
| of arguments and variables (admittedly, with the use case of
| type checking in mind). Mypy uses that for type annotations and
| checking. Here are the two main reasons I think it's useful:
|
| * To the extent that it enables some static analysis on python
| code, it has often caught a number of errors I tend to make
| while writing code
|
| * It serves as a light form of documentation that tends to stay
| more fresh because there's an automated way to check its
| consistency (run the type checker)
|
| I don't expect that it will do as perfect a job as Rust or
| C++'s type system or even that it would be there for the same
| reasons. On the other hand, it's nice to just have more info
| about what the code is expected to do in the form of type
| annotations.
| kdmytro wrote:
| Unchecked types by themselves are useful as they help my text
| editor to provide more accurate autocomplete suggestions.
| Xophmeister wrote:
| Where would you use `typing.Protocol` where you wouldn't use an
| abstract base class (or a bunch of them as mixins)? My
| understanding is that the latter is more "formal", whereas
| `Protocol` is better suited for gradual typing (i.e., moving a
| more "free-form" codebase into something MyPy won't complain
| about).
|
| The only time I've ever used `Protocol` is to define a type that
| makes it explicit that I need an object to have a `__str__`
| implementation: @runtime_checkable
| class Stringable(Protocol): """ Protocol type that
| supports the __str__ method """
| @abstractmethod def __str__(self) -> str:
| """ Return the string representation of an object """
|
| I've since learned that this is redundant, because all Python
| objects have an implicit `__str__` if one isn't specified (IIRC).
| When I _didn 't_ know this (when I implemented the above), I
| didn't use an ABC because obviously I can't guarantee all objects
| (e.g., those outside of my control) are subclasses of said ABC.
| The cases where this is true are vanishingly small, especially
| when you go big on dependency inversion.
| Conlectus wrote:
| This distinction is the same as the distinction between
| structural and nominal typing, described at the end of the
| article.
|
| Abstract base classes require everything to extend from a base-
| level object, and _also inherit it 's default implementations_.
| This is a nominal (name-based) mechanism.
|
| Protocol (structural-based) subtyping instead only declares
| what methods something has to have, without requiring it tie
| itself to either to a concrete parent class and its behaviours.
|
| Edit: since you partially address this, you ask why you'd use
| structural typing rather than an abstract base class. I'll turn
| it around and ask why you would use an abstract base class,
| which requires modifying the children accordingly, when you
| could instead use a structural type. The usual preference comes
| down to whether you want to be explicit about implementing it
| or not, and whether you want to pull in default behaviour.
| Xophmeister wrote:
| > The usual preference comes down to whether you want to be
| explicit about implementing it or not, and whether you want
| to pull in default behaviour.
|
| Thanks :) I definitely prefer things to be explicit;
| regardless of it being a "Zen of Python" mantra. That said, I
| do see the value of `Protocol` for non-annotated/legacy code,
| in that it gives you a convenient mechanism for this. I would
| be worried about it becoming a crutch, if overused, but
| definitely better than no annotations at all!
|
| (I don't understand your point regarding an ABC's default
| implementation. Isn't the point of ABCs that they're not base
| classes, but interfaces that define the methods required
| without any implementation?)
| edoceo wrote:
| Haha, I thought somehow Weird Al had made it to top.
|
| Here's the song for the curious
| https://m.youtube.com/watch?v=eOL2q8leiLw
|
| From 1985.
| mixmastamyk wrote:
| A take off on the classic Huey Lewis and the News song. Yes I
| had the Sports album, and saw Al live ~two years ago. :-D
| davepeck wrote:
| (For those living in the python + typing world, I'd definitely
| recommend trying Microsoft's pyright as an alternative to mypy.
| In my experience, it is substantially faster and produces higher
| quality error messages. It does not support mypy plugins, so --
| for instance -- Django ORM dynamism isn't well captured. But I've
| found that the Django plugin for mypy is still pretty young; when
| used on a big codebase, it both slows down type checking
| considerably _and_ regularly produces spurious errors.)
| legerdemain wrote:
| Is this not literally the distinction between an interface and an
| implementation?
|
| Like, we all know about the expression problem and how interfaces
| are hard to modify after the fact...
|
| But are test mocks and stubs not literally the prime use case for
| interfaces at the syntax level?
| morelisp wrote:
| > Is this not literally the distinction between an interface
| and an implementation?
|
| No. e.g. Java interfaces are just as nominal as classes.
|
| > But are test mocks and stubs not literally the prime use case
| for interfaces at the syntax level?
|
| Also no. If your language has structural interfaces it is often
| preferred to inheritance as a tool for polymorphism. Even in
| C++ where templates are not the easiest tool to wield it's
| often still preferable to inheritance for many reasons.
| legerdemain wrote:
| What? No. The author has a Duck, which has a quack() method.
| The quack() method is, presumably, identified by its name,
| argument types, and return type. Even in a language with
| purely nominal typing, you would make a Quacker interface,
| which is the set of things that quack() -> None, and then
| have both Duck and all of its test stubs be Quackers.
|
| This is very literally how people who write in languages
| without structural types do test stubs (and often dependency
| injection, and other mechanisms that need to select from
| among several implementations of something).
___________________________________________________________________
(page generated 2021-03-18 23:01 UTC)