[HN Gopher] PEP 810 - Explicit lazy imports
___________________________________________________________________
PEP 810 - Explicit lazy imports
Author : azhenley
Score : 225 points
Date : 2025-10-03 18:24 UTC (4 hours ago)
(HTM) web link (pep-previews--4622.org.readthedocs.build)
(TXT) w3m dump (pep-previews--4622.org.readthedocs.build)
| TheMrZZ wrote:
| Feels like a good feature, with a simple explanation, real world
| use cases, and a scoped solution (global only, pretty simple
| keyword). I like it!
| 12_throw_away wrote:
| Yeah, I think this is one of the cleanest PEPs to come around
| in quite a while, at least from the userspace perspective.
| Interested to see what happens after the traditional syntax
| bikeshedding ritual has been completed.
| oofbey wrote:
| Hopefully they learned lessons from why PEP-690 was rejected.
| I've spent quite a while trying to build this stuff for our
| codebase and it's never worked well enough to use.
| BiteCode_dev wrote:
| Agree, they really did their homework, listed edge cases, made
| practical compromises, chose not to overdo it, reworked it
| again and again quite a bit and compared it to real life
| experience.
|
| It's really beautiful work, especially since touching the back
| bone (the import system) of a language as popular as Python
| with such a diverse community is super dangerous surgery.
|
| I'm impressed.
| sedatk wrote:
| I wonder how much things would break if all imports were lazy by
| default.
| simlevesque wrote:
| Maybe nothing would break ?
|
| edit: ok well "xxx in sys.modules" would indeed be a problem
| IshKebab wrote:
| Yeah unfortunately in real world Python code people put side
| effects in their modules all the time.
| kstrauser wrote:
| In Python? I almost never see that. And I've certainly
| never signed off on a PR that did that.
| oofbey wrote:
| Maybe your code is awesome, but you're almost certainly
| using a library which does it.
| lyu07282 wrote:
| This. The most egregious example I've ever seen is the
| inflect library, takes almost 2 seconds(!) to import:
| https://github.com/jaraco/inflect/issues/212
| zahlman wrote:
| Recursively importing other modules is itself a side
| effect.
|
| In fact, all the code you see in the module is "side
| effects", in a sense. A `class` body, for example, has to
| actually run at import time, creating the class object
| and attaching it as an attribute of the module object.
| Similarly for functions. Even a simple assignment of a
| constant actually has to _run_ at module import. And all
| of these things add up.
|
| Further, if there isn't already cached bytecode available
| for the module, by default it will be written to disk as
| part of the import process. That's inarguably a side
| effect.
| eesmith wrote:
| All my code which uses import probing would fail, such as
| fallbacks: try: import module
| except ImportError: import slow_module as module
|
| Conditional support testing would also break, like having tests
| which only run if module2 is available: try:
| import module2 except ImportError: def
| if_has_module2(f): return unittest.skip("module2 not
| available")(f) else: def if_has_module2(f):
| return f @if_has_module2 class
| TestModule2Bindings(....
|
| The proto-PEP also gives an example of using
| with suppress_warnings(): import module3
|
| where some global configuration changes only during import.
|
| In general, "import this" and "import antigravity" - anything
| with import side-effects - would stop working.
|
| Oh, and as the proto-PEP points out, changes to sys.path and
| others can cause problems because of the delay between time of
| lazy import and time of resolution.
|
| Then there's code where you do a long computation then make use
| of a package which might not be present. import
| database # remember to install!! import qcd_simulation
| universe = qcd_simulation.run(seconds = 30*24*60*60)
| database.save(universe)
|
| All of these would be replaced with "import module;
| module.__name__" or something to force the import, or by an
| explicit use to __import__.
| npage97 wrote:
| You have to opt in with the lazy import keyword no matter
| what. This pep also prevents lazy import in try catch. I
| think your concern matters if 'module' itself has a lazy
| import that you want to check exists. With this you now need
| to be more rigorous in check those sub dependencies.
|
| This can already happen with non top level imports so it is
| not a necessarily a new issue, but could become more
| prevalent if there is an overall uptake in this feature for
| optional dependencies.
| eesmith wrote:
| My post was a response to the sedatk conjectural "I wonder
| how much things would break if all imports were lazy by
| default."
|
| I have zero concerns about this PEP and look forward to its
| implementation.
| zahlman wrote:
| With the LazyLoader technique I described at
| https://news.ycombinator.com/item?id=45467489 , there is no
| problem: >>> import nonexistent_module
| Traceback (most recent call last): File "<python-
| input-2>", line 1, in <module> import
| nonexistent_module File "<frozen
| importlib._bootstrap>", line 1360, in _find_and_load
| File "<frozen importlib._bootstrap>", line 1322, in
| _find_and_load_unlocked File "<frozen
| importlib._bootstrap>", line 1262, in _find_spec File
| "<python-input-0>", line 8, in find_spec
| base.loader = LazyLoader(base.loader)
| ^^^^^^^^^^^ AttributeError: 'NoneType' object has no
| attribute 'loader'
|
| The implementation should probably convert that exception
| back to ImportError for you, but the point is that the
| absence of an implementation can still be detected eagerly
| while the actual loading occurs lazily.
| eesmith wrote:
| Ahh, so you do the find first, and keep that around before
| loading.
|
| I have bad memories of using a network filesystem where my
| Python app's startup time was 5 or more seconds because of
| all the small file lookups for the import were really slow.
|
| I fixed it by importing modules in functions, only when
| needed, so the time went down to less than a second. (It
| was even better using a zipimport, but for other reasons we
| didn't use that option.)
|
| If I understand things correctly, your code would have the
| same several-second delay as it tries to resolve
| everything?
| zahlman wrote:
| Yes, if checking for a file is slow, then checking for a
| file is slow. If you need to know up front whether the
| module exists, then you can't get around using the
| "figure out whether the module exists" machinery up
| front. And if the definition of "a module exists"
| includes cases where the module is represented by a file
| whose existence you have to check for, then there's no
| getting around that, either.
|
| (Trying to do "fallback" logic with lazily-loaded modules
| is also susceptible to race conditions, of course. What
| if someone defines the module before you try to use it?)
| ashu1461 wrote:
| There will be chaos, all code bases usually have tons of side
| effects.
| Galanwe wrote:
| That seems really great, .moving imports in-line for CLI tools
| startup and test discovery has always been a pain.
| hoten wrote:
| For posterity, as the submitted link seems temporary:
| https://github.com/python/peps/pull/4622
| zahlman wrote:
| The proper permanent link is https://peps.python.org/pep-0810/
| .
| IshKebab wrote:
| Wake me up when we can import a module by relative file path.
| Spivak wrote:
| You... can? I mean in the strictest sense you're technically
| not importing by file path but if you make your folder a module
| by slapping an __init__.py in there then your relative imports
| will follow the directory tree. I think as of Python 3.3 the
| init file is optional so it will do it by default but I can't
| remember if there are still some cases where it's required. The
| only thing you can't do is go "up" to a higher directory than
| the root module.
|
| Also if that doesn't strike your fancy all of the importlib
| machinery is at your disposal and it's really not very much
| work to write an import_path() function. It's one of the
| patterns plug-in systems use and so is stable and expected to
| be used by end users. No arcane magic required.
| zahlman wrote:
| > if you make your folder a module by slapping an __init__.py
| in there then your relative imports will follow the directory
| tree.
|
| `__init__.py` has _nothing to do with_ making this work. It
| is neither necessary (as of 3.3, yes, you got it right: see
| https://peps.python.org/pep-0420/) nor sufficient (careless
| use of sys.path and absolute imports could make it so that
| the current folder hasn't been imported yet, so you can't
| even go "up into" it). The folder will already be represented
| by a module object.
|
| What `__init__.py` does is:
|
| 1. Prevents relative imports from _also_ potentially checking
| in _other_ paths.
|
| 2. Provides a space for code that runs before sub-modules,
| for example to set useful package attributes such as
| `__all__` (which controls star-imports).
| IshKebab wrote:
| > if you make your folder a module by slapping an __init__.py
| in there then your relative imports will follow the directory
| tree
|
| haha no
|
| > all of the importlib machinery is at your disposal
|
| This breaks all tooling. Awful option.
| zahlman wrote:
| Relative imports have been supported for approximately forever
| (https://stackoverflow.com/questions/72852). If you mean "by
| explicitly specifying a path string" (as opposed to a symbolic
| name), that has also been supported for approximately forever
| (https://stackoverflow.com/questions/67631). Today, the
| `importlib` standard library exposes all the steps of the
| import process -- including figuring out where the source code
| is, checking the `sys.modules` cache etc. -- and lets you hook
| into everything (in particular, you can bypass the entire
| "finder" process and turn a file path into a dummy "spec" which
| is fed to a "loader").
|
| The flexibility of this system also entails that you can in
| effect define a completely new programming language, describe
| the process of creating Python bytecode from your custom
| source, and have clients transparently `import` source in the
| other language as a result. Or you can define an import process
| that grabs code from the Internet (not that it would be a good
| idea...).
|
| If you mean "by explicitly specifying a relative path, and
| having it be interpreted according to the path of the current
| module's source code", well first you have to consider that
| _the current module isn 't required to have source code_. But
| if it does, then generally it will have been loaded with the
| default mechanism, which means the module object will have a
| `__file__` attribute with an absolute path, and you just set
| your path relative to that.
| IshKebab wrote:
| Nope. Relative imports work by relative _package_ path, which
| is not at all the same. Often when you run Python you don 't
| even have a package path.
|
| Using `importlib` is a horrible hack that breaks basically
| all tooling. You very very obviously are not supposed to do
| that.
| zahlman wrote:
| > Relative imports work by relative package path, which is
| not at all the same.
|
| Exactly. This gives you the flexibility to distribute a
| complex package across multiple locations.
|
| > Often when you run Python you don't even have a package
| path.
|
| Any time you successfully import foo.bar, you necessarily
| have imported foo (because bar is an attribute of that
| object!), and therefore bar can `from . import` its
| siblings.
|
| > Using `importlib` is a horrible hack that breaks
| basically all tooling. You very very obviously are not
| supposed to do that.
|
| It is exactly as obvious (and true) that you are not
| "supposed to", in the exact same sense, directly specify
| where on disk the source code file you want to import is.
| After all, this constrains the import process to use a
| source code file. (Similarly if you specify a .pyc
| directly.) Your relative path doesn't necessarily make any
| sense after you have packaged and distributed your code and
| someone else has installed it. It definitely doesn't make
| any sense if you pack all your modules into a zipapp.
| wrmsr wrote:
| If anyone's interested I've implemented a fairly user friendly
| lazy import mechanism in the form of context managers
| (auto_proxy_import/init) at https://pypi.org/project/lazyimp/
| that I use fairly heavily. Syntactically it's just wrapping
| otherwise unmodified import statements in a with block, so tools
| 'just work' and it can be easily disabled or told to import
| eagerly for debugging. It's powered primarily by swapping out the
| frame's f_builtins in a cext (as it needs more power than
| importlib hooks provide), but has a lame attempt at a threadsafe
| pure python version, and a super dumb global hook version.
|
| I was skeptical and cautious with it at first but I've since
| moved large chunks of my codebase to it - it's caused
| surprisingly few problems (honestly none besides forgetting to
| handle some import-time registration in some modules) and the
| speed boost is addictive.
| film42 wrote:
| I'm a fan because it's something you can explicitly turn on and
| off. For my Docker based app, I really want to verify the
| completeness of imports. Preferably, at build and test time. In
| fact, most of the time I will likely disable lazy loading
| outright. But, I would really appreciate a faster loading CLI
| tool.
|
| However, there is a pattern in python to raise an error if, say,
| pandas doesn't have an excel library installed, which is fine. In
| the future, will maintainers opt to include a bunch of unused
| libraries since they won't negatively impact startup time? (Think
| pandas including 3-4 excel parsers by default, since it will only
| be loaded when called). It's a much better UX, but, now if you
| opt out of lazy loading, your code will take longer to load than
| without it.
| the__alchemist wrote:
| Does this fix the circular imports problem that comes up if you
| don't structural your programs in a hierarchical way?
| colpabar wrote:
| This is what I thought of too. I really only know python, do
| other languages not have that issue? In python it does not seem
| like a "problem" to me - whenever I have seen circular import
| issues it is because the code is organized poorly. I worry that
| this feature will lead to devs "fixing" circular import issues
| by using lazy imports.
| matsemann wrote:
| Sometimes it's hard to avoid cyclic imports, without blaming
| the design. Like if a Parent has a Child, and the Child needs
| to know of the parent. Only way to solve that in python is to
| put everything in the same file, which also feels like bad
| deisgn.
| colpabar wrote:
| I would say in that case, the Parent and Child shouldn't
| need to know about each other - some kind of handler in a
| different file should.
|
| Although I guess that doesn't work in all cases, like
| defining foreign key relationships when using an orm (like
| sqlalchemy) for example. But in the orm case, the way to
| get around that is... lazy resolution :^)
| the__alchemist wrote:
| Interesting. I find that I always have this problem in any
| non-trivial python project, and don't find this to be due to
| poorly organized code. I have only seen this requirement in
| Python.
|
| IME circular import errors aren't due to poor organization;
| they're due to an arbitrary restriction Python has.
| Spivak wrote:
| It should, when you do lazy imports on your own it fixes the
| problem.
| zahlman wrote:
| Even with eager importing there is only a "circular import
| problem" if you try to import names `from` the modules -- as I
| pointed out a few days ago
| (https://news.ycombinator.com/item?id=45400783).
| keeganpoppen wrote:
| YES!!!
| behnamoh wrote:
| Make Python more like Haskell, LFG!
| charliermarsh wrote:
| Lazy imports have been proposed before, and were rejected most
| recently back in 2022: https://discuss.python.org/t/pep-690-lazy-
| imports-again/1966.... If I recall correctly, lazy imports are a
| feature supported in Cinder, Meta's version of CPython, and the
| PEP was driven by folks that worked on Cinder. Last time, a lot
| of the discussion centered around questions like: Should this be
| opt-in or opt-out? At what level? Should it be a build-flag for
| CPython itself? Etc. The linked post suggests that the Steering
| Council ultimately rejected it because of the complexity it would
| introduce to have two divergent "modes" of importing.
|
| I hope this proposal succeeds. I would love to use this feature.
| dheera wrote:
| Oof. I wish they could support version imports
| import torch==2.6.0+cu124 import numpy>=1.2.6
|
| and support having multiple simultaneous versions of any Python
| library installed. End this conda/virtualenv/docker/bazel/[pick
| your poison] mess
| lblume wrote:
| uv is good.
| fouronnes3 wrote:
| The mess has ended thanks to uv.
| maxloh wrote:
| You could do that with uv. # /// script
| # dependencies = [ # "requests<3", # "rich",
| # ] # /// import requests from
| rich.pretty import pprint resp =
| requests.get("https://peps.python.org/api/peps.json")
| data = resp.json() pprint([(k, v["title"]) for k, v in
| data.items()][:10])
| zahlman wrote:
| It's been explained many times before why this is not
| possible: the library _doesn 't actually have_ a version
| number. The _distribution_ of source code on PyPI has a
| version number, but the name of this is _not connected_ to
| the name of any module or package you import in the source
| code. The distribution can validly define _zero or more_
| modules (packages are a subset of modules, represented using
| the same type in the Python type system).
|
| You got three other responses before me all pointing at uv.
| _They are all wrong_ , because uv _did not_ introduce this
| functionality to the Python ecosystem. It is a standard
| defined by https://peps.python.org/pep-0723/, implemented by
| multiple other tools, notably pipx.
| giancarlostoro wrote:
| You could absolutely have this be part of the language in
| any regard. The question then becomes how does one
| implement it in a reasonable way. I think every package
| should have a __version__ property you should be able to
| call, then you could have versioned imports.
|
| In fact there's already many packages already defining
| __version__ at a package level.
|
| https://packaging.python.org/en/latest/discussions/versioni
| n...
|
| Edit: What they are solving with UV is at the moment of
| standing up an environment, but you're more concerned about
| code-level protection, where are they're more concerned
| about environment setup protection for versioning.
| BugsJustFindMe wrote:
| > _It 's been explained many times before why this is not
| possible: the library doesn't actually have a version
| number. The distribution of source code on PyPI has a
| version number, but the name of this is not connected to
| the name of any module or package you import in the source
| code._
|
| You're making the common mistake of conflating how things
| currently work with how things could work if the
| responsible group agrees to change how things work.
| Something being the way it is right now is not the same as
| something else being "not possible".
| zahlman wrote:
| No, changing this breaks the world. A huge fraction of
| PyPI becomes completely invalid overnight, and the rest
| fails the expected version checks. Not to mention that
| the language is fundamentally designed around the
| expectation that modules are singleton. I've written
| about this at length before but I can't easily find it
| right now (I have way too many bookmarks and not nearly
| enough idea how to organize them).
|
| Yes, you absolutely can create a language that has syntax
| otherwise identical to Python (or at least damn close)
| which implements a feature like this. No, you cannot just
| replace Python with it. If the Python ecosystem just
| accepted that clearly better things were clearly better,
| and started using them promptly, we wouldn't have
| https://pypi.org/project/six/ making it onto
| https://pypistats.org/top (see also
| https://sethmlarson.dev/winning-a-bet-about-six-the-
| python-2...).
| hnlmorg wrote:
| The hard part is making the change. Adding an escape
| hatch so older code still works is easy in comparison.
|
| Nobody is claiming this is a trivial problem to solve but
| its also not an impossible problem. Other languages have
| managed to figure out how to achieve this and still
| maintain backwards compatibility.
| zahlman wrote:
| You're welcome to bring a concrete proposal to, e.g.,
| https://discuss.python.org/c/ideas/6 , or ask around the
| core devs to find a PEP sponsor.
|
| Note that you will be expected to have familiarized
| yourself generally with previous failed proposals of this
| sort, and proactively considered all the reasonably
| obvious corner cases.
| hnlmorg wrote:
| Tbh you're just reinforcing mine and others point there
| that the issue isn't a technical one.
| johannes1234321 wrote:
| If vereioned imports were added to the language versioned
| library support obviously would have to become part of the
| language as well.
|
| However it isn't trivial. First problem coming to my mind:
|
| module a importing first somelib>=1.2.0 and then b and b
| then requiring somelib>1.2.1 and both being available, will
| it be the same or will I have a mess from combining?
| driggs wrote:
| > It's been explained many times before why this is not
| possible: the library doesn't actually have a version
| number.
|
| Not _possible_? Come on.
|
| Almost everyone already uses one of a small handfull of
| conventional ways to specify it, eg `__version__`
| attribute. It's long overdue that this be standardized so
| library versions can reliably be introspected at runtime.
|
| Allowing multiple versions to be installed side-by-side and
| imported explicitly would be a massive improvement.
| tkfoss wrote:
| How would version imports be handled across the codebase?
| Also, what do you gain with those over PEP 723 - Inline
| script metadata? https://packaging.python.org/en/latest/speci
| fications/inline...
| BiteCode_dev wrote:
| Especially since it is opt in, with various level of
| granularity, and a global off switch. Very well constructed
| spec given the constraints.
| forrestthewoods wrote:
| I don't want lazy imports. That's just makes performance shitty
| later and harder to debug. It's a hacky workaround.
|
| What I want is for imports to not suck and be slow. I've had
| projects where it was faster to compile and run C++ than launch
| and start a Python CLI. It's so bad.
| Alir3z4 wrote:
| I wish all imports were lazy by default.
|
| I know/heard there are "some" (which I haven't seen by the way)
| libraries that depend on import side effects, but the advantage
| is much bigger.
|
| First of all, the circular import problem will go away,
| especially on type hints. Although there was a PEP or recent
| addition to make the annotation not not cause such issue.
|
| Second and most important of all, is the launch time of Python
| applications. A CLI that uses many different parts of the app has
| to wait for all the imports to be done.
|
| The second point becomes a lot painful when you have a large
| application, like a Django project where the auto reload becomes
| several seconds. Not only auto reload crawls, the testing cycle
| is slow as well. Every time you want to run test command, it has
| to wait several seconds. Painful.
|
| So far the solution has been to do the lazy import by importing
| inside the methods where it's required. That is something, I
| never got to like to be honest.
|
| Maybe it will be fixed in Python 4, where the JIT uses the type
| hints as well /s
| f311a wrote:
| Lazy imports mean late errors. Fail fast is a good design
| principle.
| zahlman wrote:
| Top-level code should not be able to fail except in
| incredibly deterministic ways that are tested during
| development. Failing fast is not as good as not failing at
| all. Lazy imports mean the power to avoid importing things
| that don't need to be imported at all on this run. Good
| design also cares about performance to some extent. On my
| machine, asking pip to do _literally nothing_ takes several
| times as long as creating a new virtual environment
| --without-pip .
| josefx wrote:
| Python really seems like a bad fit for that. So your imports
| succeed, what now? Do they have all the functions or fields
| your program needs? Those are still resolved at the last
| possible moment. If you want to be sure your program actually
| runs you will have to write and run tests with and without
| lazy imports.
| f311a wrote:
| I don't like the idea of introducing a new keyword. We need a
| backward compatible solution. I feel like Python needs some kind
| of universal annotation syntax such as in go (comments) or in
| Rust (macros). New keyword means all parsers, lsps, editors
| should be updated.
|
| I'm pretty sure there will be new keywords in Python in the
| future that only solve one thing.
| Alir3z4 wrote:
| Me neither.
|
| Introducing new keyword has become a recent thing in Python.
|
| Seems Python has a deep scare since Python2 to Python3 time and
| is scared to do anything that causes such drama again.
|
| For me, the worst of all is "async". If 2to3 didn't cause much
| division, the async definitely divided Python libraries in 2.
| Sync and Async.
|
| Maybe if they want backward compatible solution, this can be
| done by some compile or runtime flag like they did with free
| threading no-gil.
| sigmoid10 wrote:
| Async or not in modules is a huge pain in the ass for web-
| dev, but thankfully a lot of Python still isn't web dev. As a
| Data Scientist you can live your life peacefully without ever
| worrying about this. But I can see why web devs like it, even
| though personally I really don't want to see anymore
| Javascript sneaking inty my Python. Especially when there is
| already support for IO bound concurrency elsewhere in the
| language. If I want to do JS syntax, I'll fucking use JS. And
| I really don't want to see Python go the C++ route where it
| just wants to be everything and do everything so that you end
| up with so many possible approaches to the same problem that
| two devs can't read each others code anymore.
| vvpan wrote:
| I'm relatively new to Python - how does one do concurrent
| IO without async/await?
|
| My main complaint, though, about Python async is - because
| it is opt-in I never know if I forgot a sync IO call
| somewhere that will block my worker. JS makes everything
| async by default and there is effectively no chance of
| blocking.
| blibble wrote:
| take your pick: threads, fork, non-explicit coroutines
| (gevent), io multiplexing (select/poll/epoll/kqueue),
| virtual threads/goroutines
| baq wrote:
| The same way you always did it: sync io in threads.
| Caveats are similar to async/await, but stack traces
| don't suck.
| BiteCode_dev wrote:
| Async was a soft keyword for many, many years in order to
| maintain compat. And before that, asyncio used yield to be
| even more compatible.
|
| They took a decade to solidify that. At some point, you have
| to balance evolution and stability. For a language as popular
| as Python, you can not break the world every week, but you
| can't stay put for 5 years.
| ashu1461 wrote:
| Not sure if this can be made backward compatible.
|
| Right now all the imports are getting resolved at runtime
| example in a code like below from file1 import
| function1
|
| When you write this, the entire file1 module is executed right
| away, which may trigger side effects.
|
| If lazy imports suddenly defer execution, those side effects
| won't run until much later (or not at all, if the code path
| isn't hit). That shift in timing could easily break existing
| code that depends on import-time behavior.
|
| To avoid using lazy, this there is also a proposal of adding
| the modules you want to load lazily to a global
| `__lazy_modules__` variable.
| bkettle wrote:
| I think they mean backwards-compatible syntax-wise, rather
| than actually allowing this feature to be used on existing
| code. If I'm understanding correctly they would prefer for
| the Python grammar to stay the same (hence the comment about
| updating parsers and IDEs).
|
| But I don't think I really agree, the extensible annotation
| syntaxes they mention always feel clunky and awkward to me.
| For a first-party language feature (especially used as often
| as this will be), I think dedicated syntax seems right.
| f311a wrote:
| I was talking about syntax. I'm pretty sure there will be new
| features that will require a new keyword or syntax given the
| speed of Python growth. It can be universal, for example same
| as decorator, but it can be applied anywhere.
| from lazy import make_lazy from package import module
| @make_lazy @local @nogil
|
| Let's say this syntax gets introduced in Python 3.16. The
| @nogil feature can be introduced in 3.17. If such code is
| running in Python 3.16, the @nogil marker will be ignored.
|
| The problem with new keywords is that you have to stick to
| the newest Python version every time a new keyword is added.
| Older Python versions will give a syntax error. It's a big
| problem for libraries. You need to wait for 3-5 years before
| adding it to a library. There are a lot of people who still
| use Python 3.8 from 2019.
| ashu1461 wrote:
| Understood make sense
| baq wrote:
| Yeah I like the feature but hate the keyword. Dunder lazy
| imports sounds good enough imho.
| BiteCode_dev wrote:
| They thought about backward compatibility, and offer an
| alternative syntax that use no keyword for library that want to
| activate it yet stay compact with old version. It's already in
| the spec.
| jeremyscanvic wrote:
| Really excited about this - we've recently been struggling with
| making imports lazy without completely messing up the code in
| DeepInverse https://deepinv.github.io/deepinv/
| Spivak wrote:
| I love the feature but I really dislike using the word lazy as a
| new language keyword. It just feels off somehow. I think maybe
| defer might be a better word. It is at least keeps the grammar
| right because it would be lazily. lazily import
| package.foo vs defer import package.foo
|
| Also the grammar is super weird for from imports.
| lazy from package import foo vs. from package
| defer import foo.
| the_mitsuhiko wrote:
| I like the approach of ES6 where you pull in bindings that are
| generally lazily resolved. That is IMO the approach that should
| be the general strategy for Python.
| 9dev wrote:
| The import/require schism was necessary precisely of all the
| issues with module side effects the python community is
| rediscovering now. I suppose they won't get around a similar
| solution eventually; but judging from years of discussions in
| the JS world, this is going to be dragging on for a while.
| simonw wrote:
| Love this. My https://llm.datasette.io/ CLI tool supports
| plugins, and people were complaining about really slow start
| times even for commands like "llm --help" - it turned out there
| were popular plugins that did things like import pytorch at the
| base level, so the entire startup was blocked on heavy imports.
|
| I ended up adding a note to the plugin author docs suggesting
| lazy loading inside of functions -
| https://llm.datasette.io/en/stable/plugins/advanced-model-pl... -
| but having a core Python language feature for this would be
| really nice.
| zahlman wrote:
| You can implement this from your tool today:
| https://news.ycombinator.com/item?id=45467489
|
| Note that this is global to the entire process, so for example
| if you make an import of Numpy lazy this way, then so are the
| imports of all the sub-modules. Meaning that large parts of
| Numpy might not be imported at all if they aren't needed, but
| pauses for importing individual modules might be distributed
| unpredictably across the runtime.
|
| Edit: from further experimentation, it appears that if the
| source does something like `import foo.bar.baz` then `foo` and
| `foo.bar` will still be eagerly loaded, and only `foo.bar.baz`
| itself is deferred. This might be part of what the PEP meant by
| "mostly". But it might also be possible to improve my
| implementation to fix that.
| peterfirefly wrote:
| Parse the command line and do things like "--help" without
| doing the imports.
|
| Only do imports when you know you need them -- or as an easy
| approximation, only if the easy command line options have been
| handled and there's still something to do.
| stavros wrote:
| Well yes, or you can just use the `lazy` keyword, when it
| makes it into core.
| mr_mitm wrote:
| In the llm project, plugins can modify the command line
| arguments, so it's not that simple.
| simonw wrote:
| Yea, that's the core problem here: plugins can add new CLI
| subcommands, which means they all need to be loaded on
| startup.
|
| https://llm.datasette.io/en/stable/plugins/plugin-
| hooks.html...
| aftbit wrote:
| We tend to prefer explicit top-level imports specifically because
| they reveal dependency problems as soon as the program starts,
| rather than potentially hours or days later when a specific code
| path is executed.
| maest wrote:
| Who is "we"?
| aftbit wrote:
| My current team at my current company (see bio if you're
| really interested), though I should say I'm not authorized to
| speak on behalf of my employer, so I should really say
| something more like "I".
| bityard wrote:
| I haven't fully digested the PEP but perhaps there would be a
| command-line flag or external tool for dependency validation, a
| bit like how there are external tools for type annotations?
| ashu1461 wrote:
| Right, Not sure why but a lot of code which claude generates
| also comes with local dependencies vs globally declared import
| statements. Don't promote that pattern because
|
| - It reduces visibility into a module's dependencies.
|
| - It increases the risk of introducing circular dependencies
| later on.
| zahlman wrote:
| As a counterpoint, having all the imports automatically
| deferred would instantly dramatically speed up pip for short
| tasks. $ time pip install --disable-pip-
| version-check ERROR: You must give at least one
| requirement to install (see "pip help install") real
| 0m0.399s user 0m0.360s sys 0m0.041s
|
| Almost all of this time is spent importing (and later
| unloading) ultimately useless vendored code. From my testing
| (hacking the wrapper script to output some diagnostics),
| literally about 500 modules get imported in total (on top of
| the baseline for a default Python process), including almost a
| hundred modules related to Requests and its dependencies, even
| though no web request was necessary for this command.
| aftbit wrote:
| I think this makes a ton of sense in the very specific narrow
| use case of python CLI tools. For a web app or other long-
| lived process, startup time is typically not of extreme
| concern, and having more simplicity and legibility to the
| import process seems better.
|
| That's not to say this PEP should not be accepted. One could
| always apply a no-lazy-imports style rule or disable it via
| global lazy import control.
|
| https://peps.python.org/pep-0810/#global-lazy-imports-
| contro...
| zahlman wrote:
| It saddens me to think that the use case of "python CLI
| tools" is thought of as anything like "very specific" or
| "narrow".
| stefan_ wrote:
| These tools already have a million ways to avoid this
| slowdown. Is it so much to ask their authors to care?
| rplnt wrote:
| Remember mercurial? Me neither. But what I remeber is this
| article I've read about all the hacks they had to do to achieve
| reasonable startup time for CLI in python. And the no #1 cause
| was loading the whole world you don't ever need. As I recall they
| somehow monkeypatched the interpreter to ignore imports and just
| remember their existence until they were actually needed, at
| which point the import happened. So all the dead paths were just
| skipped.
| yegle wrote:
| I recall chg was a must.
| jmward01 wrote:
| This is needed, but I don't like new keywords. What I would love,
| for many reasons, is if we could decorate statements. Then things
| like:
|
| import expensive_module
|
| could be:
|
| @lazy
|
| import expensive_module
|
| or you could do:
|
| @retry(3)
|
| x = failure_prone_call(y)
|
| lazy is needed, but maybe there is a more basic change that could
| give more power with more organic syntax, and not create a new
| keyword that is special purpose (and extending an already special
| purpose keyword)
| jmward01 wrote:
| As a side note, It would be great to have a 'preview' for HN
| comments. I updated the above because I forgot to add line
| breaks. Sigh. I bet 'preview' would stop a lot of not well
| thought out comments too.
| progval wrote:
| For what it's worth, you can already do: x =
| retry(3)(failure_prone_call)(y)
| zahlman wrote:
| And, for that matter, expensive_module =
| lazy(importlib.import_module)('expensive_module') .
| lyu07282 wrote:
| Would this statement decorator then manipulate the AST of the
| following statement or how would that work?
| waynesonfire wrote:
| why does this have to be a syntax feature and not a lazy code
| loader at the intepreter level?
| Boxxed wrote:
| Ugh...I like the idea, but I wish lazy imports were the default.
| Python allows side effects in the top level though so that would
| be a breaking change.
|
| Soooo instead now we're going to be in a situation where you're
| going to be writing "lazy import ..." 99% of the time: unless
| you're a barbarian, you basically never have side effects at the
| top level.
| zahlman wrote:
| > The standard library provides the LazyLoader class to solve
| some of these inefficiency problems. It permits imports at the
| module level to work _mostly_ like inline imports do.
|
| The use of these sorts of Python import internals is highly non-
| obvious. The Stack Overflow Q&A I found about it
| (https://stackoverflow.com/questions/42703908/) doesn't result in
| an especially nice-looking UX.
|
| So here's a proof of concept in existing Python for getting all
| imports to be lazy automatically, with no special syntax for the
| caller: import sys import threading #
| needed for python 3.13, at least at the REPL, because reasons
| from importlib.util import LazyLoader # this has to be eagerly
| imported! class LazyPathFinder(sys.meta_path[-1]): # <class
| '_frozen_importlib_external.PathFinder'> @classmethod
| def find_spec(cls, fullname, path=None, target=None):
| base = super().find_spec(fullname, path, target)
| base.loader = LazyLoader(base.loader) return base
| sys.meta_path[-1] = LazyPathFinder
|
| We've replaced the "meta path finder" (which implements the logic
| "when the module isn't in sys.modules, look on sys.path for
| source code and/or bytecode, including bytecode in __pycache__
| subfolders, and create a 'spec' for it") with our own wrapper.
| The "loader" attached to the resulting spec is replaced with an
| importlib.util.LazyLoader instance, which wraps the base
| PathFinder's provided loader. When an import statement actually
| imports the module, the name will actually get bound to a <class
| 'importlib.util._LazyModule'> instance, rather than an ordinary
| module. Attempting to access any attribute of this instance will
| trigger the normal module loading procedure -- which even
| replaces the global name.
|
| Now we can do: import this # nothing shows up
| print(type(this)) # <class 'importlib.util._LazyModule'>
| rot13 = this.s # the module is loaded, printing the Zen
| print(type(this)) # <class 'module'>
|
| That said, I don't know what the PEP means by "mostly" here.
| comex wrote:
| I don't hate it but I don't love it. It sounds like everyone will
| start writing `lazy` before essentially every single import, with
| rare exceptions where eager importing is actually needed. That
| makes Python code visually noisier. And with no plan to ever
| change the default, the noise will stay forever.
|
| I would have preferred a system where modules opt in to being
| lazy-loaded, with no extra syntax on the import side. That would
| simplify things since only large libraries would have to care
| about laziness. To be fair, in such a design, the interpreter
| would have to eagerly look up imports on the filesystem to decide
| whether they should be lazy-loaded. And there are probably other
| downsides I'm not thinking of.
| BiteCode_dev wrote:
| We heard that about types, the walrus, asyncio, dataclasses and
| so much more. But it didn't happen, if people don't need
| something (and many don't know it exists or what it does), it's
| unlikely they use it.
|
| In fact, half of the community basically uses only a modernized
| set of python 2.4 features and that's one of the beauties of
| the language. You don't need a lot to be productive, and if you
| want more, you can optionally reach for it.
|
| It has worked very well for the last 2 decades and it will
| likely work again.
| hnlmorg wrote:
| People said the same about Perl and its "there's more than
| one way to do things" ethos, which gained much criticism.
|
| Same is true for C++.
|
| In this specific case, I think a lazy load directive isn't a
| bad addition. But one does need to be careful about adding
| new language features just because you have an active
| community.
| comex wrote:
| Perhaps people won't use it. But I for one _want_ it to be
| used, and I will certainly be using it in my own code if the
| PEP is accepted. Startup time is very important to me, more
| important than the cost of making the code noisier. I just
| wish I didn 't have to make that tradeoff in the first place.
| Spivak wrote:
| I don't think this makes sense to be on the module side, the
| caller is the one with the information as to whether the module
| can or needs to be lazily loaded. There's nothing really for
| the module being imported to decide, every module can be lazily
| loaded. Even if it has side effects the caller may want to
| defer those as well.
| syncsynchalt wrote:
| I think side-effects are exactly the problem, you can't have
| the runtime default to lazy-loading all modules without
| breaking code that e.g. relies on side effects running before
| thread creation or forking.
| comex wrote:
| > Even if it has side effects the caller may want to defer
| those as well.
|
| But that's rare, and could be handled with existing
| workarounds.
|
| Normally, a module needs to be eagerly imported if and only
| if it has side effects.
| lenkite wrote:
| Wish pyproject.toml was enhanced to specify lazy loading via
| regexs.
| veber-alex wrote:
| I would gladly take a command line flag that I can pass to
| python that makes all module loading lazy.
|
| Unless you are writing scripts or very simple stuff running
| side effects when modules are loaded should be avoided at all
| cost anyway.
| xdfgh1112 wrote:
| That's already part of the PIP. There is a flag to enable
| lazy imports for all possible imports.
| charliermarsh wrote:
| The PEP includes the ability to enable (or disable) lazy
| imports globally via a command-line flag or environment
| variable, in addition to the import syntax.
| mekoka wrote:
| If everyone starts favoring lazy imports with not much fuss
| then it means that lazy should have been the default behavior
| and _eager_ is the keyword we 're missing. This isn't the first
| time Python revisits this paradigm. Many constructs that used
| to eagerly produce lists in v2 were turned into generators in
| v3 with next to no problems.
| enriquto wrote:
| what is the point of this? you can just import inside function
| definitions: def antislash(A, b):
| from numpy.linalg import solve return solve(A, b)
|
| thus numpy.linalg is only imported the first time you call the
| antislash function. Much cleaner than a global import.
|
| Ignore wrong traditions. Put _all_ imports in the innermost
| scopes of your code!
| gorgoiler wrote:
| I don't really agree with the premise:
|
| > _The dominant convention in Python code is to place all imports
| at the beginning of the file. This avoids repetition, makes
| import dependencies clear and minimizes runtime overhead._
|
| > _A somewhat common way to delay imports is to move the imports
| into functions, but this practice requires more work [and]
| obfuscates the full set of dependencies for a module._
|
| The first part is just saying the traditions exist because the
| traditions have always existed. Traditions are allowed to change!
|
| The second part is basically saying if you do your own logic-
| based lazy imports (inline imports in functions) then you're
| going against the traditions. Again, traditions are allowed to
| change!
|
| The point about the import graph being obfuscated would ring more
| true if Python didn't already provide lightning fast static
| analysis tools like ast. If you care about import graphs at the
| module level then you're probably already automating everything
| with ast anyway, at which point you just walk the whole tree
| looking for imports rather than the top level.
|
| So, really, the whole argument for a new _lazy_ keyword (instead
| of inlining giant imports where they are needed) is because
| people like to see _import pytorch_ at the top of the file, and
| baulk at seeing it -- and will refuse to even look for it! --
| anywhere else? Hmmm.
|
| What _does_ seem like a pain in the ass is having to do this kind
| of repetitive crap (which they mention in their intro):
| def f1(): import pytorch ... def f2():
| import pytorch ... def f3(): import
| pytorch ...
|
| But perhaps the solution is a pattern where you put all your
| stuff like that in your own module and it's _that_ module which
| is lazy loaded instead?
| nodesocket wrote:
| Won't this have a very noticeable performance hit on the first
| request? Thinking web frameworks like Flask and Django.
| thayne wrote:
| One thing the PEP doesn't really talk about, and that I find very
| annoying is that many python linters will complain if you don't
| put all of your imports at the top of the file, so you get lint
| warnings if you do the most obvious way to implement lazy
| imports.
|
| And that is actually a problem for more than just performance. In
| some cases, importing at the top might actually just fail. For
| example if you need a platform specific library, but only if it
| is running on that platform.
| thijsvandien wrote:
| Very unfortunate how _from json lazy import dumps_ would result
| in backward compatibility issues. It reads much better and makes
| it easier to search for lazy imports, especially if in the future
| something else becomes optionally lazy as well.
| mekoka wrote:
| Crossing my fingers that it goes through this time. Been in the
| top 3 of my Python wishlist for nearly a decade.
| nilslindemann wrote:
| [delayed]
___________________________________________________________________
(page generated 2025-10-03 23:00 UTC)