[HN Gopher] PEP 810 - Explicit lazy imports
___________________________________________________________________
PEP 810 - Explicit lazy imports
Author : azhenley
Score : 387 points
Date : 2025-10-03 18:24 UTC (1 days ago)
(HTM) web link (peps.python.org)
(TXT) w3m dump (peps.python.org)
| TheMrZZ wrote:
| Feels like a good feature, with a simple explanation, real world
| use cases, and a scoped solution (global only, pretty simple
| keyword). I like it!
| 12_throw_away wrote:
| Yeah, I think this is one of the cleanest PEPs to come around
| in quite a while, at least from the userspace perspective.
| Interested to see what happens after the traditional syntax
| bikeshedding ritual has been completed.
| oofbey wrote:
| Hopefully they learned lessons from why PEP-690 was rejected.
| I've spent quite a while trying to build this stuff for our
| codebase and it's never worked well enough to use.
| dahart wrote:
| 690 is mentioned a few times, including a FAQ note on the
| differences
|
| https://pep-previews--
| 4622.org.readthedocs.build/pep-0810/#f...
|
| Q: How does this differ from the rejected PEP 690?
|
| A: PEP 810 takes an explicit, opt-in approach instead of PEP
| 690's implicit global approach. The key differences are:
|
| Explicit syntax: lazy import foo clearly marks which imports
| are lazy.
|
| Local scope: Laziness only affects the specific import
| statement, not cascading to dependencies.
|
| Simpler implementation: Uses proxy objects instead of
| modifying core dictionary behavior.
| BiteCode_dev wrote:
| Agree, they really did their homework, listed edge cases, made
| practical compromises, chose not to overdo it, reworked it
| again and again quite a bit and compared it to real life
| experience.
|
| It's really beautiful work, especially since touching the back
| bone (the import system) of a language as popular as Python
| with such a diverse community is super dangerous surgery.
|
| I'm impressed.
| sedatk wrote:
| I wonder how much things would break if all imports were lazy by
| default.
| simlevesque wrote:
| Maybe nothing would break ?
|
| edit: ok well "xxx in sys.modules" would indeed be a problem
| IshKebab wrote:
| Yeah unfortunately in real world Python code people put side
| effects in their modules all the time.
| kstrauser wrote:
| In Python? I almost never see that. And I've certainly
| never signed off on a PR that did that.
| oofbey wrote:
| Maybe your code is awesome, but you're almost certainly
| using a library which does it.
| lyu07282 wrote:
| This. The most egregious example I've ever seen is the
| inflect library, takes almost 2 seconds(!) to import:
| https://github.com/jaraco/inflect/issues/212
| zahlman wrote:
| Recursively importing other modules is itself a side
| effect.
|
| In fact, all the code you see in the module is "side
| effects", in a sense. A `class` body, for example, has to
| actually run at import time, creating the class object
| and attaching it as an attribute of the module object.
| Similarly for functions. Even a simple assignment of a
| constant actually has to _run_ at module import. And all
| of these things add up.
|
| Further, if there isn't already cached bytecode available
| for the module, by default it will be written to disk as
| part of the import process. That's inarguably a side
| effect.
| flykespice wrote:
| Side-effect means you're changing state outside the scope
| the code is running.
|
| Sure thing you can declare globals variable and run
| anything on a module file global scope (outside funcs and
| class body), but even that 'global' scope is just an
| illusion, and everything declared there, as yourself
| said, is scoped to the module's namespace
|
| (and you can't leak the 'globals' when importing the
| module unless you explicity do so 'from foo import *'.
| Think of python's import as eval but safer because it
| doesn't leaks the results from the module execution)
|
| So for a module to have side-effect (for me) it would
| either:
|
| - Change/Create attributes from other modules
|
| - Call some other function that does side-effect
| (reflection builtins? IO stuff)
| IshKebab wrote:
| Often it's things like registering functions in a global
| registry, or patching things about how Python works (e.g.
| adding stuff to the module path).
| kstrauser wrote:
| You're right. "Side effect" here means that your
| program's behavior changes simply by importing something,
| eg by monkey patching something else.
|
| If's not:
|
| * Importing other modules.
|
| * Taking a long time to import.
|
| * Writing .pyc files.
|
| If any program can "import foo" and still execute exactly
| the same bytecode afterward as before, you can say that
| foo doesn't have side effects.
| eesmith wrote:
| All my code which uses import probing would fail, such as
| fallbacks: try: import module
| except ImportError: import slow_module as module
|
| Conditional support testing would also break, like having tests
| which only run if module2 is available: try:
| import module2 except ImportError: def
| if_has_module2(f): return unittest.skip("module2 not
| available")(f) else: def if_has_module2(f):
| return f @if_has_module2 class
| TestModule2Bindings(....
|
| The proto-PEP also gives an example of using
| with suppress_warnings(): import module3
|
| where some global configuration changes only during import.
|
| In general, "import this" and "import antigravity" - anything
| with import side-effects - would stop working.
|
| Oh, and as the proto-PEP points out, changes to sys.path and
| others can cause problems because of the delay between time of
| lazy import and time of resolution.
|
| Then there's code where you do a long computation then make use
| of a package which might not be present. import
| database # remember to install!! import qcd_simulation
| universe = qcd_simulation.run(seconds = 30*24*60*60)
| database.save(universe)
|
| All of these would be replaced with "import module;
| module.__name__" or something to force the import, or by an
| explicit use to __import__.
| npage97 wrote:
| You have to opt in with the lazy import keyword no matter
| what. This pep also prevents lazy import in try catch. I
| think your concern matters if 'module' itself has a lazy
| import that you want to check exists. With this you now need
| to be more rigorous in check those sub dependencies.
|
| This can already happen with non top level imports so it is
| not a necessarily a new issue, but could become more
| prevalent if there is an overall uptake in this feature for
| optional dependencies.
| eesmith wrote:
| My post was a response to the sedatk conjectural "I wonder
| how much things would break if all imports were lazy by
| default."
|
| I have zero concerns about this PEP and look forward to its
| implementation.
| zahlman wrote:
| With the LazyLoader technique I described at
| https://news.ycombinator.com/item?id=45467489 , there is no
| problem: >>> import nonexistent_module
| Traceback (most recent call last): File "<python-
| input-2>", line 1, in <module> import
| nonexistent_module File "<frozen
| importlib._bootstrap>", line 1360, in _find_and_load
| File "<frozen importlib._bootstrap>", line 1322, in
| _find_and_load_unlocked File "<frozen
| importlib._bootstrap>", line 1262, in _find_spec File
| "<python-input-0>", line 8, in find_spec
| base.loader = LazyLoader(base.loader)
| ^^^^^^^^^^^ AttributeError: 'NoneType' object has no
| attribute 'loader'
|
| The implementation should probably convert that exception
| back to ImportError for you, but the point is that the
| absence of an implementation can still be detected eagerly
| while the actual loading occurs lazily.
| eesmith wrote:
| Ahh, so you do the find first, and keep that around before
| loading.
|
| I have bad memories of using a network filesystem where my
| Python app's startup time was 5 or more seconds because of
| all the small file lookups for the import were really slow.
|
| I fixed it by importing modules in functions, only when
| needed, so the time went down to less than a second. (It
| was even better using a zipimport, but for other reasons we
| didn't use that option.)
|
| If I understand things correctly, your code would have the
| same several-second delay as it tries to resolve
| everything?
| zahlman wrote:
| Yes, if checking for a file is slow, then checking for a
| file is slow. If you need to know up front whether the
| module exists, then you can't get around using the
| "figure out whether the module exists" machinery up
| front. And if the definition of "a module exists"
| includes cases where the module is represented by a file
| whose existence you have to check for, then there's no
| getting around that, either.
|
| (Trying to do "fallback" logic with lazily-loaded modules
| is also susceptible to race conditions, of course. What
| if someone defines the module before you try to use it?)
| ashu1461 wrote:
| There will be chaos, all code bases usually have tons of side
| effects.
| Galanwe wrote:
| That seems really great, .moving imports in-line for CLI tools
| startup and test discovery has always been a pain.
| hoten wrote:
| For posterity, as the submitted link seems temporary:
| https://github.com/python/peps/pull/4622
| zahlman wrote:
| The proper permanent link is https://peps.python.org/pep-0810/
| .
| IshKebab wrote:
| Wake me up when we can import a module by relative file path.
| Spivak wrote:
| You... can? I mean in the strictest sense you're technically
| not importing by file path but if you make your folder a module
| by slapping an __init__.py in there then your relative imports
| will follow the directory tree. I think as of Python 3.3 the
| init file is optional so it will do it by default but I can't
| remember if there are still some cases where it's required. The
| only thing you can't do is go "up" to a higher directory than
| the root module.
|
| Also if that doesn't strike your fancy all of the importlib
| machinery is at your disposal and it's really not very much
| work to write an import_path() function. It's one of the
| patterns plug-in systems use and so is stable and expected to
| be used by end users. No arcane magic required.
| zahlman wrote:
| > if you make your folder a module by slapping an __init__.py
| in there then your relative imports will follow the directory
| tree.
|
| `__init__.py` has _nothing to do with_ making this work. It
| is neither necessary (as of 3.3, yes, you got it right: see
| https://peps.python.org/pep-0420/) nor sufficient (careless
| use of sys.path and absolute imports could make it so that
| the current folder hasn't been imported yet, so you can't
| even go "up into" it). The folder will already be represented
| by a module object.
|
| What `__init__.py` does is:
|
| 1. Prevents relative imports from _also_ potentially checking
| in _other_ paths.
|
| 2. Provides a space for code that runs before sub-modules,
| for example to set useful package attributes such as
| `__all__` (which controls star-imports).
| IshKebab wrote:
| > if you make your folder a module by slapping an __init__.py
| in there then your relative imports will follow the directory
| tree
|
| haha no
|
| > all of the importlib machinery is at your disposal
|
| This breaks all tooling. Awful option.
| zahlman wrote:
| Relative imports have been supported for approximately forever
| (https://stackoverflow.com/questions/72852). If you mean "by
| explicitly specifying a path string" (as opposed to a symbolic
| name), that has also been supported for approximately forever
| (https://stackoverflow.com/questions/67631). Today, the
| `importlib` standard library exposes all the steps of the
| import process -- including figuring out where the source code
| is, checking the `sys.modules` cache etc. -- and lets you hook
| into everything (in particular, you can bypass the entire
| "finder" process and turn a file path into a dummy "spec" which
| is fed to a "loader").
|
| The flexibility of this system also entails that you can in
| effect define a completely new programming language, describe
| the process of creating Python bytecode from your custom
| source, and have clients transparently `import` source in the
| other language as a result. Or you can define an import process
| that grabs code from the Internet (not that it would be a good
| idea...).
|
| If you mean "by explicitly specifying a relative path, and
| having it be interpreted according to the path of the current
| module's source code", well first you have to consider that
| _the current module isn 't required to have source code_. But
| if it does, then generally it will have been loaded with the
| default mechanism, which means the module object will have a
| `__file__` attribute with an absolute path, and you just set
| your path relative to that.
| IshKebab wrote:
| Nope. Relative imports work by relative _package_ path, which
| is not at all the same. Often when you run Python you don 't
| even have a package path.
|
| Using `importlib` is a horrible hack that breaks basically
| all tooling. You very very obviously are not supposed to do
| that.
| zahlman wrote:
| > Relative imports work by relative package path, which is
| not at all the same.
|
| Exactly. This gives you the flexibility to distribute a
| complex package across multiple locations.
|
| > Often when you run Python you don't even have a package
| path.
|
| Any time you successfully import foo.bar, you necessarily
| have imported foo (because bar is an attribute of that
| object!), and therefore bar can `from . import` its
| siblings.
|
| > Using `importlib` is a horrible hack that breaks
| basically all tooling. You very very obviously are not
| supposed to do that.
|
| It is exactly as obvious (and true) that you are not
| "supposed to", in the exact same sense, directly specify
| where on disk the source code file you want to import is.
| After all, this constrains the import process to use a
| source code file. (Similarly if you specify a .pyc
| directly.) Your relative path doesn't necessarily make any
| sense after you have packaged and distributed your code and
| someone else has installed it. It definitely doesn't make
| any sense if you pack all your modules into a zipapp.
| flare_blitz wrote:
| > Using `importlib` is a horrible hack that breaks
| basically all tooling. You very very obviously are not
| supposed to do that.
|
| This is an assertion that has absolutely no reasoning
| behind it. I'm not saying I disagree; I'm just saying there
| is a time and a place for importlib.
| IshKebab wrote:
| Well not if you want high quality Python code. Pylint and
| Pyright won't understand it, and those are absolutely
| critical to writing Python code that works reliably.
| zahlman wrote:
| > those are absolutely critical to writing Python code
| that works reliably.
|
| Curious how much reliable Python code was written before
| those tools existed.
|
| For that matter, curious how much was written before the
| `types` and `typing` standard library modules appeared.
| wrmsr wrote:
| If anyone's interested I've implemented a fairly user friendly
| lazy import mechanism in the form of context managers
| (auto_proxy_import/init) at https://pypi.org/project/lazyimp/
| that I use fairly heavily. Syntactically it's just wrapping
| otherwise unmodified import statements in a with block, so tools
| 'just work' and it can be easily disabled or told to import
| eagerly for debugging. It's powered primarily by swapping out the
| frame's f_builtins in a cext (as it needs more power than
| importlib hooks provide), but has a lame attempt at a threadsafe
| pure python version, and a super dumb global hook version.
|
| I was skeptical and cautious with it at first but I've since
| moved large chunks of my codebase to it - it's caused
| surprisingly few problems (honestly none besides forgetting to
| handle some import-time registration in some modules) and the
| speed boost is addictive.
| NSPG911 wrote:
| looks very interesting! i might use this for some of my
| projects as well
| film42 wrote:
| I'm a fan because it's something you can explicitly turn on and
| off. For my Docker based app, I really want to verify the
| completeness of imports. Preferably, at build and test time. In
| fact, most of the time I will likely disable lazy loading
| outright. But, I would really appreciate a faster loading CLI
| tool.
|
| However, there is a pattern in python to raise an error if, say,
| pandas doesn't have an excel library installed, which is fine. In
| the future, will maintainers opt to include a bunch of unused
| libraries since they won't negatively impact startup time? (Think
| pandas including 3-4 excel parsers by default, since it will only
| be loaded when called). It's a much better UX, but, now if you
| opt out of lazy loading, your code will take longer to load than
| without it.
| the__alchemist wrote:
| Does this fix the circular imports problem that comes up if you
| don't structural your programs in a hierarchical way?
| colpabar wrote:
| This is what I thought of too. I really only know python, do
| other languages not have that issue? In python it does not seem
| like a "problem" to me - whenever I have seen circular import
| issues it is because the code is organized poorly. I worry that
| this feature will lead to devs "fixing" circular import issues
| by using lazy imports.
| matsemann wrote:
| Sometimes it's hard to avoid cyclic imports, without blaming
| the design. Like if a Parent has a Child, and the Child needs
| to know of the parent. Only way to solve that in python is to
| put everything in the same file, which also feels like bad
| deisgn.
| colpabar wrote:
| I would say in that case, the Parent and Child shouldn't
| need to know about each other - some kind of handler in a
| different file should.
|
| Although I guess that doesn't work in all cases, like
| defining foreign key relationships when using an orm (like
| sqlalchemy) for example. But in the orm case, the way to
| get around that is... lazy resolution :^)
| the__alchemist wrote:
| Interesting. I find that I always have this problem in any
| non-trivial python project, and don't find this to be due to
| poorly organized code. I have only seen this requirement in
| Python.
|
| IME circular import errors aren't due to poor organization;
| they're due to an arbitrary restriction Python has.
| Spivak wrote:
| It should, when you do lazy imports on your own it fixes the
| problem.
| zahlman wrote:
| Even with eager importing there is only a "circular import
| problem" if you try to import names `from` the modules -- as I
| pointed out a few days ago
| (https://news.ycombinator.com/item?id=45400783).
| keeganpoppen wrote:
| YES!!!
| behnamoh wrote:
| Make Python more like Haskell, LFG!
| charliermarsh wrote:
| Lazy imports have been proposed before, and were rejected most
| recently back in 2022: https://discuss.python.org/t/pep-690-lazy-
| imports-again/1966.... If I recall correctly, lazy imports are a
| feature supported in Cinder, Meta's version of CPython, and the
| PEP was driven by folks that worked on Cinder. Last time, a lot
| of the discussion centered around questions like: Should this be
| opt-in or opt-out? At what level? Should it be a build-flag for
| CPython itself? Etc. The linked post suggests that the Steering
| Council ultimately rejected it because of the complexity it would
| introduce to have two divergent "modes" of importing.
|
| I hope this proposal succeeds. I would love to use this feature.
| dheera wrote:
| Oof. I wish they could support version imports
| import torch==2.6.0+cu124 import numpy>=1.2.6
|
| and support having multiple simultaneous versions of any Python
| library installed. End this conda/virtualenv/docker/bazel/[pick
| your poison] mess
| lblume wrote:
| uv is good.
| fouronnes3 wrote:
| The mess has ended thanks to uv.
| maxloh wrote:
| You could do that with uv. # /// script
| # dependencies = [ # "requests<3", # "rich",
| # ] # /// import requests from
| rich.pretty import pprint resp =
| requests.get("https://peps.python.org/api/peps.json")
| data = resp.json() pprint([(k, v["title"]) for k, v in
| data.items()][:10])
| zahlman wrote:
| It's been explained many times before why this is not
| possible: the library _doesn 't actually have_ a version
| number. The _distribution_ of source code on PyPI has a
| version number, but the name of this is _not connected_ to
| the name of any module or package you import in the source
| code. The distribution can validly define _zero or more_
| modules (packages are a subset of modules, represented using
| the same type in the Python type system).
|
| You got three other responses before me all pointing at uv.
| _They are all wrong_ , because uv _did not_ introduce this
| functionality to the Python ecosystem. It is a standard
| defined by https://peps.python.org/pep-0723/, implemented by
| multiple other tools, notably pipx.
| giancarlostoro wrote:
| You could absolutely have this be part of the language in
| any regard. The question then becomes how does one
| implement it in a reasonable way. I think every package
| should have a __version__ property you should be able to
| call, then you could have versioned imports.
|
| In fact there's already many packages already defining
| __version__ at a package level.
|
| https://packaging.python.org/en/latest/discussions/versioni
| n...
|
| Edit: What they are solving with UV is at the moment of
| standing up an environment, but you're more concerned about
| code-level protection, where are they're more concerned
| about environment setup protection for versioning.
| zahlman wrote:
| > In fact there's already many packages already defining
| __version__ at a package level.
|
| This only helps for those that do, and it hasn't been any
| kind of standard the entire time. But more importantly,
| that helps only the tiniest possible bit with resolving
| the "import a specific version" syntax. All it solves is
| letting the file-based import system know whether it
| found the right folder for the requested (or worse: "a
| compatible") version of the importable package. It
| doesn't solve finding the right one if this one is wrong;
| it doesn't determine how the different versions of the
| same package are positioned relative to each other in the
| environment (so that "finding the right one" can work
| properly); it doesn't solve provisioning the right
| version. And most importantly, it doesn't solve _what
| happens when there are multiple requests_ for different
| versions of the same module at runtime, which
| incidentally could happen arbitrarily far apart in time,
| and also the semantics of the code _may depend on the
| same object_ being used to represent the module in both
| places.
| BugsJustFindMe wrote:
| > _It 's been explained many times before why this is not
| possible: the library doesn't actually have a version
| number. The distribution of source code on PyPI has a
| version number, but the name of this is not connected to
| the name of any module or package you import in the source
| code._
|
| You're making the common mistake of conflating how things
| currently work with how things could work if the
| responsible group agrees to change how things work.
| Something being the way it is right now is not the same as
| something else being "not possible".
| zahlman wrote:
| No, changing this breaks the world. A huge fraction of
| PyPI becomes completely invalid overnight, and the rest
| fails the expected version checks. Not to mention that
| the language is fundamentally designed around the
| expectation that modules are singleton. I've written
| about this at length before but I can't easily find it
| right now (I have way too many bookmarks and not nearly
| enough idea how to organize them).
|
| Yes, you absolutely can create a language that has syntax
| otherwise identical to Python (or at least damn close)
| which implements a feature like this. No, you cannot just
| replace Python with it. If the Python ecosystem just
| accepted that clearly better things were clearly better,
| and started using them promptly, we wouldn't have
| https://pypi.org/project/six/ making it onto
| https://pypistats.org/top (see also
| https://sethmlarson.dev/winning-a-bet-about-six-the-
| python-2...).
| hnlmorg wrote:
| The hard part is making the change. Adding an escape
| hatch so older code still works is easy in comparison.
|
| Nobody is claiming this is a trivial problem to solve but
| its also not an impossible problem. Other languages have
| managed to figure out how to achieve this and still
| maintain backwards compatibility.
| zahlman wrote:
| You're welcome to bring a concrete proposal to, e.g.,
| https://discuss.python.org/c/ideas/6 , or ask around the
| core devs to find a PEP sponsor.
|
| Note that you will be expected to have familiarized
| yourself generally with previous failed proposals of this
| sort, and proactively considered all the reasonably
| obvious corner cases.
| hnlmorg wrote:
| Tbh you're just reinforcing mine and others point there
| that the issue isn't a technical one.
| flare_blitz wrote:
| "Talk is cheap. Show me the code."
| hnlmorg wrote:
| I'm not going to spend an entire weekend drafting a
| proposal instead of spending time with my kids, just to
| win "internet points".
|
| If you want examples then just look at one of the other
| languages that have implemented compiler / runtime
| dependency version checks.
|
| Even Go has better dependency resolution than Python, and
| Go is often the HN poster child for how not to do things.
|
| The crux of the matter is this is a solvable problem. The
| real issue isn't that it's technically impossible, is
| that it's not annoying enough of a day to day problem for
| _people who are in a position_ to influence this change.
| I'm not that person and don't aspire to be that person (I
| have plenty of other projects on my plate as it is)
| jacquesm wrote:
| In spite of the 'You're welcome to bring' this does not
| actually sound like an encouragement but more of a veiled
| statement that some non-technical reason will be found to
| shoot down the proposal if it were to be made so you
| might as well not bother.
| flare_blitz wrote:
| No, the point is that most people in this thread do not
| appreciate the complexity of implementing lazy imports.
| If you disagree, your energy is better spent talking to a
| CPython core developer about implementation details of
| making baseless assertions from an ivory tower.
|
| There are many people here who think enabling lazy
| imports is as simple as flipping a light switch. They
| have no idea what they're talking about.
| hnlmorg wrote:
| You've misread the discussion
|
| This thread was a tangent from lazy imports.
|
| And actually people do appreciate the complexities of
| changes like this. We were responding to a specific
| comment that that said "it's impossible". Saying
| something is "possible" isn't the same as saying "it's
| easy".
| jacquesm wrote:
| Reading comprehension failure on your part.
| zahlman wrote:
| It's an allusion to the fact that there is a very long
| history establishing that the problem is not as simple as
| it sounds, even if you get past the most basic issues,
| and it's hard to explain it all in a single coherent
| post.
| oskarkk wrote:
| > Not to mention that the language is fundamentally
| designed around the expectation that modules are
| singleton.
|
| Modules being singletons is not a problem in itself I
| think? This could work like having two versions of the
| same library in two modules named like library_1_23 and
| library_1_24. In my program I could hypothetically have
| imports like `import library_1_23 as library` in one
| file, and `import library_1_24 as library` in another
| file. Both versions would be singletons. Then writing
| `import library==1.23` could be working like syntax sugar
| for `import library_1_23 as library`.
|
| Of course, having two different versions of a library
| running in the same program could be a nightmare, so all
| of that may not be a good idea at all, but maybe not
| because of module singletons.
| johannes1234321 wrote:
| If vereioned imports were added to the language versioned
| library support obviously would have to become part of the
| language as well.
|
| However it isn't trivial. First problem coming to my mind:
|
| module a importing first somelib>=1.2.0 and then b and b
| then requiring somelib>1.2.1 and both being available, will
| it be the same or will I have a mess from combining?
| driggs wrote:
| > It's been explained many times before why this is not
| possible: the library doesn't actually have a version
| number.
|
| Not _possible_? Come on.
|
| Almost everyone already uses one of a small handfull of
| conventional ways to specify it, eg `__version__`
| attribute. It's long overdue that this be standardized so
| library versions can reliably be introspected at runtime.
|
| Allowing multiple versions to be installed side-by-side and
| imported explicitly would be a massive improvement.
| 0cf8612b2e1e wrote:
| I believe the charitable interpretation is that it is not
| possible without breaking an enormous amount of legacy
| code. Which does feel close enough to "not possible".
|
| _Some_ situations could be improved by allowing multiple
| library versions, but this would introduce new headaches
| elsewhere. I certainly do not want my program to have N
| copies of numpy, PyTorch, etc because some intermediate
| library claims to have just-so dependency tree.
| driggs wrote:
| What do you do _today_ to resolve a dependency conflict
| when an intermediate library has a just-so dependency
| tree?
|
| The charitable interpretation of this proposed feature is
| that it would handle this case exactly as well as the
| current situation, if the situation isn't improved by the
| feature.
|
| This feature says nothing about the automatic
| installation of libraries.
|
| This feature is absolutely _not_ about supporting
| multiple simultaneous versions of a library at runtime.
|
| In the situation you describe, there would have to be a
| dependency resolution, just like there is when installing
| the deps for a program _today_. It would be good enough
| for me if "first import wins".
| zahlman wrote:
| > What do you do today to resolve a dependency conflict
| when an intermediate library has a just-so dependency
| tree?
|
| When an installer resolves dependency conflicts, the
| project code isn't running. The installer is free to
| discover new constraints on the fly, and to backtrack. It
| is in effect all being done "statically", in the sense of
| being ahead of the time that any other system cares about
| it being complete and correct.
|
| Python `import` statements on the other hand execute
| during the program's runtime, at arbitrary separation,
| with other code intervening.
|
| > This feature says nothing about the automatic
| installation of libraries.
|
| It doesn't have to. The runtime problems still occur.
|
| I guess I'll have to reproduce the basic problem
| description from memory again. If you have modules A and
| B in your project that require conflicting versions of C,
| you need a way to load both at runtime. But the standard
| import mechanism already hard-codes the assumptions that
| i) imports are cached in a key-value store; ii) the
| values are singleton _and client code absolutely may rely
| on this for correctness_ ; iii) "C" is enough information
| for lookup. And the ecosystem is further built around the
| assumption that iv) this system is documented and stable
| and can be interacted with in many clever ways for
| metaprogramming. Changing any of this would be incredibly
| disruptive.
|
| > This feature is absolutely not about supporting
| multiple simultaneous versions of a library at runtime.
|
| You say that, but you aren't the one who proposed it. And
| https://news.ycombinator.com/item?id=45467350 says
| explicitly:
|
| > and support having multiple simultaneous versions of
| any Python library installed.
|
| Which would really be the only _reason_ for the feature.
| For the cases where a single version of the third-party
| code satisfies the entire codebase, the existing
| packaging mechanisms all work fine. (Plus they properly
| distinguish between import names and distribution names.)
| driggs wrote:
| > and support having multiple simultaneous versions of
| any Python library installed.
|
| Installed. Not loaded.
|
| The reason is to do away with virtual environments.
|
| I just want to say `import numpy@2.3.x as np` in my code.
| If 2.3.2 is installed, it gets loaded as the _singleton_
| runtime library. If it 's not installed, load the closest
| numpy available and print a warning to stderr. If a
| transient dependency in the runtime tree wants an
| incompatible numpy, tough luck, the best you get is a
| warning message on stderr.
|
| You _already_ have the A, B, C dependency resolution
| problem you describe _today_. And if it 's not caught at
| the time of _installing_ your dependencies, you see the
| failure at runtime.
| mardifoufs wrote:
| I know I'm missing something but wouldn't it be possible to
| just throw an import error when that happens? Would it even
| break anything? If I try:
|
| import numpy==2.1
|
| And let's say numpy didn't expose a version number in a
| standard (which could be agreed upon in a PEP) field, then
| it would just throw an import exception. It wouldn't break
| any old code. And only packages with that explicit field
| would support the pinned version import.
|
| And it wouldn't involve trying to extract and parse
| versions from older packages with some super spotty
| heuristics.
|
| But it would make new code impossible to use with older
| versions of python, and older packages, but that's already
| the case.
|
| Maybe the issue is with module name spacing?
| zahlman wrote:
| > And let's say numpy didn't expose a version number in a
| standard (which could be agreed upon in a PEP) field,
| then it would just throw an import exception. It wouldn't
| break any old code. And only packages with that explicit
| field would support the pinned version import.
|
| Yes, this part actually is as simple as you imagine. But
| that means in practical terms that you can't _use_ the
| feature until _at best_ the next release of Numpy. If you
| want to say for example that you need at least version 2
| (breaking changes, after all), well, there are already 18
| published packages that meet that requirement but are
| unable to communicate that in the new syntax. This can to
| my understanding be fixed with post releases, but it 's a
| serious burden for maintainers and most projects are just
| not going to do that sort of thing (it bloats PyPI, too).
|
| And more importantly, that's only one of _many_ problems
| that need to be solved. And by far the simplest of them.
| jacquesm wrote:
| > It's been explained many times before why this is not
| possible: the library doesn't actually have a version
| number.
|
| That sounds like it is absolutely fixable to me, but more
| of a matter of not having the will to fix it based on some
| kind of traditionalism. I've used python, a lot. But it is
| stuff like this that is just maddeningly broken for no good
| reason at all that has turned me away from it. So as long
| as I have any alternative I will avoid python because I've
| seen way too many accidents on account of stuff like this
| and many lost nights of debugging only to find out that an
| easily avoidable issue became - once again - the source of
| much headscratching.
| zahlman wrote:
| > a matter of not having the will to fix it based on some
| kind of traditionalism
|
| Do you know what happens when Python _does_ summon the
| will to fix obviously broken things? The Python 2- >3
| migration happens. (Perl 6 didn't manage any better,
| either.) Now "Python 3 is the brand" and the idea of
| version 4 can only ever be entertained as a joke.
| jacquesm wrote:
| Yes, good point. Compared to how the Erlang community has
| handled decades of change Python does not exactly deserve
| the beauty prize. The lack of forethought - not to be
| confused with a lot of hot air - on some of these
| decisions is impressive. I think that the ability to
| track developments in near realtime is in conflict with
| that though. If you want your language to be everything
| to everybody then there will be some broken bones along
| the way.
| tkfoss wrote:
| How would version imports be handled across the codebase?
| Also, what do you gain with those over PEP 723 - Inline
| script metadata? https://packaging.python.org/en/latest/speci
| fications/inline...
| gigatexal wrote:
| Really what is the headache with virtual environments?
| They've been solved. Use UV or python's built in venv creator
| and you're good to go.
|
| uv venv --seed --python=3.12 && source .venv/bin/activate &&
| pip3 install requests && ...
| dheera wrote:
| That's messy.
|
| I should be able to do "python foo.py" and everything
| should just work. foo.py should define what it wants and
| python should fetch it and provide it to foo. I should be
| able to do "pyc foo.py; ./foo" and everything should just
| work, dependencies balled up and statically included like
| Rust or Go. Even NodeJS can turn an entire project into one
| file to execute. That's what a modern language _should_
| look and work like.
|
| The moment I see "--this --that" just to run the default
| version of something you've lost me. This is 2025.
| gigatexal wrote:
| You mean this?
|
| #!/usr/bin/env -S uv run --script # # /// script #
| requires-python = ">=3.12" # dependencies = ["httpx"] #
| ///
|
| import httpx
|
| print(httpx.get("https://example.com"))
|
| https://docs.astral.sh/uv/guides/scripts/#improving-
| reproduc...
|
| There are also projects py2exe pyinstaller iirc and
| others that try to get the whole static binary thing
| going.
|
| You're trying imo to make Python into Golang and if
| you're wanting to do that just use Golang. That seems
| like a far better use of your time.
| oivey wrote:
| Oof. This feature request has nothing to do with lazy
| imports. It's also solved far more cleanly with inline script
| metadata.
| _ZeD_ wrote:
| NO! I don't want my source code filled with this crap.
|
| I don't want to lose multiple hours debugging why something
| did go wrong because I am using three versions of numpy and
| seven of torch at the same time and there was a mixup
| BiteCode_dev wrote:
| Especially since it is opt in, with various level of
| granularity, and a global off switch. Very well constructed
| spec given the constraints.
| flare_blitz wrote:
| I also hope this proposal succeeds, but I'm not optimistic.
| This will break tons of code and introduce a slew of footguns.
| Import statements fundamentally have side effects, and when and
| how these side effects are applied will cause mysterious
| breakages that will keep people up for many nights.
|
| This is not fearmongering. There is a reason why the only
| flavor of Python with lazy imports comes from Meta, which is
| one of the most well-resourced companies in the world.
|
| Too many people in this thread hold the view of "importing
| {pandas, numpy, my weird module that is more tangled than an
| eight-player game of Twister} takes too long and I will gladly
| support anything that makes them faster". I would be willing to
| bet a large sum of money that most people who hold this opinion
| are unable to describe how Python's import system works, let
| alone describe how to implement lazy imports.
|
| PEP 690 describes a number of drawbacks. For example, lazy
| imports break code that uses decorators to add functions to a
| central registry. This behavior is crucial for Dash, a popular
| library for building frontends that has been around for more
| than a decade. At import-time, Dash uses decorators to bind a
| JavaScript-based interface to callbacks written in Python. If
| these imports were made lazy, Dash would break. Frontends used
| by thousands, if not millions of people, would immediately
| become unresponsive.
|
| You may cry, "But lazy imports are opt-in! Developers can
| choose to opt-out of lazy imports if it doesn't work for them."
| What if these imports were transitive? What if our frontend
| needed to be completely initialized before starting a critical
| process, else it would cause a production outage? What if you
| were a maintainer of a library that was used by millions of
| people? How could you be sure that adding lazy imports wouldn't
| break any code downstream? Many people made this argument for
| type hints, which is sensible because type hints have no effect
| on runtime behavior*. This is not true for lazy imports; import
| statements exist in essentially every nontrivial Python
| program, and changing them to be lazy will fundamentally alter
| runtime behavior.
|
| This is before we even get to the rest of the issues the PEP
| describes, which are even weirder and crazier than this. This
| is a far more difficult undertaking than many people realize.
|
| ---
|
| * You _can_ make a program change its behavior based on type
| annotations, but you 'd need to explicitly call into typing
| APIs to do this. Discussion about this is beyond the scope of
| this post.
| WD-42 wrote:
| Some of these worries make sense, but wouldn't it be
| relatively trivial to pass a flag to the interpreter or
| something similar in order to force all imports to evaluate,
| as in the current behavior? But to be a bit cheeky if some of
| these issues cause serious production outages for you it
| might be time to consider moving on from a scripting language
| altogether.
| flare_blitz wrote:
| The issue is that some imports can be made lazy and some
| cannot. A binaristic all-or-nothing approach does not
| address the issue. (I also think that there is zero basis
| to claim that adding such a flag is trivial, since there's
| no reference implementation of this flavor of lazy
| imports.)
|
| What if we have a program where one feature works only when
| lazy imports are enabled and one feature only when lazy
| imports are disabled?
|
| This is not a contrived concern. Let's say I'm a maintainer
| of an open-source library and I choose to use lazy imports
| in my library. Because I'm volunteering my time, I don't
| test whether my code works with eager imports.
|
| Now, let's say someone comes and builds an application on
| top of this library. It doesn't work with lazy imports for
| some unknown reason. If they reach for a "force all
| imports" flag, their application might break in another
| mysterious way because the code they depend on is not built
| to work with eager imports. And even if my dependency
| doesn't break, what about all the other packages the
| application may depend on?
|
| The only solution here would be for the maintainer to
| ensure that their code works with both lazy and eager
| imports. However, this imposes a high maintenance cost and
| is part of the reason why PEP 690 was rejected. (And if
| your proposed solution was "don't use libraries made by
| random strangers on the Internet", boy do I have news for
| you...)
|
| My point is that many things _will_ break if migrated to
| lazy imports. Whether they should have been written in
| Python in the first place is a separate question that isn't
| relevant to this discussion.
| uncletoxa wrote:
| Theoretically the implementation may use the approach "as
| lazy as possible". Traverse lazy imports until you
| encounter a regular one. I doubt it will make much
| difference, but at least it gives an option.
| paffdragon wrote:
| Maybe the package that requires lazy can somehow declare
| that requirement, so another package that tries to force
| not lazy will fail early and realize it needs to replace
| this dependency with something compatible or change its
| ways. It definitely adds complexity, though.
| cuu508 wrote:
| Or check at runtime if it's running with the lazy import
| feature active. Then instead of breaking in mysterious
| ways in production it would crash on startup, during
| development.
| fastasucan wrote:
| They are not entitled to hold the opinion that their imports
| takes too long, if they dont know the inner workings of
| pythons import system? Do you listen to yourself?
| flare_blitz wrote:
| That's not what I said. Nobody is "entitled" or "not
| entitled" to hold certain opinions. Please reread my
| original comment carefully.
| ActorNightly wrote:
| Nothing wrong with that statement.
|
| Right now in python, you can move import statement inside a
| function. Lazy imports at top level are not needed. All
| lazy imports do is make you think less about what you are
| writing. If you like that, then just vibe code all of your
| stuff, and leave the language spec alone.
| zahlman wrote:
| > This will break tons of code
|
| I don't see how. It adds a new, entirely optional syntax
| using a soft keyword. The semantics of existing code do not
| change. Yes, yes, you anticipated the objection:
|
| > What if these imports were transitive? ... How could you be
| sure that adding lazy imports wouldn't break any code
| downstream?
|
| I would need to see concrete examples of how this would be a
| realistic risk in principle. (My gut reaction is that top-
| level code in libraries shouldn't be doing the kinds of
| things that would be problematic here, in the first place. In
| my experience, the main thing they do at top level is just
| eagerly importing everything else for convenience, or to
| establish compatibility aliases.)
|
| But if it were, clearly that's a breaking change, and the
| library bumps the major version and clients do their usual
| dependency version management. As you note, type hints work
| similarly. And "explicitly calling into typing APIs" is more
| common than you might think;
| https://pypistats.org/packages/pydantic exists pretty much to
| do exactly this. It didn't cause major problems.
|
| > Import statements fundamentally have side effects, and when
| and how these side effects are applied will cause mysterious
| breakages that will keep people up for many nights.
|
| They do have side effects that can be arbitrarily complex.
| But someone who opts in to changing import timing and
| encounters a difficult bug can just roll back the changes. It
| shouldn't cause extended debugging sessions unless someone
| really needs the benefits of the deferral. And people in that
| situation will have been hand-rolling their own workarounds
| anyway.
|
| > Too many people in this thread hold the view of "importing
| {pandas, numpy, my weird module that is more tangled than an
| eight-player game of Twister} takes too long and I will
| gladly support anything that makes them faster".
|
| I don't think they're under the impression that this
| necessarily makes things _faster_. Maybe I haven 't seen the
| same comments you have.
|
| Deferring imports absolutely would allow, for example, pip to
| do trivial tasks faster -- because it could avoid importing
| unnecessary things _at all_. As things currently stand, a
| huge fraction of the vendored codebase will get imported
| pretty much no matter what. It 's analogous to tree shaking,
| but implicitly, at runtime and without actually removing
| code.
|
| Yes, this could be deferred to explicitly chosen times to get
| more or less the same benefit. It would also be more work.
| f33d5173 wrote:
| This is a new syntax, so it is opt-in. The new syntax can be
| conceived as syntax sugar that lets you rewrite
| def my_func(): import my_mod
| my_mod.do_stuff()
|
| as lazy import my_mod def my_func():
| my_mod.do_stuff()
|
| Ie, with lazy, the import happens at the site of usage. Since
| clearly this is code that could already be written, it only
| breaks things in the sense that someone could already write
| broken code. Since it is opt in, if using it breaks some
| code, then people will notice that and choose not to rewrite
| that code using it.
| flare_blitz wrote:
| This is already addressed in my comment above.
| tomy0000000 wrote:
| Just curious. What changed?
|
| From merely browsing through a few comments, people have mostly
| positive opinions regarding this proposal. Then why did it fail
| many times, but not this time? What drives the success behind
| this PEP?
| forrestthewoods wrote:
| I don't want lazy imports. That's just makes performance shitty
| later and harder to debug. It's a hacky workaround.
|
| What I want is for imports to not suck and be slow. I've had
| projects where it was faster to compile and run C++ than launch
| and start a Python CLI. It's so bad.
| dataangel wrote:
| imports run arbitrary Python code, making them faster is the
| same as making every other language feature faster
| forrestthewoods wrote:
| Can you show me a profile that demonstrate the slowness is
| because imports are running large amounts of arbitrary Python
| code? Maybe they shouldn't do that. Maybe libraries should be
| pushed to not run expensive arbitrary code execution at
| import time.
| Alir3z4 wrote:
| I wish all imports were lazy by default.
|
| I know/heard there are "some" (which I haven't seen by the way)
| libraries that depend on import side effects, but the advantage
| is much bigger.
|
| First of all, the circular import problem will go away,
| especially on type hints. Although there was a PEP or recent
| addition to make the annotation not not cause such issue.
|
| Second and most important of all, is the launch time of Python
| applications. A CLI that uses many different parts of the app has
| to wait for all the imports to be done.
|
| The second point becomes a lot painful when you have a large
| application, like a Django project where the auto reload becomes
| several seconds. Not only auto reload crawls, the testing cycle
| is slow as well. Every time you want to run test command, it has
| to wait several seconds. Painful.
|
| So far the solution has been to do the lazy import by importing
| inside the methods where it's required. That is something, I
| never got to like to be honest.
|
| Maybe it will be fixed in Python 4, where the JIT uses the type
| hints as well /s
| f311a wrote:
| Lazy imports mean late errors. Fail fast is a good design
| principle.
| zahlman wrote:
| Top-level code should not be able to fail except in
| incredibly deterministic ways that are tested during
| development. Failing fast is not as good as not failing at
| all. Lazy imports mean the power to avoid importing things
| that don't need to be imported at all on this run. Good
| design also cares about performance to some extent. On my
| machine, asking pip to do _literally nothing_ takes several
| times as long as creating a new virtual environment
| --without-pip .
| josefx wrote:
| Python really seems like a bad fit for that. So your imports
| succeed, what now? Do they have all the functions or fields
| your program needs? Those are still resolved at the last
| possible moment. If you want to be sure your program actually
| runs you will have to write and run tests with and without
| lazy imports.
| f311a wrote:
| I don't like the idea of introducing a new keyword. We need a
| backward compatible solution. I feel like Python needs some kind
| of universal annotation syntax such as in go (comments) or in
| Rust (macros). New keyword means all parsers, lsps, editors
| should be updated.
|
| I'm pretty sure there will be new keywords in Python in the
| future that only solve one thing.
| Alir3z4 wrote:
| Me neither.
|
| Introducing new keyword has become a recent thing in Python.
|
| Seems Python has a deep scare since Python2 to Python3 time and
| is scared to do anything that causes such drama again.
|
| For me, the worst of all is "async". If 2to3 didn't cause much
| division, the async definitely divided Python libraries in 2.
| Sync and Async.
|
| Maybe if they want backward compatible solution, this can be
| done by some compile or runtime flag like they did with free
| threading no-gil.
| sigmoid10 wrote:
| Async or not in modules is a huge pain in the ass for web-
| dev, but thankfully a lot of Python still isn't web dev. As a
| Data Scientist you can live your life peacefully without ever
| worrying about this. But I can see why web devs like it, even
| though personally I really don't want to see anymore
| Javascript sneaking inty my Python. Especially when there is
| already support for IO bound concurrency elsewhere in the
| language. If I want to do JS syntax, I'll fucking use JS. And
| I really don't want to see Python go the C++ route where it
| just wants to be everything and do everything so that you end
| up with so many possible approaches to the same problem that
| two devs can't read each others code anymore.
| vvpan wrote:
| I'm relatively new to Python - how does one do concurrent
| IO without async/await?
|
| My main complaint, though, about Python async is - because
| it is opt-in I never know if I forgot a sync IO call
| somewhere that will block my worker. JS makes everything
| async by default and there is effectively no chance of
| blocking.
| blibble wrote:
| take your pick: threads, fork, non-explicit coroutines
| (gevent), io multiplexing (select/poll/epoll/kqueue),
| virtual threads/goroutines
| baq wrote:
| The same way you always did it: sync io in threads.
| Caveats are similar to async/await, but stack traces
| don't suck.
| BiteCode_dev wrote:
| Async was a soft keyword for many, many years in order to
| maintain compat. And before that, asyncio used yield to be
| even more compatible.
|
| They took a decade to solidify that. At some point, you have
| to balance evolution and stability. For a language as popular
| as Python, you can not break the world every week, but you
| can't stay put for 5 years.
| ashu1461 wrote:
| Not sure if this can be made backward compatible.
|
| Right now all the imports are getting resolved at runtime
| example in a code like below from file1 import
| function1
|
| When you write this, the entire file1 module is executed right
| away, which may trigger side effects.
|
| If lazy imports suddenly defer execution, those side effects
| won't run until much later (or not at all, if the code path
| isn't hit). That shift in timing could easily break existing
| code that depends on import-time behavior.
|
| To avoid using lazy, this there is also a proposal of adding
| the modules you want to load lazily to a global
| `__lazy_modules__` variable.
| bkettle wrote:
| I think they mean backwards-compatible syntax-wise, rather
| than actually allowing this feature to be used on existing
| code. If I'm understanding correctly they would prefer for
| the Python grammar to stay the same (hence the comment about
| updating parsers and IDEs).
|
| But I don't think I really agree, the extensible annotation
| syntaxes they mention always feel clunky and awkward to me.
| For a first-party language feature (especially used as often
| as this will be), I think dedicated syntax seems right.
| f311a wrote:
| I was talking about syntax. I'm pretty sure there will be new
| features that will require a new keyword or syntax given the
| speed of Python growth. It can be universal, for example same
| as decorator, but it can be applied anywhere.
| from lazy import make_lazy from package import module
| @make_lazy @local @nogil
|
| Let's say this syntax gets introduced in Python 3.16. The
| @nogil feature can be introduced in 3.17. If such code is
| running in Python 3.16, the @nogil marker will be ignored.
|
| The problem with new keywords is that you have to stick to
| the newest Python version every time a new keyword is added.
| Older Python versions will give a syntax error. It's a big
| problem for libraries. You need to wait for 3-5 years before
| adding it to a library. There are a lot of people who still
| use Python 3.8 from 2019.
| ashu1461 wrote:
| Understood make sense
| baq wrote:
| Yeah I like the feature but hate the keyword. Dunder lazy
| imports sounds good enough imho.
| BiteCode_dev wrote:
| They thought about backward compatibility, and offer an
| alternative syntax that use no keyword for library that want to
| activate it yet stay compact with old version. It's already in
| the spec.
| jeremyscanvic wrote:
| Really excited about this - we've recently been struggling with
| making imports lazy without completely messing up the code in
| DeepInverse https://deepinv.github.io/deepinv/
| Spivak wrote:
| I love the feature but I really dislike using the word lazy as a
| new language keyword. It just feels off somehow. I think maybe
| defer might be a better word. It is at least keeps the grammar
| right because it would be lazily. lazily import
| package.foo vs defer import package.foo
|
| Also the grammar is super weird for from imports.
| lazy from package import foo vs. from package
| defer import foo.
| est wrote:
| second this.
|
| Just might as well add `defer` keyword like Golang.
| the_mitsuhiko wrote:
| I like the approach of ES6 where you pull in bindings that are
| generally lazily resolved. That is IMO the approach that should
| be the general strategy for Python.
| 9dev wrote:
| The import/require schism was necessary precisely of all the
| issues with module side effects the python community is
| rediscovering now. I suppose they won't get around a similar
| solution eventually; but judging from years of discussions in
| the JS world, this is going to be dragging on for a while.
| simonw wrote:
| Love this. My https://llm.datasette.io/ CLI tool supports
| plugins, and people were complaining about really slow start
| times even for commands like "llm --help" - it turned out there
| were popular plugins that did things like import pytorch at the
| base level, so the entire startup was blocked on heavy imports.
|
| I ended up adding a note to the plugin author docs suggesting
| lazy loading inside of functions -
| https://llm.datasette.io/en/stable/plugins/advanced-model-pl... -
| but having a core Python language feature for this would be
| really nice.
| zahlman wrote:
| You can implement this from your tool today:
| https://news.ycombinator.com/item?id=45467489
|
| Note that this is global to the entire process, so for example
| if you make an import of Numpy lazy this way, then so are the
| imports of all the sub-modules. Meaning that large parts of
| Numpy might not be imported at all if they aren't needed, but
| pauses for importing individual modules might be distributed
| unpredictably across the runtime.
|
| Edit: from further experimentation, it appears that if the
| source does something like `import foo.bar.baz` then `foo` and
| `foo.bar` will still be eagerly loaded, and only `foo.bar.baz`
| itself is deferred. This might be part of what the PEP meant by
| "mostly". But it might also be possible to improve my
| implementation to fix that.
| dahart wrote:
| Note the PEP does have a FAQ entry that mentions reasons they
| believe this proposed solution might be preferable to
| LazyLoader
|
| https://pep-previews--
| 4622.org.readthedocs.build/pep-0810/#f...
|
| Q: Why not use importlib.util.LazyLoader instead?
|
| A: LazyLoader has significant limitations:
|
| Requires verbose setup code for each lazy import.
|
| Has ongoing performance overhead on every attribute access.
|
| Doesn't work well with from ... import statements.
|
| Less clear and standard than dedicated syntax.
| zahlman wrote:
| Thanks for pointing it out!
|
| > Has ongoing performance overhead on every attribute
| access.
|
| I would have expected so, but in my testing it seems like
| the lazy load does some kind of magic to replace the proxy
| with the real thing. I haven't properly dug into it,
| though. It appears this point is removed in the live
| version (https://peps.python.org/pep-0810).
|
| > Doesn't work well with from ... import statements.
|
| Hmm. The PEP doesn't seem to explain how reification works
| in this case. Per the above it's a solved problem for
| modules; I guess for the from-imports it could be made to
| work essentially the same way. Presumably this involves the
| proxy holding a reference to the namespace where the import
| occurred. That probably has a lot to do with restricting
| the syntax to top level. (Which is the opposite of how
| we've seen soft keywords used before!)
|
| > Requires verbose setup code for each lazy import.
|
| > Less clear and standard than dedicated syntax.
|
| If you want to use it in a fine-grained way, then sure.
| wslh wrote:
| Is it another potential solution (until PEP 810 is accepted)
| to override the NameError exception, decide if it was
| triggered by an unloaded package from a list, and then
| running again that line of code? I understand the
| inefficiency of this solution (e.g. the same line could
| trigger NameError several times and you need to run it again
| until all modules are loaded) but this is a good
| brainstorming thread.
| zahlman wrote:
| That sounds very unpleasant. However nicely you wrapped it
| up, you'd still be referring to the code for that process
| everywhere that the NameError could occur.
| peterfirefly wrote:
| Parse the command line and do things like "--help" without
| doing the imports.
|
| Only do imports when you know you need them -- or as an easy
| approximation, only if the easy command line options have been
| handled and there's still something to do.
| stavros wrote:
| Well yes, or you can just use the `lazy` keyword, when it
| makes it into core.
| loloquwowndueo wrote:
| You can "just" use a feature which does not exist yet? How
| is that something you "just" do?
| Y_Y wrote:
| lazy from __future__ import __lazy_import__
| zahlman wrote:
| I think Guido's time machine will need some serious
| overclocking to handle that one!
| mr_mitm wrote:
| In the llm project, plugins can modify the command line
| arguments, so it's not that simple.
| simonw wrote:
| Yea, that's the core problem here: plugins can add new CLI
| subcommands, which means they all need to be loaded on
| startup.
|
| https://llm.datasette.io/en/stable/plugins/plugin-
| hooks.html...
| pests wrote:
| Could you cache the help doc after first full loaded run
| and only regenerate when new plugins are added / updated?
| simonw wrote:
| It's not just help - the plugins need to be imported so
| the root level CLI tool knows what to do if you type "llm
| subcommand ..." where that subcommand is defined by a
| plugin.
| Inufu wrote:
| In that case the CLI only needs to import the plugin that
| defined that sub command, not all plugins?
| simonw wrote:
| It doesn't know which plugin defines a subcommand until
| it imports the plugin's module.
|
| I'm happy with the solution I have now, which is to
| encourage plugin authors not to import PyTorch or other
| heavy dependencies at the root level of their plugin
| code.
| peterfirefly wrote:
| > It doesn't know which plugin defines a subcommand until
| it imports the plugin's module.
|
| That might be considered a design mistake -- one that
| should be easy to migrate away from.
|
| You won't need to do anything, of course, if the lazy
| import becomes available on common Python installs some
| day in the future. That might take years, though.
| zahlman wrote:
| Just FWIW, a trick that I'm planning to use for PAPER:
| first I make separate actual commands -- `paper-foo`,
| `paper-bar` etc. that are each implemented as separate
| top-level scripts that can import only what they need.
| Later, the implementation of `paper foo` has the main
| `paper` script dynamically look up `paper-foo`. (Even a
| `subprocess.call` would work there but I'd like to avoid
| that overhead)
| whatevaa wrote:
| Or require plugins to be competently written.
|
| Bad performing third party plugins are user error.
| Neywiny wrote:
| I think really the problem is that packages like pytorch take
| so long to import. In my work I've tried a few packages (not AI
| stuff) that do a _lot_ of work on import. It 's actually quite
| detrimental because I have to setup environment variables to
| pass things that should be arguments of a setup function in.
| All things considered a python module shouldn't take any
| noticeable time to import
| aftbit wrote:
| We tend to prefer explicit top-level imports specifically because
| they reveal dependency problems as soon as the program starts,
| rather than potentially hours or days later when a specific code
| path is executed.
| maest wrote:
| Who is "we"?
| aftbit wrote:
| My current team at my current company (see bio if you're
| really interested), though I should say I'm not authorized to
| speak on behalf of my employer, so I should really say
| something more like "I".
| bityard wrote:
| I haven't fully digested the PEP but perhaps there would be a
| command-line flag or external tool for dependency validation, a
| bit like how there are external tools for type annotations?
| ashu1461 wrote:
| Right, Not sure why but a lot of code which claude generates
| also comes with local dependencies vs globally declared import
| statements. Don't promote that pattern because
|
| - It reduces visibility into a module's dependencies.
|
| - It increases the risk of introducing circular dependencies
| later on.
| zahlman wrote:
| As a counterpoint, having all the imports automatically
| deferred would instantly dramatically speed up pip for short
| tasks. $ time pip install --disable-pip-
| version-check ERROR: You must give at least one
| requirement to install (see "pip help install") real
| 0m0.399s user 0m0.360s sys 0m0.041s
|
| Almost all of this time is spent importing (and later
| unloading) ultimately useless vendored code. From my testing
| (hacking the wrapper script to output some diagnostics),
| literally about 500 modules get imported in total (on top of
| the baseline for a default Python process), including almost a
| hundred modules related to Requests and its dependencies, even
| though no web request was necessary for this command.
| aftbit wrote:
| I think this makes a ton of sense in the very specific narrow
| use case of python CLI tools. For a web app or other long-
| lived process, startup time is typically not of extreme
| concern, and having more simplicity and legibility to the
| import process seems better.
|
| That's not to say this PEP should not be accepted. One could
| always apply a no-lazy-imports style rule or disable it via
| global lazy import control.
|
| https://peps.python.org/pep-0810/#global-lazy-imports-
| contro...
| zahlman wrote:
| It saddens me to think that the use case of "python CLI
| tools" is thought of as anything like "very specific" or
| "narrow".
| stefan_ wrote:
| These tools already have a million ways to avoid this
| slowdown. Is it so much to ask their authors to care?
| hiq wrote:
| This sounds like something that should be covered by tests?
| rplnt wrote:
| Remember mercurial? Me neither. But what I remeber is this
| article I've read about all the hacks they had to do to achieve
| reasonable startup time for CLI in python. And the no #1 cause
| was loading the whole world you don't ever need. As I recall they
| somehow monkeypatched the interpreter to ignore imports and just
| remember their existence until they were actually needed, at
| which point the import happened. So all the dead paths were just
| skipped.
| yegle wrote:
| I recall chg was a must.
| jmward01 wrote:
| This is needed, but I don't like new keywords. What I would love,
| for many reasons, is if we could decorate statements. Then things
| like:
|
| import expensive_module
|
| could be:
|
| @lazy
|
| import expensive_module
|
| or you could do:
|
| @retry(3)
|
| x = failure_prone_call(y)
|
| lazy is needed, but maybe there is a more basic change that could
| give more power with more organic syntax, and not create a new
| keyword that is special purpose (and extending an already special
| purpose keyword)
| jmward01 wrote:
| As a side note, It would be great to have a 'preview' for HN
| comments. I updated the above because I forgot to add line
| breaks. Sigh. I bet 'preview' would stop a lot of not well
| thought out comments too.
| progval wrote:
| For what it's worth, you can already do: x =
| retry(3)(failure_prone_call)(y)
| zahlman wrote:
| And, for that matter, expensive_module =
| lazy(importlib.import_module)('expensive_module') .
| lyu07282 wrote:
| Would this statement decorator then manipulate the AST of the
| following statement or how would that work?
| _ZeD_ wrote:
| good luck with that syntax - it would be as possible as passing
| an inline-defined def as parameter instead of a lambda
| waynesonfire wrote:
| why does this have to be a syntax feature and not a lazy code
| loader at the intepreter level?
| Boxxed wrote:
| Ugh...I like the idea, but I wish lazy imports were the default.
| Python allows side effects in the top level though so that would
| be a breaking change.
|
| Soooo instead now we're going to be in a situation where you're
| going to be writing "lazy import ..." 99% of the time: unless
| you're a barbarian, you basically never have side effects at the
| top level.
| zahlman wrote:
| > The standard library provides the LazyLoader class to solve
| some of these inefficiency problems. It permits imports at the
| module level to work _mostly_ like inline imports do.
|
| The use of these sorts of Python import internals is highly non-
| obvious. The Stack Overflow Q&A I found about it
| (https://stackoverflow.com/questions/42703908/) doesn't result in
| an especially nice-looking UX.
|
| So here's a proof of concept in existing Python for getting all
| imports to be lazy automatically, with no special syntax for the
| caller: import sys import threading #
| needed for python 3.13, at least at the REPL, because reasons
| from importlib.util import LazyLoader # this has to be eagerly
| imported! class LazyPathFinder(sys.meta_path[-1]): # <class
| '_frozen_importlib_external.PathFinder'> @classmethod
| def find_spec(cls, fullname, path=None, target=None):
| base = super().find_spec(fullname, path, target)
| base.loader = LazyLoader(base.loader) return base
| sys.meta_path[-1] = LazyPathFinder
|
| We've replaced the "meta path finder" (which implements the logic
| "when the module isn't in sys.modules, look on sys.path for
| source code and/or bytecode, including bytecode in __pycache__
| subfolders, and create a 'spec' for it") with our own wrapper.
| The "loader" attached to the resulting spec is replaced with an
| importlib.util.LazyLoader instance, which wraps the base
| PathFinder's provided loader. When an import statement actually
| imports the module, the name will actually get bound to a <class
| 'importlib.util._LazyModule'> instance, rather than an ordinary
| module. Attempting to access any attribute of this instance will
| trigger the normal module loading procedure -- which even
| replaces the global name.
|
| Now we can do: import this # nothing shows up
| print(type(this)) # <class 'importlib.util._LazyModule'>
| rot13 = this.s # the module is loaded, printing the Zen
| print(type(this)) # <class 'module'>
|
| That said, I don't know what the PEP means by "mostly" here.
| redleader55 wrote:
| Can you explain why does "threading" needs to be loaded? The
| rest seems decently straightforward, but what initialization
| from threading is required and why only in 3.13+?
| comex wrote:
| I don't hate it but I don't love it. It sounds like everyone will
| start writing `lazy` before essentially every single import, with
| rare exceptions where eager importing is actually needed. That
| makes Python code visually noisier. And with no plan to ever
| change the default, the noise will stay forever.
|
| I would have preferred a system where modules opt in to being
| lazy-loaded, with no extra syntax on the import side. That would
| simplify things since only large libraries would have to care
| about laziness. To be fair, in such a design, the interpreter
| would have to eagerly look up imports on the filesystem to decide
| whether they should be lazy-loaded. And there are probably other
| downsides I'm not thinking of.
| BiteCode_dev wrote:
| We heard that about types, the walrus, asyncio, dataclasses and
| so much more. But it didn't happen, if people don't need
| something (and many don't know it exists or what it does), it's
| unlikely they use it.
|
| In fact, half of the community basically uses only a modernized
| set of python 2.4 features and that's one of the beauties of
| the language. You don't need a lot to be productive, and if you
| want more, you can optionally reach for it.
|
| It has worked very well for the last 2 decades and it will
| likely work again.
| hnlmorg wrote:
| People said the same about Perl and its "there's more than
| one way to do things" ethos, which gained much criticism.
|
| Same is true for C++.
|
| In this specific case, I think a lazy load directive isn't a
| bad addition. But one does need to be careful about adding
| new language features just because you have an active
| community.
| comex wrote:
| Perhaps people won't use it. But I for one _want_ it to be
| used, and I will certainly be using it in my own code if the
| PEP is accepted. Startup time is very important to me, more
| important than the cost of making the code noisier. I just
| wish I didn 't have to make that tradeoff in the first place.
| Spivak wrote:
| I don't think this makes sense to be on the module side, the
| caller is the one with the information as to whether the module
| can or needs to be lazily loaded. There's nothing really for
| the module being imported to decide, every module can be lazily
| loaded. Even if it has side effects the caller may want to
| defer those as well.
| syncsynchalt wrote:
| I think side-effects are exactly the problem, you can't have
| the runtime default to lazy-loading all modules without
| breaking code that e.g. relies on side effects running before
| thread creation or forking.
| comex wrote:
| > Even if it has side effects the caller may want to defer
| those as well.
|
| But that's rare, and could be handled with existing
| workarounds.
|
| Normally, a module needs to be eagerly imported if and only
| if it has side effects.
| lenkite wrote:
| Wish pyproject.toml was enhanced to specify lazy loading via
| regexs.
| zahlman wrote:
| I don't follow your reasoning. `pyproject.toml` has nothing
| to do with what happens at runtime. It's about building and
| packaging the code. It also doesn't say anything related to
| the modules that will be imported at runtime. It deals in the
| names of _distributions_ , which are completely independent
| of `import` statements.
| veber-alex wrote:
| I would gladly take a command line flag that I can pass to
| python that makes all module loading lazy.
|
| Unless you are writing scripts or very simple stuff running
| side effects when modules are loaded should be avoided at all
| cost anyway.
| xdfgh1112 wrote:
| That's already part of the PIP. There is a flag to enable
| lazy imports for all possible imports.
| charliermarsh wrote:
| The PEP includes the ability to enable (or disable) lazy
| imports globally via a command-line flag or environment
| variable, in addition to the import syntax.
| _ZeD_ wrote:
| > I would gladly take a command line flag that I can pass to
| python that makes all module loading lazy.
|
| oh, you want a "break my libraries" flag? :D
|
| seriously, in theory lazy imports may be "transparent" for
| common use cases, but I've saw too many modules rely on the
| side effects of the importing, that I understand why they
| needed to make this a "double opt in" feature
| zahlman wrote:
| You can do it today with a few lines of code, although the
| implementation I show is not particularly robust (since it
| works by metaprogramming the import system, of course other
| code could interfere):
| https://news.ycombinator.com/item?id=45467489
| mekoka wrote:
| If everyone starts favoring lazy imports with not much fuss
| then it means that lazy should have been the default behavior
| and _eager_ is the keyword we 're missing. This isn't the first
| time Python revisits this paradigm. Many constructs that used
| to eagerly produce lists in v2 were turned into generators in
| v3 with next to no problems.
| johnfn wrote:
| Yes, there were totally no problems in the transition from 2
| to 3 :)
| mekoka wrote:
| The paradigm I'm alluding to is eager to lazy, not 2 to 3.
| E.g. dict.items() in v3 works like dict.iteritems() in v2.
| enriquto wrote:
| what is the point of this? you can just import inside function
| definitions: def antislash(A, b):
| from numpy.linalg import solve return solve(A, b)
|
| thus numpy.linalg is only imported the first time you call the
| antislash function. Much cleaner than a global import.
|
| Ignore wrong traditions. Put _all_ imports in the innermost
| scopes of your code!
| og_kalu wrote:
| That's a hack that forces you to duplicate and hide imports.
| The tradition is to specify imports at the top because it's
| that much better readability wise.
| gorgoiler wrote:
| I don't really agree with the premise:
|
| > _The dominant convention in Python code is to place all imports
| at the beginning of the file. This avoids repetition, makes
| import dependencies clear and minimizes runtime overhead._
|
| > _A somewhat common way to delay imports is to move the imports
| into functions, but this practice requires more work [and]
| obfuscates the full set of dependencies for a module._
|
| The first part is just saying the traditions exist because the
| traditions have always existed. Traditions are allowed to change!
|
| The second part is basically saying if you do your own logic-
| based lazy imports (inline imports in functions) then you're
| going against the traditions. Again, traditions are allowed to
| change!
|
| The point about the import graph being obfuscated would ring more
| true if Python didn't already provide lightning fast static
| analysis tools like ast. If you care about import graphs at the
| module level then you're probably already automating everything
| with ast anyway, at which point you just walk the whole tree
| looking for imports rather than the top level.
|
| So, really, the whole argument for a new _lazy_ keyword (instead
| of inlining giant imports where they are needed) is because
| people like to see _import pytorch_ at the top of the file, and
| baulk at seeing it -- and will refuse to even look for it! --
| anywhere else? Hmmm.
|
| What _does_ seem like a pain in the ass is having to do this kind
| of repetitive crap (which they mention in their intro):
| def f1(): import pytorch ... def f2():
| import pytorch ... def f3(): import
| pytorch ...
|
| But perhaps the solution is a pattern where you put all your
| stuff like that in your own module and it's _that_ module which
| is lazy loaded instead?
| og_kalu wrote:
| >The second part is basically saying if you do your own logic-
| based lazy imports (inline imports in functions) then you're
| going against the traditions. Again, traditions are allowed to
| change!
|
| No, they are saying the tradition is there for a reason.
| Imports at the beginning of the file makes reasoning about the
| dependencies of a module much easier and faster. I've had to
| deal with both and I sure as hell know which I'd prefer. Lazy
| imports by functions is a sometimes necessary evil and it would
| be very nice if it became unnecessary.
| fastasucan wrote:
| >The second part is basically saying if you do your own logic-
| based lazy imports (inline imports in functions) then you're
| going against the traditions.
|
| Think you need to read again.
| nodesocket wrote:
| Won't this have a very noticeable performance hit on the first
| request? Thinking web frameworks like Flask and Django.
| thayne wrote:
| One thing the PEP doesn't really talk about, and that I find very
| annoying is that many python linters will complain if you don't
| put all of your imports at the top of the file, so you get lint
| warnings if you do the most obvious way to implement lazy
| imports.
|
| And that is actually a problem for more than just performance. In
| some cases, importing at the top might actually just fail. For
| example if you need a platform specific library, but only if it
| is running on that platform.
| bcoates wrote:
| I don't think there is any solution for that but "fix your
| broken linter".
| thayne wrote:
| It isn't just one though. Every linter I've used has warned
| about that.
|
| Probably because PEP 8 says
|
| > Imports are always put at the top of the file, just after
| any module comments and docstrings, and before module globals
| and constants
| BHSPitMonkey wrote:
| Ruff doesn't do this, and in fact even lets you specify
| modules that _must_ not be imported at the top level
| (banned-module-level-imports = [...])
|
| I banished the worst/heaviest libraries to this list at my
| workplace and it's been really helpful at keeping startup
| times from regressing.
| phainopepla2 wrote:
| It is annoying, but most linters will accept a `#noqa E402`
| comment to ignore it
| zahlman wrote:
| >most linters will accept
|
| So, they agreed on a common system of linting error codes? Is
| that documented somewhere?
| thijsvandien wrote:
| Very unfortunate how _from json lazy import dumps_ would result
| in backward compatibility issues. It reads much better and makes
| it easier to search for lazy imports, especially if in the future
| something else becomes optionally lazy as well.
| mekoka wrote:
| Crossing my fingers that it goes through this time. Been in the
| top 3 of my Python wishlist for nearly a decade.
| nilslindemann wrote:
| This is the wrong syntax, comparable to how "u" strings were the
| wrong syntax and "b" strings are the right syntax.
|
| They make that, what should be the default, a special case. Soon,
| every new code will use "lazy". The long term effect of such
| changes is a verbose language syntax.
|
| They should have had a period where one, if they want lazy
| imports, has to do "from __future__ import lazy_import". After
| that period, lazy imports become the default. For old-style
| immediate imports, introduce a syntax: "import now foo" and "from
| foo import now bar".
|
| All which authors of old code would have to do is run a provided
| fix script in the root directory of their code.
| nilslindemann wrote:
| In short, grandiose change, wrong syntax.
| zahlman wrote:
| I think your assessment of what's "the right/wrong" syntax is
| fair. But the transition you describe takes a long time, even
| now that the community has figured out a "deprecation cycle"
| process that seems satisfactory (i.e. won't lead to another
| Python 3.0 situation).
|
| > All which authors of old code would have to do is run a
| provided fix script in the root directory of their code.
|
| As I recall, `lib2to3` didn't do a lot to ease tensions. And
| `six` is still absurdly popular, mainly thanks to `python-
| dateutil` still attempting to support 2.7.
| nilslindemann wrote:
| Honestly, I don't care for 2 to 3 or six (which, btw, has
| just 1k stars on GitHub, compare to e.g. 90k stars for
| FastAPI). Someone who bases their code on 2.7 in 2025 must
| expect to run into trouble.
|
| If people do not run such upgrade scripts, it must be
| documented better.
|
| Python is free. In order to also stay elegant, they should
| say to their users: "We expect from you to run an upgrade
| script on your code once per Python upgrade"
| dataangel wrote:
| This would be a huge deal for Python startup time *if* it was
| applied to all the standard library packages recursively. Right
| now importing asyncio brings in half the standard library through
| transitive imports.
| zahlman wrote:
| It's bad, but it's not _that_ bad. $ time
| python -c '' # baseline real 0m0.020s user
| 0m0.015s sys 0m0.005s $ time python -c 'import
| sys; old = len(sys.modules); import asyncio;
| print(len(sys.modules) - old)' 104 real
| 0m0.076s user 0m0.067s sys 0m0.009s
|
| For comparison, with the (seemingly optimized) Numpy included
| with my system: $ time python -c 'import sys;
| old = len(sys.modules); import numpy; print(len(sys.modules) -
| old)' 185 real 0m0.124s user 0m0.098s
| sys 0m0.026s
| romanows wrote:
| Kinda related, I wish there was an easy way to exclude
| dependencies at pip-install time and mock them at runtime so an
| import doesn't cause an exception. Basically a way for me to
| approximate "extras" when the author isn't motivated to do it for
| me, even though it'd be super brittle.
| zahlman wrote:
| This sounds doable, actually. You'd want to pre-install (say,
| from a local wheel) a matching dummy dependency where the
| metadata claims that it's the right version of whatever package
| (so the installer will just see that the dependency is "already
| satisfied" and skip it), but the actual implementation code
| just exposes a hook to your mocking system.
|
| Doesn't work if version resolution decides to upgrade or
| downgrade your installed package, so you need to make sure the
| declared version is satisfactory, too.
| throwaway81523 wrote:
| We need some kind of unexec revival in Linux so we don't have to
| resort to crap like this. Maybe CRIU based. Worst case, some
| Python specific hack. But Python's import system is wacky enough
| without lazy loading. This sounds like combining the worst of
| multiple worlds.
| jacquesm wrote:
| Lazy imports are a great way to create runtime errors far into
| the operation of a long lived service. Yes, it gives the
| superficial benefit of 'fast startup', but that upside is negated
| by the downside of not being sure that once something runs it
| will run to completion due to a failed import much further down
| the line. It also allows for some interesting edge cases with the
| items that are going to be imported no longer being what is on
| the tin at the time the program is started.
| og_kalu wrote:
| That's fine, because this is still a genuine problem in need of
| a solution. It's not just about startup time for the sake of it
| (not that this is even a superficial concern - python startup
| time with large dependencies quickly gets awful). Large
| projects can have hefty dependencies that not every user will
| use. And bundling it all for everyone can sometimes be
| intractable. The work arounds people use already have the sort
| of problems you're talking about, on top of being diabolical
| and hacky. Not having to duplicate and hide imports in
| functions alone would be a big improvement. It's not like it
| isn't being proposed as an optional language feature.
| BHSPitMonkey wrote:
| An automated test mitigates the risk you describe, and is well
| worth the tradeoff for fast startup.
|
| I don't consider startup time "superficial" at all; I work in a
| Django monolith where this problem resulted in each and every
| management command, test invokation, and container reload
| incurring a 10-15sec penalty because of just a handful of
| heavy-to-import libraries used by certain tasks/views.
| Deferring these made a massive difference.
| d_burfoot wrote:
| Looks good to me. I use a tab-completion trick where the tab-
| completer tool calls the script I'm about to invoke with special
| arguments, and the script reflects on itself and responds with
| possible completions. But because of slow imports, it often takes
| a while for the completion to respond.
|
| I could, and sometimes do, go through all the imports to figure
| out which ones are taking a long time to load, but it's a chore.
| bcoates wrote:
| I think they're understating the thread safety risks here. The
| import is going to wind up happening at a random nondeterministic
| time, in who knows what thread holding who knows what locks
| (aside from the importer lock).
|
| Previously, if you had some thread hazardous code at module
| import time, it was highly likely to only run during the single
| threaded process startup phase, so it was likely harmless. Lazy
| loading is going to unearth these errors in the most inconvenient
| way (as Heisenbugs)
|
| (Function level import can trigger this as well, but the top of a
| function is at least a slightly more deterministic place for
| imports to happen, and an explicit line of syntax triggering the
| import/bug)
| Too wrote:
| This is a great compromise given how much would break if this was
| the default. Making this the default would be better in the long-
| run, require taming the abuse of side-effects in imports too,
| win-win.
|
| If one could dream, modules should have to explicitly declare
| whether they have side-effects or not, with a marker at the top
| of the module. Not declaring this and trying anything except
| declaring a function or class should lead to type-checker errors.
| Modules declaring this pure-marker then automatically become
| lazy. Others should require explicit "import_with_side_effects"
| keyword. __pure__ = True import logging
| import threading app = Flask() # <<
| ImpureError sys.path.append() # << ImpureError
| with open(.. ) # << ImpureError
| logging.basicSetup() # << ImpureError if foo:
| # << ImpureError (arguable) @app.route("/foo")
| # Questionable def serve(): # OK
| ... serve() # << ImpureError
| t = threading.Thread(target=serve) # << ImpureError
|
| All of this would be impossible today, given how much the Python
| relies on metaprogramming. Even standard library exposes
| functions to create classes on the fly like Enum and dataclasses,
| that are difficult to assert as either pure or impure. With more
| and more of the ecosystem embracing typed Python, this
| metaprogramming is reducing into a less dynamic subset. Type
| checkers and LSPs must have at least some awareness of these
| modules, without executing them as plain python-code.
| ajb wrote:
| Given all the problems people are mentioning, it seems like this
| proposal is on the wrong side. There should be an easy way for a
| module to declare itself to be lazy loaded. The module author,
| not the user, is the one who knows whether lazy loading will
| break stuff.
| Grikbdl wrote:
| It's also simpler. It could even be a package level thing
| similar to typed.py marker. I don't want to pepper literally
| all my modules with loads of explicit lazy keywords.
| zahlman wrote:
| > There should be an easy way for a module to declare itself to
| be lazy loaded.
|
| It can just implement lazy loading itself today, by using
| module-level __getattr__ (https://docs.python.org/3/reference/d
| atamodel.html#customizi...) to overwrite itself with a private
| implementation module at the appropriate time. Something like:
| # foo.py def __getattr__(name): # clean up the
| lazy loader before loading # this way it's cleaned up
| if the implementation doesn't replace it, # and not
| scrubbed if it does global __getattr__ del
| __getattr__ import sys self =
| sys.modules[__name__] from . import _foo #
| "star-import" by adding names that show up in __dir__
| self.__dict__.update(**{k: getattr(_foo, k) for k in
| _foo.__dir__()}) # On future attribute lookups,
| everything will be in place. # But this time, we need
| to delegate to the normal lookup explicitly return
| getattr(self, name)
|
| Genericizing this is left as an exercise.
| pjjpo wrote:
| I wonder if this proposal suffers an because of Python's
| extremely generous support period and perhaps the ship has
| sailed.
|
| - lazy imports are a hugely impactful feature
|
| - lazy imports are already possible without syntax
|
| This means any libraries that get large benefit from lazy imports
| already use import statements within functions. They can't really
| use the new feature since 3.14 EoL is _2030_, forever from now.
| The __lazy_modules__ syntax preserves compatibility only but
| being eager, not performance - libraries that need lazy imports
| can't use it until 2030.
|
| This means that the primary target for a long time is CLI
| authors, which can have a more strict python target and is
| mentioned many times in the PEP, and libraries that don't have
| broad Python version support (meaning not just works but works
| well), which indicates they are probably not major libraries.
|
| Unless the feature gets backported to 2+ versions, it feels not
| so compelling. But given how it modifies the interpreter to a
| reasonable degree, I wonder if even any backport is on the table.
| Grikbdl wrote:
| In at least the scientific python environment, there's "SPEC0"
| in which a lot of the de facto core libraries have basically
| agreed to support the last three versions of Python, no more.
|
| For other libraries they can of course choose as they want, but
| generally I don't think it's so common for libraries to be as
| generous with the support length as cpython.
| shiandow wrote:
| I don't quite understand why they forbid this from being used in
| functions.
|
| I mean I get why that makes the most sense in most scenarios, but
| it is not as if the problem of having to choose between declaring
| dependencies up front or deferring expensive imports until needed
| does not happen in functions.
|
| Take for instance a function that quickly fails because the
| arguments are incorrect, it might do a whole bunch of imports
| that only make sense for that function but which are made
| immediately obsolete.
|
| It feels like it is forbidden just because someone thought it
| wasn't a good coding style but to me there is no obvious reason
| it couldn't work.
| Hendrikto wrote:
| Python imports are already way too complicated. I don't think
| even more syntax should be added.
| hhthrowaway1230 wrote:
| doesn't this make the language a little unpredictable in terms of
| loading times? requiring to touch all parts to fully load the
| app?
| amdsn wrote:
| I think a feature like this sees best use in short lived
| programs (where startup time is a disproportionate percentage
| of total run time) and programs where really fast startup is
| essential. There are plenty of places where I could imagine
| taking advantage of this in my code at work immediately, but I
| share your concern about unpredictability when libraries we use
| are also making use of it. It wouldn't be fun to have to dive
| into dependencies to see what needs to be touched to trigger
| lazy imports at the most convenient time. Unless I am
| misunderstanding and a normal import of a module means that all
| of its lazy imports also become non-lazy?
| f33d5173 wrote:
| It would be interesting if instead you added a syntax whereby a
| module could declare that it supported lazy importing. Maybe even
| after running some code with side effects that couldn't be done
| lazily. For one thing, this would have a much broader performance
| impact, since it would benefit all users of the library, not just
| those who explicitly tagged their imports as lazy. For another,
| it would minimize breakage, since a module author knows best
| whether, and which parts of, their module can be lazily loaded.
|
| On the other hand, it would create confusion for users of a
| library when the performance hit of importing a library was
| delayed to the site of usage. They might not expect, for example,
| a lag to occur there. I don't think it would cause outright
| breakage, but people might not like the way it behaved.
| croemer wrote:
| The link target should be https://peps.python.org/pep-0810/ not a
| preview build for a PR. @dang
| wmichelin wrote:
| This seems unnecessary if we can run imports from functions,
| right? This feels like a layer of indirection and a potential
| source for confusion. Rather than implicitly importing a library
| when the variable is first used, why don't you just explicitly do
| it?
|
| Edit
|
| > A somewhat common way to delay imports is to move the imports
| into functions (inline imports), but this practice requires more
| work to implement and maintain, and can be subverted by a single
| inadvertent top-level import. Additionally, it obfuscates the
| full set of dependencies for a module. Analysis of the Python
| standard library shows that approximately 17% of all imports
| outside tests (nearly 3500 total imports across 730 files) are
| already placed inside functions or methods specifically to defer
| their execution. This demonstrates that developers are already
| manually implementing lazy imports in performance-sensitive code,
| but doing so requires scattering imports throughout the codebase
| and makes the full dependency graph harder to understand at a
| glance.
|
| I think this is a really weak foundation for this language
| feature. We don't need it.
| tremon wrote:
| On the contrary, I think "everyone is already doing this in
| their own way, so we will offer it as a language feature, which
| brings these extra benefits" is one of the stronger arguments
| for introducing a language feature.
| wsve wrote:
| > Rather than implicitly importing a library when the variable
| is first used, why don't you just explicitly do it?
|
| I think it's on you to explain why that's a better approach for
| _everyone 's_ use cases instead of this language feature
___________________________________________________________________
(page generated 2025-10-04 23:01 UTC)