[HN Gopher] Why SciPy builds for Python 3.12 on Windows are a mi...
___________________________________________________________________
Why SciPy builds for Python 3.12 on Windows are a minor miracle
Author : todsacerdoti
Score : 122 points
Date : 2023-11-08 20:24 UTC (2 hours ago)
(HTM) web link (labs.quansight.org)
(TXT) w3m dump (labs.quansight.org)
| pklausler wrote:
| I'm very glad that f18 worked (so far as we know). I haven't
| tried building SciPy for any platform yet myself, much less
| Windows.
| gamache wrote:
| One nit to pick: as far as I am aware, "aarch64" and "arm64" are
| the same thing. Am I off?
| nerdponx wrote:
| What I see in Python is that "aarch64" usually refers to Linux
| and "arm64" usually refers to MacOS ARM. I don't know enough
| about these things to understand why they have different names.
| mschuster91 wrote:
| They are the same thing but there used to be two competing LLVM
| implementations on the backend side.
|
| [1] https://www.phoronix.com/news/MTY5ODk
| gamache wrote:
| Thanks, that is very helpful!
| stefan_ wrote:
| Obligatory: https://lkml.org/lkml/2012/7/15/133
| cozzyd wrote:
| the build system churn in python is really hard to keep up with.
|
| I'm also curious about performance numbers on Windows (though, to
| first order, it doesn't matter... anything serious is probably
| running on a Linux machine).
| nerdponx wrote:
| Fortunately it should be slowing down now. The big transition
| was getting everyone on board with PEP 517, especially
| converting legacy Setuptools projects.
| lmm wrote:
| I've been hearing something like this about Python builds
| every two or three years for decades. How long until the next
| "big transition" is "converting legacy PEP 517 projects"?
| drtournier wrote:
| https://xkcd.com/2347/
| josh-sematic wrote:
| Also https://xkcd.com/1987/
| tedunangst wrote:
| > In particular, Meson was going to refuse to accept the
| MSVC+gfortran combination that was in use in conda-forge.
|
| This sounds like a bug? The point of a build tool is to run the
| commands you tell it, not tell you, sorry dave.
| rbancroft wrote:
| The article is well-written and detailed, however I was taken
| aback by the claim that meson is 'widely used for C & C++
| projects'. I've come across bazel more often than meson. I
| guess because meson is written in python it seemed like a good
| choice for SciPy, and it worked out in the end so
| congratulations.
|
| But yeah, I think CMake is still the gold standard despite all
| its quirks, complexities and problems.
| bsder wrote:
| Meson is a front end.
|
| Edit: Caution: This statement is incorrect--It can generate
| CMakefiles as a backend. It can also generate build files for
| MSVC. It can also generate standard Makefiles.
|
| Edit: Corrected downstream. CMake can make Ninja files which
| Meson also makes. I got this backwards but kept the edit so
| people won't get confused.
|
| CMake is _NOT_ a gold standard. CMake is an agglomerative
| disaster.
|
| For example: try getting CMake to accept zig as your
| compiler. "Oh, your compiler command has a space in it? So
| sorry. I'm going to put everything after the space in _all
| manner of weird places_. Some correct--some broken--some
| totally random. " If you're _lucky_ CMake crashes with an
| inscrutable message. If you 're unlucky, you wind up with a
| compiler command that fails in bizarre ways and no way to
| figure out why CMake is doing what it did.
|
| This is my experience with CMake _every damn time_ --some
| absolutely inscrutable bug pops up until I figure out how to
| route around it. If I'm _really_ unlucky, I have to file a
| bug report with CMake as I _can 't_ route around it.
|
| Sure, if some unfortunate soul has beaten CMake into
| submission and produced a functioning CMakefile, CMake works.
| If _YOU_ are the poor slob having to create that CMakefile,
| you are in for worlds and worlds and worlds of pain.
| bonzini wrote:
| Note that Meson does not generate CMake files. It generates
| MSVC or ninja files.
| bsder wrote:
| Sorry. You are correct. I forgot that it's CMake that can
| now generate Ninja files.
| wenc wrote:
| Projects that use Meson
|
| https://mesonbuild.com/Users.html
|
| Includes Gimp, Gtk+, nautilus, Postgres, qemu, Wayland...
| gavinhoward wrote:
| I agree.
|
| Disclaimer: author of a soon-to-be released Meson competitor.
|
| But one obscure, little gem of a rant that taught me what you
| just said is [1].
|
| tl;dr: Build systems should run the commands you _tell_ them to
| run, period. Because sometimes, the programmer actually does
| know what he 's doing.
|
| I am ashamed to admit that before I read that comment, I was
| thinking about making my build system magical. But after
| reading that comment, I realized that "magic" is why people
| _hate_ build systems.
|
| [1]: https://ofekshilon.com/2016/08/30/cmake-
| rants/#comment-29273
| aidenn0 wrote:
| Meson does more than run the command you tell it, it can also
| synthesize those commands (so you can support MSVC/gcc/clang
| without writing build rules). If you ask it to synthesize a
| command for a combination it doesn't know about, of course it's
| going to tell you "sorry dave"
| tedunangst wrote:
| Yeah, sure, 40 years ago make had implicit rules for building
| .o files from .c files so you didn't have to write them, but
| if you told it your compiler was going to be bananac, it
| wouldn't refuse to run.
| bonzini wrote:
| It's the MSVC linker that would complain. The problem is that
| the C runtimes used by MSVC and gfortran (whose own runtime
| library is written in C) are not ABI-compatible. The hack that
| numpy used was to link the Fortran objects into a DLL to add a
| level of indirection (the import library) that would pacify
| MSVC.
|
| So there was some extra work needed to create these DLLs.
| Either in the build description files, or in Meson. The SciPy
| people didn't want to implement this indirection in either
| place, and the Meson developers were not eager to help them
| either (they did help in general, for example with Fortran and
| Cython support; but they don't want to provide footguns)
| because it was indeed a hack. It only worked because the
| Fortran side didn't use files that were opened on the Python/C
| side, for example.
|
| https://web.archive.org/web/20180711144501/https://pav.iki.f...
| bsder wrote:
| Thank you for the clarification. Meson refusing to do
| something and the developers not deeming that a bug seemed
| really unusual.
| bonzini wrote:
| Oh, Meson _is_ definitely a "sorry Dave" kind of build
| system. In many cases it's extremely opinionated, though I
| have only found a couple cases that get into "infuriating"
| territory.
|
| It does compensate by generally preserving a lot more
| sanity than its competitors, and having a readable and
| maintainable description of the build system.
| wenc wrote:
| What an amazing read. Now I know why my pip installs are failing
| in 3.12 but we now have a brighter future ahead.
|
| Also while I love Python it's helpful to understand why Python
| packaging is a (manageable) mess. It's because of non
| standardization of build tools for C/C++/fortran and the
| immensity of the ecosystem, nothing to do with Python itself.
| It's part irreducible complexity.
|
| It's a miracle it works at all.
| IshKebab wrote:
| It's _also_ to do with Python itself.
| wenc wrote:
| How?
| cdavid wrote:
| Yes, that's a fundamental reason python packaging is a mess.
| Python success is largely due to the availability of key mixed
| language packages. No other mainstream language package manager
| has to deal with this.
|
| For example, cargo for rust, which is great, can assume to
| package mostly rust-only code. And while it is compiled, the
| language "owns" the compiler, which means building from sources
| as distribution strategy works. I don't know how/if cargo can
| deal with e.g. fortran out of the box, but I doubt cargo on
| windows would work well if top cargo packages required fortran
| code.
|
| The single biggest improvement for python ecosystem was the
| standardisation of a binary package format, wheel. It is only
| then that the whole scientific python ecosystem started to
| thrive on windows. But binary compatibility is a huge PITA,
| especially across languages and CPUs.
| bee_rider wrote:
| I'm under the impression that the best BLASes are mostly C (MKL,
| Blis, and OpenBlas). I wonder how far they could get with just C
| and Python.
|
| I wonder if they'd just go with libflame instead, if they started
| today.
|
| Of course there's lots of other functionality in scipy; iterative
| stuff, sparse stuff, etc etc, so maybe Fortran is unavoidable
| (although, Fortran is a great language, I'm glad the tooling
| situation is at least starting to improve on Windows).
| cdavid wrote:
| The question of removing fortran from scipy has happened a few
| times, and never got anywhere for the reasons you gave. A lot
| of scipy itself contains fortran code that would take man years
| to rewrite.
|
| Several key parts using fortran have been removed, once it
| became possible. For example fft stuff.
| d3m0t3p wrote:
| In their table about compilers, isn't AArch64 and ARM64 the same
| ?
| tempay wrote:
| Yes, though different operating systems use different naming.
| On Linux it's `aarch64` and on macOS+Windows it's `arm64`.
| Grimburger wrote:
| > GCC..there are ways to make it work on Windows (through cygwin
| or MinGW)
|
| I assume it would be working on WSL also? Which seems like a
| simple solution to the mess that is developing on Windows,
| especially in Python.
| nabakin wrote:
| Funny that this shows up so soon after my work laptop's anti-
| virus blocked my scipy install.
| FrustratedMonky wrote:
| Maybe move away from Python altogether?
| bonzini wrote:
| Replace it with what, even without considering that this is sci
| _py_?
| mkoubaa wrote:
| Great idea! Why don't you go suggest to the developers of
| sciPY??
| mattbillenstein wrote:
| Very naive question, but are the semantics of Fortran so
| different that it can't be translated to C first and then
| compiled using a C compiler? Perhaps maintained in C going
| forward?
|
| I can't imagine there are a lot of Fortran folks around
| maintaining these old libraries - they must need maintenance no?
| googl-free wrote:
| yeah, fortran has native arrays
| lldb wrote:
| We have this already - "f2c" has been around for decades.
| mkoubaa wrote:
| Fortran is a higher level language than C. There are plenty of
| fortran developers.
|
| It's awful for application development, but that isn't it's
| niche
___________________________________________________________________
(page generated 2023-11-08 23:00 UTC)