[HN Gopher] Moving SciPy to the Meson Build System
___________________________________________________________________
Moving SciPy to the Meson Build System
Author : stockholmsin
Score : 85 points
Date : 2021-07-25 22:48 UTC (2 days ago)
(HTM) web link (labs.quansight.org)
(TXT) w3m dump (labs.quansight.org)
| rkrzr wrote:
| > Really there were only two viable candidates: CMake and Meson.
|
| Would be great to hear why those were the only two candidates
| that they considered (and not e.g. Bazel or Nix).
| chpatrick wrote:
| Nix works at a higher level and usually invokes CMake, make,
| Meson, etc. It doesn't have its own C/C++ build system.
| prpl wrote:
| SciPy is a multi platform library that is distributed, most
| famously on pypi but in many other ways (conda, spack, rpm,
| deb, etc...). How can nix handle that without requiring the
| user to install nix?
| chromatin wrote:
| Yes, perhaps it is true, and it is obvious to build experts,
| but as a novice in this area (build systems), I would love for
| some more exposition.
| flohofwoe wrote:
| One typical problem for C/C++ build systems is lack of proper
| Windows and MSVC toolchain support. Most C/C++ build systems
| originated in the UNIX world, and either only support GCC-
| compatible toolchains or if they support MSVC then only as an
| afterthought. Trying to get this stuff running in a Windows
| environment is often a mix of frustation and plain hilarity.
| CMake is one of the few build tools that gets Windows and
| MSVC support right.
| rjsw wrote:
| Haven't read the article but meson will run on a lot more
| systems than bazel.
| ihnorton wrote:
| Nix is likely a nonstarter because as far as I can tell it does
| not natively support Visual Studio and MSVC.
|
| I suspect Bazel was ruled out because it requires the JVM and
| it has limited open source uptake relative to CMake (huge open
| source userbase), and Meson (limited presence in open source
| scientific software, but adopted by GNOME and systemd).
| mumblemumble wrote:
| I can't speak for scipy, but I ruled Bazel out for a project
| that included Python components a few years ago. It was great
| for C, C++, and Java projects, but, at least at the time,
| there was no official Python support. I looked into third-
| party options and the feasibility of building it myself. Most
| of what I found was a few different conference talks and blog
| posts where the presenter was enthusiastically talking about
| how they'd spent a year trying to get it to work and it's not
| quite there yet but they're feeling really really confident
| that they'll turn a corner sometime soon. But I could never
| find any subsequent evidence that the presenter's team
| actually had turned that corner.
|
| Based on that, I just sort of assumed that, in addition to
| all the fairly well-documented up-front challenges that these
| folks had identified and were talking about, there must also
| be some impassable barrier lurking around in there that
| nobody finds until they've already sunk a lot of time and
| money into trying to get it working. I don't know what that
| is, and I'm not curious enough to spend the better part of a
| year trying to find it for myself. I ended up choosing
| Gradle.
| prpl wrote:
| What was your artifacts and what did you choose instead?
| [deleted]
| dfinninger wrote:
| Our team started moving to Bazel two years ago with 0.21 or
| something. Python support is much, much better these days.
| We're building Python with requirements and packing it into
| Docker containers.
|
| It went from "maybe run stdlib Python in an activated venv"
| to actually working.
| mumblemumble wrote:
| That's great to hear. Yeah, 2 years ago would have been
| after I made this decision. Looking forward to taking
| another look at Bazel; I strenuously wanted it to be the
| one.
| nonameiguess wrote:
| nix-build relies on symlinks, which seemingly can't work on
| Windows. And it's not really a build system. Bazel relies on a
| JVM and it's really nice to not have one of those on your
| system if you don't absolutely need it.
|
| Depending on how minimal you want to get, I think native build
| systems where you're not just building for yourself, but
| distributing source bundles that need to be built by others,
| have to start the discussion with Autotools, CMake, and Meson.
| The reason is just that they're already required by everything
| else anyway, so you're not adding any burden to the downstream
| packagers by using them.
|
| If you're just building for yourself, have at it, get as
| bespoke and obscure as you want, but SciPy isn't being built
| and deployed primarily on NumFOCUS's own servers. It's being
| distributed as a tarball to its users who mostly build it
| themselves. You should make every effort to use a build system
| they're likely to already have and already be familiar with.
|
| In particular, much of the focus here seems aimed at users who
| want to cross-compile for embedded Linuxes running on ARM. I
| doubt those people want to pull in a JVM.
|
| Kind of reminds me of when I was working for a geointelligence
| agency charged with building and integrating third-party ground
| processing algorithms into a user-facing web tasking framework.
| We have to retrieve and build the dependencies, too. xerces-c,
| Armadillo, MKL, no problem. Just keeping downloading, either
| cmake .. or configure, make, then make install, over and over.
|
| Then one of them requires TensorFlow and it becomes a two-month
| research project trying to figure out how to build it (sorry
| Google, but the military does not trust your pre-built
| binaries).
| ihnorton wrote:
| This is interesting. It will certainly give a boost to Meson,
| which is in part a _better CMake_ (the compilation model is very
| similar). It probably makes a lot of sense for SciPy given the
| alternatives, but from an ecosystem perspective it 's not clear
| whether it is better enough to justify the churn. For example,
| caching seems like a secondary concern and outsourced to Linux-
| specific technologies. This is unfortunate given the implications
| of build time for individual and team (CI) productivity, as well
| as the environmental considerations of redundant compilation at
| scale.
|
| I haven't followed Meson closely in about 3 years, but I also got
| the sense that Windows support sometimes lagged. If that's true,
| it's going to be a tough sell for the many large C++ projects who
| adopted CMake almost exclusively due to its support for Windows
| and Visual Studio.
| misnome wrote:
| This is interesting to read. I've used CMake on-and-off for mixed
| python/C++ code bases, and in particular https://scikit-
| build.readthedocs.io/en/latest/ which aims to integrate with
| cmake (lightly mentioned here but not evaluated as "getting away
| from setuptools".
|
| But I have never been satisfied with the solutions for out-of-
| tree builds with python - being able to debug-in-place extensions
| _and_ eventually install in a single package structure seems to
| be somewhat in conflict with the way that Python allows you to
| structure packaging, without lots of environment variable hacks.
| kvnhn wrote:
| > Cross-compiling will become possible. For years we've told
| people "sorry, distutils wasn't really made for cross-compiling,
| let us know if you have any luck". As a result we've completely
| ignored users on some exotic platforms, and also spent a lot of
| time fighting with different CI systems to do native builds.
|
| As a user (not a developer) of SciPy, this is the big win. My
| "exotic platform" is embedded Linux distributions via
| Buildroot[1]. This opens the door to many downstream libraries
| becoming available as well, such as pandas and scikit-learn.
|
| [1]: https://buildroot.org/
| tr33house wrote:
| How does Meson compare to other build systems like bagel?
| qznc wrote:
| It is Cmake with Python syntax.
| kortex wrote:
| That doesn't strike me as a good characterization. Especially
| because both Python and Cmake are quite imperative. Meson is
| declarative with an emphasis on immutability.
|
| It's "cmake in role only" - both are build systems, that's
| where the similarity falls apart.
|
| The syntax is very pythonic, however, I think that's true of
| many ergonomic centric formats.
| SAI_Peregrinus wrote:
| CMake is two smaller systems in a trenchcoat, with a creepy
| stalker following along.
|
| There's the 3.0+ CMakeLists.txt syntax, which is a nice
| declarative target-based system.
|
| There's the *.cmake language syntax for finding
| dependencies and other scripting, which is a C'thonian
| horror mix of imperative and declarative bits that the
| trench coat tries to hide.
|
| Then there's the creepy pre-3.0 stalker, the old syntax for
| both. Non-target based, imperative bits sneaking in
| everywhere, makes Autotools look sane.
| tomn wrote:
| Meson internals have nothing to do with cmake, and the syntax
| is not python.
|
| If you want to make that comparison, meson is more
| restrictive than cmake, but tends to have more functionality
| built-in to make up for it.
|
| bazel is more all-encompasing than either, as it's more about
| reproducible and distributed builds than a "better makefile"
| type system.
| xvilka wrote:
| It would be great to port Meson from Python to pure C for
| portability. There is Boson[1] attempt to do exactly this but
| it's just a first step towards the goal.
|
| [1] https://sr.ht/~bl4ckb0ne/boson/
| nrclark wrote:
| This is great. I used to support an line of Linux-based realtime
| audio products, and our engineers wanted me to include Scipy in
| the internal developer builds. It turned out to be a really tough
| ask, because of Scipy's build system. The result was a tech-debt
| blemish on an otherwise excellent codebase.
|
| Scipy is very, very hard to cross-compile. Like, almost
| impossible. I eventually had to compile it on-target and store
| the outputs for inclusion in later images.
|
| Distutils in general is wretched at cross-compilation tasks,
| should be retired.
| ho_schi wrote:
| I'm using Meson for a project using C++ with Gtkmm,
| Libmicrohttpd, catch2 and other libraries. My experience is good
| neglecting some minor issues. Meson is fast, well readable if
| written well, uses system libraries by default and allows
| flexible usage.
|
| On the other side I'm using Java with Maven and it is - a big
| burden. It's build in dependency retrieval system also isn't
| helping. Maybe I just don't like XML - because it is not human-
| readable ;)
| 0x138d5 wrote:
| Any particular reason you use Maven instead of Gradle?
| zaphirplane wrote:
| Not op Each gradle task is artistically handcrafted written
| to capture the life experience and pain the author has
| experienced in their life
|
| :) it devolves into a custom shell script
| hawski wrote:
| I'm sorry to pick a nit, but could you expand on "well readable
| if written well"? I would say that a Makefile or even a
| CMakeLists.txt are readable if written well.
| mumblemumble wrote:
| For my part, I find that being written well is necessary but
| not sufficient for readability in a Makefile.
|
| The other big one is that you need to have a strong
| familiarity with Make, and use it often. Familiar because
| there's no way a person who hasn't actually been through some
| form of Make documentation in detail can even guess at what
| things like $@, $?, and $^ do, or accurately decipher its
| macro replacement syntax, or any of that. And use it often
| because, even if you were deeply familiar with Make in the
| past, if you haven't touched it in a few years, you're
| unlikely to reliably remember it without help.
| vips7L wrote:
| I find pom.xml much more readable than deciphering a giant
| script file like build.gradle or CMakeLists.txt. 99% of the
| time I automatically know how the build works because its
| convention.
| mumblemumble wrote:
| For me, Gradle's readability depends heavily on how you use
| it.
|
| If you use it Maven-style, with relatively simple build
| scripts and all the complexity pushed out to plugins, it's
| very readable. A fair bit more readable than Maven, in my
| opinion, but I wouldn't care to argue that point in
| particular.
|
| If you use it the way Stack Overflow tells you to do it, so
| that your build scripts are basically glorified ad-hoc Groovy
| programs, then, yeah, it's just impossible.
|
| Unfortunately, since the only way to really understand any of
| Gradle is to understand all of Gradle, and understanding all
| of Gradle is a huge effort, it's kind of a catch-22. For most
| people, the only really sensible way to use it is to Google
| for advice and then do what Stack Overflow tells you to do.
| But that invariably leads to a result that is not even
| remotely sensible.
|
| Long story short, my hot take is this. Maven is write-only.
| Gradle (ideal usage) is read-only. Gradle (normal usage) can
| be neither read nor written.
| pjmlp wrote:
| On top of that each Gradle release changes a couple of DSL
| stuff, and the Android plugin is an hopeless case.
| jkbbwr wrote:
| Meson is okay, it solved a few problems for us on our deployment
| but the creator has a few hard decisions and style choices that
| don't really gel well with other projects. We are tempted to go
| back to cmake.
___________________________________________________________________
(page generated 2021-07-27 23:02 UTC)