[HN Gopher] Zb: An Early-Stage Build System
___________________________________________________________________
Zb: An Early-Stage Build System
Author : zombiezen
Score : 122 points
Date : 2024-09-19 19:16 UTC (3 hours ago)
(HTM) web link (www.zombiezen.com)
(TXT) w3m dump (www.zombiezen.com)
| evanjrowley wrote:
| This looks really exciting and I absolutely must give it a try.
| Well done! At face value the vision and design choices appear to
| be great.
| zombiezen wrote:
| Thank you! <3
| ramon156 wrote:
| I'd definitely write a build systen in lua, looks promising!
| Iceland_jack wrote:
| I appreciate the link to https://dl.acm.org/doi/10.1145/3236774
| o11c wrote:
| Definitely interesting, but it's flat-out wrong about the
| limitations of `make`.
|
| In particular, the `release.txt` task is trivial by adding a
| dummy rule to generate and include dependencies; see
| https://www.gnu.org/software/make/manual/html_node/Remaking-...
| (be sure to add empty rules to handle the case of deleted
| dynamic dependencies). You can use hashes instead of file
| modification times by adding a different kind of dummy rule.
| The only downside is that you have to think about the
| performance a little.
|
| I imagine it's _possible_ for a project to have some kind of
| dynamic dependencies that GNU make can 't handle, but I dare
| say that any such dependency tree is hard to understand for
| humans too, and thus should be avoided regardless. By contrast,
| in many other build tools it is _impossible_ to handle some of
| the things that are trivial in `make`.
|
| (if you're not using GNU make, _you_ are the problem; do not
| blame `make`)
| packetlost wrote:
| One request that I would make of a project like this is to
| support distributed builds out of the box. Like, really basic
| support for identical builder hosts (this is much easier now than
| in the past with containers) and caching of targets. Otherwise,
| this looks great! Big fan of the choice of Lua, though the
| modifications to strings might make it difficult to onboard new
| users depending on how the modification was made.
| zombiezen wrote:
| Yup, remote building and caching is on my radar. I expect it
| will work much in the same way Nix does now, although I'm being
| a bit more deliberate in creating an RPC layer so build
| coordinators and other such tools are more straightforward to
| build.
|
| The string tweak is transparent to users broadly speaking. IME
| with Nix this thing works the way people expect (i.e if you use
| a dependency variable in your build target, it adds a
| dependency).
| Glacia wrote:
| Xmake?
| bsnnkv wrote:
| You had my interest at Windows support! I'll carve out some time
| this weekend to see if I can write a build for komorebi
| zombiezen wrote:
| Nice! It might be a little too rough until I've got a working C
| compiler for Windows: https://github.com/256lights/zb/issues/6
| (and Linux for that matter:
| https://github.com/256lights/zb/issues/30)
| steeleduncan wrote:
| Looks great, Nix-with-Lua that also supports Windows would be
| amazing. Two questions if I may
|
| - Does this sandbox builds the way flakes do?
|
| - What is MinGW used for on Windows? Does this rely on the MinGW
| userland, or is it just because it would be painful to write a
| full bootstrap for a windows compiler while also developing Zb?
|
| Also, its great to see the live-bootstrap in there. I love the
| purity of how Guix's packages are built, and I like the idea Zb
| will be that way from the start
| zombiezen wrote:
| Nix sandboxes derivation runs on Linux even without flakes, and
| I'm planning on implementing that, yes:
| https://github.com/256lights/zb/issues/29 and
| https://github.com/256lights/zb/issues/31
|
| MinGW is used to build Lua using cgo. I'd like to remove that
| part, see https://github.com/256lights/zb/issues/28 I haven't
| started the userspace for Windows yet
| (https://github.com/256lights/zb/issues/6), but I suspect that
| it will be more "download the Visual C++ compiler binary from
| this URL" than the Linux source bootstrap.
|
| Yeah, I'm happy with live-bootstrap, too! I tried emulating
| Guix's bootstrap, but it depended a little too much on Scheme
| for me to use as-is. live-bootstrap has mostly worked out-of-
| the-box, which was a great validation test for this approach.
| laurentlb wrote:
| I'd like to know more about the "Support for non-determinism" and
| how that differs from other build systems. Usually, build systems
| rerun actions when at least one of the inputs has changed. Are
| non-deterministic targets rerun all the time?
|
| Also, I'm curious to know if you've considered using Starlark or
| the build file syntax used in multiple other recent build systems
| (Bazel, Buck, Please, Pants).
| zombiezen wrote:
| (Hi! I recognize your name from Bazel mailing lists but I
| forget whether we've talked before.)
|
| I'm mostly contrasting from Nix, which has difficulty with
| poisoning cache when faced with non-deterministic build steps
| when using input-addressing (the default mode). If zb
| encounters a build target with multiple cached outputs for the
| same inputs, it rebuilds and then relies on content-addressing
| to obtain build outputs for subsequent steps if possible. (I
| have an open issue for marking a target as intentionally non-
| deterministic and always triggering this re-run behavior:
| https://github.com/256lights/zb/issues/33)
|
| I'll admit I haven't done my research into how Bazel handles
| non-determinism, especially nowadays, so I can't remark there.
| I know from my Google days that even writing genrules you had
| to be careful about introducing non-determinism, but I forget
| how that failure mode plays out. If you have a good link (or
| don't mind giving a quick summary), I'd love to read up.
|
| I have considered Starlark, and still might end up using it.
| The critical feature I wanted to bolt in from Nix was having
| strings carrying dependency information (see
| https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb...
| for a description of the feature). In my prototyping, this was
| pretty simple to bolt on to Lua, but I'm not sure how
| disruptive that would be to Starlark. Nix configurations tend
| to be a bit more complex than Bazel ones, so having a more
| full-featured language felt more appropriate. Still exploring
| the design space!
| msvan wrote:
| As a current Nix user, what I would really like is a statically
| typed language to define builds. Recreating Nix without
| addressing that feels like a missed opportunity.
| zombiezen wrote:
| The Lua VSCode extension adds a type system that works really
| well IME
| 0cf8612b2e1e wrote:
| There are Lua flavors with typing. Teal is one I have heard
| that compiles down to regular Lua like a typescript
| Rucadi wrote:
| For me the killer feature is Windows Support, Ericsson is doing
| a great job bringing nix into Windows, but the process it's
| understandably slow, If this project is similar enough to nix
| that I can kind-off translate easily the zb derivations to nix
| derivations, I'm willing to use it in windows (It's not like
| nix has windows programs in the nixpkgs either way I have to
| bring them in my own).
|
| The problem for me is that I see no benefit on using this over
| nix language (which I kinda like a lot right now)
| droelf wrote:
| We're working on rattler-build (https://github.com/prefix-
| dev/rattler-build/) - which is a build system inspired by
| Apko / conda-build and uses YAML files to statically define
| dependencies. It works really well with pixi (our package
| manager) but also any other conda compatible package managers
| (mamba, conda).
|
| And it has Windows support, of course. It can also be used to
| build your own distribution (e.g. here is one for a bunch of
| Rust utilities: https://github.com/wolfv/rust-forge)
| hamandcheese wrote:
| > Ericsson is doing a great job bringing nix into Windows
|
| Is this Ericsson... the corporation? Windows support for nix
| is something I don't hear much about, but if there is
| progress being made (even slowly) I'd love to know more.
| Rucadi wrote:
| John Ericson (@Ericson2314)
|
| You can read a post on that here:
| https://lastlog.de/blog/libnix_roadmap.html
| mikepurvis wrote:
| Whoa, nifty. Can you speak more to the interop issues with Nix?
| I've been working on a pretty large Nix deployment in the
| robotics space for the past 3ish years, and the infrastructure
| side is the biggest pain point:
|
| * Running a bare `nix build` in your CI isn't really enough-- no
| hosted logs, lack of proper prioritization, may end up double-
| building things.
|
| * Running your own instance of Hydra is a gigantic pain; it's a
| big ball of perl and has compiled components that link right into
| Nix internals, and architectural fiasco.
|
| * SaaS solutions are limited and lack maturity (Hercules CI is
| Github-only, nixbuild.net is based in Europe and last I checked
| was still missing some features I needed).
|
| * Tvix is cool but not ready for primetime, and the authors
| oppose flakes, which is a deal-breaker for me.
|
| Something that's a barebones capable of running these builds and
| could be wrapped in a sane REST API and simple web frontend would
| be very appealing.
| Rucadi wrote:
| Why are flakes such a deal-breaker? While not ideal, you can
| still tag your versions in the .nix file instead of the
| lockfile.
|
| I even had to avoid flakes in a system I developed used by ~200
| developers since it involved a non-nixos OS and it involved
| user secrets (Tokens etc...) So with flakes I had to keep track
| of the secrets (and was a pain point, since they obviously
| didn't have to push them into the git repo) but nix flakes
| doesn't handle well omitting files on git (it ignores them also
| on nix commands). In the end, the workarounds were too messy
| and had to drop flakes entirely.
| flurie wrote:
| I've been wondering idly if it's possible for Nix to support
| the Bazel Remote Execution API that seems to be catching on[1]
| more generally.
|
| [1] https://github.com/bazelbuild/remote-apis?tab=readme-ov-
| file...
| zombiezen wrote:
| Tracking issue is https://github.com/256lights/zb/issues/2
|
| The hurdles to interop I see are:
|
| - Nixpkgs is not content-addressed (yet). I made a conscious
| decision to only support content-addressed derivations in zb to
| simplify the build model and provide easier-to-understand
| guarantees to users. As a result, the store paths are different
| (/zb/store instead of /nix/store). Which leads to... - Nix
| store objects have no notion of cross-store references. I am
| not sure how many assumptions are made on this in the
| codebases, but it seems gnarly in general. (e.g. how would GC
| work, how do you download the closure of a cross-store object,
| etc.) - In order to obtain Nixpkgs derivations, you need to run
| a Nix evaluator, which means you still need Nix installed. I'm
| not sure of a way around this, and seems like it would be a
| hassle for users.
|
| I have experienced the same friction in build infra for Nix. My
| hope is that by reusing the binary cache layer and introducing
| a JSON-RPC-based public API (already checked in, but needs to
| be documented and cleaned up) for running builds that the
| infrastructure ecosystem will be easier.
| imiric wrote:
| Happy to see someone inspired by Nix, but wanting to carve their
| own path. Nix popularized some powerful ideas in the Linux world,
| but it has a steep learning curve and a very unfriendly UI, so
| there is plenty of room for improvement there.
|
| I'm not sure if Lua is the right choice, though. A declarative
| language seems like a better fit for reproducibility. The goal of
| supporting non-deterministic builds also seems to go against
| this. But I'm interested to know how this would work in practice.
| Good luck!
| hinkley wrote:
| If you design it like SCons, it'll look imperative but behave
| more declaratively.
|
| If I understand the architecture correctly, the imperative
| calls in the config file don't actually run the build process.
| They run a Builder Pattern that sets up the state machine
| necessary for the builds to happen. So it's a bit like LINQ in
| C# (but older).
|
| I have no idea how that plays out single-step debugging build
| problems though. That depends on how it's implemented and a lot
| of abstractions (especially frameworks) seem to forget that
| breakpoints are things other people want to use as well.
| zombiezen wrote:
| That's accurate (unless the config file attempts to read
| something from the build process, that will trigger a build).
|
| It's a good point about debugging build problems. This is an
| issue I've experienced in Nix and Bazel as well. I'm not
| convinced that I have a great solution yet, but at least for
| my own debugging while using the system, I've included a `zb
| derivation env` command which spits out a .env file that
| matches the environment the builder runs under. I'd like to
| extend that to pop open a shell.
| kortex wrote:
| > The goal of supporting non-deterministic builds also seems to
| go against this.
|
| I think this is actually a great escape hatch. Supporting non-
| deterministic builds means more folks will be able to migrate
| their existing build to zb. Postel's law and all that.
| IshKebab wrote:
| Interesting. I feel like I would have gone with Starlark over
| Lua, but I guess it's good to have options.
|
| Does it support sandboxing?
| zombiezen wrote:
| Not yet, but I've hacked up most of the Linux sandboxing:
| https://github.com/256lights/zb/issues/29
|
| I want to introduce Windows sandboxing, too, but I'm not as
| confident about how to do that:
| https://github.com/256lights/zb/issues/31
| zombiezen wrote:
| Oh and as for Starlark, I went into more detail over in this
| thread: https://news.ycombinator.com/item?id=41596426
| israrkhan wrote:
| You need bazel if you need starlark & sandboxing
| alxmng wrote:
| Did you consider writing a nicer language that compiles to Nix? A
| "friendly" tool on the outside with Nix inside.
| zombiezen wrote:
| Yup, that was how I built the prototype:
| https://www.zombiezen.com/blog/2024/06/zb-build-system-proto...
|
| The last commit using that approach was
| https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9...
| if you want to try it out. And actually, I haven't touched the
| Lua frontend much since I swapped out the backend: the .drv
| files it writes are the same.
|
| The motivation behind replacing the backend was content-
| addressibility and Windows support, which have been slow to be
| adopted in Nix core.
| Rucadi wrote:
| I don't think nix is that awful, while there are some tasks
| that are more difficult or can be a little bit verbose (if you
| want to play a lot with the attribute sets / lists or string
| manip) When using nix most of the time you'll end up just
| writing bash or using it as a templating language.
| droelf wrote:
| Cool that this space is getting more attention - I just came from
| the reproducible builds summit in Hamburg. We're working on
| similar low level build system tools with rattler-build and pixi.
| Would love to have a chat and potentially figure out if
| collaboration is possible.
| zombiezen wrote:
| Cool! Contact info is in my profile and on my website. :)
| greener_grass wrote:
| I'm excited by this!
|
| Quick question: if the build graph can be dynamic (I think they
| call it monadic in the paper), then does it become impossible to
| reason about the build statically? I think this is why Bazel has
| a static graph and why it scales so well.
| zombiezen wrote:
| According to Build systems a la carte, "it is not possible to
| express dynamic dependencies in [Bazel's] user-defined build
| rules; however some of the pre-defined build rules require
| dynamic dependencies and the internal build engine can cope
| with them by using a restarting task scheduler, which is
| similar to that of Excel but does not use the calc chain." (p6)
|
| IME import-from-derivation and similar in Nix is usually used
| for importing build configurations from remote repositories.
| Bazel has a repository rule system that is similar:
| https://bazel.build/extending/repo
|
| So to answer your question: yes from the strictest possible
| definition, but in practice, I believe the tradeoffs are
| acceptable.
| Ericson2314 wrote:
| You should look at Nix's experimental dynamic derivations,
| which provide functionality entirely at the level of
| derivation language / store layer.
| zombiezen wrote:
| Interesting! Thanks, hadn't seen that yet. (For anyone else
| curious, the RFC is here:
| https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-
| dyn...)
| kortex wrote:
| How do you pronounce "Zb"? Zee-bee?
| zombiezen wrote:
| Heh, I think I need to add something to the README. I've been
| pronouncing it as "zeeb" in my head as in the first syllable
| Zebesian Space Pirate from Metroid, but TIL that that's
| canonically "Zay-bay-zee-uhn" so idk.
|
| Naming is hard.
| Ericson2314 wrote:
| Nice to see Windows support. We/I are working on that with
| upstream Nix too.
|
| Also I hope we can keep the store layer compatible. It would be
| good to replace ATerm with JSON, for example. We should
| coordinate that!
___________________________________________________________________
(page generated 2024-09-19 23:00 UTC)