[HN Gopher] Reflecting on the Shake Build System
___________________________________________________________________
Reflecting on the Shake Build System
Author : gbrown_
Score : 69 points
Date : 2021-11-06 08:03 UTC (14 hours ago)
(HTM) web link (neilmitchell.blogspot.com)
(TXT) w3m dump (neilmitchell.blogspot.com)
| autarch wrote:
| He also has two follow-up posts on his blog about build systems.
| They're both worth reading.
|
| Small Project Build Systems -
| https://neilmitchell.blogspot.com/2021/09/small-project-buil...
|
| Huge Project Build Systems -
| https://neilmitchell.blogspot.com/2021/09/huge-project-build...
| davidm-d wrote:
| I used shake for some simple build scripts and to work on a
| predecessor of the Haskell Language Server. Due to it's lack of
| hermetic builds I don't think it's really appropriate for any
| project you're going to share with more than one person.
|
| That being said I think it's an absolutely amazing Haskell
| library for building complex systems with scheduling jobs and
| caching of intermediate results. Once you use it you get the
| sense that every large codebase has an ad hoc poorly specified
| build system nestled inside of it. I described the experience of
| using it briefly here:
|
| https://4ta.uk/p/shaking-up-the-ide
| ur-whale wrote:
| > Due to it's lack of hermetic builds
|
| Care to explain what that means or post a ref?
| gravypod wrote:
| For a build system to be hermetic compilations (or other
| things) should only see declared dependencies. The least
| hermetic build system is Make. If you have a target that
| depends on `a.h` nothing is stoping you from reading `b.h`.
| More hermetic build systems, like Bazel, will make it so
| `b.h` isn't visible at all during the build process of it
| isn't declared as an input.
|
| This has nice properties that allow you to be sure your build
| rules are correct, caching your outputs based on inputs, and
| because you can cache things you can now remotely execute
| things safer.
| ur-whale wrote:
| Understood, and in other words:
|
| hermetic == sandbox which only contains declared deps
|
| I assume bazel implements this via containers?
|
| Does that include compilers and OS?
|
| Also, IMO, this only provides a false sense of safety
| unless you archive absolutely _everything_ that 's inside
| the sandbox.
| gravypod wrote:
| > hermetic == sandbox which only contains declared deps
|
| Yes, exactly. It also can include other deps like
| networking. A perfect hermetic build will (for most of
| the build process) not have access to the network.
|
| > I assume bazel implements this via containers?
|
| Sort of. Bazel's code predates "containers" by ~6 years.
| Bazel is the build system used at Google (sprinkle on a
| large amount of asterisks here).
|
| > Also, IMO, this only provides a false sense of safety
| unless you archive absolutely everything that's inside
| the sandbox.
|
| Yes, that is what a Bazel monorepo does. You can also
| include your language's runtime and standard library as
| part of the build process. See Dropbox's repo for more
| details on how they do this with Python:
| https://github.com/dropbox/dbx_build_tools
|
| Some more details about what Google does are available
| here: https://opensource.google/docs/thirdparty/#
|
| Essentially, _everything_ is built from source (more
| asterisks here). Since everything is built from source
| using Bazel you can guarantee that at a certain view of
| the source repository you can build a reproducible copy
| of all software in the repo (more asterisks).
|
| One other nice property is you can guarantee that all
| software in the repo is using a compatible version of
| each dep:
| https://opensource.google/docs/thirdparty/oneversion/
|
| The downside is that if you want to upgrade a library
| that has a breaking change this requires you update a lot
| of code. The upside is that doing that isn't actually
| that hard with the tooling available and you can
| guarantee that you can link together any two libraries in
| the repo (more asterisks) and have the build succeed
| without any dep version conflicts. Since you can do this
| some langauges with runtimes (like JVM language) can see
| massive speed boosts because they only need to compile
| changed libraries and can run a `java -cp ... your.jar`
| to load up deps. You can check out rules_docker for some
| examples of how this can be used.
| _peeley wrote:
| I haven't looked into Bazel at all, other than knowing
| it's a build system used at Google. Sounds a lot like
| Nix's philosophy of totally sandboxed/declarative package
| builds, although instead of needing a monorepo with every
| piece of software you can just grab a reproducible build
| from nixpkgs.
| nikita2206 wrote:
| This is so great, I wish Bazel had a section in its
| README with this info. Thank you
| gravypod wrote:
| If you have further questions ask them and I can see
| about writing a full blog post which touches on
| everything I know.
| mgaunard wrote:
| Guy is now working on Buck.
|
| In the meantime I'll stick to Bazel thnak you very much.
| de_keyboard wrote:
| Buck is (in my view) better designed than Bazel, particularly
| around how outputs are identified, but the inertia is with
| Bazel at this point. Too bad.
| gravypod wrote:
| I have similar feelings except with please.build.
| twic wrote:
| A detailed example of dynamic dependencies would be nice. Does
| this just mean a file dependency where you don't know the exact
| set of filenames ahead of time? Or does it mean dynamic topology?
| Quekid5 wrote:
| Both
| thrower123 wrote:
| Anybody outside of Standard Chartered ever used this?
| Vosporos wrote:
| Yep, we use Shake at my work. :)
| agumonkey wrote:
| happily ?
| WJW wrote:
| The build system for GHC uses this.
___________________________________________________________________
(page generated 2021-11-06 23:02 UTC)