[HN Gopher] Show HN: FlakeHub Cache: Fast, secure, configurable....
___________________________________________________________________
Show HN: FlakeHub Cache: Fast, secure, configurable. A new take on
Nix caching
Author : grhmc
Score : 45 points
Date : 2024-03-13 14:38 UTC (8 hours ago)
(HTM) web link (determinate.systems)
(TXT) w3m dump (determinate.systems)
| grhmc wrote:
| Hey folks, DetSys's cofounder here. Eelco, the team, and I have
| been working hard on FlakeHub Cache for some time now. We're
| starting to round out the toolkit you need to take Nix to work.
| Here to answer questions!
| mtlynch wrote:
| I've been trying to gradually introduce Nix more into my
| development.[0]
|
| One roadblock I haven't been able to overcome is integrating Nix
| in CI. In my case, I use CircleCI. I know there are Nix-centric
| CI platforms, but I've invested a lot into CircleCI and would
| like to stay.
|
| The problem I hit is that if I switch from something like a
| sqlfluff Docker container to a Nix Docker container, my build job
| goes from 15s to 2m+ because now I have to download several gigs
| within the CI job and build my Nix environment from scratch.[1]
|
| I tried Cachix, but the best I could get was bringing the job
| down to 55s (a roughly 3x performance penalty from an off-the-
| shelf Docker container).[2]
|
| How does FlakeHub Cache's performance in CI compare to the
| default Nix cache or to Cachix?
|
| [0] https://mtlynch.io/notes/nix-dev-environment/
|
| [1] https://github.com/cachix/cachix/issues/579
|
| [2]
| https://github.com/cachix/cachix/issues/579#issuecomment-175...
| grhmc wrote:
| I'd love to find out how it compares! We're not quite ready to
| support CircleCI, but when we are -- want to do a test? You
| could hop on to our discord:
| https://determinate.systems/discord
| zupo wrote:
| What worked great for me for speeding up CI was using Cachix +
| namespace.so. This works so well I moved all my personal and
| our company CI to GitHub Actions.
|
| With "it works so well", I mean:
|
| * CI is cheap because namespace.so runners are faster & cheaper
| than GitHub Actions
|
| * the UX is exactly the same as using vanilla GitHub Actions,
| namespace.so's runner is a drop-in replacement
|
| * the CI starts up very fast because `/nix/store` is stored on
| namespace.so's "Cached Volumes" which are like virtual disks
| attached to the CI runner, so there is nothing to download to
| prime the cache, Nix can just start running immediately
|
| * https://github.com/cachix/cachix-action uploads whatever Nix
| builds to Cachix in the background so from this point on if any
| of our devs pulls the latest code, they don't have to build any
| derivations locally, and lose time waiting, everything is
| pulled automatically from Cachix to their machines, they can
| get right to work. Or if I have to purge the Cache Volume for
| some reason, nothing needs to be rebuilt, just pulled from
| Cachix.
|
| It's taken a few years to get this dialed in, but now that it
| works, it's sooo good!
| biggestlou wrote:
| I'd be quite curious to see how using the Magic Nix Cache
| Action compares to your setup in terms of performance:
| https://github.com/DeterminateSystems/magic-nix-cache-action
|
| (full disclosure: Determinate Systems employee)
| zupo wrote:
| IIUC Magic Nix Cache uses Github Actions Cache, which means
| there is waiting for the cache contents to be loaded onto
| the runner instance. This can in no way be as fast as what
| namespace.so does, which is mounting a disk image with
| /nix/store.
| unshavedyak wrote:
| I'm going to have to look into this mounting option. I
| wanted to try making volumes very similar to this for
| reusing existing /target directories for internal company
| Rust builds. Sounds like Namespace could make this very
| easy to try out.
|
| Just need to figure out what the cache size limitations
| are..
| biggestlou wrote:
| To be clear, I would _not_ expect it to be as fast! But I
| do wonder what the percentage drop-off would be and, if
| small enough, if that would make staying on Actions more
| compelling than a bespoke solution.
| madjam002 wrote:
| I don't know about CircleCI but if you can persist the
| /nix/store then that will give you the performance improvements
| you're looking for.
|
| I use Gitlab CI and with self hosted runners I can make sure
| the /nix/store is persistent across CI jobs and then my builds
| take ~5 seconds if no code has changed, 5 seconds of which is
| just nix build evaluating all my nix files and realising no
| work needs to be done.
| grhmc wrote:
| The problem people often run in to when persisting the nix
| store is that it can be very large, and updates often. This
| adds a significant tax on setup/teardown time and sometimes
| storage costs. The completely free and unsubsidized Magic Nix
| Cache's design alleviates that on GitHub Actions by
| persisting each path individually in the actions cache.
| FlakeHub Cache brings a for-cost component to that, which
| exposes the cache to your workstation and production
| infrastructure.
| tracnar wrote:
| What I've been doing for now is to pre-build the container to
| include the Nix environment in the nix store. I still run nix
| in the CI so I only have to rebuild the container from time to
| time, when the env has changed a lot. It's certainly not an
| ideal solution but it works without special support.
|
| You indeed need some kind of nix-aware runner (e.g. with a
| persistent /nix) if you want both the flexibility and
| performance...
| hinkley wrote:
| I am this close to building my own CI pipeline. Building a new
| CI pipeline is a really bad idea, I would be the first to tell
| you, and it's what I've been telling myself for five plus years
| now, but it's becoming harder to ignore the disparity between
| off the shelf tools that solve parts of the CI/CD problem and
| the ones that are actually used in garbage like Bamboo.
|
| I think the fact that there is no bidding process for agents is
| a mistake that we need to correct. It's a little used concept
| in distributed computing, but a powerful one when applicable,
| and I think the preconditions for a fast build, with all of the
| docker images and other bootstrapping pipelines is making that
| cost harder to ignore. We are de facto using heterogeneous
| networks of worker processes now, and it's starting to reach 8
| Fallacies level.
|
| You build a distributed queue for response time or throughput,
| and we are using the throughput model, more or less, while
| CI/CD is mostly about response time.
___________________________________________________________________
(page generated 2024-03-13 23:01 UTC)