https://blog.replit.com/super-colliding-nix-stores

Replit Features Jobs Blog Pricing Jam
[                    ]
Sign up [?]
<-- Back to blog Edit on Replit [?]

Super Colliding Nix Stores: Nix Flakes for Millions of Developers

Thu May 25 2023 by Ryan Mulligan

stacks

We've teamed up with Obsidian Systems and Tweag to enable Nix to
merge multiple (possibly-remote) Nix stores, bringing Nix Flakes and
development environment portability to millions of Replit users.

Nix is an open-source cross-platform package manager and build tool
that lets you access the most up-to-date and complete repository of
software packages in the world. Nix's approach to package building
ensures that software and development environments will always work
the same way, no matter where it is deployed.

Replit has bet big on Nix because we believe in a future where
developers are free from the drudgery of setting up their development
and production environments. A world where onboarding a co-worker is
as simple as forking a Repl. When you use Nix to describe your
environment, you don't have to redo your setup everywhere you want
your code to work. As Mitchell Hashimoto, founder of Hashicorp, said:

    one big benefit is that once you adopt Nix, you can get a
    consistent environment across development (on both Linux and
    Mac), CI, and production.

Since early last year, all new Repls have been powered by Nix.
Fast-forward to today, we're providing nearly a million software
artifacts for instant installation, all without counting against your
Repl's storage limits.

But that was just the beginning, we want to give you more power to
configure Repls how you would like and to increase Repl portability
with other platforms. We want to give you access to an even larger
collection of Nix packages: ones that were published years ago, and
the latest ones published today. We also want you to be able to use
Nix Flakes on Replit: it shouldn't require extra work to configure
development environments on every platform. Write a Nix Flake once
and have the same reproducible environment everywhere.

To achieve this, we're going to need a way to merge Nix stores! But
first let's dive into the details of how we use Nix to create
reproducible development environments.

Development environments at Replit

Every Repl has a replit.nix file which allows you to install tens of
thousands of packages available in the Nixpkgs collection. For
example, here is the one for C++:

{ pkgs }: {
  deps = [
    pkgs.clang_12
    pkgs.ccls
    pkgs.gdb
    pkgs.gnumake
  ];
}

It installs a compiler (clang), language server (ccls), debugger
(gdb), and build tool (make). When this file is changed, we add all
these packages to your Repl's shell environment. Behind the scenes we
do something close to what would happen on your computer if you
copied the following into a default.nix file and ran nix-shell in a
terminal in the same directory.

{pkgs ? import <nixpkgs> {}}:

pkgs.mkShell {
  packages = [
    pkgs.clang_12
    pkgs.ccls
    pkgs.gdb
    pkgs.gnumake
  ];
}

When you run nix-shell, Nix evaluates this code and determines that
it needs to build a shell environment as a package. For example, if I
run nix-build in the same directory, I see:

$ nix-build
this derivation will be built:
  /nix/store/ni73sa8sh3jl99nz06z3af5kkp9xl1ws-nix-shell.drv
building '/nix/store/ni73sa8sh3jl99nz06z3af5kkp9xl1ws-nix-shell.drv'...
building
/nix/store/ylhzwsmznmjw8g39wvysg1rsh9ld2il7-nix-shell

Nix has figured out what to build and wrote a detailed reproducible
build plan in the Derivation file /nix/store/
ni73sa8sh3jl99nz06z3af5kkp9xl1ws-nix-shell.drv into the /nix/store/
directory and then it followed the build plan producing the file /nix
/store/ylhzwsmznmjw8g39wvysg1rsh9ld2il7-nix-shell which is a shell
script that describes what environment variables to set to produce
this development environment. Here's a modified excerpt of that file:

declare -x PATH=
"/nix/store/7c4759gi42c2bhfgxixbq1hilv0m1g4i-clang-wrapper-12.0.1/bin:
/nix/store/k33vhsd3js5ri12prl1q309fs5l73c1p-clang-12.0.1/bin:
/nix/store/lyvhsvwp2pzy74fkcn7qbs5vcgy5d7vl-glibc-2.37-8-bin/bin:
/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin..."
declare -x SHELL=
"/nix/store/rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin/bash"

The script sets a bunch of environment variables to make sure your
development environment has exactly what it needs in it.

When it builds the shell environment successfully, it also adds an
entry to a SQLite database at /nix/var/nix/db/db.sqlite:

sqlite>
select *
from ValidPaths
where
path = "/nix/store/ylhzwsmznmjw8g39wvysg1rsh9ld2il7-nix-shell";

333859|/nix/store/ylhzwsmznmjw8g39wvysg1rsh9ld2il7-nix-shell|
sha256:450a0340bb5cf76d350c1c7bee48c3403c3df6076c3d66f592f31499ac4f2f5e|
1684944022|
/nix/store/ni73sa8sh3jl99nz06z3af5kkp9xl1ws-nix-shell.drv|7936|1||

Nix records this information in the database for both
file-system-consistency reasons and performance reasons.

There's one last step for setting up Repl development environments
that we'll get to after we learn about the overlay filesystem we use
to enable instant package installation.

How our overlay filesystem currently works

If we let you use Nix the exact same way you do on your local
computer, you'd use a lot of space and initially download a lot from
the Nix caches, which can take many minutes.

Instead, we have ~16 TB Persistent Disks (Big Disks) on Google Cloud
that we attach to each machine that serves Repls. The Big Disks have
a Nix Store (a collection of packages in a /nix/store directory,
alongside the SQLite database) containing almost a million packages.
Like the nix-shell example above, the SQLite database has a row for
each package.

current Nix store overlayfs setup

Inside Repls, the Big Disk is mounted in an overlay filesystem stack.
The Big Disk is a lower disk and an upper scratch disk lets Nix
builds happen inside the Repl. Critically, this upper disk is not
persistent.

Under this setup, if we persisted the upper store, you wouldn't see
the updated lower store database the next time we added a package to
the Big Disk.

This means that currently whenever you do a regular Nix build (or
anything with Nix Flakes) inside a Repl, the build result is not
saved between sessions and is limited to the size allotted to the
scratch disks.

Revisiting development environments

Since the upper store is not persistent, how do we avoid rebuilding a
Repl's development environment every session? Caching.

We save all the environment variables to .cache/replit/nix/env.json
along with metadata that helps us determine if the cache is stale.
When a Repl starts, we look for this cache, and if it is not stale we
use it instead of rebuilding the Nix environment. See our previous
post Faster Nix Repl Startup for more details.

Without a persistent upper store, everything we cache has to be
present on the Big Disk. If you want to add a new Nix package to the
Big Disk, you update nixpkgs-replit (Replit's nixpkgs overlay) and
wait for the disks to be rebuilt and deployed.

Unlocking the full power of Nix

If we could persist the upper store, we'd bring the full power of Nix
to Replit. Here are some of the benefits:

Configuration reusability: projects with existing Nix configurations
can bring them to Replit without modification. Repls with standard
Nix configuration can be downloaded to your local computer and work.
Write one development environment configuration, reuse everywhere
(local, Repls, CI, and production deployments).

Better caching: your development environment can be cached and ready
to go when your Repl starts. You can even cache your build artifacts
across different versions of your code.

Nix Flakes: get access to the improved user experience and code
sharing provided by Nix Flakes.

Using other Nix projects: not all Nix code is available in nixpkgs,
this unlocks access to the wider ecosystem.

Build your project: Nix is a general build tool with lots of
applications. It can build projects in almost any language including
Python, Go, Rust, and C++. It can also build containers, and virtual
machine images.

Layered Store to the rescue!

By modifying Nix to be aware of the layered overlay filesystem, we
can mix instant installation with the full power of persistent Nix
builds. Rather than using an overlay filesystem for /nix/store and
the database, we only overlay the /nix/store directories.

new Nix store overlayfs setup

When Nix is ready to build, it first looks into the lower store
database to see if it was already built. If available, it skips
building and adds an entry to the upper store database indicating the
package is available. Through the power of the overlay filesystem,
the build products are already in place!

With the overlay filesystem between the upper and lower databases
removed, we can persist the upper store and add packages to the lower
store database without interfering with the upper store database.

Other applications for the Layered Store

The Layered Store is an interesting primitive that we think could be
useful in a variety of situations:

  * Using a network filesystem to serve a Nix store
  * A build farm or CI server that maintains a fast-access upper Nix
    store while still having access to a networked filesystem lower
    store for reusing big or expensive build products.
  * Anything composing more than 2 stores (2 stores is the upper
    limit of an overlay filesystem without a Layered Store)

We'd love to hear your ideas for how you'd use it.

Development

To make this a reality, we've teamed up with Obsidian Systems and
Tweag. In the coming weeks, we will be developing the Layered Store
features, releasing a Nix Community RFC, and working to upstream it
into Nix.

Want to be one of the first ones to try it out? Add yourself to our
beta list.

Work at Replit

Are you interested in Nix and would love to see it in the hands of
more people? Come work with us on making instantaneous software setup
and deployment a reality for everyone.

                     Replit Logo Follow @Replit

Legal

Terms and services Privacy Subprocessors DPA US Student DPA

Replit

Blog About Jobs Teams Pricing

Handy links

Create a repl Docs Status Page Python Packages Import From Glitch

Social media

Facebook Twitter Tiktok Instagram

Languages

Clojure Haskell Kotlin QBasic Forth LOLCODE BrainF Emoticon Bloop
Unlambda CoffeeScript Scheme APL Lua Ruby Roy Python Node.js
JavaScript Deno (beta) Golang C++ C C# F# HTML, CSS, JS Rust Swift
Python (with Turtle) Basic (beta) R Bash Crystal Julia Elixir Nim
Dart Reason Node.js Tcl Erlang TypeScript Pygame Love2D Emacs Lisp
(Elisp) PHP Web Server SQLite Java PHP CLI Pyxel Raku Scala (beta)
Nix (beta)

Copyright (c) 2023 Replit, Inc. All rights reserved.