Native Linux Builder is now generally available

Native Linux Builder is a feature of Determinate Nix that enables you to build Linux derivations on macOS using macOS’s built-in Virtualization framework. That means no remote builder, Docker container, or Linux virtual machine that you need to start, stop, and manage yourself. You just run nix build and Determinate Nix handles the rest:

nix build "https://flakehub.com/f/NixOS/nixpkgs/0#legacyPackages.x86_64-linux.ponysay"

When we introduced it with Determinate Nix 3.8.4, we released it as a developer preview, which meant that we required potential users to request access. Today, we’re happy to announce that Native Linux Builder is now generally available, which means that it’s enabled for every macOS user of Determinate Nix who’s logged in to FlakeHub. As before, it requires no configuration or setup to use once you install Determinate Nix.

And because the builder targets both ARM and x86 Linux (aarch64-linux and x86_64-linux), you can build things like NixOS systems and Docker images for either architecture right on your Mac. Here’s an example for an x86 Linux Docker image:

nix build "https://flakehub.com/f/DeterminateSystems/fh-fetch-example/0#dockerImages.x86_64-linux.server"
docker load < result

Getting the builder

All you need is Determinate Nix on macOS and a FlakeHub account. If you don’t yet have Determinate Nix installed, you can install it on macOS using our graphical installer:

Install Determinate Nix on macOS

With support for Apple Silicon (aarch64-darwin)

If you’re on an older version of Determinate Nix, you can upgrade with a single command:

sudo determinate-nixd upgrade

Signing up for FlakeHub is free, and once you’ve signed up you can log in using Determinate Nixd:

determinate-nixd auth login

To confirm that you’re logged in, check your status:

determinate-nixd status

How Native Linux Builder is changing the game

The feedback we received during the developer preview was overwhelmingly positive. Many teams were able to outright eliminate dedicated remote builders and long-running local virtual machines from their Nix setups, providing some highly welcome simplification.

Some use cases we saw in the wild:

• As mentioned above, building Open Container Initiative (OCI) images, including Docker images, without needing to push the work to CI or a Linux box first.
• Building full NixOS system configurations on a Mac and running them on local VMs or even pushing them to servers.
• Running nix flake check against the Linux outputs of a flake before pushing, rather than waiting on CI to find out that something broke.
• Iterating on Linux-only packages without leaving macOS.

If Native Linux Builder has unlocked something not listed here, we’d love to hear about it.

Disabling the builder

The builder is on by default, but if you’d rather handle Linux builds some other way, for example with your own remote builders, you can turn it off in your Determinate Nixd configuration at /etc/determinate/config.json:

{
  "builder": {
    "state": "disabled"
  }
}

The same builder block is also where you can adjust how much memory (memoryBytes) and how many CPUs (cpuCount) the builder gets. See the configuration docs for the full set of parameters.

Common problems

Thanks to everyone who sent us bug reports during the developer preview. We’ve solved virtually all of the issues that you reported. The few that remain come down to a handful of problems, which the sections below cover. Our Native Linux Builder troubleshooting guide covers all of these in more detail, and it’s the place to go first if you run into problems with the builder.

The builder isn’t available

If Determinate Nix tells you that it can’t build a Linux derivation, you may see an error like this:

error: Cannot build '/nix/store/...-nixos.conf.drv'.
       Reason: required system or feature not available
       Required system: 'aarch64-linux' with features {}
       Current system: 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

This almost always means that you’re not logged in to FlakeHub or that you’re running an older version of Determinate Nix. If determinate-nixd status says that you’re logged in and you’re up to date, try logging out and back in:

determinate-nixd auth logout
determinate-nixd auth login

When it prompts you for a FlakeHub token, you can reuse your existing one:

cat /nix/var/determinate/token | pbcopy

If that doesn’t do the trick, restart Determinate Nixd:

sudo launchctl kickstart -k system/systems.determinate.nix-daemon

To check whether the builder is working, build a small Linux derivation from source:

nix build --print-build-logs --substituters "" \
  "https://flakehub.com/f/DeterminateSystems/minimal-stdenv/0.1#packages.x86_64-linux.default"

See Addressing common problems in the troubleshooting guide for more.

Case sensitivity and hash mismatches

This is the biggest known issue. Native Linux Builder runs your build inside a Linux virtual machine, but it writes the outputs directly to your host’s Nix store, and on macOS the Nix store lives on an APFS volume that’s case insensitive by default. Linux has no such restriction, and some Linux packages ship files whose names differ only in case. The Linux kernel headers are the classic example: xt_CONNMARK.h and xt_connmark.h are two different files on Linux but the same file on a case-insensitive volume. When a build writes files like these, one overwrites the other, and you usually see the result as a hash mismatch. Nix’s special handling for macOS’s case-insensitive filesystem doesn’t help here because the Linux builder directly mounts the host’s store path.

We’re planning on making the Nix store for Determinate Nix case sensitive on macOS, but we’re not quite ready to ship it by default. You can set up a case-sensitive Nix store today, but it means uninstalling Determinate Nix, which deletes your entire Nix store, and then reinstalling it with a case-sensitive volume:

/nix/nix-installer uninstall
curl -fsSL https://install.determinate.systems/nix | sh -s -- install macos --case-sensitive --extra-conf 'use-case-hack = false'

The troubleshooting guide’s section on hash mismatches shows you how to verify that the new store is case sensitive. If reinstalling isn’t an option for you, you can fetch affected derivations from a binary cache or build them on a remote Linux builder instead.

Permission denied when copying files

Some builds fail with Permission denied errors when they run cp --no-preserve=mode or similar commands to copy outputs into the Nix store. Native Linux Builder mounts the host’s Nix store into the virtual machine using the Virtualization framework’s virtio file system driver, and cp --no-preserve=mode interacts badly with that driver when the destination is the host’s store. The fix is to change the affected derivation so that it doesn’t rely on --no-preserve=mode, for example by splitting the command into a cp followed by a chmod, as we did in this pull request to Crane. If you hit this in Nixpkgs or elsewhere, let us know and we’ll help track it down.

Builds stall on fixed-output derivations

Native Linux Builder gets network access for fixed-output derivations (FODs), which download sources, from macOS’s built-in Dynamic Host Configuration Protocol (DHCP) server. That server hands out leases that last one hour, and we can’t release a lease when the virtual machine shuts down. So if you build 253 FODs within a one-hour window, the 254th stalls until a lease expires. You can check whether you’ve hit this by looking for many 192.168.64.* entries in the lease file:

sudo grep 192.168.64 /var/db/dhcpd_leases | sort -V

If you’re up in the 200s, and nothing else on your Mac depends on the built-in DHCP server, moving that file out of the way frees up new leases:

sudo mv /var/db/dhcpd_leases /var/db/dhcpd_leases.bak

We’re working on a proper fix. You can follow along in this issue.

Builds run out of memory

By default, the builder’s virtual machine has access to up to 8 GiB of memory. If you’re building derivations that need more than that, raise the limit by setting builder.memoryBytes in your Determinate Nixd configuration.

Let us know how it goes

Making macOS a first-class platform for Nix has long been one of our top priorities, and building Linux derivations on a Mac without extra setup is a big part of that. If you run into something that isn’t covered here or in the troubleshooting guide, open an issue or come say hi on Discord.