Determinate Nix: the recent past and the shining future

Wow! What a difference a few months can make. Although we released Determinate Nix 3.0 a bit over a half a year ago, the past four months have seen a noticeable intensification in rolling out features that we’ve long wanted to provide to our customers and to Nix users more broadly.

In this post, I’d like to take a deep breath, step back, and provide a recap of these recent changes for the sake of those who maybe haven’t been keeping up (yet). That includes descriptions of major features like parallel evaluation, the native Linux builder, and lazy trees, as well as of less splashy but nonetheless vital quality-of-life features. After that, I’ll tease some features that we have in the works that you’ll absolutely want to keep on your radar, including flake schemas, configurable flakes, and sparse flake.lock files.

Parallel evaluation

In Determinate Nix version 3.11.1, we introduced parallel evaluation for the operations behind some common Nix commands. Parallel evaluation makes Nix much faster in some scenarios by distributing work across multiple processors. We started with these commands:

• nix search
• nix flake check
• nix flake show
• nix eval --json

And we plan to continue parallelizing Nix operations until we run out. To try it out, install Determinate Nix and update your configuration.

Parallel evaluation is one of those quiet features that makes Determinate Nix better without you even realizing it—better both in CI and on your workstation.

If you’re already using Determinate Nix, run determinate-nixd version to see if the parallel-evaluation feature is listed as enabled.

The native Linux builder

In Determinate Nix version 3.8.4, we released the native Linux builder, which you can use to build Linux derivations on macOS with zero manual configuration or setup. That means no remote builder, no local VM that you need to manually start and stop, just Nix talking to macOS’s built-in Virtualization framework.

With the native Linux builder, this command builds ponysay for x86 Linux on macOS:

nix build "https://flakehub.com/f/NixOS/nixpkgs/0#legacyPackages.x86_64-linux.ponysay"

Now, you can’t run that package on macOS, of course. But you can also build more useful things, like Docker images…

nix build "https://flakehub.com/f/DeterminateSystems/fh-fetch-example/0#dockerImages.x86_64-linux.server"
docker load < result

…and even NixOS systems:

nix build "https://flakehub.com/f/DeterminateSystems/garage-door/0#nixosConfigurations.turner.config.system.build.toplevel"

Over time, macOS users will forget that this wasn’t always possible—and that’s precisely our intention.

If you’re already using Determinate Nix, run determinate-nixd version to see if the native-linux-builder feature is listed as enabled. If not and you’re eager to try it out, reach out to us at support@determinate.systems and include your FlakeHub username.

Lazy trees

In Determinate Nix version 3.6.7, we released lazy trees, a long-desired feature that significantly improves the performance of Nix with flakes. In essence, lazy trees offer much faster and less resource-intensive evaluation in many standard flake usage scenarios, especially in larger repositories like monorepos.

In practice, we’ve seen lazy trees reduce wall time for evaluations by 3x or more and disk usage of 20x or more in some cases. These reductions stem from Nix being much more parsimonious about file copying, using a virtual filesystem to gather file state prior to copying to the Nix store. Like parallel evaluation, this is one of those quiet features that makes things imperceptibly better all the time.

This feature has now been rolled out to 100% of Determinate Nix users, so if you install Determinate Nix now, you should immediately see the benefits.

Quality-of-life features

In addition to the blockbuster features, mentioned above, the past few months have seen a wide variety of less flashy features that have nonetheless pushed the Nix envelope forward substantially.

• Build-time flake inputs. This change enables you to declare flake inputs as build-time inputs, which means that they’re fetched only when building (whereas Nix’s default behavior is to fetch them upon evaluation). This feature is a solid performance win for flakes that fetch a lot of inputs like external Git repos or trees.
• Parallel Git cache unpacking. Previously, unpacking sources into the user’s Git cache happened serially, but this has been parallelized in Determinate Nix. As a result, Git cache unpacking now frequently takes less than half the time it used to, and sometimes as low as 1/4th the time.
• Background copying of sources to the store: Previously, Nix’s evaluator would pause evaluation when it needed to add files to the Nix store. In Determinate Nix, this is no longer the case. Instead, it copies in the background, which allows evaluation to proceed. This improves evaluation times in many scenarios.
• Reduced Nix daemon requests. Previously, the Nix client needed to perform thousands of Nix daemon requests that our team determined to be unnecessary—requests that were effectively stealing time and resources. The fix involved caching store path validity data within a single evaluation and it eliminates tons of daemon queries; in one sample evaluation, we saw 12,000 requests removed.
• Parallel garbage collection. Previously, Nix’s “marking” processor for the evaluator’s garbage collector was single threaded. In Determinate Nix, the evaluator uses multiple threads, which means that garbage collection now happens much more quickly, especially when dealing with the large heap created by things like Nixpkgs and NixOS. To give an example, this change has cut over five seconds from nix search on nixpkgs, from 24.3 seconds to 18.9 seconds.
• The builtins.parallel function. This new function enables users to parallelize parts of their own Nix expressions. Using builtins.parallel can drastically improve the performance of many projects, especially those using import-from-derivation (IFD).

Upcoming changes to Determinate Nix

As you can see, a lot has come down the pike recently but we have every intention to take advantage of this accumulated momentum. Although we don’t yet have any firm delivery dates, you can expect to see all of these features—and more—in Determinate Nix down the road:

• Customize flake output types with flake schemas
• Pass values to flake outputs with configurable flakes
• Sparse flake.lock file

Flake schemas

Flake schemas will enable you to provide custom flake output types to supplement current standard outputs like devShells, packages, formatter, and checks. Imagine being able to instruct Nix how to enumerate and check flake outputs like darwinConfigurations for nix-darwin, homeConfigurations for Home Manager, dockerImages for Docker containers, and so on.

We have a set of preliminary schemas for some of these outputs but we still have work to do in Determinate Nix itself to enable it to handle them. But we expect this to significantly improve flakes’ discoverability, provide a new way to encode best practices surrounding flakes, and to make Nix more adaptable to new domains.

Configurable flakes

In their current form, Nix can’t provide any “external” information when building flake outputs (packages, NixOS configurations, and so on). Nix considers the expressions in the flake complete and so you need to bake every desired output into the flake itself.

Configurable flakes will enable you to pass values into your Nix expressions and thereby to create on-demand flake outputs, and it will enable you to do so in a type-safe, deterministic way.

Eelco Dolstra has built a prototype of configurable flakes and even gave a conference talk on it. We expect configurability to make flakes dramatically more flexible and to open up broad new horizons in what flakes can provide—potentially even paving the way for formats like TOML and JSON to play a more central role in flake outputs.

Sparse lockfiles

Currently, any time Nix creates a flake.lock for a flake, it creates entries for all flake inputs—even flake inputs that aren’t actually used in any Nix expressions in the flake. With sparse lockfiles, Nix would only create entries for inputs that are actually used. Think of it as a rough analogy to lazy trees.

We expect this feature to make flake.lock files more neat and tidy and thus to cut superfluous information out of some of Nix’s flake-based operations.

How to upgrade or install

If you already have Determinate Nix installed, you can upgrade to the latest version with one Determinate Nixd command:

sudo determinate-nixd upgrade

If you don’t yet have Determinate Nix installed, you can install it on macOS using our graphical installer:

Install Determinate Nix on macOS now

Apple Silicon and Intel

On Linux:

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | \
  sh -s -- install --determinate

On NixOS, we recommend using our dedicated NixOS module.