A new chapter for the Nix language, courtesy of WebAssembly

The Nix language is a domain-specific language for describing software configurations. Although it’s Turing complete, it was never really intended as a general-purpose language. For complex programming tasks, it lacks the conveniences of modern languages like Rust.

The Nix language is also a fully interpreted language without any kind of just-in-time compilation, so it’s not all that well suited for computationally intensive tasks. In most cases this isn’t much of a blocker for Nix users, but it does become a problem when you need to do something in Nix that isn’t provided as a builtin function in the language.

To give an example, suppose that you need to parse a YAML file in Nix to extract some configuration data. Unfortunately, Nix has no builtin YAML parser. So you have a few possibilities:

• Write a YAML parser in Nix. This is a pretty daunting, not-so-fun task because Nix is not a great language for this kind of string processing. The resulting parser will also be rather slow and memory hungry.

• Add a YAML parser to Nix as a builtin function. This has to be written in C++, but it does allow you to reuse any existing YAML parser library for C++. But you’re going to have a hard time getting this accepted upstream. The main reason is that YAML is complex, while the Nix language is intended to be reproducible across releases. So updating the YAML parser dependency could cause differences in evaluation results across Nix versions, which has been a real problem with builtins.fromTOML. And even if you do get your new builtin function accepted, it’s going to be a while before it makes it into a release and everybody can use it.

• Write a Nix plugin. This is similar to the previous approach—in that the plugin would need to be written in C++—except that you don’t need to get it accepted upstream. But now you do need to ensure that everybody who uses a Nix expression that calls your YAML parser has the plugin installed. This means that Nix flakes using it are no longer self-contained, and there is no convenient mechanism to declare that a flake requires a specific plugin.

• Use “import-from-derivation” (IFD), that is, do the YAML parsing using any language or tool of your choice and run it inside a derivation, and then import the result. Here’s an example:

  builtins.fromJSON (
    runCommand
      "parse-yaml"
      { src = ./input.yaml; }
      ``...run some command that converts $src from YAML into JSON...``)

  But IFD is an expensive mechanism, as realising the derivation may require downloading and building a lot of dependencies. It also breaks the separation between evaluating and building configurations, so an operation like nix flake show may unexpectedly start downloading and building lots of stuff.

  IFD is particularly unsuited when you want to do a traversal over a large source tree (for example to discover dependencies of source files), since it requires the entire source tree to be copied to the Nix store—even with lazy trees.

Determinate Nix now has a better way to extend the Nix language: through the power of WebAssembly.

WebAssembly

WebAssembly (Wasm) was created for pretty much the same reason it’s attractive for Nix: to allow JavaScript programs in web browsers to offload computationally expensive tasks to a more performant language. Wasm is a low-level binary instruction format that can be compiled from many high-level languages, including Rust, C++, and Zig. It is designed to be fast, portable, and secure. It has many implementations, including several that can be embedded in C++, such as Wasmtime and WasmEdge.

WebAssembly has a precisely defined semantics: a call to a WebAssembly function will always produce the same result when executed, as long as it has no access to impure external functions (“host functions” in Wasm parlance). For instance, WebAssembly by default has no access to a source of random numbers. This is critically important to Nix, as it is intended to be reproducible. Without it, Wasm functions could break the purity of the language.

builtins.wasm

The builtins.wasm function allows you to call a WebAssembly function from Nix. Here is an example of calling a Wasm function that computes the nth Fibonacci number:

nix-repl> builtins.wasm { path = ./nix_wasm_plugin_fib.wasm; function = "fib"; } 33
warning: 'nix_wasm_plugin_fib.wasm' function 'fib': greetings from Wasm!
5702887

So to call a Wasm function, you need to provide the path to the Wasm module and the name of the function you want to call. The Wasm function takes a single Nix value as input (in this case 33), and returns a single Nix value as output. These values, however, can be arbitrarily complex Nix values, such as attribute sets.

Wasm modules can be written in any language for which there is a compiler that targets Wasm. nix_wasm_plugin_fib.wasm was written in Rust. Here is its source code:

use nix_wasm_rust::{warn, Value};

#[no_mangle]
pub extern "C" fn fib(arg: Value) -> Value {
    warn!("greetings from Wasm!");

    fn fib2(n: i64) -> i64 {
        if n < 2 {
            1
        } else {
            fib2(n - 1) + fib2(n - 2)
        }
    }

    Value::make_int(fib2(arg.get_int()))
}

nix_wasm_rust is a support crate that provides Rust wrappers around the Wasm host functions that Nix makes available to Wasm modules. The type Value represents a (possibly not yet evaluated) Nix value. The call arg.get_int() makes a host function call to Nix to check that the value arg evaluates to an integer and return its value. Conversely, Value::make_int() creates a new Nix integer value. There are similar functions to access or construct other Nix data types, including attribute sets and lists. The macro warn!() calls a host function that prints out a message to stderr.

YAML in Nix

Using builtins.wasm, adding support for YAML is pretty trivial, since Rust already has a crate for parsing and generating YAML. Here is fromYAML implemented in Rust:

use nix_wasm_rust::{Type, Value};
use yaml_rust2::{Yaml, YamlLoader};

#[no_mangle]
pub extern "C" fn fromYAML(arg: Value) -> Value {
    Value::make_list(
        &YamlLoader::load_from_str(&arg.get_string())
            .unwrap()
            .iter()
            .map(yaml_to_value)
            .collect::<Vec<_>>(),
    )
}

fn yaml_to_value(yaml: &Yaml) -> Value {
    match yaml {
        Yaml::Integer(n) => Value::make_int(*n),
        Yaml::String(s) => Value::make_string(s),
        Yaml::Array(array) => {
            Value::make_list(&array.iter().map(yaml_to_value).collect::<Vec<_>>())
        }
        Yaml::Hash(hash) => Value::make_attrset(...),
        ...
    }
}

Performance

Nix uses Wasmtime, a Wasm runtime written in Rust that features a just-in-time code generator named Cranelift. The resulting code is much faster than equivalent Nix code. For example, here is Fibonacci in Nix:

# command time nix eval --expr 'let fib = n: if n < 2 then 1 else fib (n - 1) + fib (n - 2); in fib 40'
165580141
77.52user 1.66system 1:19.33elapsed 99%CPU (0avgtext+0avgdata 4570812maxresident)k

And here we are using the Rust Wasm version shown above:

# command time nix eval --impure --expr 'builtins.wasm { path = ./nix_wasm_plugin_fib.wasm; function = "fib"; } 40'
warning: 'nix_wasm_plugin_fib.wasm' function 'fib': greetings from Wasm!
165580141
0.31user 0.02system 0:00.33elapsed 100%CPU (0avgtext+0avgdata 30076maxresident)k

79.33 seconds to 0.33 seconds, a 240x speedup! It’s worth noting that the 0.33 seconds includes the code generation overhead, which Nix could cache on disk across invocations but currently doesn’t. Not only that, but Nix uses much less memory using the Wasm version: 30 MB instead of 4.5 GB, a 151x reduction.

It’s not all great, however. Wasm calls have a non-trivial overhead due to the need to create a new Wasm instance for every call. We can’t reuse instances between calls to the same function, because then the function could do impure things like maintain a global counter. We do use Wasmtime’s pre-instantiation feature to parse and compile Wasm modules only once per Nix process. On an Intel i7-1260P, Nix can do around 123,000 Wasm calls per second. By contrast, it can do around 2.8 million “native” function calls per second. Thus, Wasm is best used for larger tasks.

Managing Wasm functions

Wasm modules are often small enough that you can commit them into your Git repositories directly. For example, the compiled Wasm module for parsing and generating YAML is 180 KiB—probably still an acceptable size for adding to a repository like Nixpkgs.

Alternatively, you can fetch the Wasm module at evaluation time like this:

builtins.wasm {
  path = builtins.fetchurl https://.../nix_wasm_plugin_fib.wasm;
  function = "fib";
} 33

If you’re using flakes, you can use the file flake input type to fetch a single Wasm module via HTTP. This allows you to update the Wasm dependency automatically using nix flake update.

Finally, you could use import-from-derivation to declaratively build the Wasm module from source. But then you’re back to using import-from-derivation, which somewhat defeats the purpose!

Status

builtins.wasm is currently an experimental feature in Determinate Nix. There is also a PR to add it to upstream Nix.

If you want to give builtins.wasm a try, either install Determinate Nix or add the Determinate Nix CLI to your shell session:

nix shell github:DeterminateSystems/nix-src

To get a set of example Wasm functions from the nix-wasm-rust repo, run:

nix build github:DeterminateSystems/nix-wasm-rust

Then test whether it works:

nix eval --extra-experimental-features wasm-builtin \
  --impure --raw --expr \
  'builtins.wasm { path = ./result/nix_wasm_plugin_mandelbrot.wasm; function = "mandelbrot"; } { width = 60; }'

If you want to write Wasm functions in Rust, the nix-wasm-rust crate provides you with everything you need to interface with Nix. We have a blog post on compiling Rust to Wasm using Nix that you may find useful. For other languages, please consult the Wasm Host Interface documentation in the Determinate Nix manual. This interface is subject to change, which is the main reason builtins.wasm is still experimental. We welcome your feedback on writing Nix Wasm functions—in particular, please let us know if you run into limitations with the host interface.

Extending the Nix language isn’t the only application of Wasm in Nix. Wasm also enables platform-independent derivation builders, which also opens up many compelling possibilities. But that’s a topic for another blog post.

The Nix language has its detractors but it’s nonetheless provided a stable foundation for Nix for many years. With Nix usage pushing ever upward, now feels like an opportune—and exciting—time to push beyond some of the language’s historical limitations and see what the Nix ecosystem does with it.

Acknowledgments

Wasm support is the result of a collaboration between Determinate Systems and Shopify. Surma at Shopify developed the first prototype and wrote a function for running JavaScript in Nix via Wasm.