Deploying Nix-built containers to Kubernetes

While it’s great to fall in love with Nix and want to use it to build and configure just about everything, let’s face it: few of us are in a position to do so, especially in the workplace. “Hey team, let’s Nixify our entire stack!” is unlikely to endear you to your coworkers at daily standup. Fortunately, Nix is good at enough things that you can reap many of its potential benefits by incorporating it into only a subset of an already existing software pipeline.

In this post, I’ll provide a concrete example of this by using Nix inside a fairly standard DevOps pipeline that builds Docker images and deploys them to Kubernetes in CI. Nix does a lot of heavy lifting in this scenario but it doesn’t do everything: it doesn’t stand up any infrastructure and it doesn’t handle deployment. That’s all left to popular, trusted tools and platforms from outside the Nix universe. I hope that this example inspires you to find ways to incrementally introduce Nix into your own stacks.

The deployment scenario

In my example scenario, which you can find in the nix-to-kubernetes repo under the Determinate Systems organization, I use Nix for two things:

• To create cross-platform development environments for both local machines and the continuous integration level (which runs on GitHub Actions).
• To build a Go web service into Docker images that can be deployed on Kubernetes (or most other container orchestration platforms).
• All of that is configured in the repo’s flake, which amounts to a few dozen lines. What Nix doesn’t do here:
• Stand up any infrastructure. Instead, Terraform uses an HCL configuration to handle that.
• Deploy the container. Instead, kubectl deploys the container to Kubernetes, which is configured using a straightforward YAML file.

The benefits of Nix

What would this scenario look like without Nix? I suspect it would usually look something like this:

• Contributors would be on their own to create their development environment. The README would perhaps list necessary dependencies—Go, Terraform, kubectl, and others—but there’d be no guarantee that the “works on my machine” problem would be overcome the way it can with Nix.
• The CI pipeline would use third-party Actions, which are problematic for reasons I lay out here.
• The Docker CLI and a Dockerfile would be used to build Docker images. Dockerfiles are a “good enough” construct for defining container build logic, but they’re not reproducible by default (the same apt-get install command can yield different results on different days, for example) and their procedural nature—COPY this, ADD that, RUN this—is a source of great consternation for many. Contrast that approach with this highly concise build logic in Nix.

By contrast, using Nix here—a relatively straightforward Nix flake—provides a standardized, cross-platform dev environment that’s synced with the CI environment as well as a fully declarative Docker image build, a major improvement over the known brittleness of procedural Dockerfiles.

Implications

This project shows that Nix can not just unobtrusively but quite comfortably live alongside tools and processes that have zero awareness of Nix. I certainly could have “Nixified” other parts of this pipeline but deliberately held back.

Introducing Nix into an organization can be a daunting process, especially if you want to boil the proverbial ocean and convert everything in sight—developing, building, deploying—to Nix. But what I hope to have shown here is that you can introduce Nix in a gentle, piecemeal way that yields immediate benefits while introducing few if any drawbacks. Using Nix to run a service locally and package it as a Docker image is a low-risk way to introduce Nix into a multifaceted software pipeline. So go head, pass this post along to your teammates or, even better, CTO or director of engineering. If you—or your superiors—have any lingering questions, feel more than free to reach out to us.