The future of software is Nix and we at Determinate Systems want to have a role in building that future. Today, I want to talk about Determinate, and I want to speak to you from a more personal place than you might be accustomed to here on our blog.
Determinate, to live up to the promise of our industry
What if every machine had a modern Nix with flakes enabled out of the box? And the cache just worked? And private repositories were easy to use? And Nix was up to date? And updating your dependencies took little to no effort? And the Nix store didn’t occasionally take up your entire disk, with no clear path to solving that?
What if adopting Nix were straightforward? What if adopting Nix didn’t involve solving a zillion annoying puzzles? What if adopting Nix didn’t irritate your IT or security teams?
That’s our vision.
To make a smooth path from “Nix could solve my dev, build, and configuration management problems” to “Nix is solving these problems” and in a way that doesn’t involve turning someone from a developer to “the Nix person.”
For users to pick up Nix and be effective and deploy critical software and not have to spend every waking moment solving unrelated puzzles. For users to have components that fit together in straightforward ways to let them actually do the thing that is important—where the important thing is not toying with Nix.
A future where software is maintainable and securable and we can take big leaps and still be safe and confident it’ll be fine. Where we can take risks and undo it. Where folks building critical infrastructure or deploying to space feel confident choosing Nix. Where Nix is mature enough for their use case. Where security teams know where to find clearly communicated, timely, advisories?
That’s why we’re here. To make more Nix users. To make those users happier and more productive. To make critical infrastructure safer.
How I got here
I knew Nix was the future in 2016 when I found Nix and switched all my personal systems and work infrastructure to it within just a few weeks. And I know this now more than ever. I can feel it in my bones.
Immutable infrastructure is the way
By 2014 I had been burned by declarative configuration management tools like Chef and Puppet. That experience spurred me to convince the startup where I worked to aggressively adopt immutable infrastructure.
With Chef, I had experienced severe drift not just from unmanaged config changes but also because the world around us was changing. I’d come in the next day, push out a change with Chef, and the unmanaged configuration met the new managed configuration and kaboom: production outage.
I’d also been burned by the Docker’s promise of reproducibility. The company where I worked at the time was in the process of aggressively adopting Docker for all of its deployments. Every merge to one of our dozens of repositories created fresh Docker images.
We pushed those images to a registry and then kicked off a CI/CD pipeline that would use Packer to build a brand new AMI. We’d then do a blue/green-style rolling deployment to our autoscaling groups to finalize the deploy.
It was brilliant.
I started talking at conferences about the importance of immutable infrastructure and about how incredibly stable and resilient our deployments and systems were.
Chef and Docker weren’t enough for great immutable infrastructure
But our CI/CD pipeline was slow. And it broke—a lot. We learned two things:
- Trying to manage individual packages or general system state with Chef was really hard. An annoying but solvable part of this came from resources like users where, if you added a user then wanted to remove them, you had to explicitly write that “remove” code. The bigger issue was that steps like “install nginx” could do different things on different servers that were running applying updates at the same time. I’d seen that several times, where we’d be deploying changes at the same time the apt mirrors were updating.
Dockerfiles weren’t reproducible because the entire world around you was changing. Files would disappear or change from URLs. Base images would change randomly. It was common practice to runapt-get updatein your image, and who knows what that was doing?
At conferences, I started talking about how Docker images were not enough because of how much the world was changing under our feet. How Chef and Puppet and similar tools weren’t enough because of how much of the system was unmanaged. How the root of this problem is how blind we were as operators were to the details of our software and systems.
I remember paraphrasing Carl Sagan a lot:
If you want to reproducibly build a server from scratch, you first have to pin the universe.
I’ve seen these words before…
In January 2016 I was complaining to a close friend about how slow it was to build a new AMI with Chef. She pointed me to the NixOS website. I read it—and I was Not Impressed™️.
Beyond the boring website, it said words I’d seen before. Those words I’d read about Chef and Docker. “Declarative.” “Reliable.” “Roll back.”
Yeah, yeah. Those words were heavily in vogue at the time. I’d heard them before, and they were lies.
But my friend persisted. “I don’t know, Graham… You should probably try it…”
I tried it
By the time I found NixOS, I knew what a system needed to look like if it were to be reproducible, declarative, effective, and to actually accomplish these goals we were trying to achieve as an industry:
- Strictly enforcing that all inputs are hashed.
- Eliminating network access during the build.
- Enabling you to build the system you want instead of trying to change a running system.
And here it was. Nix made sense. Nix was fast.
Nix was declarative. Nix was reliable. Nix’s rollbacks really did work.
I “got it.” I was hooked. This was the future.
I had to make this the future. With all love and respect to the Chef and Docker ecosystems, which taught me so much and to this day extend great generosity: I never wanted to use either tool again.
All in
I tried NixOS on a Wednesday and by Saturday I had erased macOS and switched to NixOS. It wasn’t easy! Within a few weeks, I’d replaced thousands of lines of Chef cookbooks with a few hundred lines of Nix expressions and a Bash script to build and upload AMIs to AWS.
The next step for me was fixing the obvious problem: Nix was hard to adopt.
Packaging was hard
A major problem was packaging software in the first place. So much of the software ecosystem around Nix hadn’t even started pinning dependencies or writing out version locks.
In a sense, the world was starting to realize what Eelco had identified over ten years prior: that pinning versions and hashing dependencies was not just worth doing, but critical to software security.
This has been a big focus of the industry’s work in the last fifteen plus years, but at the time there was so much to do. We’re in a different place than we were at the time. The industry has moved “towards” Nix in so many ways, cutting the packaging difficulty from “impossible” to “easy” for so many languages.
Documentation was hard
The docs were written largely by academics or folks so deeply steeped in Nix their writing was impenetrable to outsiders. This has improved since but there is still plenty to be done.
This problem was not just practical “how-to” documentation. Nix didn’t have great documentation to communicate and teach why Nix was important and why its rules were important. It struggled to educate the world about the new way of working that it demanded and about the incredible gifts it had bestowed on its users.
I would not have fallen in love with it if I had not already made the conceptual leaps I’d already made.
Persisting was so rewarding
Everything about Nix was a hugely rewarding puzzle. Every successful build was a huge dopamine hit, rewarding the challenges of navigating the Nix libraries and fixing build problems. Every pull request to Nixpkgs was quickly reviewed, merged, and celebrated by a relatively small group of dedicated folks who loved this tech as much as I did.
Even more “in”
After a couple years of intensely contributing to the project in my spare time, I quit my job to do full-time Nix consulting. Consulting taught me some important things.
Consulting revealed that Nix’s flexibility made it too hard
As a Nix consultant, I would often drop into a project and team that had more or less successfully and effectively adopted Nix. Invariably, that team used Nix in a way that was wholly their own and completely unlike that of any other team I’d worked with.
Their Nix champion had followed a choose-your-own-adventure game of writing scripts and tools and .nix files to adopt Nix at their company.
But this was not their fault.
There were no standard processes, no workflows, no off-the-shelf techniques to adopt and be successful.
On top of that, a lot of Nixpkgs’ interfaces were undocumented and hard to discover.
Consulting revealed that Nix just wasn’t quite mature enough
Nix’s flexibility meant joining a new team meant learning their own unique approach.
As I worked with more and more serious use cases, I noticed a schism in expectations and reality. These problems compounded as I moved up into increasingly sensitive and critical use cases.
They loved what Nix got them but they would squirm when we talked about how pull requests were merged. How casual the review process was. How casual the “maintainer” role was treated. How casually security patching was handled.
It was uncomfortable to beat around the bush and say “well, historically it has been fine …” and “folks are usually careful about updates…”
And for security patching, corporate users had security and compliance objectives that didn’t run on vibes alone. They needed to know what was vulnerable and what wasn’t. They needed to know when security updates were available and how to get them.
Let’s get mature
I became obsessed with maturing the project.
I wrote OfBorg to catch more contributor errors before they merged.
I started the NixOS security team and ran weekly security round-ups until our data source literally shut down.
I wrote security advisories.
I rewrote the Nix installer to be more defensive and reliable.
I worked like crazy to make the Nix installer use the daemon for better security, and to be more defensive and reliable.
The bones of that installer are still the foundation of today’s upstream Nix installer.
To be clear, while I did start and lead these (and other) initiatives and my fingerprints of effort and love are all over the project, I didn’t do any of this alone. Thank you, to the entire community, for which all of this would be impossible.
The world needs Nix
I remember the moment I realized I was white-knuckling the project into maturing. Something in Nixpkgs broke. It wasn’t a particularly big deal but it did need to be treated seriously and fixed.
Someone made a comment like “good thing this isn’t being used by banks or whatever!” At that moment, I was working with financial firms using Nix in their banking and trading platforms. I was dizzied by the disparity between how important the project was compared to the concrete reality of the project.
The next 100 years of computing needs to start with Nix. Our world’s infrastructure should not be stuck in a “don’t touch it, it works” mode, but it is.
And I don’t just mean low-stakes systems. Our entire world is software, and as an industry we’re failing the world. Critical infrastructure running power grids and nuclear reactors and robots and scientific research is woefully insecure because it is simply too hard and scary to maintain.
Nix can, and does, solve many of these problems in very real terms today.
I felt at the time that it was a huge tragedy that Nix wasn’t mature enough to be used in those scenarios. I felt frustrated and occasionally angry that so many people saw the sparkling diamond in front of us but were afraid of reaching for the big picture. This stuff matters.
I love Nix. I hate that Nix doesn’t have consistent interfaces.
Nix was so obvious and important. And painful.
The inconsistencies really stacked up. Not just across teams and projects and repositories, but across environments.
Every new target was a fresh batch of misery. Setting up CI, or deploying to a new machine or cloud provider or bare metal server was a fresh batch of tiresome questions:
- How do I onboard my team members?
- How do I set up my cache in CI? Or on the new server?
- How do I deploy to this machine?
- How do I scale up and down quickly?
- How do I get private repositories in there?
- Why do I have to SSH in as root to deploy?
- How do I manage my NixOS infrastructure that feels like 2015 and not 2005?
It just went on and on and the answers invariably involved writing way too many Bash scripts predicated on hopes and dreams. Maintenance was annoying and the trade-offs were deeply sub-optimal.
It is mind numbing how badly we were missing the table stakes of similar ecosystems. I wanted a developer experience that didn’t involve banging together hundreds of lines of copy-pasted Bash scripts to be good.
Nix’s raison d’être is to make it possible to move software from one computer to another and have it keep working. The promise was there, it worked, but the reality was occasionally misery.
Okay, fine, flakes
I didn’t care for adopting flakes, but only because they were experimental. My thinking evolved.
I came around after realizing that flakes cut down on inconsistencies by making a standard evaluation model and input/output interface for projects. Flakes are a first, small step, but a foundationally important step.
But a portion of the community has not made that leap yet. I still think that even if flakes are occasionally annoying, or even if there are flaws in their design, they’re predictable, improve build performance, add stability, and facilitate collaboration across developers and teams. They make Nix viable, and delightful, at work.
There are blog posts out there that describe how to get flakes without flakes. They’re wrong because the interface is the point. Everything else is details.
Today, we still find ourselves sitting on the most powerful technology of our lifetimes and we can’t even decide on small steps in the direction of making it easier for folks to adopt it.
Determinate, to live up to the promise of our industry
So here we are. We’re building it. We’re unabashedly making Nix the best way to build and deploy software to the most important targets in the world.
Our work means flakes are enabled. That a new user has a one-step process to get private repositories and access to their team’s cache. We’ve made tools for tracking and updating dependencies.
Our documentation, tools, and workflow are designed top to bottom to make adopting Nix dramatically more straightforward. We’re working with IT and security teams to make Nix a welcome addition. We’re deliberately cutting out puzzles and bash scripts for deployment and access control in dev, test and prod.
We’re building that smooth path from “Nix could solve these problems” to “Nix is solving these problems”. We’re making it possible to use Nix for critical software, with components that fit together in straightforward ways. We’re helping teams focus on the valuable problems they’re solving.
We’re enabling teams to safely take big leaps with confidence and safety.
That’s why we’re here. That’s our vision.
To make amends and live up to the promise our industry has made to the world.
