Hydra is the Nix-based continuous integration system for the NixOS project. It is designed around evaluating a project’s Nix expressions and walking the graph of build jobs. Hydra is a fantastic tool for building small and large software collections. It is also a great tool for orchestrating releases.
Hydra’s API includes dynamic links that point to the most recent build of a job. Using this interface, deployment tools can query Hydra for the most recent artifact to deploy.
Fetching the Latest Build
Starting from a jobset named myapp:main
that builds
your application with this hydra.nix
file:
The Hydra API includes a “latest” URL to find the most recent, successful build
of the myapp
job. You can find this by visiting the jobset’s root page,
clicking Jobs, clicking myapp
, then the “Links” tab. In my case, the URL is
https://demo.cloudscalehydra.com/job/myapp/main/myapp/latest.
Fetching this URL will redirect to the completed build:
Note: There is another useful URL that links to the most recent passing build of the job with the additional requirement that there must not be any queued jobs left in the evaluation. This is useful for maximizing the amount of builds that are cached, but does not imply that all the jobs passed. That URL is the “latest-finished” URL: https://demo.cloudscalehydra.com/job/myapp/main/myapp/latest-finished.
Deploying Software and Servers from Hydra
You could imagine a deployment process that consumes the most recent build of
myapp
and automatically updates a local symlink:
Indeed, if you’re running NixOS, you could build entire NixOS system configurations in Hydra and deploy to your clients the same way:
Monitoring Build Status
Hydra exports Prometheus metrics for every job:
You can track and page on failed
, or monitor
completion_time
to ensure that you never let a project go more than a few days without a
completed build.
Gating Releases on Tests
The myapp
example above will work great for some projects, but sometimes the
software or system you’re deploying is much more complicated.
It may not be practical to run all of your software’s test validation in a single build.
In this case you want to deploy myapp
, but you want to gate on some other
build jobs succeeding too.
One solution is to make one job that depends on your other jobs. You can do
that by adding a job to your hydra.nix
that lists the other jobs in a text
file:
This method will create a job, release_gate
, which only passes if myapp
builds and runs. It also poses a problem: how do you get from release_gate
to
myapp
? One option could be parsing the contents of this release_gate file,
but that is fairly ugly. Another problem is when a test fails, Hydra doesn’t
tell you a lot about what went wrong:
If your software has one or two tests, this might work, but this will be tedious with more than a handful of jobs you want to gate on.
Aggregate Jobs
Hydra’s Aggregate Jobs is a special type of job that addresses both of these problems.
Rewriting our previous example with an aggregate job involves
making a list of “constituents” (the jobs you depend on) and setting the _hydraAggregate
attribute:
This build task is trivial and the build product itself isn’t useful. The
valuable part is the proof that all our important dependencies built
successfully. Our release process can check to see if the release_gate
build
finished, and proceed if it did.
Hydra displays aggregate jobs differently. The build page for an Aggregate Job lists the named constituent jobs and their statuses:
The page for the job itself also shows the constituent jobs and their status history:
Using the latest-finished
URL, you can get the most recent build
where all the tests passed, and then fetch that build’s constituents:
Applying a little bit more jq
we can get the exact
path to the myapp
build:
Find out more about how Determinate Systems is transforming the developer experience around Nix
Scaling Aggregate Jobs
For aggregate jobs that have a very large evaluation graph, the evaluator’s memory footprint can exhaust the host’s available memory. This is especially easy if your constituents include a lot of NixOS tests, or your jobset is evaluating a lot of NixOS system closures.
Since Hydra’s main design motivation is to be NixOS’s CI system and NixOS’s
tested
job depends on a lot of NixOS tests, Hydra has developed a
small extension to Nix’s semantics to allow for more efficient aggregate jobs,
allowing Nix’s garbage collector to free memory early.
Changing our example hydra.nix a little, we get the same behavior but allow the evaluator’s garbage collector to free some memory. Instead of listing derivations as constituents, Hydra allows you to specify constituent jobs using the job’s name as a string:
Hydra’s evaluator will notice that the listed constituents are not derivations and are in fact regular strings. It will then look up these attributes in the list of jobs in the jobset and rewrite the derivation, substituting the plain string with the derivation path.
Note that while Hydra will be much more efficient at evaluating the
release_gate
job, Nix and other tools will not be able to evaluate and build
the release gate in the same way.
An Aggregate of All Jobs
If you wanted your release gate to depend on all of the build jobs passing, a little bit of Nix can automatically create an aggregate job of all the other jobs:
Recap
Hydra is uniquely capable of building Nix projects, and using Hydra’s Aggregate
Jobs provides deeper insight into the state and health of your project. Almost
all of the data on https://status.nixos.org comes
from Hydra’s Prometheus exporter, and you might notice the “Hydra job for
tests” link goes to an aggregate jobset. The NixOS project has used aggregate
jobs and the latest-finished
URLs to manage releasing expression for years.
Using Nix together with Hydra’s unique feature set and API can give good visibility in to your test suite, and allows you to deploy with confidence.
If you’d like a managed Hydra server as a service, check out the first product we’re building: Cloudscale Hydra.