How to Use Hydra as your Deployment Source of Truth

Hydra is the Nix-based continuous integration system for the NixOS project. It is designed around evaluating a project’s Nix expressions and walking the graph of build jobs. Hydra is a fantastic tool for building small and large software collections. It is also a great tool for orchestrating releases.

Hydra’s API includes dynamic links that point to the most recent build of a job. Using this interface, deployment tools can query Hydra for the most recent artifact to deploy.

Fetching the Latest Build

Starting from a jobset named myapp:main that builds your application with this hydra.nix file:

1
{ nixpkgs ? <nixpkgs> }:
2
let
3
  pkgs = import nixpkgs { };
4
in
5
{
6
  myapp = pkgs.writeShellScript "hello" "${pkgs.hello}/bin/hello";
7
}

The Hydra API includes a “latest” URL to find the most recent, successful build of the myapp job. You can find this by visiting the jobset’s root page, clicking Jobs, clicking myapp, then the “Links” tab. In my case, the URL is https://demo.cloudscalehydra.com/job/myapp/main/myapp/latest.

Fetching this URL will redirect to the completed build:

$ curl --location \
    --header "Accept: application/json"
    https://demo.cloudscalehydra.com/job/myapp/main/myapp/latest \
      | jq .
{
  "id": 70,
  "finished": 1,
  "stoptime": 1634228145,
  "timestamp": 1634228145,
  "buildstatus": 0,
  "buildproducts": {},
  "project": "myapp",
  "system": "x86_64-linux",
  "buildmetrics": {},
  "job": "myapp",
  "starttime": 1634228145,
  "nixname": "hello",
  "buildoutputs": {
    "out": {
      "path": "/nix/store/c3l2x6fwl8cjkmfz6ilqcgczk13w8bk2-hello"
    }
  },
  "drvpath": "/nix/store/gf5wshlqfk1xa4i6ibk0z1l3j441c7vl-hello.drv",
  "jobset": "main",
  "jobsetevals": [
    296
  ],
  "priority": 100,
  "releasename": null
}

Note: There is another useful URL that links to the most recent passing build of the job with the additional requirement that there must not be any queued jobs left in the evaluation. This is useful for maximizing the amount of builds that are cached, but does not imply that all the jobs passed. That URL is the “latest-finished” URL: https://demo.cloudscalehydra.com/job/myapp/main/myapp/latest-finished.

Deploying Software and Servers from Hydra

You could imagine a deployment process that consumes the most recent build of myapp and automatically updates a local symlink:

$ app_path=$(curl --location \
    --header "Accept: application/json" \
    https://demo.cloudscalehydra.com/job/myapp/main/myapp/latest \
    | jq -r .buildoutputs.out.path)
$ nix-env -p /nix/var/nix/profiles/production --set "$app_path"

Indeed, if you’re running NixOS, you could build entire NixOS system configurations in Hydra and deploy to your clients the same way:

$ system_path=$(curl --location \
    --header "Accept: application/json" \
    https://demo.cloudscalehydra.com/job/servers/main/$(hostname)/latest \
    | jq -r .buildoutputs.out.path)
$ nix-env --profile /nix/var/nix/profiles/system --set "$system_path"
$ /nix/var/nix/profiles/system/bin/switch-to-configuration switch

Monitoring Build Status

Hydra exports Prometheus metrics for every job:

$ curl https://demo.cloudscalehydra.com/job/myapp/main/myapp/prometheus
# HELP hydra_job_completion_time The most recent job's completion time
# TYPE hydra_job_completion_time counter
hydra_job_completion_time{project="myapp",jobset="main",job="myapp"} 1634228145
# HELP hydra_job_failed Record if the most recent version of this job failed (1 means failed)
# TYPE hydra_job_failed gauge
hydra_job_failed{project="myapp",jobset="main",job="myapp"} 0

You can track and page on failed, or monitor completion_time to ensure that you never let a project go more than a few days without a completed build.

Gating Releases on Tests

The myapp example above will work great for some projects, but sometimes the software or system you’re deploying is much more complicated.

It may not be practical to run all of your software’s test validation in a single build.

In this case you want to deploy myapp, but you want to gate on some other build jobs succeeding too.

One solution is to make one job that depends on your other jobs. You can do that by adding a job to your hydra.nix that lists the other jobs in a text file:

1
{ nixpkgs ? <nixpkgs> }:
2
let
3
  pkgs = import nixpkgs { };
4
in
5
rec {
6
  myapp = pkgs.writeShellScript "hello" "${pkgs.hello}/bin/hello";
7
  myapp_test_does_it_run = pkgs.runCommand "test-hello" { } ''${myapp} > $out'';
8
  release_gate = pkgs.writeText "test-dependencies" ''
9
    ${myapp}
10
    ${myapp_test_does_it_run}
11
  '';
12
}

This method will create a job, release_gate, which only passes if myapp builds and runs. It also poses a problem: how do you get from release_gate to myapp? One option could be parsing the contents of this release_gate file, but that is fairly ugly. Another problem is when a test fails, Hydra doesn’t tell you a lot about what went wrong:

If your software has one or two tests, this might work, but this will be tedious with more than a handful of jobs you want to gate on.

Aggregate Jobs

Hydra’s Aggregate Jobs is a special type of job that addresses both of these problems.

Rewriting our previous example with an aggregate job involves making a list of “constituents” (the jobs you depend on) and setting the _hydraAggregate attribute:

1
{ nixpkgs ? <nixpkgs> }:
2
let
3
  pkgs = import nixpkgs { };
4
in
5
rec {
6
  myapp = pkgs.writeShellScript "hello" "${pkgs.hello}/bin/hello";
7
  myapp_test_does_it_run = pkgs.runCommand "test-hello" { } ''${myapp} > $out'';
8

9
  release_gate = pkgs.runCommand "test-dependencies"
10
    {
11
      _hydraAggregate = true;
12
      constituents = [
13
        myapp
14
        myapp_test_does_it_run
15
      ];
16
    } "touch $out";
17
}

This build task is trivial and the build product itself isn’t useful. The valuable part is the proof that all our important dependencies built successfully. Our release process can check to see if the release_gate build finished, and proceed if it did.

Hydra displays aggregate jobs differently. The build page for an Aggregate Job lists the named constituent jobs and their statuses:

The page for the job itself also shows the constituent jobs and their status history:

Using the latest-finished URL, you can get the most recent build where all the tests passed, and then fetch that build’s constituents:

$ build_id=$(curl --silent --location \
    --header "Accept: application/json" \
    https://demo.cloudscalehydra.com/job/myapp/main/release_gate/latest-finished | jq .id
    $ curl --silent \
    --header "Accept: application/json" \
    "https://demo.cloudscalehydra.com/build/$build_id/constituents" | jq
    [
    {
    "job": "myapp",
    "drvpath": "/nix/store/gf5wshlqfk1xa4i6ibk0z1l3j441c7vl-hello.drv",
    "id": 70,
    "releasename": null,
    "priority": 100,
    "timestamp": 1634228145,
    "finished": 1,
    "starttime": 1634228145,
    "system": "x86_64-linux",
    "buildmetrics": {},
    "jobset": "main",
    "nixname": "hello",
    "buildstatus": 0,
    "buildoutputs": {
      "out": {
        "path": "/nix/store/c3l2x6fwl8cjkmfz6ilqcgczk13w8bk2-hello"
      }
    },
    "project": "myapp",
    "jobsetevals": [
      296,
    303,
    307,
    310,
    313,
    314,
    315
    ],
    "buildproducts": {},
    "stoptime": 1634228145
    },
{
  "jobset": "main",
  "nixname": "test-hello",
  "buildstatus": 0,
  "buildoutputs": {
    "out": {
      "path": "/nix/store/zvs873fz97pwc91dk76dvzmnilj6mvis-test-hello"
    }
  },
  "project": "myapp",
  "jobsetevals": [
    314
  ],
  "buildproducts": {},
  "stoptime": 1634230682,
  "job": "myapp_test_does_it_run",
  "drvpath": "/nix/store/d36l0zyp5vxiqyird8m9p2jivzf0k67z-test-hello.drv",
  "releasename": null,
  "id": 78,
  "priority": 100,
  "timestamp": 1634230682,
  "finished": 1,
  "starttime": 1634230682,
  "buildmetrics": {},
  "system": "x86_64-linux"
}
]

Applying a little bit more jq we can get the exact path to the myapp build:

$ build_id=$(curl --silent --location \
    --header "Accept: application/json" \
    https://demo.cloudscalehydra.com/job/myapp/main/release_gate/latest-finished | jq .id
    $ curl --silent \
    --header "Accept: application/json" \
    "https://demo.cloudscalehydra.com/build/$build_id/constituents" \
    | jq -r '.[] | select(.job == "myapp") | .buildoutputs.out.path'
    /nix/store/c3l2x6fwl8cjkmfz6ilqcgczk13w8bk2-hello

Scaling Aggregate Jobs

For aggregate jobs that have a large evaluation graph, the evaluator’s memory footprint can exhaust the host’s available memory. This is especially common if your constituents include a lot of NixOS tests, or your jobset is evaluating a lot of NixOS system closures.

Since Hydra’s main design motivation is to be NixOS’s CI system and NixOS’s tested job depends on a lot of NixOS tests, Hydra has developed a small extension to Nix’s semantics to allow for more efficient aggregate jobs, allowing Nix’s garbage collector to free memory early.

Changing our example hydra.nix a little, we get the same behavior but allow the evaluator’s garbage collector to free some memory. Instead of listing derivations as constituents, Hydra allows you to specify constituent jobs using the job’s name as a string:

1
{ nixpkgs ? <nixpkgs> }:
2
let
3
  pkgs = import nixpkgs { };
4
in
5
rec {
6
  myapp = pkgs.writeShellScript "hello" "${pkgs.hello}/bin/hello";
7
  myapp_test_does_it_run = pkgs.runCommand "test-hello" { } ''${myapp} > $out'';
8

9
  release_gate = pkgs.runCommand "test-dependencies"
10
    {
11
      _hydraAggregate = true;
12
      constituents = [
13
        "myapp"
14
        "myapp_test_does_it_run"
15
      ];
16
    } "touch  $out";
17
}

Hydra’s evaluator will notice that the listed constituents are not derivations and are in fact regular strings. It will then look up these attributes in the list of jobs in the jobset and rewrite the derivation, substituting the plain string with the derivation path.

Note that while Hydra will be much more efficient at evaluating the release_gate job, Nix and other tools will not be able to evaluate and build the release gate in the same way.

An Aggregate of All Jobs

If you wanted your release gate to depend on all of the build jobs passing, a little bit of Nix can automatically create an aggregate job of all the other jobs:

1
{ nixpkgs ? <nixpkgs> }:
2
let
3
  pkgs = import nixpkgs { };
4
  jobs = rec {
5
    myapp = pkgs.writeShellScript "hello" "${pkgs.hello}/bin/hello";
6
    myapp_test_does_it_run = pkgs.runCommand "test-hello" { } ''${myapp} > $out'';
7
  };
8
in
9
jobs // {
10
  release_gate = pkgs.runCommand "test-dependencies"
11
    {
12
      _hydraAggregate = true;
13
      constituents = builtins.attrNames jobs;
14
    } "touch  $out";
15
}

Recap

Hydra is uniquely capable of building Nix projects, and using Hydra’s Aggregate Jobs provides deeper insight into the state and health of your project. Almost all of the data on https://status.nixos.org comes from Hydra’s Prometheus exporter, and you might notice the “Hydra job for tests” link goes to an aggregate jobset. The NixOS project has used aggregate jobs and the latest-finished URLs to manage releasing expression for years.

Using Nix together with Hydra’s unique feature set and API can give good visibility in to your test suite, and allows you to deploy with confidence.

If you’d like a managed Hydra server as a service, check out the first product we’re building: Cloudscale Hydra.