Announcing: Terraform Provider Hydra

I’m excited to announce Determinate Systems’ first open source release: terraform-provider-hydra, a Terraform provider for managing Hydra projects and jobsets.

Standardize jobset configuration. Manage your Hydra jobsets with the rest of your critical infrastructure. Know your jobsets are configured correctly. Apply standard PR workflows to approve changes and ensure operational consistency.

Reduce risk exposure. Exchange highly privileged users and manual changes for automation and auditable logs. Remove high-level create-project and admin permissions from most users, and deploy configuration changes through Terraform instead.

Reduce configuration overhead. Directly manage your jobsets with the rest of your infrastructure in a consistent and predictable way. Save Hydra’s generated declarative jobsets for when you truly need dynamically generated jobsets.

We’re big fans of Hydra. Its native understanding of Nix closures makes it a first class build system for Nix projects. The scheduler and built-in distributed build support allows for distributing work to an automatically scaled build pool of builders.

Now you can configure your Hydra with the same tool you use to manage the rest of your infrastructure.

Declarative configuration, at the CI level

Hydra deployments are usually a central root of trust. Using projects and jobsets, Hydra can provide a lookup table and audit log, mapping use cases to tested revisions of code. However Hydra does not provide an audit log of configuration changes to a jobset. The input configuration is the keystone to the validity of that mapping.

Using Terraform as the source of configuration truth solves a number of problems presented by hand-managed configuration:

1. Managing inputs by hand poses a risk of misconfiguration and drift between jobsets. As configuration changes over time, older jobsets are less likely to be updated.
2. Permission to modify a jobset is only granted to administrators and the “owner” of a project. Configuring a jobset across a team requires granting high level, administrative-level access to the entire team.
3. Changes are unlikely to be noticed. The jobset could be changed to point to an alternative source repository or branch without anyone noticing. The lack of an audit log or other guarantees makes it difficult to notice changes to jobsets which are not actively monitored.

Audits, robust access control, and effective delegation

Managing your Hydra’s configuration through Terraform resolves all of these issues at once. Inputs are tracked and managed holistically, across the entire server, from the same source of truth. Users no longer need their own administrative credentials: any changes should go through a Continuous Delivery pipeline with its own credentials, just like everyone else. Additionally, any divergence of a Terraform managed Jobset or Project is identified and corrected on the next run of Terraform.

Get started

Your Hydra will need to be running version 6e537671 or later, which is already available in the nixos-unstable, nixos-unstable-small, nixos-20.09, and nixos-20.09-small channels.

Then, install the provider as documented in the project’s README.

From there, we recommend starting by importing your existing configuration.

Migrating an existing instance

Clone the repository to get started:

git clone https://github.com/DeterminateSystems/terraform-provider-hydra.git
cd terraform-provider-hydra/tools
nix-shell

Then, run the generator:

[nix-shell]$ mkdir network
[nix-shell]$ ./generator.sh https://hydra.example.com ./network ./network/imports.sh

The generator will create one file per project in ./network, and ./network/imports.sh will contain a list of terraform import commands to execute.

Execute those commands:

bash ./network/imports.sh

and then you’re done! terraform plan should show no differences to be applied.

What’s next

Take a look at the code, give it a try. Let us know what you think!

Since May 5th, we have continuously run the importer against hydra.nixos.org and checked the resulting plan is clean. So far so good: each configuration change has worked flawlessly with the provider and the importer. We hope the NixOS infrastructure team will adopt this provider and manage the public Hydra with code.

Since our founding just a few months ago, we’ve made significant investments into Hydra’s long term stability and maintainability. We see Hydra as a cornerstone of the Nix ecosystem, and we’re planning to continue our investment over the long term. Look out in the near future for performance improvements for large deployments.