Recently I was helping someone get bootstrapped on their Axum project, and they were getting a bit frustrated with the lack of context about the goings-on in their application.
While Axum and its ecosystem are well instrumented, by default most of it is not well surfaced for beginners.
It can be a bit intimidating figuring out how to put the pieces together and use them.
Once we set up tracing and color_eyre our logging and errors went from looking like this:
To something with a little bit more context:
While this isn’t a complete and fully mature solution suitable for an industrial application, it is a good starting point for them that smoothly evolve into something more mature.
Often the first place I start with a binary project is the command line interface, so we’ll start there.
Instrumentation can have many knobs, this project used clap, so we took the existing Cli struct and added an instrumentation field:
Using #[clap(flatten)] here means we can define all our instrumentation options in a separate struct, while still appearing in the --help as one might expect:
Tracing offers use the choice of several built in logging styles, or even our own logging style if we so choose!
It’s straightforward to pass on this choice to the user, though we do need to create Logger enum to represent that choice:
The Logger can then get used in our Instrumentation struct, which we’ll build up over the next few code blocks:
This interface offers users the ability to specify ‘traditional’ verbosity options such as -v or, my favorite, -vvvvvvv through the use of clap::ArgAction::Count.
It also permits the use of more tracing-specific options like the Logger we created above, or the option of any number of tracing_subscriber::filter::Directive. We’ll explore those a bit together once we get things running.
We can then attach some functions to this struct that set up a Registry which stores span data, such as the data defined in #[tracing::instrument] calls.
In a setup() function we’ll build a Registry and compose it with several layers, including a ErrorLayer and an EnvFilter layer we configure from the knobs made available in the Cli struct (as well as some conventional environment variables):
Then we can go ahead and define the various format layers, along with any customization we wanted to do.
Later, your project may choose to integrate something like tracing_opentelemetry and route that data to a service like Honeycomb.
Unfortunately, this does not work out of the box with eyre::Report.
It’s also forbidden for us (as demo) to implement IntoResponse for Report due to the Orphan Rule, so we must create a wrapper. While we do that, we might as well create a crate-specific DemoError as well, which can be later used to define user-facing error messages and status codes.
Now we can use that new Result in our routes, and return errors either ad-hoc via eyre::eyre, or structured through the error type we created with thiserror.
This is also a good opportunity to decorate your routes with a tracing::instrument attribute.
Consider using skip_all and explicitly defining any fields if you are dealing complex route arguments!
Eg. #[tracing::instrument(skip_all, fields(uid = context.user.uid))].
Find out more about how Determinate Systems is transforming the developer experience around Nix
At the start, and during prototyping, you can use eyre::eyre to write ad-hoc errors.
As the application grows, you’ll be able to work with thiserror and use downcast_ref (like we did in the instrumentation.rs code) and other tools available in eyre::Report to respond with a more structured and informative messages to the web browser.
After painting our user interface and performing the ritualistic type dancing, it’s finally time to update main() with the code to get everything working.
At the top of main() we want to install color_eyre’s error hooks, so that any errors after that are fully styled and integrated.
After that, we can call setup() on the Instrumentation instance created by clap to initialize tracing.
TraceLayer offers opportunities to attach our own functions where we can create spans or emit events.
There exist defaults (tower_http::trace::{DefaultMakeSpan, DefaultOnRequest}) which can be modified using a builder API, these have the rather unfortunate quality of being part of the tower_http crate, not our own, so because the defaults we configured in the Instrumentation::filter_layer() function they won’t normally be enabled.
Defining our own functions allows us to build our own base span, as well as ensure these spans are part of our own crate.
If your project opts to use the defaults, consider altering Instrumentation::filter_layer() to also set some default for tower_http.
We can create a few test routes a fallback route, one reporting no error, another reporting an eyre::eyre based error, and a route reporting a DemoError.
Due to our error handling code we can use DemoError variants to return specific, user-facing messages and status codes, while more ad-hoc or developer/operator facing errors can be eyre::eyre based.
If you’re following along you may have run cargo add for some of the dependencies above, the specific examples shown utilized these features and crates:
One feature you may wish to enable the track-caller on color-eyre package, it will show the locations of errors in the output using the track_caller attribute.
The reported location is not always accurate, but often it’s helpful.
At this point, we can run our binary and visit the homepage:
We can run the application again with different directives and get some more info from one of the packages, for example hyper which does the bulk of the HTTP work:
That’s a lot of output! We can also use log directives to drill down into specific parts of the code, this can be quite useful for debugging. Here we isolate hyper events to parse_headers:
Or how about only logging requests for a specific URL? We can specify a filter like demo[request{uri=/}]=trace and visit several pages, observing we only see logs for the one we filtered for:
Of the different logger options, my personal favorite is the pretty logger which creates a cozy, human readable experience:
The tracing spans also appear in our errors, let’s take a look at those. We can observe these errors when we visit the respective /error urls:
Visiting /error/demo we can observe the browser returning A spooky occured as we intended.
The combination of thiserror, tracing, and color_eyre provides a solid starting point for a budding project. The ability to instrument code with spans, see those spans in errors, then filter based on those spans in logging messages enables greater insight while diagnosing issues and authoring new features. As your project grows, these same tools offer a smooth path to adoption of standards like OpenTelemetry.
While researching this article I bumped into the really lovely article by @carlosmv about a many of the same ideas! They go in detail about how to use tracing_appender and some other things which are not covered here, but don’t discuss error handling as much. If you want to dig deeper into this topic, that article is a wonderful place to visit after this!
Ana is a hacker working in the Rust and Nix ecosystems.
She's from Lək̓ʷəŋən territory in the Pacific Northwest, and holds a B.Sc. in Computer Science from the University of Victoria.
She takes care of a golden retriever named Nami with her partner.