Deriving Traits in Rust with Procedural Macros

Procedural macros in Rust are a really compelling feature that I didn’t understand until recently. There are a few gotchas, but they make it super easy to implement custom #[derive()] expansions for implementing traits with a single line of code. Let’s dive in.

Before the advent of procedural macros two years ago in Rust 1.15, it was historically really difficult to create your own derive implementations to automatically implement traits for your types. serde and friends have long supported it, but for the average user, it remained just out of reach. I didn’t know what procedural macros meant until I read Alex Crichton’s excellent post on them, last month.

I had a really simple trait that I wanted to derive for my types. In my nfty command line utility, I’m using askama, which is something similar to Jinja for Python in its syntax. Askama is an amazing Rust implementation that does most of the templating work at compile-time. Syntax errors in your templates prevent compilation, the templates are compiled into your binary/library, and at runtime, rendering does no template language processing; an AST is compiled at Rust compile time.

I could gush all day about how awesome this is, but the implication is that if your program compiles, your templates are valid, and performance is incredible.

I wrote a pretty simple trait which extends Askama template structs to include a write method to write the template to the filesystem using a given std::path::Path. It’s pretty simple and looks like this:

use askama::Template;
use std::io::Result;
use std::path::Path;

pub trait WritableTemplate: Template {
    fn write(&self, path: &Path) -> Result<()>;
}

The implementation of this trait for a given type looks kind of like this:

use askama::Template;

use std::fs;

use std::io;
use std::io::prelude:*;

use std::path::Path;

#[derive(Template)]
#[template(path = "demo.j2")]
pub struct MyTemplate {}

impl WritableTemplate for MyTemplate {
    fn write(&self, path: &Path) -> io::Result<()> {
        let mut file = io::BufWriter(fs::File::create(path)?);
        file.write(self.render.unwrap().trim().as_bytes())?;

        Ok(())
    }
}

So far so good, yeah? Take a Path and render and write the template output to that Path.

The problem becomes the fact that I need to repeat this trait implementation by hand for all of my Template types. We can do better, thanks to procedural macros.

The first gotcha that I ran into was the fact that procedural macros must live in their own crate, if you’re creating a binary crate. This was kind of frustrating, but the overhead turned out to be pretty minimal.

I created a sub-crate using the Cargo workspaces feature, which allows you to host multiple crates in a single repository. Since I’m not planning on exposing this crate to crates.io or anywhere else, there’s not much to do. I did the following to create the new layout for the new crate:

mkdir -p nfty-derive/src
touch nfty-derive/Cargo.toml nfty-derive/lib.rs

I next needed to enable the workspaces feature in my root Cargo.toml and add the sub-crate as a dependency. My full Cargo.toml is kind of long, but here are the changes I had to make:

[package]
# ...
edition = "2018"
# ...

[dependencies]
# ...
nfty-derive = { path = "nfty-derive", version = "0.1.0" }
# ...

[workspace]

That’s basically it: set edition to 2018, add a reference to the local crate, and create an empty [workspace] section.

Next, let’s define nfty-derive/Cargo.toml:

[package]
name = "nfty-derive"
version = "0.1.0"
edition = "2018"

[lib]
proc-macro = true

[dependencies]
askama = "0.7.2"
syn = "0.15.23"
quote = "0.6.10"

We make the crate a library crate and set proc-macro to true to enable procedural macros for this library crate. We also declare some dependencies, which isn’t anything surprising. The syn and quote crates provide the ability to quote and parse the Rust syntax as an AST. This is why procedural macros are so amazing: you can literally parse and interact with the source code as if you were extending rustc!

Let’s take a first pass at the procedural macro:

extern crate proc_macro;

use crate::proc_macro::TokenStream;

use quote::quote;

use syn::DeriveInput;

#[proc_macro_derive(WritableTemplate)]
pub fn writable_template_derive(input: TokenStream) -> TokenStream {
  let input = parse_macro_input!(input as DeriveInput);

  // get the name of the type we want to implement the trait for
  let name = &input.ident;

  let expanded = quote! {
    impl crate::project::templates::WritableTemplate for #name {
      fn write(&self, dest: &::std::path::Path) -> ::std::io::Result<()> {
        let mut file = ::std::io::BufWriter::new(::std::fs::File::create(dest)?);
        file.write(self.render().unwrap().as_bytes())?;

        Ok(())
      }
    }
  };

  TokenStream::from(expanded)
}

There’s a few things that might seem weird here. First, why extern crate? Didn’t this die in the 2018 edition? The answer is yes, but no. For procedural macro crates, these lines still need to explicitly exist.

Next, we use the #[proc_macro_derive(...)] annotation to tell Rust that we’re creating a #[derive(...)] macro for something called WritableTemplate. Note how there is no definition of the trait in this crate. This is important: the derive definition is separate from the actual trait implementation itself. Since we’re not interested in reuse and this is kind of a single-purpose crate, we simply expand to crate::project::templates::WritableTemplate, which is where the trait is defined in the parent crate.

#name references the variable above, name, and refers to the name of the struct/type we’re deriving for.

Everything else should look really straightforward. The quote! macro parses the Rust within its brackets to templatize the Rust within. We return a proc_macro::TokenStream back to the compiler so it can be rendered during compilation.

This code works, and we can now accomplish our goal:

use askama::Template;

// need this for the Write trait
use std::io::prelude::*;

#[derive(Template, WritableTemplate)]
#[template(path = "demo.j2")]
pub struct MyTemplate {}

AWESOME. We’ve done it!

Actually, not yet. What happens if we have a more complicated type?

// ...
pub struct MyTemplate2<'a> {
  title: &'a str,
}

If we try to derive WritableTemplate for this type, compilation will fail. Why? Well, the rendered code by our procedural macro doesn’t include any generic type parameters, which are now present (i.e. <'a>) and compilation fails.

It was kind of hard to find, but poking around, I discovered that it’s necessary to get a handle to the various generic type attributes and include them in my procedural macro. I ultimately arrived at this:

// ...
#[proc_macro_derive(WritableTemplate)]
pub fn writable_template_derive(input: TokenStream) -> TokenStream {
  // ...
  let name = &input.ident;
  let generics = input.generics;
  let (impl_generics, ty_generics, where_clause) = generics.split_for_impl();

  let expanded = quote! {
    impl #impl_generics crate::project::template::WritableTemplate for #name #ty_generics #where_clause {
      // ...
    }
  };

  TokenStream::from(expanded)
}

. Now, types with generic arguments and where clauses will work. We’ve made it, right?

In a sense, yes. However, I wanted to add a call to trim() on the output of the template to kill surrounding whitespace. Should be straightforward, right?

Well, surprise surprise, it wasn’t. I changed the render line to look like this in the procedural macro definition:

file.write(self.render().unwrap().trim().as_bytes())?;

All of a sudden, I got some weird error at compile time about a recursion limit being exceeded. What? We only added a single call to String::trim() and this broke our compilation? The compiler asked me to configure the recursion limit for my procedural macro crate by adding the #![recursion_limit = "128"], but this didn’t seem right. Was this really the solution?

I asked around on #rust on the Mozilla IRC servers (can’t recommend this enough, people are super helpful!) and yeah, configuring the recursion limit was the answer. Underneath everything, a stringify! macro call was being added and it appears that by default, the recursion limit in macros is set to a very conservative value. I added the crate attribute, recompiled, and everything just worked™.

The final nfty-derive/src/lib.rs looks like this:

#![recursion_limit = "128"]

extern crate proc_macro;
#[macro_use]
extern crate quote;
#[macro_use]
extern crate syn;

use crate::proc_macro::TokenStream;
use syn::DeriveInput;

#[proc_macro_derive(WritableTemplate)]
pub fn writable_template_derive(input: TokenStream) -> TokenStream {
    let input = parse_macro_input!(input as DeriveInput);

    // type name
    let name = &input.ident;

    // generics
    let generics = input.generics;
    let (impl_generics, ty_generics, where_clause) = generics.split_for_impl();

    let expanded = quote! {
        impl #impl_generics crate::project::templates::WritableTemplate for #name #ty_generics #where_clause {
            fn write(&self, dest: &::std::path::Path) -> ::std::io::Result<()> {
                let mut file = ::std::io::BufWriter::new(::std::fs::File::create(dest)?);
                file.write(self.render().unwrap().trim().as_bytes())?;

                Ok(())
            }
        }
    };

    TokenStream::from(expanded)
}

Calling the derive macro looks like this:

use askama::Template;

use chrono::{Datelike, NaiveDate, Utc};

use std::io::prelude::*;

#[derive(Template, WritableTemplate)]
#[template(path = "licenses/APACHE.j2")]
pub struct ApacheLicense<'a> {
  pub author: &'a str,
  pub date: NaiveDate,
}

Finally, calling the write method looks like this:

let license = ApacheLicense { author: "Naftuli Kay", date: Utc::now().date().naive_utc() };
license.write(&project_dir.join("LICENSE-APACHE")).expect("unable to write template");

It wasn’t as simple as I would have hoped, but it’s really not that bad! The utility of being able to one-liner derive arbitrary traits for types means that I’ve cut out a lot of code duplication. Thanks, as always, to the incredible Rust community for all the blog posts, documentation, and support that helped me get here

EDIT

A couple users on users.rust-lang.org and /r/rust pointed out some corrections to the post:

1. Procedural macros have been in Rust since 1.15, which is two years old at this point, not in the 2018 edition.
2. I had originally external crate syn and external crate quote in the macro crate. This is not necessary and these lines have been dropped from the post.
3. There’s another way to do all of this without derive macros, which I’ll discuss now.

Instead of using a derive on each struct I want WritableTemplate to apply to, there’s a more global way of doing things using the type system:

impl <T: Template> WritableTemplate for T {
    fn write(&self, path: &Path) -> io::Result<()> {
        // ...
    }
}

This is a big hammer, but it’ll basically automatically implement WritableTemplate with the given write implementation for any type which is an askama::Template. If it’s desirable to not go fully global, it’s possible to introduce a WritableTemplateMarker trait which is empty and serves as an indicator to mark certain types as needing WritableTemplate implemented for them. See the Reddit comments and forum post for more info.