Blue/Green Deployments with Route 53

Blue-green deployments with Route 53 involve transferring traffic from one endpoint to another using weighted DNS records. We can automate this relatively easily via the AWS CLI and jq, though there are a few things that are left to be desired: the syntax is quite verbose for updating DNS records.

Finding Your Hosted Zone

We can find a hosted zone id by its name using the following AWS CLI script, tranformed with jq:

$ aws route53 list-hosted-zones | jq -r --arg name domain.com. \
     '.HostedZones[] | select(.Name == $name and .Config.PrivateZone == false) | .Id | ltrimstr("/hostedzone/")'
147MAIJCWVL9PC

Do note that the trailing . is important due to our jq matching. Now that we have our hosted zone, we can use it to discover our records.

Discovering Record Identifiers

Weighted record sets have identifiers so that they can be uniquely referenced and updated. Let’s discover those records and their identifiers.

$ aws route53 list-resource-record-sets --hosted-zone-id 147MAIJCWVL9PC \
    --query "ResourceRecordSets[?Name=='www.domain.com.']" | jq .
[
  {
    "Name": "www.domain.com.",
    "Type": "CNAME",
    "Weight": 100,
    "TTL": 1,
    "SetIdentifier": "blue",
    "ResourceRecords": [
      { "Value": "blue.domain.com" }
    ]
  },
  {
    "Name": "www.domain.com.",
    "Type": "CNAME",
    "Weight": 0,
    "TTL": 1,
    "SetIdentifier": "green",
    "ResourceRecords": [
      { "Value": "green.domain.com" }
    ],
  }
]

We have now obtained our record sets with their identifiers. Unfortunately, due to the nature of what the AWS API and CLI expect, we’ll need these full literal values to make updates.

Shifting Traffic

Now that we have obtained our two record set identifers and their weights, we can begin to transfer traffic. It is unfortunately necessary to send all DNS record related data in the request, so it is not possible to omit things:

$ aws route53 change-resource-record-sets --hosted-zone-id 147MAIJCWVL9PC --change-batch '
{
  "Comment": "{ blue: 90, green: 10 }",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "www.domain.com.",
        "Type": "CNAME",
        "TTL": 60,
        "Weight": 90,
        "SetIdentifier": "blue",
        "ResourceRecords": [{ "Value": "blue.domain.com" }]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "www.domain.com.",
        "Type": "CNAME",
        "TTL": 60,
        "Weight": 10
        "SetIdentifier": "green",
        "ResourceRecords": [{ "Value": "green.domain.com" }]
      }
    }
  ]
}'

In this change set that we have requested, we have shifted to 90% traffic to blue, 10% traffic to green. The output from this command is:

{
  "ChangeInfo": {
    "Id": "/change/C1F72YWO0VLCD8",
    "Status": "PENDING",
    "Comment": "{ blue: 90, green: 10 }",
    "SubmittedAt": "2017-02-01T22:35:52.221Z"
  }
}

We can use this value to then wait for the change to fully propagate:

$ aws route53 wait resource-record-sets-changed --id "/change/C1F72YWO0VLCD8"

Once this has completed, the update will have been propagated to all Amazon nameservers. This process can be looped in order to complete the deployment, shifting traffic in batches until completed.

It is important in addition to waiting for nameserver synchronization that DNS TTLs be observed; if your TTL is 60 seconds, you must wait at least a minute after synchronization has been achieved across nameservers so that clients will respect the changes to DNS, if the clients behave , which with DNS isn’t always guaranteed.