PSA: Don't Break Public APIs

The date is December 1st, 2016. Amazon announces an interesting new feature for CloudFront allowing running Lambda functions at CloudFront edge locations. This is a powerful addition to the AWS arsenal for running code in locations geographically closest to users. This feature is a “preview” feature, and it’s opt-in only.

What was not mentioned, however is this change to the CloudFront API. Namely, Amazon added a field to DefaultCacheBehavior objects in CloudFront Distributions which is documented as being not required, but is nevertheless required, resulting in the following error message if UpdateDistribution is called:

InvalidArgument: The parameter Lambda function associations is required.

Their documentation states:

LambdaFunctionAssociations
A complex type that contains zero or more Lambda function associations for a cache behavior.

Type: LambdaFunctionAssociations

Required: No

Emphasis on “no” is mine.

Of course, the reality is that this parameter is required, and not passing that XML element breaks all API calls, as seen and documented in this Terraform bug report. A simple hack works around the issue by always creating an empty <LambdaFunctionAssociations> block for every request:

diff --git a/builtin/providers/aws/cloudfront_distribution_configuration_structure.go b/builtin/providers/aws/cloudfront_distribution_configuration_structure.go
index b891bd26b..1eff7689f 100644
--- a/builtin/providers/aws/cloudfront_distribution_configuration_structure.go
+++ b/builtin/providers/aws/cloudfront_distribution_configuration_structure.go
@@ -261,6 +261,9 @@ func expandCacheBehavior(m map[string]interface{}) *cloudfront.CacheBehavior {
                MinTTL:               aws.Int64(int64(m["min_ttl"].(int))),
                MaxTTL:               aws.Int64(int64(m["max_ttl"].(int))),
                DefaultTTL:           aws.Int64(int64(m["default_ttl"].(int))),
+               LambdaFunctionAssociations: &cloudfront.LambdaFunctionAssociations{
+                       Quantity: aws.Int64(0),
+               },
        }
        if v, ok := m["trusted_signers"]; ok {
                cb.TrustedSigners = expandTrustedSigners(v.([]interface{}))

A full fix is forthcoming from the Terraform community thankfully, but this isn’t Terraform’s problem. Amazon broke the interface to this API without warning and in contrast to their documentation which says that the field isn’t required. This change would have broken their CLI if not for a fix in botocore, and even appears to have broken some of their web interface for origins for a distribution; configuring an S3 origin is broken and doesn’t appear to work for adding origin access identities for S3 origins.

All of this added up to finding myself in a predicament: I had tampered with my origin configuration on CloudFront and I had no way of returning to a sane state. I couldn’t use Terraform to revert, the CLI was very hard to work with, and I couldn’t use the web interface to revert.

I was able to ultimately get around the issue by manually compiling Terraform myself after patching the source code. After recompiling, I was able to apply changes again and get my distribution working. Again, Terraform is not at fault here, it’s entirely Amazon’s fault for breaking a public API.

Lessons Learned

API breakage, whether we like it or not, happens.

However, the fact that Amazon could release software that would break things like this and release documentation contrary to the actual functionality, all without some testing alarms going off, engenders a serious violation of trust for me as a user of Amazon Web Services.

It should go without saying to developers of REST APIs that if you introduce backwards incompatible changes, you must bump the API version in the URL.

Amazon, please update your documentation or please make LambdaFunctionAssociations a truly optional field in DefaultCacheBehavior. In the meantime, everyone should scramble and try to work around this API breakage.