How to manage S3 Objects in CloudFormation templates

Background

You can create and manage the full lifecycle of an S3 bucket within a CloudFormation template. But there is no resource type that can create an object in it.

How to write one?

We'll build a solution upon Custom Resources, which can add support for arbitrary resources using a Lambda function as a handler for the lifecycle. As the baseline, the solution is built on the basics detailed in the previous article. Check that out for an introduction on how to write custom resources.

Related

How to manage custom CloudFormation resources with Lambda

CloudFormation supports a lot of resources out of the box. But sometimes they are not enough.

Let's see how a simple implementation would look like!

Book

S3 and CloudFront Signed URLs

How to handle files in an AWS serverless environment

Template

Obviously, we'll need a bucket to store the objects:

Resources:
  Bucket:
    Type: AWS::S3::Bucket

And as we'll use the Lambda function to write into this bucket, we need to amend the execution role:

Policies:
...
- PolicyName: s3-bucket
  PolicyDocument:
    Version: '2012-10-17'
    Statement:
    - Effect: Allow
      Action:
      - 's3:PutObject'
      - 's3:GetObject'
      - 's3:DeleteObject'
      Resource:
      - !Sub '${Bucket.Arn}/*'

Then a custom resource with a custom type, something like this:

Resources:
  S3File:
    Type: Custom::S3File
    Properties:
      ServiceToken: ...

As for Properties, we need the Bucket, the Content, and the parts of the Key. It's best practice not to define the key in its entirety but leave space for some random part. Let's define the key as a prefix and a suffix, which yield the full key using this scheme: {KeyPrefix}-{Random}{KeySuffix}.

Resources:
  S3File:
    Type: Custom::S3File
    Properties:
      ServiceToken: ...
      Bucket: !Ref Bucket
      KeyPrefix: "examplekey"
      KeySuffix: ".txt"
      Content: "Hello world"

Now that we have everything needed on the template side, let's move on to the actual implementation!

Lambda code

First, we need the AWS SDK and the S3 object to interface with S3:

const AWS = require("aws-sdk");
const s3 = new AWS.S3();

At this point, there is no need to supply any credentials, as it will be used from the Lambda execution role.

Constructing the Key

Since we only have the prefix and a suffix of the key, we need to construct it. Between these two, let's have a pseudo-random part. For that, a hash function is a good candidate: concatenate the parameters, take the hash of the resulting string and get only the first few characters.

What should be part of the hash?

The StackId should be included, so that different stacks have different keys. Failing to do so will make copying and deploying the same template problematic.

Then the LogicalResourceId should be also included, so that if two resources in a single template have the same properties they won't clash.

This would be a suitable implementation for calculating the random part:

const randomPart = require("crypto").createHash("sha1").update([StackId, LogicalResourceId].join(";")).digest("hex").substring(0, 7);

Why not use a truly random string?

The problem with a truly random as opposed to a pseudo-random part is that it makes the function non-idempotent. If there is a problem, CloudFormation may call this function several times, and if it creates different resources you might end up orphaned ones that will be left when you delete the stack. In most cases, hashes are a safer solution.

Why clashing names are a problem?

When you have multiple CloudFormation resources that map to the same underlying resource, deleting one of them will delete the resource for all of them. This is a situation that is very hard to recover from. Also, if you rename a resource in the template, CloudFormation will issue a delete, easily resulting in the above situation.

Construct the Key

Now that we have everything for the key, let's construct it:

return `${KeyPrefix}-${randomPart}${KeySuffix}`;

Handle the lifecycles

Now that we have everything in place, let's see how to handle the 3 lifecycle events: Create, Update, and Delete.

Since there is no way to rename an object, Create and Update works the same:

await s3.putObject({
	Body: Buffer.from(Content, 'binary'),
	Bucket,
	Key
}).promise();

Delete, on the other hand, issues a deleteObject:

await s3.deleteObject({
	Bucket,
	Key
}).promise();

PhysicalResourceId

The PhysicalResourceId is what identifies the object itself. In S3, the bucket and the key must be unique, so this is a good candidate for the id:

{
	...
	PhysicalResourceId: `${Bucket}/${Key},
	...
}

Outputs

Since the key is calculated, it should be added as a return value. To achieve this, add a Data: {Key} to the response body.

Then this can be referenced in the template, allowing other resources and stacks to dynamically link to the object:

Outputs:
  File:
    Value: !GetAtt S3File.Key

Conclusion

In this post, you've learned how to use custom CloudFormation resources to add support for S3 objects. While this is a simplified implementation that does not support all aspects of S3, it is a robust implementation that can be a baseline that you can adapt to your specific use-case.