How to manage custom CloudFormation resources with Lambda

Background

CloudFormation is a powerful tool, and has broad support to other AWS services. Essentially, it is the main part of the infrastructure-as-code concept within the AWS environment.

It is great and supports many services out of the box. But what if you need something that it has no support?

This is when custom resources come in handy. They let you define a Lambda function that handles the lifecycle of the resource, be it any type, even outside AWS.

Let's see how to make it work and a minimal config that you'll need to get started with custom resources!

The resource

To define a custom resource, use the prefix Custom:: and give it a name. It can have parameters and outputs, just like any other regular resource:

Resources:
  Custom:
    Type: Custom::CustomResource
    Properties:
      Param1: val1
      Param2: val2

Until now, it looks like a regular resource with a logical name (resource name) of Custom and a type of Custom::CustomResource.

Let's see what you'll need to make a Lambda function manage its lifecycle!

The Lambda function

CloudFormation

On the CloudFormation side, you'll need a Lambda function and an execution role for it. This part is the same as any other Lambda function.

The first part, the execution role defines the trust policy that allows the Lambda service to assume it, as well as a set of policies the function has access to:

Resources:
  CustomResourceLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
      - PolicyName: allow-logs
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - 'logs:*'
            Resource: arn:aws:logs:*:*:*

This part is standard for all kinds of functions.

Then we need the Lambda resource itself, defining the Runtime, the Role, the Code, and the Handler:

Resources:
  CustomResourceLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: handler.index
      Role: !GetAtt CustomResourceLambdaExecutionRole.Arn
      Code: function/
      Runtime: nodejs8.10

As the code is defined as a local path, you'll need to run aws cloudformation package ... to zip and upload the code to S3.

Until this point, this is exactly how you'd define any Lambda function.

To connect the function to the custom resource, pass its Arn as the ServiceToken parameter:

Resources:
  Custom:
    Type: Custom::CustomResource
    Properties:
      ServiceToken: !GetAtt CustomResourceLambdaFunction.Arn
        ...

This is the part that wires the resource to the function.

The Lambda code

Now that we have everything set up on the CloudFormation side, let's move on to the actual code.

An important concept is that the resource handling is asynchronous. The function gets a responseURL and it needs to send a PUT with some required parameters. This signals CloudFormation that the resource is ready (or there is a problem).

At the bare minimum, you'll need to send the request with these parameters:

• Status: "SUCCESS" or "FAILED"
• PhysicalResourceId: This identifies the underlying resource
• Data: The Output of the function
• and you need to pass back some parameters: StackId, RequestId, and LogicalResourceId

For the full code on how to send the request and provide all the parameters, see here. It is based on the AWS sample code.

Handling lifecycle

Now that everything required is in place, let's see how to handle each lifecycle transition!

Create

When the resource is created, for example the first time you deploy the template, it gets a Create event. In this case, the event looks like this:

{
	"RequestType": "Create",
	"ServiceToken": "...",
	"ResponseURL": "...",
	"StackId": "...",
	"RequestId": "...",
	"LogicalResourceId": "Custom",
	"ResourceType": "Custom::CustomResource",
	"ResourceProperties": {
		"ServiceToken": "...",
		"Param2": "val2",
		"Param1": "val1"
	}
}

The "RequestType": "Create" is what makes it a Create event.

Also, take a look at the ResourceProperties section. It contains the Properties from the CloudFormation template. The !Ref-s and other functions are resolved by the time the Lambda is called, so if you referenced other resources in the template, you can use them here. As an example, this is how you'd reference an S3 bucket.

Delete

When you remove the resource from the template, the lifecycle event is a "Delete". The event is almost the same as the "Create", but the RequestType is "Delete". Use this to remove any underlying resource that is associated with this custom resource.

Update

When there is a change in the template for the custom resource, you'll get an "Update". In this step, you'll not only get the current Properties, but also the previous ones, so you know what is changed.

For example, by changing Param2 from val2 to val3:

"ResourceProperties": {
	"ServiceToken": "...",
	"Param2": "val3",
	"Param1": "val1"
},
"OldResourceProperties": {
	"ServiceToken": "...",
	"Param2": "val2",
	"Param1": "val1"
}

Use the OldResourceProperties to know what is modified.

In some cases, you can simply delete the old resource and handle the event as a Create. This is usually the case for stateless resources that can not be changed.

Handling outputs

If you send some values with the response in the Data object, those will be available in the template.

For example, if you want to send a parameter named Out, use Data: {Out: ...}. Then you can reference it in the template with the !GetAtt Custom.Out function:

Outputs:
  File:
    Value: !GetAtt Custom.Out

PhysicalResourceId

This is the underlying identifier of the resource. It is important to understand how it works as it can cause a lot of unexpected behavior.

As a rule of thumb, it should be different for different underlying resources. For example, if you have a GitHub repository, it should be the name. If you manage an S3 object, it should be the full path. For both cases, the id uniquely represents the resource and makes sure that different ones make different ids.

You also need to send it back when you send back the response.

Its lifecycle is the following:

• For "Create", you return a PhysicalResourceId.
• For "Update", you have a choice:
  • If you send the same, nothing else happens
  • If you send a different one, there will be a "Delete" event for the old resource

Let's see an example!

Our custom resource has 2 parameters:

Resources:
  Custom:
    Type: Custom::CustomResource
    Properties:
      Param1: val1
      Param2: val2

Let's calculate the PhysicalResourceId from only Param1:

PhysicalResourceId: `${Param1}`,

In this case, changing Param2 updates the same resource, but changing Param1 creates a new one (from CloudFormation's perspective). This is roughly the case when you have an S3 Object and Param1 is the key, while Param2 is the contents.

By chaning Param2 to val3, there will an "Update":

"ResourceProperties": {
	"ServiceToken": "...",
	"Param2": "val3",
	"Param1": "val1"
},
"OldResourceProperties": {
	"ServiceToken": "...",
	"Param2": "val2",
	"Param1": "val1"
}

But setting Param1: val4, after this "Update":

"ResourceProperties": {
	"ServiceToken": "...",
	"Param2": "val3",
	"Param1": "val4"
},
"OldResourceProperties": {
	"ServiceToken": "...",
	"Param2": "val3",
	"Param1": "val1"
}

There will also be a "Delete" with the old parameters:

"ResourceProperties": {
	"ServiceToken": "...",
	"Param2": "val3",
	"Param1": "val1"
},

Keep this in mind when you decide on a PhysicalResourceId.

What should you use then?

It should be different for a different resource, and same for the same resource. If you manage an S3 Object, it can be the bucket and the key of the object. If the resource is a GitHub gist, use the id returned from GitHub.

Conclusion

Custom resources are powerful additions to CloudFormation. They open support for all sorts of resources, and not just what it supports out of the box.

Handled with care, you can define any resource type, even third-party ones. They require some boilerplate, especially considering the Lambda function, and should be tested extensively.

But using them gives you the freedom to define anything. Want to create a Mailchimp campaign from CloudFormation? Or a GitHub repository? A Slack channel? These are all possible, and they can be made first-class citizens.