CloudFormation CLI workflows

How to use CloudFormation from the terminal

11 mins
I have a lot of challenges when it comes to AWS, but I bet your pain points are entirely different than mine. I'd love to hear what keeps you up at night. It would be great to hear from you by filling out this form. Thanks in advance!

CloudFormation from the terminal

Despite my ambivalent feeling about CloudFormation I use it a lot, but managing stacks through the Console is a pain. Fortunately, this service enjoys the same CLI support most other ones do, so it is just a matter of scripting to make it more developer-friendly.

There are several projects that provide a better experience from the terminal, but for simple needs I prefer simple tools. Here you can find a couple of what I use regularly for various common tasks.

Almost all the scripts below use jq, which you can usually install with a simple apt install -y jq, along with the AWS CLI and some basic tools available on most distributions.

Note: In all the examples I use $STACK_NAME variable as the current stack name. You can set it by defining it as follows:

export STACK_NAME=stack1

Regions

Since CloudFormation is region-specific, each region provides an isolated environment. To change the region, either specify it for the aws command:

aws --region us-west-1 cloudformation ...

or define the AWS_DEFAULT_REGION environment variable:

export AWS_DEFAULT_REGION="us-west-1"

Deploy

The most used use-case is to deploy a template file to a new stack or update an existing one. You can use the deploy command for these tasks:

aws cloudformation deploy \
	--template-file cloudformation.yml \
	--capabilities CAPABILITY_IAM \
	--stack-name $STACK_NAME

Here’s how it looks like:

It needs these parameters at the minimum:

  • template-file
  • stack-name

In this case, I defined cloudformation.yml as the template and $STACK_NAME as the name of the new stack.

Capabilities

You may notice the --capabilities argument of the above command. This is used to explicitly acknowledge that the stack creates IAM resources. There are 2 possible values: CAPABILITY_IAM and CAPABILITY_NAMED_IAM. The latter is required when you define a name for an IAM resource, for example, the RoleName:

LambdaExecutionRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: my_role

Parameters

When a stack requires parameters, you can supply them like this:

--parameter-overrides "PARAMETER1=value1" "PARAMETER2=value2"

Changesets

Under the hood, the deploy command does 2 things: it creates a changeset that describes what to do, then executes it. To skip the second one, use this flag:

--no-execute-changeset

Instead of a full deploy cycle, it stops when the changeset is created:

Waiting for changeset to be created..
Changeset created successfully. Run the following command to review changes:
aws cloudformation describe-change-set --change-set-name <arn>

After the changeset is ready, view it with the command:

aws cloudformation describe-change-set \
	--change-set-name <arn>

It gives back a lot of information, and usually, it’s enough to see what resources are changed. To filter for this, define a query:

aws cloudformation describe-change-set \
	--change-set-name <arn> \
	--query "Changes[].ResourceChange"

It gives back a smaller JSON describing how each resource will be affected when the changeset is executed:

[
    {
        "Action": "Remove",
        "LogicalResourceId": "LambdaApiGatewayInvoke",
        "PhysicalResourceId": "stack1-LambdaApiGatewayInvoke-1H6XSIBWPT9MB",
        "ResourceType": "AWS::Lambda::Permission",
        "Scope": [],
        "Details": []
    },
    {
        "Action": "Add",
        "LogicalResourceId": "LambdaApiGatewayInvoke2",
        "ResourceType": "AWS::Lambda::Permission",
        "Scope": [],
        "Details": []
    }
]

Review changes before execution

It is a good practice to review what will be changed before actually executing them, especially when updating a critical stack. Let’s construct such script!

It needs to perform 4 steps:

  • Create a new changeset (deploy --no-execute-changeset)
  • Get the changeset Arn (list-change-sets)
  • Describe the changeset (describe-change-set)
  • Cleanup (delete-change-set)

The 2 missing steps are (2) and (4).

To get the last changeset’s Arn for a given stack:

aws cloudformation list-change-sets \
	--stack-name $STACK_NAME | \
	jq -r '.Summaries |
		sort_by(.CreationTime) |
		.[-1].ChangeSetId
	'

And to delete a changeset:

aws cloudformation delete-change-set \
	--change-set-name $CHANGESET

With the above building blocks, this rather complex script can check what a template would change:

CHANGESET=$( \
		aws cloudformation deploy \
			--template-file cloudformation.yml \
			--capabilities CAPABILITY_IAM \
			--stack-name $STACK_NAME \
			--no-execute-changeset > /dev/null && \
		aws cloudformation list-change-sets \
			--stack-name $STACK_NAME | \
		jq -r '.Summaries |
			sort_by(.CreationTime) |
			.[-1].ChangeSetId') && \
aws cloudformation describe-change-set \
	--change-set-name $CHANGESET \
	--query "Changes[].ResourceChange" && \
aws cloudformation delete-change-set \
	--change-set-name $CHANGESET

It looks like this in action:

Events

To get the events related to a given stack, use this command:

aws cloudformation describe-stack-events \
	--stack-name $STACK_NAME

It gives back a large JSON, detailing what happened to each resource:

{
	"StackEvents": [
		"StackId": "arn:aws:cloudformation:us-west-1:757253985935:stack/stack1/22059b00-6a88-11e9-b3cd-50d5caff88fd",
		"EventId": "3b11a760-6a88-11e9-ba0c-02e6681d586c",
		"StackName": "stack1",
		"LogicalResourceId": "stack1",
		"PhysicalResourceId": "arn:aws:cloudformation:us-west-1:757253985935:stack/stack1/22059b00-6a88-11e9-b3cd-50d5caff88fd",
		"ResourceType": "AWS::CloudFormation::Stack",
		"Timestamp": "2019-04-29T14:08:15.025Z",
		"ResourceStatus": "CREATE_COMPLETE"
	},
	{
		"StackId": "arn:aws:cloudformation:us-west-1:757253985935:stack/stack1/22059b00-6a88-11e9-b3cd-50d5caff88fd",
		"EventId": "LambdaApiGatewayInvoke-CREATE_COMPLETE-2019-04-29T14:08:12.548Z",
		"StackName": "stack1",
		"LogicalResourceId": "LambdaApiGatewayInvoke",
		"PhysicalResourceId": "stack1-LambdaApiGatewayInvoke-1H6XSIBWPT9MB",
		"ResourceType": "AWS::Lambda::Permission",
		"Timestamp": "2019-04-29T14:08:12.548Z",
		"ResourceStatus": "CREATE_COMPLETE",
		"ResourceProperties": "{\"FunctionName\":\"arn:aws:lambda:us-west-1:757253985935:function:stack1-LambdaFunction-FLV1VGO5J08S\",\"Action\":\"lambda:InvokeFunction\",\"SourceArn\":\"arn:aws:execute-api:us-west-1:757253985935:v2510wqve1/*/*/*\",\"Principal\":\"apigateway.amazonaws.com\"}"
	},
...
}

This contains all the informations, but it is hard to get an overview what happened to what. To extract the valuable data and format it to a columnar format, use this snippet:

aws cloudformation describe-stack-events \
	--stack-name $STACK_NAME | \
jq -r '.StackEvents[] |
	"\(.Timestamp | sub("\\.[0-9]+Z$"; "Z") | fromdate | strftime("%H:%M:%S") ) \(.LogicalResourceId) \(.ResourceType) \(.ResourceStatus)"
' | column -t

This shows a table with less information but that is easier to get a glimpse of the events:

14:08:15  stack1                  AWS::CloudFormation::Stack   CREATE_COMPLETE
14:08:12  LambdaApiGatewayInvoke  AWS::Lambda::Permission      CREATE_COMPLETE
14:08:07  ApiGatewayDeployment    AWS::ApiGateway::Deployment  CREATE_COMPLETE
14:08:07  ApiGatewayDeployment    AWS::ApiGateway::Deployment  CREATE_IN_PROGRESS

Note: timezone is not supported by jq at the moment, so it shows the time in UTC.

Watch events

To see how things unfold near real-time, use watch:

watch "aws cloudformation \
	describe-stack-events \
		--stack-name $STACK_NAME | \
	jq -r '.StackEvents[] |
		\"\\(.Timestamp | sub(\"\\\\.[0-9]+Z$\"; \"Z\") | fromdate | strftime(\"%H:%M:%S\") ) \\(.LogicalResourceId) \\(.ResourceType) \\(.ResourceStatus)\"
	' | column -t"

Deploy and watch the events

When I deploy something using the AWS Console, I just press refresh every other second to see how the resources are appearing at my behest. The beauty of the CLI is that I don’t need to manually click.

To deploy a stack and get the events refreshed automatically, use:

(aws cloudformation deploy \
	--template-file cloudformation.yml \
	--capabilities CAPABILITY_IAM \
	--stack-name $STACK_NAME > /dev/null & \
) && watch "aws cloudformation describe-stack-events \
	--stack-name $STACK_NAME | \
		jq -r '.StackEvents[] |
			\"\\(.Timestamp | sub(\"\\\\.[0-9]+Z$\"; \"Z\") | fromdate | strftime(\"%H:%M:%S\") ) \\(.LogicalResourceId) \\(.ResourceType) \\(.ResourceStatus)\"
		' | column -t"

This is how it looks like in action:

Isn’t that awesome?

List

To get a list of the stacks in the current region, use:

aws cloudformation list-stacks | jq -r '.StackSummaries[] |
	select(.StackStatus != "DELETE_COMPLETE") |
	"\(.StackName) \(.LastUpdatedTime) \(.StackStatus)"
' | column -t

Deleted stacks are shown for some time so we need to filter them out.

stack2  2019-04-29T14:32:00.614Z  CREATE_COMPLETE
stack1  2019-04-29T14:07:39.159Z  CREATE_COMPLETE

As usual, this can be watched, which can be useful when you deploy/delete multiple stacks:

watch "aws cloudformation list-stacks | \
	jq -r '.StackSummaries[] |
		select(.StackStatus != \"DELETE_COMPLETE\") | \"\(.StackName) \(.LastUpdatedTime) \(.StackStatus)\"
	' | column -t"

Delete

Stack deletion is easy with this script:

aws cloudformation delete-stack --stack-name $STACK_NAME

Combined with watching, it looks like this in action:

List resources

To list the resources managed in a stack, use:

aws cloudformation list-stack-resources \
	--stack-name $STACK_NAME | \
	jq -r '.StackResourceSummaries[] |
		"\(.LogicalResourceId) \(.ResourceType) \(.ResourceStatus)"
	' | column -t

This prints in a columnar format, extracted from the JSON the AWS CLI returns:

ApiGateway              AWS::ApiGateway::RestApi     CREATE_COMPLETE
ApiGatewayDeployment    AWS::ApiGateway::Deployment  CREATE_COMPLETE
ApiGatewayMethod        AWS::ApiGateway::Method      CREATE_COMPLETE
LambdaApiGatewayInvoke  AWS::Lambda::Permission      CREATE_COMPLETE
LambdaExecutionRole     AWS::IAM::Role               CREATE_COMPLETE
LambdaFunction          AWS::Lambda::Function        CREATE_COMPLETE
ProxyResource           AWS::ApiGateway::Resource    CREATE_COMPLETE

As usual, it can be watched:

watch "aws cloudformation list-stack-resources \
	--stack-name $STACK_NAME | \
	jq -r '.StackResourceSummaries[] |
		\"\(.LogicalResourceId) \(.ResourceType) \(.ResourceStatus)\"
	' | column -t"

Deploy and watch resources

To watch the resources during a deploy, you can combine the above script with the deployment:

(aws cloudformation deploy \
	--template-file cloudformation.yml \
	--capabilities CAPABILITY_IAM \
	--stack-name $STACK_NAME > /dev/null & \
) && watch "aws cloudformation list-stack-resources \
	--stack-name $STACK_NAME | \
	jq -r '.StackResourceSummaries[] |
		\"\(.LogicalResourceId) \(.ResourceType) \(.ResourceStatus)\"
	' | column -t"

With this you can see how the deployment unfolds:

Outputs

When you want to use the outputs of a stack, the AWS CLI returns them as an array of objects. To extract an output based on its name, use jq’s select:

aws cloudformation describe-stacks \
	--stack-name $STACK_NAME | \
	jq -r '.Stacks[0].Outputs[] |
		select(.OutputKey == "URL") |
		.OutputValue
	'

This returns the value of the parameter:

07 May 2019