AWS Config notifications with CloudWatch Events

Background

Recently I got a question about how to set up a mechanism to automatically respond to a resource being marked as non-compliant by a Config Rule. AWS recently rolled out a feature for that, but it's so new at the time of writing even Terraform had no support for it.

But this is not a new functionality. You could get notifications when a resource becomes non-compliant without this new remediation feature, you just had to use a different service, namely CloudWatch Events. With that, you can listen to changes and forward the notification to all sorts of other services such as Lambda, SNS, SQS. And with them, you can implement auto-remediation.

Compliance status change notifications

So, you have a Config Rule set up and want to know when it reports something is non-compliant. How to do that?

CloudWatch Events publishes an event when a change occurs (COMPLIANT <=> NON_COMPLIANT) with all sorts of details. You just need to specify an Event Rule and filter for the Config Rules Compliance Change event and the Config Rule Arn:

{
  "source": [
    "aws.config"
  ],
  "detail-type": [
    "Config Rules Compliance Change"
  ],
  "detail": {
    "configRuleARN": [
      "<config rule>"
    ]
  }
}

The whole event object looks like this:

{
  "version": "0",
  "id": "...",
  "detail-type": "Config Rules Compliance Change",
  "source": "aws.config",
  "account": "...",
  "time": "2019-08-13T13:17:14Z",
  "region": "eu-central-1",
  "resources": [],
  "detail": {
    "resourceId": "terraform-20190813125905278500000001",
    "awsRegion": "eu-central-1",
    "awsAccountId": "...",
    "configRuleName": "config-rule-f6815e21a2b9a86e",
    "recordVersion": "1.0",
    "configRuleARN": "...",
    "messageType": "ComplianceChangeNotification",
    "newEvaluationResult": {
      "evaluationResultIdentifier": {
        "evaluationResultQualifier": {
          "configRuleName": "config-rule-f6815e21a2b9a86e",
          "resourceType": "AWS::S3::Bucket",
          "resourceId": "terraform-20190813125905278500000001"
        },
        "orderingTimestamp": "2019-08-13T13:16:58.303Z"
      },
      "complianceType": "COMPLIANT",
      "resultRecordedTime": "2019-08-13T13:17:13.829Z",
      "configRuleInvokedTime": "2019-08-13T13:17:13.653Z"
    },
    "oldEvaluationResult": {
      "evaluationResultIdentifier": {
        "evaluationResultQualifier": {
          "configRuleName": "config-rule-f6815e21a2b9a86e",
          "resourceType": "AWS::S3::Bucket",
          "resourceId": "terraform-20190813125905278500000001"
        },
        "orderingTimestamp": "2019-08-13T13:04:23.345Z"
      },
      "complianceType": "NON_COMPLIANT",
      "resultRecordedTime": "2019-08-13T13:04:38.860Z",
      "configRuleInvokedTime": "2019-08-13T13:04:38.689Z"
    },
    "notificationCreationTime": "2019-08-13T13:17:14.527Z",
    "resourceType": "AWS::S3::Bucket"
  }
}

The important parts are detail.resourceId and detail.newEvaluationResult.complianceType. The former is the resource, in this case a bucket, the latter is COMPLIANT / NON_COMPLIANT, indicating the result of the validation.

The event pattern above matches both the resource becoming compliant and non-compliant. To handle only one direction you can add a filter to the detail.newEvaluationResult.complianceType in the pattern.

Forward the event

With CloudWatch events you can set up all kinds of notifications CloudWatch supports. For auto-remediation, you might want to add a Lambda function that does something to rectify the situation. You can set up SNS notifications so that you get an email or a Slack notification.

Test setup

Let's see how it works in action!

For a test setup, I have a simple Config Rule: it checks if a given bucket has SSE (encryption) enabled. This is an AWS-provided rule, so I just need to reference it.

Then I'll use an SQS queue as the event target and a CLI script to see the events as they come.

Infrastructure

This part sets up a bucket and a Config Rule scoped to the bucket only. To reproduce this example you'll need Config enabled for the region for config rules to work.

resource "aws_s3_bucket" "bucket" {
	force_destroy = "true"
}

resource "aws_config_config_rule" "rule" {
	name = "config-rule-${random_id.id.hex}"

	source {
		owner             = "AWS"
		source_identifier = "S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED"
	}

	scope {
		compliance_resource_id = aws_s3_bucket.bucket.id
		compliance_resource_types = ["AWS::S3::Bucket"]
	}
}

The scope makes sure that only the specified bucket is checked so that I have a controlled test environment.

CloudWatch Event Rule

The next thing is to set up the Event Rule. The configRuleARN filter makes sure that only the above rule will trigger a notification.

resource "aws_cloudwatch_event_rule" "compliance_change" {

  event_pattern = <<PATTERN
{
	"source": [
		"aws.config"
	],
	"detail-type": [
		"Config Rules Compliance Change"
	],
	"detail": {
		"configRuleARN": [
			"${aws_config_config_rule.rule.arn}"
		]
	}
}
PATTERN
}

Event target

As a target for the events there is an SQS queue (see the linked source for the full example) and a queue policy applied so that CloudWatch can send messages. This part wires them together:

resource "aws_cloudwatch_event_target" "sqs" {
	rule = "${aws_cloudwatch_event_rule.compliance_change.name}"
	target_id = "SQS"
	arn = "${aws_sqs_queue.queue.arn}"
	sqs_target {
		message_group_id = "1"
	}
}

Outputs

Finally, the bucket and the queue are exported to use them in scripts:

output "bucket" {
	value = aws_s3_bucket.bucket.id
}

output "queue" {
	value = aws_sqs_queue.queue.id
}

Testing

Listen for messages

To monitor the SQS queue for new messages I use the following script:

while sleep 1; do \
	(MSG=$(aws sqs receive-message --queue-url $(terraform output queue)); \
		[ ! -z "$MSG" ] && echo "$MSG" | jq -r '.Messages[] | .ReceiptHandle' \
		| (xargs -I {} aws sqs delete-message --queue-url $(terraform output queue) --receipt-handle {}) \
		&& echo "$MSG") \
	| jq -r '.Messages[] | .Body | fromjson |
		"\(.time): \(.detail.resourceId) => \(.detail.newEvaluationResult.complianceType)"' \
; done

This continually polls the queue and outputs the resource and the compliance status to the console.

Trigger the rule

The bucket is NON_COMPLIANT by default (no default encryption set). To change this, I need to set the bucket encryption. And to make it non-compliant again, delete it. The following script toggles the compliance status by querying the current encryption and sets or deletes it:

aws s3api get-bucket-encryption --bucket $(terraform output bucket) \
	&& aws s3api delete-bucket-encryption --bucket $(terraform output bucket) \
	|| aws s3api put-bucket-encryption \
		--bucket $(terraform output bucket) \
		--server-side-encryption-configuration \
			'{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'

Results

Even though the Config Rule is event-driven, it still takes a few minutes for the message to appear. But then I can observe when the bucket is becoming compliant and non-compliant:

2019-08-13T13:01:40Z: terraform-20190813125905278500000001 => COMPLIANT
2019-08-13T13:04:39Z: terraform-20190813125905278500000001 => NON_COMPLIANT

Conclusion

The above example uses an SQS queue to print compliance changes of a resource to the console. Instead of just logging, you can define a Lambda function to automatically fix the problem. This opens the way for writing security rules that are guaranteed to be enforced.