AWS: How to get notified on compromised credentials
Get an email when someone tries to use your keys
Motivation
When you create Access Keys, you are basically blind how they are used. You can see the effects, like new instances starting or things disappearing, but to see how the key is used you need a separate service. CloudTrail records all calls to the AWS APIs and logs some metadata, such as the caller, the result, and a few others. Using this information you can get an insight into how your keys are used, and whether they have landed in the wrong hands.
One of the most important things is to know when something is denied. During normal operations, there should be no denials as everything is tested and can work in one way only. But when an attacker gets access to a key he usually tries some innocuous calls to see if it works. If you followed the least privilege principle, this call will most likely be denied. And this is the chance to detect the compromise. Without CloudTrail you won't get any indication that something is wrong. But with some preparations, you could get a notification and then you can act on it.
Metric Filters
The solution described in this post follows the principles on how to detect IAM user logins. CloudTrail entries are delivered into CloudWatch Logs and you can use a Metric Filter to produce a numerical value how many events match a given filter. This, in turn, is pushed into CloudWatch, where an Alarm can send a notification when it crosses a threshold, typically when it is greater than 0.
Alternative approaches
An alternative approach uses CloudWatch Events to collect and forward API calls to a Lambda which can act on the suspicious ones. You can also use CloudWatch Events directly to filter on the access denied events and publish a notification without a Lambda.
These are both valid approaches that correct some of the shortcomings of the Metric Filters.
Why did I choose the Metric Filters then?
There are two fundamental problems with CloudWatch Events.
First, there is no easy way to rate-limit it. If you do some heavy processing when an Access Key is used in a way it shouldn't have been used, then an attacker can flood your account with such attempts and, due to the cost, you can not really do anything than disable it.
The second issue is that CloudWatch events are region-specific, while CloudWatch Logs can collect CloudTrail events from all regions. An attacker is likely to use the most recently added region hoping you don't cover all of them. A region-agnostic approach is a more future-proof way of detection then.
Setting up the notifications
First, you need CloudTrail enabled and a CloudWatch Logs Group set up as the target. Second, you need a Metric Filter that counts the events where
the errorCode is AccessDenied. For the Filter Pattern, use:
{$.errorCode = "AccessDenied"}
This pushes a metric with a chosen name and namespace. In effect, when there is an attempt to use a key that is denied, the metric will be 1. To get a notification on this occasion, a traditional CloudWatch Alarm can be used. Simply set up to treat missing data as OK, and alert when the metric is greater than 0.
Using the CloudFormation template
Of course, there is a ready-to-deploy CloudFormation template that does all the heavy lifting of setting up the resources for you, You need to have CloudTrail set up to deliver the logs into CloudWatch Logs, and have an SNS topic that sends you an email.
Then deploy the template, provide the required parameters, wait a bit and you are all set up.
Limitations
There are two problems with this approach. First and foremost, you'll only get a notification after a significant amount of time as CloudTrail delays logging by 15-20 minutes (!). In some cases, this can be a problem, but if you set up permissions properly and you require MFA for the Access Keys, then you should be fine.
The other problem is that you'll only get information about the existence of a denied attempt but without any details. For the full CloudTrail event you need to go to the CloudWatch Logs console and filter the logs.
Fortunately, the URL for this seems to be fairly constant and thus the CloudFormation template outputs it. When you need to see the events just click on the link, or bookmark it in advance.
Conclusion
Detection is a cornerstone of a secure environment and access key leakage is one of the worst security incidents. This template provides a deploy-and-forget solution that can alert you when there is a problem with your keys.
