How hard it is to disable an IAM user when it does something suspicious?

I wanted to automatically disable an IAM user when it does something suspicious. Since this IAM user is used by a script I know that when it deviates that is a good indicator that it was compromised and I need to investigate.

How hard could it be?

Well, it turned out to be a frustrating experience. CloudTrail records events done by users, so this should be easy to setup. But then I started to encounter problems:

Only the first CloudTrail event is free, so I did not want to create more than one trail
CloudTrail sends events to EventBridge but only for the current region, which is not enough

I have an organizational trail in the management account. Let's see how easy it is to send these events into a member account!

CloudTrail can send events to a CloudWatch log then I could set up a subscription filter. This worked for a PoC but ultimately there is a limit of 2 for subscription filters for a log group. So this was a no-go
Otherwise it writes to S3, so I had to have a Lambda reading the objects as CloudTrail writes them

At this point I had a Lambda that got all CloudTrail events and filters out the interesting ones: ones with AccessDenied error, GetCallerIdentity, and ConsoleLogin. That should be a good start.

EventBus Rules can send events based on a filter, so forwarding these events into an EventBus seems like a good idea. So so far the chain is: CloudTrail => S3 => Lambda => EventBus.

But how can I send these events to the member account? Well, an EventBus Rule, of course. So I created an EventBus in the member account.

Next issue: a CloudFormation stack can't create an EventBus Rule in a different region. Interestingly, it is possible to create cross-account but not cross-region. So I needed an EventBus in the target region as well and set up a Rule to forward events there.

Then the very last step is to set up a Rule to filter events for the IAM user(s) and set up a CloudWatch alarm that calls a Lambda that attaches the DenyAll policy to the user.

Since I wanted everything managed by CloudFormation I ended up with an enormous amount of stacks:

(mgmt acc us-east-1) CloudTrail + Lambda + EventBus
(mgmt acc us-east-1) EventBus Rule to forward events to the member account
(member acc us-east-1) EventBus to receive events from the mgmt account
(member acc us-east-1) EventBus Rule to forward events to the regional EventBus
(member acc eu-west-1) The target stack with the IAM users and an EventBus to receive events to

What makes it a particularly annoying experience is that there are so many small limitations that make a simpler solution impossible:

CloudTrail should support filtering by events so that the whole management account => member account part could be saved
Or: the default EventBus should receive all CloudTrail events not just ones for the current region
EventBus Rule should be allowed to be cross-region. That would have saved me one EventBus
EventBridge Pipes don't support SNS as a source and also it's not clear if that supports cross-region and cross-account pipes

I wrote about my frustrations in this article: https://advancedweb.hu/cloudtrails-horrible-developer-experience/.

Overall, I'm fairly happy with this solution, but I feel that it would be so much easier if AWS supported some basic features around CloudTrail.