Encryption options for S3 objects

Encryption in the cloud is a coin with two faces. On one side it's all about algorithms, compliance requirements, dedicated encryption hardware, and third-party audits. For example, the AWS documentation says that enabling S3 encryption does this:

Server-side encryption protects data at rest. Amazon S3 encrypts each object with a unique key. As an additional safeguard, it encrypts the key itself with a master key that it rotates regularly. Amazon S3 server-side encryption uses one of the strongest block ciphers available to encrypt your data, 256-bit Advanced Encryption Standard (AES-256).

When you use this feature, you are protected by the best practices and the best available algorithms.

The other face of the cloud encryption coin is what is observable. If I enable this encryption, what change can I see? Is there a request that is now denied? Or do I get the encrypted data for an operation instead of the plaintext?

When I think about the cloud I always imagine a big box with a lot of levers and lights. I can read the documentation to know how it works on the inside but I'm also interested in what changes if I push a particular button? In other words, if AWS decides one day that it no longer encrypts the objects, would I notice?

This article is about what are the observable benefits of encrypting S3 objects in different ways AWS supports.

No encryption

Let's establish a baseline first! If you put an object into an S3 bucket, it is private. If an IAM entity (user, role, service, anonymous user) has no permission to read it, the S3 service denies access:

aws s3api get-object --bucket <bucket> --key <key> /dev/stdout

Access Denied

The get-object operation goes to the S3 API which is protected by IAM. This means whether a request is allowed or not is determined by IAM policies.

The easiest way to give permission to read the object is to attach a policy to the IAM user or role:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": "s3:GetObject",
			"Resource": "<bucket>/*"
		}
	]
}

With this attached, the get-object returns the object:

aws s3api get-object --bucket <bucket> --key <key> /dev/stdout

This gives 2 outcomes. First, the user has no permission to access the object in which case the operation is denied. And second, the user has the necessary permission, in which case they can read the contents.

SSE-S3

Let's turn on server-side encryption and see what changed! The first thing we see is that the Console shows that the object is indeed encrypted:

This is also called "seamless encryption" as it's a push-button solution. There are no keys to manage and no change in how the object is stored or retrieved. Everything is managed inside the service.

The same command gets the object:

aws s3api get-object --bucket <bucket> --key <key> /dev/stdout

And whether the contents or an Access Denied is returned is determined by the IAM policies.

There is no observable difference between SSE-S3 and no encryption at all.

SSE-KMS

The SSE-KMS encryption scheme moves encryption out from the S3 service into KMS, AWS's dedicated encryption service. This is again transparent, as they communicate behind the scene. When you put an object with this encryption, the S3 service asks KMS to generate a key, then it encrypts the object with it, also storing the encrypted key next to the data. When you retrieve this object, S3 asks KMS to decrypt the key, then uses that to encrypt the object itself. This process is called "envelope encryption" and it reduces the number of calls to the dedicated hardware (KMS).

There are two types of KMS encryptions: one that uses an AWS managed key (the key is called CMK in KMS's terminology), and one that uses a customer managed key.

SSE-KMS with an AWS managed CMK

This key is created for you when you start using SSE-KMS and you can not manage it (hence the name "AWS managed"). On the S3 Console, the object is reported as encrypted with an AWS-KMS master key:

Apart from this text, the object works like before. The command to retrieve it is the same:

aws s3api get-object --bucket <bucket> --key <key> /dev/stdout

And also there is no change in the outcomes. When the user has access via IAM policies, the object is returned, otherwise it's an Access Denied error.

There is an observable difference though but in an entirely different part of AWS. Using KMS incurs costs for every encryption and decryption and that shows up in the monthly bill.

SSE-KMS with a customer managed CMK

KMS allows creating and managing your own keys. This allows adding more keys apart from the AWS managed ones and you can use them to encrypt S3 objects.

On the KMS Console, you see the keys you manage and you can add more here:

Then when you upload the object you can specify this key to use for encryption. On the S3 Console, you'll see that object is encrypted with this key:

The command to get the object is still not changed:

aws s3api get-object --bucket <bucket> --key <key> /dev/stdout

But now there is an observable difference in access control. If you try the above command with the user that has the s3:GetObject permission it returns an "Access Denied" error. This is because a user needs access to the KMS key as well as to the object. This is another IAM policy.

To give access, you can add kms:Decrypt permission to the user. This policy also allows encryption rights so that the user can also upload objects encrypted with this key.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"kms:ReEncrypt",
				"kms:GenerateDataKey",
				"kms:Encrypt",
				"kms:DescribeKey",
				"kms:Decrypt"
			],
			"Resource": "arn:aws:kms:eu-central-1:<accountid>:key/8308b9ef-5487-48d1-bcc4-1d1bceac02bb"
		}
	]
}

In effect, the user now needs access to the object and the key to get the object.

s3:GetObject	kms:Decrypt	Result
-	-	-
✓	-	-
-	✓	-
✓	✓	✓

Other than the requirement of more permissions, the result is still either the plaintext contents of the object or an "Access Denied" error. The observable difference is the requirement of the additional permissions.

When is it useful? This type of KMS encryption adds a data-centric permission in addition to the service-centric one. You can use the same key to encrypt data in Redshift, DynamoDB, and other databases and you have a central place to manage access to the data in all places. This is especially important when it's the same data, for example when you archive it from DynamoDB to S3. In this case, if you mistakenly give access to either of the services, the KMS key still protects the data itself.

But this comes with additional costs. Each customer-managed CMK costs $1/month on top of the KMS encryption/decryption costs.

SSE-C

In this encryption scheme, you provide the encryption key when uploading the object and AWS promises to forget it immediately. As a result, you need to send the same key when you retrieve the object.

This changes the command to upload the object. In addition to the bucket and the key, you also need to define the algorithm, the key, and the MD5 of the key. This is an example request that uses the gL6RsUG2fPElqDyMghs1yCrRJMJFLgR9MN key for the encryption:

aws s3api put-object \
	--bucket <bucket> \
	--key <key> \
	--body text.txt \
	--sse-customer-algorithm "AES256" \
	--sse-customer-key gL6RsUG2fPElqDyMghs1yCrRJMJFLgR9MN/Z8vjALUI= \
	--sse-customer-key-md5 XaroTmmABjK75669+kj/xw==

Strangely, the S3 Console does not report the object as Server-side encrypted:

Since getting the object now requires the encryption key, you need to specify that in the request. If you don't, the S3 service reports an error:

An error occurred (InvalidRequest) when calling the GetObject operation: The object was stored
using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object.

To get the object, use the same arguments. This returns the contents of the object as the encryption is still managed by the service.

aws s3api get-object \
	--bucket <bucket> \
	--key <key> \
	--sse-customer-algorithm "AES256" \
	--sse-customer-key gL6RsUG2fPElqDyMghs1yCrRJMJFLgR9MN/Z8vjALUI= \
	--sse-customer-key-md5 XaroTmmABjK75669+kj/xw== \
	/dev/stderr > /dev/null

contents

To check that an invalid key does not work, let's use a different one. In this case, S3 returns an "Access Denied" error.

aws s3api get-object \
	--bucket <bucket> \
	--key <key> \
	--sse-customer-algorithm "AES256" \
	--sse-customer-key gi6MmyZp6Vvz+gf7r4s349nXjcPUUU0JH5gwfYufsqs= \
	--sse-customer-key-md5 qqHzshF3ZcyKffalGwi9AQ== \
	/dev/stderr > /dev/null

Access Denied

So, what is SSE-C in practical terms?

You set a key when you upload the object, then you need to use the same key to get it later. "Server-side encryption with customer-provided encryption keys" is a fancy name for this. More colloquially, it's called a password.

Conclusion

In all the cases the response was in plaintext, there is no way to access the encrypted data when using server-side encryption in S3. As a result, there is no observable evidence that encryption happens at all. Because of this, it's better to think about how using these schemes affect access control instead.

With SSE-S3 and SSE-KMS when using the AWS managed CMK, access control is the same as for non-encrypted objects.

With SSE-KMS when using a customer managed CMK an additional layer of access control is enabled: the access to the KMS key. This enables inter-service permission control of data.

And finally, SSE-C allows a key to be set during the uploading and every retrieval needs the same key. This is effectively password-protection.