Cacheable S3 signed URLs

Learn how to sign URLs in a cache-friendly way and save money on bandwidth

Tamás Sallai

5 mins

Want to learn AWS serverless development? Click here

The problem with signed URLs

Let's say you have an image on a customers-only corner of your website. To ease the bandwidth requirements on your servers, you decided to move the image to S3 and use signed URLs to selectively grant access to it. This can be a small thing like an avatar, or a bigger one, for example, some promotional content.

This works great, the users see the images without downloading them from your servers directly.

But when you test your page and open the Network tab you see that the same image is downloaded over and over again when navigating between the pages:

This wastes bandwidth on the clients' side and money on yours.

What is going on here?

How S3 Signed URLs work

Signed URLs provide secure a way to distribute private content without streaming them through the backend. Learn how they work and how to use them.

Book

Building GraphQL APIs with AWS AppSync

How to design, implement, and deploy GraphQL-based APIs on the AWS cloud

Repeatability

The signature on the signed URL is time-based. The first image the user downloads has the URL like this:

https://signedurls-truncate-test-1556886762.s3.amazonaws.com/cat.jpg?
	AWSAccessKeyId=XXXXX&
	Expires=1556887750&
	Signature=XXXXXX&
	x-amz-security-token=XXXXX

But when he comes back later the URL changes:

https://signedurls-truncate-test-1556886762.s3.amazonaws.com/cat.jpg?
	AWSAccessKeyId=XXXXX&
	Expires=1556887796&
	Signature=YYYYYY&
	x-amz-security-token=YYYYY

As you can see the Expires parameter is different and with it the Signature and the security token. And because the URL is different, the browser treats them as two separate images.

When you sign an URL, the AWS SDK gets the current time and calculates the expiration based on that. And since it has a precision of one second the user will get different URLs every time:

This is not a problem during one-time downloading of large files, the primary use-case of signed URLs, but when the same user might repeatedly request the same file it can be a problem.

Let's see what we can do to remedy this!

Cache-friendly signing

The idea is to round the time the URLs are signed, effectively traveling back in time, so that ones created close together will be exactly the same. As an illustration, if the backend signs the URL at 11:05 it behaves as if it signed it at 11:00. If the client comes back and requests the image at 11:07, round that back to 11:00 so that the two URLs will match exactly. And when the same client comes back at 11:35, he'll get a different URL:

With this technique, the browser does not need to download the files every time from S3 but use the cached version:

This obviously changes the real expiration time of a signed URL. A URL signed at 11:05 dated back to 11:00 will be valid for 5 minutes less than one that was signed at 11:00.

If you keep the default expiration of 15 minutes for S3 URLs and round to the last 10-minute mark, the effective validity will fluctuate between 5-15 minutes.

Are S3 signed URLs secure?

The potential security problems with file distribution based on signed URLs

Implementation

By default, creating a signed URL using the JS SDK is by calling the s3.getSignedUrl function:

const AWS = require("aws-sdk");

const s3 = new AWS.S3();

// caching is not supported
const url = s3.getSignedUrl(
	"getObject",
	{
		Bucket: ...,
		Key: ...,
	}
);

To round the time down to the last 10-minute mark, you need to make the AWS SDK think that the current time is an earlier instant. To achieve this, timekeeper provides a nice API:

const AWS = require("aws-sdk");
const tk = require("timekeeper");

const s3 = new AWS.S3();

// round the time to the last 10-minute mark
const getTruncatedTime = () => {
	const currentTime = new Date();
	const d = new Date(currentTime);

	d.setMinutes(Math.floor(d.getMinutes() / 10) * 10);
	d.setSeconds(0);
	d.setMilliseconds(0);

	return d;
};

// cache-friendly signing
const url = tk.withFreeze(getTruncatedTime(), () => {
	return s3.getSignedUrl(
		"getObject",
		{
			Bucket: ...,
			Key: ...,
		}
	);
});

Fortunately, the s3.getSignedUrl is synchronous, so freezing the time does not interfere with other time-dependent parts of the codebase. But keep in mind not to use the callback or the Promise version of the function as those do not come with the same guarantees.

Validity and rounding

Make sure that the expiration time is greater than the rounding.

If, for example, the validity of the signature is 15 minutes but you round down to the last 30 minutes, an URL signed at 11:17 is already expired:

The default signature validity is 15 minutes making rounding to the last 10-minute mark a good option. Don't use the same value for both, as that would create a race condition if the signature is done close to the end of the period.

The effective validity is [(expiration - rounding), expiration], for example for the scenario above (the signature is valid for 15 minutes, rounding is to 10-minute marks), the effective interval is 5-15 minutes.

If the default seems too low, just increase the two values. An expiration time of 7 hours and rounding to 6 hours gives plenty of time for the browser to use the cached version.

Credentials

The other changing part of a signed URL is the credentials but it's more involved to stabilize. To see how to do it, see this article:

Stable S3 signed URLs

How to make signed URLs cacheable

Conclusion

When you use signed URLs in a way that a single user might download the file multiple times, make sure you use cache-friendly signatures. With a little bit of planning, they are easy to set up, and they play well with CloudFront caches also, making the user experience even better.