Lambda development toolbox: Set environment variables locally

Code is available on GitHub

An npm package is available

Want to learn AWS serverless development? Click here

Local Lambda development challenges

Lambda development usually starts with writing a handler function and deploying it to an AWS account. If you use a standard runtime, such as node14.x, the code the Lambda environment runs is just plain Javascript which you could run on your machine as well. But when the Lambda service invokes the function it does a few additional things, such as it assumes the execution role and sets the environment variables assigned to the function and a few other ones. This makes it hard to run the handler locally.

For example, the function might be managed by Terraform (or CloudFormation, or AWS CDK, or AWS SAM) and get the identifier for an S3 bucket via the environment:

resource "aws_lambda_function" "lambda" {
	# ...
	environment {
		variables = {
			BUCKET = aws_s3_bucket.bucket.id
			KEY    = aws_s3_bucket_object.object.id
		}
	}
}

Also, it might receive permissions to read this bucket via the execution role:

data "aws_iam_policy_document" "lambda_exec_role_policy" {
	statement {
		actions   = ["s3:GetObject"]
		resources = ["${aws_s3_bucket.bucket.arn}/*"]
	}
}

resource "aws_iam_role_policy" "lambda_exec_role" {
	role   = aws_iam_role.lambda_exec.id
	policy = data.aws_iam_policy_document.lambda_exec_role_policy.json
}

resource "aws_lambda_function" "lambda" {
	# ...
	role    = aws_iam_role.lambda_exec.arn
}

Let's say you want to invoke this function locally for rapid development, by starting a HTTP server and interface with the Lambda handler:

// main.js is the lambda handler, exporting the handler function
const main = require("./main");

const http = require("http");
const port = 3000;

const processResponse = (res) => async (lambdaResponse) => {
	if (lambdaResponse.headers) {
		Object.entries(lambdaResponse.headers).forEach(([k, v]) => res.setHeader(k, v));
	}
	res.statusCode = lambdaResponse.statusCode;
	res.end(lambdaResponse.body);
};

const server = http.createServer((req, res) => main.handler().then(processResponse(res)));

server.listen(port, (err) => {
	if (err) {
		return console.log(err);
	}else {
		console.log(`server is listening on ${port}`);
	}
});

While the above code calls the Lambda handler correctly, the missing environment prevents the function from running correctly:

$ node src/index.js
server is listening on 3000
(node:27191) UnhandledPromiseRejectionWarning: MultipleValidationErrors: There were 2 validation errors:
* MissingRequiredParameter: Missing required key 'Bucket' in params
* MissingRequiredParameter: Missing required key 'Key' in params
    at ParamValidator.validate (/home/ubuntu/workspace/local-lambda-environment/examples/src/node_modules/aws-sdk/lib/param_validator.js:40:28)

The usual next step here is to get the values from the live function and call the handler with those:

BUCKET=bucket-abc node index.js

But this approach has two problems:

• It's a pain to go to the Console and copy-paste the parameters, especially when they change
• The permissions are not replicated

Book

Building GraphQL APIs with AWS AppSync

How to design, implement, and deploy GraphQL-based APIs on the AWS cloud

Reproducing the environment

I've tried to solve this problem and came up with 25-lines of Bash script. While that worked, copying that script to all project when I needed it proved to be just difficult enough to be impractical. Also, it required a permanent entry to the execution role's assume role policy.

Related

How to reproduce a Lambda function's environment variables locally

How to reproduce the environment for efficient local Lambda development

So, I made a Node CLI script that is easy-to-use and just a single line. To run the above script with the important environment variables straight from the live environment, use:

npx set-lambda-env-vars "node src/index.js"

It does multiple things:

• It reads the environment variables set to the function (BUCKET, KEY, etc.)
• It adds a statement to the execution role's trust policy with a 10 minutes time limit, then assumes it, and sets the keys (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
• It reads the region and sets the AWS_REGION variable
• Then it calls the command with the environment set

This is the full command:

npx set-lambda-env-vars [-f|--function] <command>

If you use Terraform then it tries to read the local state for any Lambda functions. If there is only one then it uses that if no --function argument is passed.

You can run a shell so that it won't need to make AWS calls every time:

$ npx set-lambda-env-vars bash
$ env
AWS_ACCESS_KEY_ID=ASIA...
AWS_SESSION_TOKEN=...
AWS_SECRET_ACCESS_KEY=...
BUCKET=...
KEY=...
AWS_REGION=...

And for a full developer experience, combine it with a file watcher:

$ npx set-lambda-env-vars bash
$ npx nodemon --exec "node src/index.js"

This reproduces the environment from the live Lambda function and restarts the local server whenever there is a change.

With just two lines, it is probably the most developer-friendly approach I can think of.