How to remove the RDS master user password from the Terraform state

Sensitive data in the state

The Terraform state file usually does not contain sensitive data. When you create a Lambda function, it will contain its configuration, its ARN, the configured role, and the appended policies, but nothing that would constitute sensitive information. This is because Lambda uses process-based credentials, so when the Lambda function needs access to protected resources, the Lambda service generates temporary credentials based on the IAM role. So if somebody gets access to the Terraform state file, they only get some insight into the infrastructure but nothing that would compromise security.

RDS works differently as it uses passwords to connect to the database and that (the master_password attribute) goes into the state file in plain-text:

When you add an RDS cluster to your infrastructure, the state file suddenly becomes a sensitive document and that requires special handling. If an attacker can access it, they might be able to connect to the database and read/write data. That's why the documentation recommends encryption or remote state with access control.

But then, it seems like it makes life a lot harder in many cases. So, instead of securing the state file, is there a way to remove the sensitive information instead?

Change the db password

The idea is that if the database password is changed after the stack is deployed, the password in the state file is no longer sensitive data. The initial password is still there, but it can not be used for connecting to the database.

Combined with Secrets Manager, which is a requirement if you use the Data API, it is possible to implement process-based credentials handling. While the database still uses a password, all the clients only need access to the secret to get the current value.

The process then:

• Store the password in Secrets Manager and configure all clients to use that
• After the deployment:
  • Change the database password
  • Update the secret value

There are multiple ways to implement this. In this article, we'll look into 2 solutions: automatic password rotation and the local-exec provisioner.

Automatic password rotation

Secrets Manager supports automatic password rotation using a Lambda function. It is a multi-step process but once set up it's fully automatic and supports scheduling regular password expiration.

This is "the correct" solution as it not only changes the password after deployment but also keeps it rotated on a regular basis. If an old version of the password is compromised, it does not affect security anymore.

But it is complicated to set up. Secret rotation requires multiple steps and while AWS provides click-to-deploy solutions, it is still not easy. Those solutions all rely on a direct connection to the database and setting up everything properly is a complicated process. Check out this article for the details.

Related

How to set up Amazon RDS password rotation with Terraform

Set up Secrets Manager password rotation with a VPC and a Lambda function

Local-exec provisioner

A simpler solution is to change the password during the deployment in a way that is not persisted in the state file. For this, a null resource with a local-exec provisioner is a good solution.

With this combination, you can define commands that Terraform will run locally as part of a terraform apply and can use data from other resources with automatic dependency resolution.

The drawback is that it requires software installed on the local machine, in this case the AWS CLI and jq. This can be especially problematic on CI machines.

The structure is simple: add a resource, add a provisioner, then define the commands and the environment:

resource "null_resource" "change-db-pass" {
  provisioner "local-exec" {
    command = <<EOT
...commands
EOT
    environment = {
			...env variables
    }
  }
}

Let's break down the steps for the password change!

Environment

Since the script needs to change the password of the cluster then update the secret, it needs to know the ARN of both:

resource "null_resource" "change-db-pass" {
  provisioner "local-exec" {
    command = <<EOT
...
EOT
    environment = {
      SECRET_ARN = aws_secretsmanager_secret_version.db-pass-val.secret_id
			RDS_CLUSTER_ID = aws_rds_cluster.cluster.id
    }
  }
}

Generate the password

There are many ways to generate random strings, and even Secrets Manager provides an API for that. While usually it's a best practice to generate secrets locally, in this case it's not a problem as it will be sent to AWS in plain-text in a later step.

PW=$(aws secretsmanager get-random-password --password-length 40 --exclude-characters '/@"\\'\'| jq -r .'RandomPassword')

RDS has some limits on what characters can be included in the password. The --exclude-characters argument tells Secret Manager to not use the forbidden ones.

Get the current secret

The secret contains more information and the script should leave the other fields alone. Since an update is an all-or-nothing operation, it needs to fetch the current value and change the password field:

aws secretsmanager get-secret-value \
	--secret-id $SECRET_ARN | jq -r '.SecretString' |
	jq -rM --arg PW "$PW" '.password = $PW'

Update the secret

With the secret value constructed, update it:

aws secretsmanager put-secret-value \
	--secret-id $SECRET_ARN \
	--secret-string "$(aws secretsmanager get-secret-value \
		--secret-id $SECRET_ARN | jq -r '.SecretString' |
		jq -rM --arg PW "$PW" '.password = $PW')"

Change the database password

Finally, change the master user password to the generated value:

aws rds modify-db-cluster --db-cluster-identifier $RDS_CLUSTER_ID --master-user-password "$PW" --apply-immediately

The final script

resource "null_resource" "change-db-pass" {
  provisioner "local-exec" {
    command = <<EOT
PW=$(aws secretsmanager get-random-password --password-length 40 --exclude-characters '/@"\\'\'| jq -r .'RandomPassword')
aws secretsmanager put-secret-value \
	--secret-id $SECRET_ARN \
	--secret-string "$(aws secretsmanager get-secret-value \
		--secret-id $SECRET_ARN | jq -r '.SecretString' |
		jq -rM --arg PW "$PW" '.password = $PW')"
aws rds modify-db-cluster --db-cluster-identifier $RDS_CLUSTER_ID --master-user-password "$PW" --apply-immediately
EOT
    environment = {
      SECRET_ARN = aws_secretsmanager_secret_version.db-pass-val.secret_id
			RDS_CLUSTER_ID = aws_rds_cluster.cluster.id
    }
  }
}

Now the Terraform state does not contain sensitive information.