How to read values from SSM Parameter Store and Secrets Manager with AppSync HTTP data source
Send signed requests to AWS services and retrieve protected values
Storing secrets in AWS
For IAM credentials, AWS provides a secret-less way to retrieve and use keys. This is how Lambda functions and EC2 instances automagically have access to services defined in an IAM policy. You only need to define the permissions, attach it to the function or the instance, and everything else is handled in the background. This process uses the same Access Key ID and Secret Access Key as any other AWS calls, but these keys are transient: they are retrieved when needed and stay only in memory. As a result, you don't need to hardcode a secret value anywhere in your infrastructure, and especially not in code.
But that's only for one specific type of secret. For others, such as a private key to sign a JWT, a token for an external service, a password, or a random but secret value a backend service requires, storing and securely retrieving them is still a problem.
In AWS, 2 services are specifically for storing secret values: SSM Parameters Store and Secrets Manager. Both work similarly: they store the secret value and offer a way to read the value while access is determined by IAM policies. Whenever something needs the secret value, it uses its IAM role or any other source of IAM credentials to send a signed request and receive the value from the secret store. An important aspect here is that the secret value lives only in memory, the service does not store it permanently.
This process also works in an AppSync resolver thanks to the HTTP data source and its "hidden" feature that it supports signing the requests using the AWS signature algorithm.
In this article we'll look into how to read a secret value from SSM Parameters Store and then how to do the same with Secrets Manager. Reading secrets directly in a resolver is especially useful in a pipeline resolver, as the next steps has access to the secret value. We'll explore this use-case later.
SSM Parameters Store
The value is stored as a parameter:
The HTTP data source has a role with ssm:GetParameter access to this parameter:
Then the HTTP data source is configured with the Role, the SSM endpoint for the parameter's region, and signing options:
resource "aws_appsync_datasource" "parameter_store" {
api_id = aws_appsync_graphql_api.appsync.id
name = "ssm"
service_role_arn = aws_iam_role.appsync.arn
type = "HTTP"
http_config {
endpoint = "https://ssm.${data.aws_arn.ssm_parameter.region}.amazonaws.com"
authorization_config {
authorization_type = "AWS_IAM"
aws_iam_config {
signing_region = data.aws_arn.ssm_parameter.region
signing_service_name = "ssm"
}
}
}
}
The endpoint and the signing region are determined where the parameter is located and not by the AppSync API. The above example extracts the region from the ARN
of the parameter so it supports inter-region secrets as well. Finally, the signing service name is ssm as the signature is for the SSM service.
The resolver that sends the request needs to configure the other parts (documentation):
{
"version": "2018-05-29",
"method": "POST",
"params": {
"headers": {
"Content-Type" : "application/x-amz-json-1.1",
"X-Amz-Target" : "AmazonSSM.GetParameter"
},
"body": {
"Name": "${aws_ssm_parameter.value.name}",
"WithDecryption": true
}
},
"resourcePath": "/"
}
The X-Amz-Target defines the operation as the request goes to /. Then the body contains the arguments for the operation, in this case the name of
the parameter and that it will be decrypted if it's encrypted.
The result has a JSON body. This response mapping template extracts the value:
#if ($ctx.error)
$util.error($ctx.error.message, $ctx.error.type)
#end
#if ($ctx.result.statusCode < 200 || $ctx.result.statusCode >= 300)
$util.error($ctx.result.body, "StatusCode$ctx.result.statusCode")
#end
$util.toJson($util.parseJson($ctx.result.body).Parameter.Value)
Note the error handling. The $ctx.error is only defined if there was a problem with the request but not when the response indicates an error. That's why
the second check to the status code is needed: it throws an error if the status is not 2XX.
To test it, send a GraphQL request to the AppSync API:
query MyQuery {
ssm_parameter
}
The result is a JSON with the stored value:
{
"data": {
"ssm_parameter": "secret"
}
}
Secrets Manager
Values in Secrets Manager works almost the same. The value:
Then a permission to read this value:
The HTTP data source is configured for the endpoint, the role, and the signature values:
resource "aws_appsync_datasource" "secret" {
api_id = aws_appsync_graphql_api.appsync.id
name = "secret"
service_role_arn = aws_iam_role.appsync.arn
type = "HTTP"
http_config {
endpoint = "https://secretsmanager.${data.aws_arn.secret.region}.amazonaws.com"
authorization_config {
authorization_type = "AWS_IAM"
aws_iam_config {
signing_region = data.aws_arn.secret.region
signing_service_name = "secretsmanager"
}
}
}
}
The same considerations apply here: the endpoint and the signing region is determined by where the secret is located, and the service name is secretmanager.
The resolver that sends a request to retrieve the value (documentation):
{
"version": "2018-05-29",
"method": "POST",
"params": {
"headers": {
"Content-Type" : "application/x-amz-json-1.1",
"X-Amz-Target" : "secretsmanager.GetSecretValue"
},
"body": {
"SecretId": "${aws_secretsmanager_secret.secret.id}"
}
},
"resourcePath": "/"
}
Again, the X-Amz-Target defines the operation, and the body contains the parameters.
The response mapping template checks for errors then extracts the value:
#if ($ctx.error)
$util.error($ctx.error.message, $ctx.error.type)
#end
#if ($ctx.result.statusCode < 200 || $ctx.result.statusCode >= 300)
$util.error($ctx.result.body, "StatusCode$ctx.result.statusCode")
#end
$util.toJson($util.parseJson($ctx.result.body).SecretString)
Testing it with a GraphQL query:
query MyQuery {
secret
}
Gives the value in a JSON:
{
"data": {
"secret": "secret value"
}
}
For more info, see this blog post.
Rate limits
When depending on a separate service, you'll inherit the quotas of that service. In this case, while AppSync does not have a documented upper limit on the rate of requests (only an adjustable "Rate of request tokens"), SSM and Secrets Manager have.
Especially in the case of SSM, the free tier for parameters provide a relatively low limit on the number of requests: 40/sec. There is a paid higher throughput tier that allows up to 3000 requests/sec for $0.05/10k requests that should cover pretty much all use-cases.
Similarly to SSM's paid tier, Secrets Manager has a high throughput limit: 5000/s. This should be enough for everything.
If you want to stay under the limit of the free tier of SSM, you can use a Lambda function that provides a rudimentary cache that drastically decreases the number of requests sent to the SSM service.
Logs
The main concept behind using a specialized storage to store sensitive values is that these values are not stored anywhere else. When a component needs the value, it uses its credentials to retrieve it, then only store it in memory.
Guess what happens when you turn on resolver logging for the AppSync API:
Yeah, the secret value is in the logs in plain-text. Worse still, this is not controllable by the resolver itself as AppSync logs what the data source returned which contains the response body.
Because of this, be extra cautious when you read secrets in an AppSync resolver not to turn on logging in the future.
Conclusion
With the combination of the HTTP data source and its ability to sign the requests, it is possible to retrieve values stored in SSM Parameters Store or Secrets Manager directly with an AppSync resolver. This provides an easy and generally safe way to call external services that rely on an access token. But watch out for resolver logging as that can leak the secrets.
