Programmatically retrieving secrets from Parameter Store and Secrets Manager

1. Problem statement

Most of the time, we use parameters and secrets in our applications. Apps need connection URLs, and database credentials or have to call 3rd-party endpoints protected with API keys.

We can integrate these parameters and secrets in our code in different ways. Some of them are better, and others are not so good.

Below I’ll discuss some ways we can make them available in our Lambda functions. I’ll refer to passwords, connection URLs, and API keys as secrets in the rest of this writing to be brief.

Although I’ll use Lambda functions in the examples, we can transfer the concepts to other compute resources, like EC2 instances, and ECS or EKS containers.

2. Hardcoding

The easiest way is to bake the secrets in the code, which is a strict no in most cases for security and DRY reasons. Even if the data is not confidential, we might still need it in multiple places. When the data changes, we’ll have to change it everywhere, which is not a good practice.

So we are better off adding the secrets to a central place like Parameter Store or Secrets Manager.

3. Using Parameter Store

Parameter Store is great because we can park our secrets there for free in many cases. The task is to retrieve the secret from there and make it available in the Lambda function code.

Although the rest of the blog entry uses Parameter Store as an example, the same concepts will apply to Secrets Manager items.

3.1. Deployment time

We can add dynamic references to our CloudFormation templates. This way, the secret’s value will be available at deployment time. We can store it as an environment variable in the function code.

An example in a SAM template can look like this:

Resources:
  DeployTimeFunction:
    Type: AWS::Serverless::Function
    Properties:
      # other properties here
      Environment:
        Variables:
          MY_PLAIN_STRING: '{{resolve:ssm:/test/plain-secret:1}}'
          MY_SM_SECRET: '{{resolve:secretsmanager:test/sm-secret}}'

We can do the same with CDK:

export class LambdaSecretsCdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const fn = new lambda.NodejsFunction(this, 'DeployTimeFunction', {
      // other properties here
      environment: {
        MY_PLAIN_STRING: ssm.StringParameter.fromStringParameterName(
          this,
          'ParameterStoreString',
          '/test/plain-secret'
        ).stringValue,
        MY_SM_SECRET: cdk.SecretValue.secretsManager('test/sm-secret', {
          jsonField: 'name',
        }).toString(),
      },
    });

    // and more code here
  }
}

Since SAM and CDK use CloudFormation in the background, the same concepts apply.

The advantage of this method is that we don’t need to worry about Parameter Store or Secrets Manager permissions because CloudFormation will handle everything for us. The secret will be readily available, and we can immediately refer to it in the code:

const { MY_SM_SECRET, MY_PLAIN_STRING } = process.env;

export const handler = async (event) => {
  // use MY_SM_SECRET and MY_PLAIN_STRING in the handler code
}

A disadvantage of this method is that if we change the secret’s value, we’ll need to redeploy the stack and change the version number in the infrastructure code. Also, anyone with access to the Lambda function can see the value in the console.

CloudFormation currently doesn’t support dynamic Parameter Store secure string resolution. So if we want to use secure strings in our code, we must choose one of the following methods.

3.2. Run time - own code

Another (and more secure) way is to retrieve secrets at run time.

We can do the heavy lifting and write the code ourselves. For example, we can have a helper module to get the secret using the SDK, and cache it like this:

// helper.mjs
import { SSMClient, GetParameterCommand } from '@aws-sdk/client-ssm';

const client = new SSMClient();

let secret;

export const getParameter = async ({ name, withDecryption = false }) => {
  if (secret) {
    console.log('Returning secret from cache');
    return secret;
  }

  const input = {
    Name: name,
    ...(withDecryption && { WithDecryption: withDecryption }),
  };
  const command = new GetParameterCommand(input);

  const parameter = await client.send(command);
  secret = parameter.Parameter.Value;
  console.log('Getting the secret from Parameter Store');
  return secret;
}

We can easily use it in the function handler:

// index.mjs
import { getParameter } from './helpers/ssm.mjs';

export const handler = async (event) => {
  try {
    const myParameter = await getParameter({
      name: '/test/my-secret', withDecryption: true,
    });
    // do something with myParameter
  } catch (error) {
    console.error('Error: ', error.message);
    throw error;
  }
}

The helper module caches the secret and makes it available throughout the execution environment’s lifetime. It implies that the code will call Parameter Store only once for each Lambda function lifecycle. The helper module is reusable across multiple functions and applications.

The function’s execution role must have the relevant IAM permissions. If we allow the ssm:GetParameters, ssm:GetParameter and ssm:GetParametersByPath actions in the role’s policy, the function will be able to retrieve various types and numbers of parameters. If we choose to encrypt the secret with a customer-managed KMS key (i.e., not the default AWS-managed key), we must add the kms:Decrypt permission to the policy, too.

3.3. Using a Lambda extension

We can use the AWS Parameters and Secrets Lambda Extension to get the secret from Parameter Store.

It’s a fun and easy way to retrieve and cache parameters and secrets. I wrote a post about the extension earlier, so I won’t repeat it here.

3.4. Using Lambda Powertools

Lambda Powertools is a library that provides several utilities to make us write less code and follow best practices. These utilities include logging, metrics, tracing, retrieving parameters, and other helpers.

We can use the Parameters utility to get secrets from Parameter Store and Secrets Manager. In addition to obtaining them from the services, we can set a TTL different from the default 5 seconds. We can also create our parameter store provider.

For our use case, a sample code with Lambda Powertools can look like this:

import { getParameter } from '@aws-lambda-powertools/parameters/ssm';

export const handler = async (event) => {
  try {
    const secret = await getParameter(
      '/test/my-secret', { decrypt: true, maxAge: 300 },
    );
    // do something with the secret from Parameter Store
  } catch (error) {
    console.error('Error: ', error.message);
    throw error;
  }
}

The second argument of the getParameter function is a config object, where we can set caching with the maxAge (in seconds) or decryption with the decrypt (boolean) properties. We can also use the POWERTOOLS_PARAMETERS_MAX_AGE and POWERTOOLS_PARAMETERS_SSM_DECRYPT environment variables to adjust these settings.

We must add the relevant permissions to the function’s execution role, as discussed above.

For more information on Lambda Powertools, please see the Further reading section.

4. Summary

Parameters and secrets are part of our application codes. The best practice is to store them in purpose-built services, from which we can get and share them among multiple applications.

We can retrieve the secrets at deployment or run time. If we choose run-time retrieval, we can write our code or use the prebuilt logic of the AWS Parameters and Secrets Lambda Extension or Lambda Powertools.

5. Further reading

Using Parameter Store parameters in AWS Lambda functions - (Almost) everything about the extension

Retrieving parameters and secrets with Powertools for AWS Lambda (TypeScript) - More information about getting secrets from Parameter Store

Creating IAM policies - The title of the documentation page says it all