Generate and retrieve secrets from Secrets Manager using the CLI

I discussed how secrets can be stored in AWS Parameter Store in an earlier post.

But Parameter Store can’t do everything what users might need from a good secrets manager service, so AWS created another to manage secrets and credentials.

1. What is Secrets Manager?

AWS Secrets Manager helps users store, rotate and retrieve encrypted database credentials, API keys and other secrets.

2. Some features

Secrets Manager automatically encrypts sensitive information using AWS Key Management Service (KMS).

Secrets like database credentials consist of not only the password in Secrets Manager but the full set of related pieces of information like user name, host URL, database name or port. They are stored together as one secret in JSON format and when the secret is fetched from Secrets Manager, the application can retrieve all the necessary parts of the secret.

I can also enable the rotation of credentials in Secrets Manager. I can set the service to rotate my credentials and if so, how often.

RDS, Redshift and the MongoDB compatible DocumentDB credentials are rotated by a built-in Lambda function.

If I want to rotate other secrets like API keys, I’ll have to provide the code for the function that manages the rotation of the secret.

Because Secrets Manager supports rotation, it will store different versions of the same secret. Versions are managed by the service but when I have to provide the Lambda function which does the rotation (see last paragraph), I’ll be responsible for managing the different versions in the code. Rotation and versioning will be discussed more in detail in a future post.

3. Pre-requisites, cost

Below I’ll create a simple set of database credentials in Secrets Manager. If you decide to follow along, there will be some pre-requisites and cost-related information coming in this section.

You’ll need to have a free AWS account, user credentials for programmatic access to AWS services and the AWS CLI installed on your computer with the relevant permissions to work with Secrets Manager (admin access will do if you work from your own account).

Secrets Manager comes with a 30-day trial period, which starts from the creation of the first secret. If you haven’t created any secrets in your account yet, executing the commands below won’t cost you anything.

I have already had some secrets (I mean in Secrets Manager) so I’ll pay 40 cents per secret per month pro-rated and I’ll also have to pay for the API calls (retrieving, updating and deleting the secrets). The latter item is negligible because it costs 5 cents per 10 000 API calls per month.

4. Create a secret

Let’s start with creating a secret!

As I mentioned above, a secret consists of the encrypted data and any information needed to manage the secret.

Secrets can be created by calling the CreateSecret API. A good way to discover the API and the options to call the API with is the open up a terminal window and type the following:

aws secretsmanager create-secret --generate-cli-skeleton

This command will return a JSON object containing the possible options and it looks like this:

{
  "Name": "",
  "ClientRequestToken": "",
  "Description": "",
  "KmsKeyId": "",
  "SecretBinary": null,
  "SecretString": "",
  "Tags": [
    {
        "Key": "",
        "Value": ""
    }
  ]
}

One way to use this object is to create a JSON file with these keys and refer to the file when executing the command in the terminal. I’ll write about this cool feature soon in another post.

I’ll instead use this skeleton to discuss the various options this command offers and will define the values for the necessary options in the terminal.

4.1. Options used

As it was stated above, a secret might not only consist of a key/value pair but can have multiple pairs. It makes sense to me to create a JSON file called secret-creds.json in the project folder that contains all information I need for the secret and refer to this file in the CLI command:

[
  {
    "username": "admin"
  },
  {
    "password": "hjshf3j4Hd7DNei9"
  },
  {
    "host": "database.123456789012.us-west-2.rds.amazonaws.com"
  },
  {
    "dbname": "ProductionDatabase"
  }
]

This secret contains the credentials (user name, password, host and database name) for an RDS database instance.

Referring to this JSON file is much easier than typing the JSON-format in the terminal.

I can now submit the create-secret CLI command under the secretsmanager domain:

aws secretsmanager create-secret --name ProductionDatabaseSecrets \\
--description 'Credentials for the production database' --secret-string file://secret-creds.json

The name of the secret will be ProductionDatabaseSecrets and I can provide an optional description to the secret, which will be displayed when the secrets are retrieved either in the Console or CLI.

I’ll use the secret-string property to define my database credentials because the format of the secret is JSON. I also have the option to store binary data as the value of the secret-binary option, but both secret-string and secret-binary can’t go together.

4.2. KMS key ID

One of the unused options in the above command is kms-key-id.

I didn’t specify any value here because I wanted Secrets Manager to use the default AWS-managed CMK (Customer Master Key). This key is generated by AWS for each service that uses encryption.

If I wanted to use a custom CMK key that I generate in KMS (which gives me greater control over the key), I would need to specify the ID of the key here.

4.3. Other options

The client-request-token option ensures that the secret remains unique. This is a unique ID (a UUID) and it’s automatically generated when I use the CLI to create the secret.

I’ve already mentioned secret-binary, which is used when secrets are stored as a buffer. This option cannot be used in the Console.

4.4. API response

The response to the CreateSecret API call looks like this:

  {
    "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:ProductionDatabaseSecrets-LVwoDz",
    "Name": "ProductionDatabaseSecrets",
    "VersionId": "ec124a81-e5a4-4288-882d-f2dd4e03e493"
  }

I’ll get the ARN and VersionId of the secret (both are generated by Secrets Manager) along with the Name, which I defined when the secret was created.

5. Some useful APIs

Now that I successfully created a secret with my database credentials, I might want to check what secrets I have.

5.1. List secrets

I can do this by calling the ListSecrets API:

aws secretsmanager list-secrets

Executing this command will result in a response similar to this:

{
  "SecretList": [
    {
      "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:ProductionDatabaseSecrets-LVwoDz",
      "Name": "ProductionDatabaseSecrets",
      "Description": "Credentials for the production database",
      "LastChangedDate": 1576392192.873,
      "SecretVersionsToStages": {
        "ec124a81-e5a4-4288-882d-f2dd4e03e493": [
            "AWSCURRENT"
        ]
      }
    }
  ]
}

If I had more than one secrets in my account in the region, I’d see more secrets in the SecretList array.

I often want to retrieve the metadata for a specific secret. I can use the DescribeSecret API like this:

aws secretsmanager describe-secret --secret-id ProductionDatabaseSecrets

The secret-id option is required otherwise Secrets Manager won’t know which secret I want to get more information about. I can put in the name (as in this example) or the ARN of the secret as the value of secret-id. I choose the name here because it’s shorter and I can remember it.

The response is very similar to the ListSecrets response, the only difference is that I’ll get the details only for the secret I specified:

{
  "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:ProductionDatabaseSecrets-LVwoDz",
  "Name": "ProductionDatabaseSecrets",
  "Description": "Credentials for the production database",
  "LastChangedDate": 1576392192.873,
  "VersionIdsToStages": {
    "ec124a81-e5a4-4288-882d-f2dd4e03e493": [
        "AWSCURRENT"
    ]
  }
}

5.2. Get the value of the secret

If I want to see the value of the secret, I can retrieve it through the GetSecretValue API:

aws secretsmanager get-secret-value --secret-id ProductionDatabaseSecrets

This command (as its name implies) will return the value of the secrets (the content of the JSON file I created above):

{
  "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:ProductionDatabaseSecrets-LVwoDz",
  "Name": "ProductionDatabaseSecrets",
  "VersionId": "ec124a81-e5a4-4288-882d-f2dd4e03e493",
  "SecretString": "[\n  {\n    \"username\": \"admin\"\n  },\n  {\n    \"password\": \"hjshf3j4Hd7DNei9\"\n  },\n  {\n    \"host\": \"database.123456789012.us-west-2.rds.amazonaws.com\"\n  },\n  {\n    \"dbname\": \"ProductionDatabase\"\n  }\n]\n",
  "VersionStages": [
      "AWSCURRENT"
  ],
  "CreatedDate": 1576392192.867
}

The SecretString key has the value I store in the secret and the VersionStages key shows that this is the current version of the secret.

6. Clean up

I don’t want to pay more than 40 cents for this hands-on exercise, so I’ll delete the secret from Secrets Manager.

Not very surprisingly the DeleteSecret API can help me with this:

aws secretsmanager delete-secret --secret-id ProductionDatabaseSecrets --recovery-window-in-days 7

The new option that I can specify is the recovery-window-in-days.

Recovery window is a time period during which I can change my mind and restore the secret calling the RestoreSecret API. Secrets Manager gives me this option so that I can manage the credentials in the applications and services that use this secret.

The recovery window can span from 7 to 30 days with the default value being 30. I specify 7 here partly because this database doesn’t exist in real life and I also want to reduce cost.

The response will show me the date of deletion, which is the end of the recovery window:

{
  "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:ProductionDatabaseSecrets-LVwoDz",
  "Name": "ProductionDatabaseSecrets",
  "DeletionDate": 1576998150.034
}

If I call ListSecrets now, I won’t see ProductionDatabaseSecrets in the response, although it will still be recoverable.

If I’m very confident that I don’t want to keep this secret anymore, I can specify the force-delete-without-recovery option instead of the recovery-window-in-days:

aws secretsmanager delete-secret --secret-id ProductionDatabaseSecrets --recovery-window-in-days

In this case the secret will be deleted immediately and it will be lost forever.

7. Summary

Secrets Manager is a powerful service to store sensitive database credentials and API keys.

Secrets Manager uses KMS keys to encrypt secrets, which are stored and can be retrieved in JSON format.

The secret consists of multiple key/value pairs, for example, for a database it can contain the user name, password, host and database name as well.

I won’t post next week and posts will resume after Christmas.

Thanks for reading and see you next time.