Store and Rotate API Keys with AWS Secrets Manager

There are many options when it comes to managing API keys in AWS. It’s a spectrum of good and bad with trade-offs in performance, cost, security, and complexity.

At the bad end of the spectrum are simple solutions like hardcoded strings and configuration files committed to version control.

Environment variables are somewhere in the middle, and you’ll find Parameter Store and Secrets Manager at the good end.

This post isn’t intended to be a comparison of solutions. Instead, I’m simply going to show you how to store, access, and automatically rotate API keys using AWS Secrets Manager, which was introduced in April 2018.

A photo I took of Alcatraz last year

AWS Secrets Manager

Secrets Manager is relatively new, so you may not have heard of it before.

Obviously, it’s a secrets management service. It enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

Using Secrets Manager, you can secure, audit, and manage secrets used to access resources in the AWS Cloud, on third-party services, and on-premises.

That’s all you need to know going into this. I’ll build up your knowledge along the way.

Using Secrets Manager

Let’s start by looking at how you’d use Secrets Manager from your code. The diagram below shows a Lambda function that calls Secrets Manager to get the API key it needs to access an external API.

The GetSecretValue command has a single required parameter named SecretId. Secrets can be identified using their name or ARN.

When you create a secret, you give it a name on to which AWS appends a hyphen followed by six random characters (code in the ARN below).

arn:aws:secretsmanager:<region>:<accountId>:secret:<name>-<code>

Regardless of whether you use the secret’s name or ARN as SecretId, you don’t have to include the hyphen and code (though AWS recommend you do).

Secrets can have a string or binary value. The response of GetSecretValue will have either a SecretString or SecretBinary property, respectively.

The AWS console only supports string values containing plaintext or JSON formatted key-value pairs. Binary secrets must be managed via the API.

Secret Recipe

To understand secret rotation, you need to know what secrets are made of, so let’s take a look. I’ll go into more detail after a quick summary.

❤️ Metadata

• ARN & Name: Unique secret identifiers.
• Description: A human-friendly description of the secret.
• KMSKeyId: (Optional) The ARN of the KMS key that Secrets Manager uses to protect the secret.
• Rotation Configuration: (Optional) How frequently the secret is automatically rotated and which Lambda function is used to perform the rotation.
• Timestamps: Last accessed, last changed, and last rotated timestamps.
• Tags: Key-value pairs just like many other AWS services.

🧡 Versions

• ID: Unique version identifier. Currently a UUID by default.
• Staging Labels: List of strings.
• Secret Value: A string or binary value.

Versions and Stages

A secret can have one or more versions, each of which contains a secret value. In fact, it’s the versions that contain the value and not the secret itself.

A secret must have at least one version, and one of its versions must have the AWSCURRENT staging label. Therefore, when you create a new secret, Secrets Manager automatically creates a version and gives it the AWSCURRENT label.

Each version can have up to 20 labels, but only one version can have each label at a time. Another way to look at it is that each stage can only have one version at a time. It’s a one-to-one mapping.

In the GetSecretValue example at the top of this post, we could have additionally passed in either VersionId or VersionStage. Secrets Manager would have returned the value of the version with the given ID or stage label. Since we didn’t, the default version with the AWSCURRENT label was returned.

Creating Secrets

You can create secrets using the CLI, API, CloudFormation, or the console. Humans are visual creatures, so I’ll show you how to do it with the console.

Let’s walk through a simple example to store an OAuth access token and refresh token.

In the Secrets Manager console, click Store a new secret on the right.

Select Other type of secrets (e.g. API key). Then add your access token and refresh token. You’ll need to click Add row or use the Plaintext view. You can store up to 7168 bytes in each secret.

KMS encryption is out of scope here, so leave DefaultEncryptionKey selected. The default means Secrets Manager creates and manages a new encryption key for each account in each region.

Finally, click Next to move on.

You need to give the secret a unique name. It only needs to be unique to the account and region, not globally like S3 buckets.

It’s recommended that you choose a naming convention and stick to it. In the placeholder text, we see prod/AppBeta/Mysql which is using a slash to create a hierarchy. This is great for simple namespacing, but it's better to use tags for anything more complex.

Tags are great for things like grouping secrets, cost allocation and tracking, or recording ownership. You can even use IAM policies to restrict access to only secrets with particular tags.

A good name, description, and tags can help developers discover existing secrets instead of creating new ones.

Clicking Next takes you to the last page where you can configure automatic rotation. You can’t configure automatic rotation yet since you haven’t created a Lambda function, but let’s look at the options.

With Enable automatic rotation selected, you can choose a rotation interval of 30, 60, or 90 days. Alternatively, you select Custom and enter a number between 1 and 365 days. You can’t have an interval of less than a day at the moment.

Next, we’ll go into detail on the Lambda function and how Secrets Manager invokes it during the rotation process.

The last page allows you to review the configuration and provides some code examples for accessing the secret’s value from your application.

If you configured automatic rotation, the secret will be rotated immediately upon clicking Store.

The Rotation Function

Secrets Manager decides when to rotate your secret based on the interval you configure. Rotation happens randomly during a 24 hour period. That is, if you select a 7-day interval, your secret will be rotated every 144–168 hours.

There are four steps in the rotation process. Secrets Manager will invoke your Lambda function sequentially to perform each one.

AWS provide sample function implementations and a general template.

Let’s walk through a rotation of the prod/foo secret below. This secret has a single version aaaaaaaaaaaaaaaaaaaaaaaa with the label AWSCURRENT.

Step 1: createSecret

When Secrets Manager decides its time to rotate your secret, it generates a new version ID, a UUID like bdb9c291-afa2–4435–822b-6240dc732caf. To make things simpler, we’ll use bbbbbbbbbbbbbbbbbbbbbbbb.

Secrets Manager then invokes your Lambda function with the following input.

{
    "SecretId": "arn:aws:secretsmanager...secret:prod/foo-C8F3BL",
    "ClientRequestToken": "bbbbbbbbbbbbbbbbbbbbbbbb",
    "Step": "createSecret"
}

Step is createSecret and the newly generated ID is passed in as ClientRequestToken. SecretId is the ARN of the prod/foo secret.

On this step, your Lambda function needs to do two things:

Create a new secret value

The exact implementation will differ by API, but let’s say our access tokens are created by sending an HTTP POST to the /tokens resource of the API. In our example, the HTTP request must contain a client_id and client_secret.

You should store the client_id and client_secret in a separate secret, sometimes called a master secret. You then include the ARN or name of the master secret inside your main secret (prod/foo):

{
    "token": "11111111",
    "master_secret_id": "aws:arn:secretsmanager..."
}

This keeps the client credentials isolated and safe. It also allows your Lambda function to be reused to rotate other secrets.

Your function needs to:

1. Get the current secret value using GetSecretValue, passing SecretId.
2. Read master_secret_id from SecretString.
3. Fetch the master secret using GetSecretValue, passing master_secret_id.
4. Read client_id and client_secret from the master secret.
5. Call the API’s /tokens endpoint, passing client_id and client_secret.
6. Extract the new token from the HTTP response.

Store the new secret value

Now that you have the new access token, you use the PutSecretValue command to create a new version of the secret.

This command takes the SecretId and ClientRequestToken from your function’s input. SecretString will contain the new token and a copy of master_secret_id (for the next rotation). Lastly, the VersionStages list will contain one label, AWSPENDING.

You can see below that ClientRequestToken became the new version’s ID.

Step 2: setSecret

When rotating an API key, you usually won’t need to do anything for this step.

The input to your function will look like this. Note that only step has changed. That’s the case for all invocations.

{
    "SecretId": "arn:aws:secretsmanager...secret:prod/foo-C8F3BL",
    "ClientRequestToken": "bbbbbbbbbbbbbbbbbbbbbbbb",
    "Step": "setSecret"
}

The setSecret step is used for rotating other kinds of secrets. For example, when rotating database credentials, you generate a new password in createSecret, and create the database user with that password during setSecret.

In our scenario, there is nothing to do here.

Step 3: testSecret

Again, only the step property of the input will have changed:

{
    "SecretId": "arn:aws:secretsmanager...secret:prod/foo-C8F3BL",
    "ClientRequestToken": "bbbbbbbbbbbbbbbbbbbbbbbb",
    "Step": "testSecret"
}

This step is your opportunity to ensure the new secret works as expected. In our case, we should test that we can successfully call the API with the new token.

To ensure the new secret is 100% working, AWS recommends you perform every action your application will. Obviously, some actions are destructive, so just do what you can to be confident.

Your function needs to:

1. Get the new/pending secret value using GetSecretValue, passing SecretId and ClientRequestToken. By passing ClientRequestToken you are saying you want the value of the secret version you created in createSecret.
2. Read token from SecretString.
3. Call the API using token to ensure it works as expected.

Step 4: finishSecret

The input to this final step is like the others. Only step has changed.

{
    "SecretId": "arn:aws:secretsmanager...secret:prod/foo-C8F3BL",
    "ClientRequestToken": "bbbbbbbbbbbbbbbbbbbbbbbb",
    "Step": "finishSecret"
}

In this step, your function will move the AWSCURRENT label to the new version using the UpdateSecretVersionStage command.

Until now, clients have been reading version aaaaaaaaaaaaaaaaaaaaaaaa because it had the AWSCURRENT label. Subsequent GetSecretValue calls will now return bbbbbbbbbbbbbbbbbbbbbbbb.

UpdateSecretVersionStage has four request properties:

• VersionStage — The label you want to move. AWSCURRENT in this case.
• SecretId — The ARN or name of prod/foo. Get this from the input.
• MoveToVersionId — The ID of the version we want to move AWSCURRENT to. In this case, it’s bbbbbbbbbbbbbbbbbbbbbbbb (ClientRequestToken in the input).
• RemoveFromVersionId — The ID of the version AWSCURRENT is currently assigned to: aaaaaaaaaaaaaaaaaaaaaaaa.

Note that you’re not given the ID of version aaaaaaaaaaaaaaaaaaaaaaaa, so you’ll need to use the DescribeSecret command to get it. That command takes a SecretId and returns the metadata and version details described in the Secret Recipe section further up.

Specifically, the response will have a VersionIdsToStages property. This is a mapping from version ID to a list of staging labels.

{
    "aaaaaaaaaaaaaaaaaaaaaaaa": [ "AWSCURRENT" ],
    "bbbbbbbbbbbbbbbbbbbbbbbb": [ "AWSPENDING" ]
}

Following the call to UpdateSecretVersionStage, our example secret will look like this:

Note that aaaaaaaaaaaaaaaaaaaaaaaa has been automatically given the AWSPREVIOUS label. This helps to identify the last good version in case you need to roll back.

The Next Rotation

Subsequent rotations are all the same, but you may be wondering what happens to the old versions.

After the next rotation, you’ll have three versions. Since there is a one-to-one mapping between versions and labels, only bbbbbbbbbbbbbbbbbbbbbbbb can have AWSPREVIOUS, leaving aaaaaaaaaaaaaaaaaaaaaaaa without a label.

aaaaaaaaaaaaaaaaaaaaaaaa will not appear in the DescribeSecret response, but can still be read. However, Secrets Manager will eventually delete versions without labels.

Pricing Considerations

There is a 30-day free trial period when you first start using Secrets Manager, so there’s no reason not to give it a go.

After that, you’ll pay $0.40 per secret per month. It’s important to note that this is per secret not per secret version, so rotating and storing previous versions will not cost extra.

Every 10,000 API calls will cost you $0.05. It’s recommended that you cache secrets where possible. AWS offers a Java and JDBC caching library which caches secrets for 1 hour, which is good guidance if you use another language.

When using Lambda, retrieve your secrets outside the handler method and reuse them on subsequent invocations.

Conclusion

Secrets Manager has a short but steep learning curve. In my opinion, it’s worth it for the robust automatic rotations, on-demand rotations, fine-grained security, and ability to store complex secrets as JSON.

I hope this post has helped you in some way. Please let me know if you still have any questions. I’ll happily help and update my post.

For more like this, please follow me on Medium and Twitter.