How Secrets Manager Schedules Automatic Rotations

Update 2/Feb/2022: AWS just announced a change to Secrets Manager that allows you to limit secret totations to specific hours on specific days:

AWS Secrets Manager now supports rotation windows

Not long ago, I wrote a detailed guide on how to Store and Rotate API Keys with AWS Secrets Manager. In that guide, I briefly mentioned that secrets could be automatically rotated with an interval between 1 and 365 days.

Imagine you set the interval to 1 day. On which day, hour, and minute will the secret be rotated? The answer is a little strange, so let me explain.

Photo Malvestida Magazine via Unsplash

The Secrets Manager documentation contains a decent explanation. However, I didn’t interpret it correctly the first time I read it.

Secrets Manager schedules the date by adding the rotation interval (number of days) to the actual date of the last rotation. The service chooses the hour within that 24-hour date window randomly. The minute is also chosen somewhat randomly, but is weighted towards the top of the hour and influenced by a variety of factors that help distribute load.

Day

When your secret finishes rotating, Secrets Manager adds your rotation interval to the current UTC date. The result is the day on which your secret will be next rotated.

Let’s look at an example where the rotation interval is 1 day. If the secret rotation completes at 2019–04–25 17:43 UTC, its next rotation will be on 2019–04–26. Secrets Manager added one day and ignored the time part.

Hour

Next, a random hour on the rotation day is chosen. The reason for randomness is mostly likely to reduce the load on Secrets Manager by spreading out rotations. If this was configurable, rotations would probably be grouped around popular times like midnight.

Continuing the previous example, we now know the next rotation will be between 2019–04–26 00:?? UTC and 2019–04–26 23:?? UTC.

Minute

Lastly, the minute is chosen somewhat randomly, but is weighted towards the top of the hour and influenced by a variety of factors that help distribute load.

Take a look at the last two weeks of automatic rotations of one of my secrets and you’ll see that the minute is actually quite consistent (which could just be a coincidence).

2019-04-25 08:07 UTC
2019-04-24 15:05 UTC
2019-04-23 20:07 UTC
2019-04-22 22:07 UTC
2019-04-21 20:07 UTC
2019-04-20 15:08 UTC
2019-04-19 15:07 UTC
2019-04-18 08:07 UTC
2019-04-17 04:07 UTC
2019-04-16 17:07 UTC
2019-04-15 20:07 UTC
2019-04-14 16:07 UTC
2019-04-13 19:08 UTC
2019-04-12 05:07 UTC

One Day Is Not 24 Hours

When I first read the documentation, I thought Secrets Manager would add 24 hours (my 1 day interval) to the last rotation time, and rotate the secret again before that 24 hours is up. That’s not the case.

Take a look at the rotations of my secret with the time since last rotation added. There is between 11 and 38 hours between each rotation!

2019-04-25 08:07 UTC (~17 hours)
2019-04-24 15:05 UTC (~18 hours)
2019-04-23 20:07 UTC (~22 hours)
2019-04-22 22:07 UTC (~26 hours)
2019-04-21 20:07 UTC (~28 hours)
2019-04-20 15:08 UTC (~24 hours)
2019-04-19 15:07 UTC (~31 hours)
2019-04-18 08:07 UTC (~28 hours)
2019-04-17 04:07 UTC (~11 hours)
2019-04-16 17:07 UTC (~21 hours)
2019-04-15 20:07 UTC (~28 hours)
2019-04-14 16:07 UTC (~20 hours)
2019-04-13 19:08 UTC (~38 hours)
2019-04-12 05:07 UTC

To completely understand this, let’s look at an example over a couple of days.

A secret with a 1 day interval gets rotated at 2019-04-25 23:45. Secrets Manager adds a day and gets 2019-04-26 for the next rotation. It then randomly chooses an hour and minute.

If it chooses 23:45 again, there will be 24 hours between rotations. If it choose 00:45, there will only be 1 hour between rotations!

Imagine it chooses 00:45 and does the rotation after only one hour, then starts calculating the next rotation. Of course, it adds a day and decides it will be on 2019-04-27. This time, however, it chooses 23:45 which results in a whopping 47 hours between rotations!

Conclusion

Due to the way Secrets Manager schedules rotations, a 1 day rotation interval will result in your secret being rotated every 0 to 48 hours.

Secrets Manager is a great service. It knows how to rotate database passwords already, it can store multiple values in string or binary format, and it implements safe rotations via its four stages. Automatic rotation scheduling is an unfortunate weak point.

In my case, it was okay if the secret is rotated after up to 48 hours, but this could be a deal-breaker for you. I may have a solution, though.

Alternative Rotation Scheduling

I think there is a way to keep the benefits of Secrets Manger, while getting more granularity, predictability, and control over your rotations:

You could trigger the rotation yourself.

1. Create a CloudWatch Events Rule That Triggers on a Schedule (docs):
   This could be every 24 hours (more predictable), less than 24 hours (more granular), or anything you can represent with a Schedule Expression, such as “the 1st of each month” (more control).
2. Use AWS Lambda with Amazon CloudWatch Events (docs):
   Subscribe a Lambda function the the CloudWatch Events rule to invoke the function on a schedule.
3. Call the RotateSecret API Action (docs):
   Trigger rotation of the secret by calling the Secrets Manager API from the Lambda function.

For more like this, please follow me on Medium and Twitter.