Another Way to Trigger a Lambda Function Every 5–10 Seconds
This is a follow up to my recent post titled How to Trigger a Lambda Function Every 5–10 Seconds. I recommend you read that too, but it’s not required.
As is often the case in software, this problem has more than one possible solution. One awesome thing about serverless is how we can glue services together in creative ways to solve the same problem differently. Jeremy Thomerson did just that when he described an alternate solution.
Jeremy proposed using a Step Function to provide the second-level precision instead of the Lambda/SQS combination used in my previous post. The Step Function itself is similarly triggered by an EventBridge rule.
Like last time, I’ve set this up and analysed the results.
Overview: serverless.yml
I once again used Serverless Framework to put everything together, but this time I’ve added the Serverless Step Functions plugin to make life easier.
I’ll go into more detail next, but the big changes are that everyminute is now a Step Function (instead of a Lambda function) and the SQS queue is gone.
The solution is cleaner overall (it has a few more lines of YAML, but the code in everyminute.js is gone).
Step Function: everyminute
Let me give you something better to look at instead of YAML. This is a screenshot from AWS’ recently released “Step Functions Workflow Studio”.
I’ve built this to be as comparable as possible to the Lambda/SQS solution, but you could do it other ways (see Enhancements).
I’m using a Pass state to initialise an array containing 6 numbers. The numbers themselves aren’t actually important, I’m just using them to make the Map state loop 6 times.
Importantly, I’ve set
MaxConcurrencyto1on the Map state, otherwise it would run multiple threads and invoke the Lambda function in parallel.
The Map state iterator waits 10 seconds using a Wait state, then invokes the interval Lambda function asynchronously (it doesn’t wait for it to complete before moving on to the next loop).
Function: interval
I’ve removed the SQS event trigger and the reserved concurrency, but the code is unchanged; it still simply logs its input and the current time.
How accurate is it?
EventBridge
The results in my previous post show that EventBridge rules appear to trigger on the same second of every minute, give or take ~500 milliseconds.
Step Functions
The Wait state offers second-level accuracy and the AWS console shows it waits for exactly 10,000 milliseconds. State transitions take ~10ms.
The following graph shows how far away from “every 10 seconds” the invocations were over an hour. I’d call this within a second, usually 500ms.
Most invocations seem to have happened ~50ms late. If they weren’t late, they were 250–500ms early (with a couple of larger spikes).
The graph below compares these results to the previous ones. Very similar!
Cost
Lambda & SQS
I didn’t actually publish the cost of the other solution, so let’s look at it now.
The cost is made up of:
everyminuteinvocation time +everyminutesending messages to SQS +- Lambda polling SQS for
interval.
I’m omitting the tiny SQS storage cost and assuming a 30-day month.
The everyminute function had an average billed duation of 25ms and I was running it at 256MB of memory. 43,200 invocations a month, priced at $0.000000004 per millisecond, means $0.004536/month for everyminute invocation time.
everyminute sends 6 messages to SQS each time it’s invoked (259,200 messages a month). If we ignore the free tier and use $0.40 per million requests, everyminute sending messages to SQS costs $0.103687/month.
Lambda long polls with five processes. Let’s say four of them never pick up messages, so they each make 3 requests per minute. One process picks up a message every 10 seconds, so it makes 6 requests per minute. Polling requests are priced the same as sending messages, so Lambda polling SQS for interval costs $0.31104 per month.
In total, that’s $0.419263/month. I’ll call that 42 cents.
Step Functions
As Jeremy said in his tweet, using an Express Step Functions is the cheapest option. The Express pricing model charges $0.000001 for each execution and $0.00001667 per GB-second its running.
I have the Step Function being executed every minute (43,200/month) which costs $0.0432.
Small Step Functions like this one use 64MB of memory, so $0.000001042 per second. Since the Step Function is effectively running constantly, that means it costs $2.700864/month.
In total, that’s $2.744064/month. I’ll call that $2.75.
For comparison, a Standard Step Function could cost ~$13/month.
Enhancements
The most interesting enhancement is one Jeremy already pointed out: Express Step Functions can run for up to 5 minutes. That means you could change the EventBridge rule to trigger every 5 minutes instead of every minute. The Step Function would then invoke the Lambda function 30 times instead of 6 times.
This would save 3 cents on per-execution costs. In my opinon, this isn’t worth it for 3 reasons (one per cent?):
- If an execution fails for unexpected reasons, I’d prefer to retry it the next minute instead of having up to 5 minutes without invoking my function.
- Running the Step Function right up to the 5 minute limit seems risky. It is hard to reason about the Wait states, transition times, and Lambda service response times to make sure it all fits.
- Due to transition times and Lambda service response times, I think that the longer the Step Function runs, the more the invocation times would move around. I like the reset given by starting a new execution.
Maybe you could do 3 or 4 minutes if you really wanted, but I still don’t think it’s worth it.
Similar to the previous solution, if you wanted to fan this out to invoke multiple functions, you could make the Step Function directly publish an SNS message instead of invoking just one function directly. Be aware that this will add some latency.
Conclusion
This solution is great! Sure, it costs 6.5 times more, but it’s on a small enough scale that I suspect the extra $2.33 isn’t going to break the bank. If you wanted to invoke your function even more often, the price gap would close very slightly.
The accuracy is slightly poorer, but it’s not far off. There are less moving parts and no custom Lambda code to miss bugs in. Finally, I think this solution is a little easier to understand.
Which one should you use? Either, really. If cost (or accuracy?) is very important to you then go with the previous solution. Otherwise, use this Step Function solution.
All of the code shown here is available on GitHub.
