Does Maximum Concurrency Solve the Lambda+SQS Issue?
In January 2023, AWS announced a new for Lambda called Maximum Concurrency. Shortly after the announcement, I was asked if it’s a solution to the “SQS overpull” problem that I first wrote about way back in 2019.
The answer is yes, it is a solution!
However, You still need to be careful. This blog post will discuss the things you must still watch out for.
You can find my original post and June 2019 follow-up here:
What has changed
AWS Lambda uses event source mappings to pull messages from your SQS queue and invoke your Lambda function. Since their launch in 2018, they have gained features such as filtering and batching. This week, AWS gave them a new setting called maximum concurrency.
When you set maximum concurrency to N, the event source mapping will only invoke your function N times concurrently.
When the maximum concurrency is reached, the event source mapping will stop reading messages from your SQS queue until an invocation finishes.
This means the event source mapping no longer reads messages that it won’t be able to send to your function. Therefore, there shouldn’t be any unprocessed sent back to the queue.
How well does it work?
The announcement blog post shares a GitHub repo containing a SAM app.
The app deploys a Lambda function that simply sleeps for 10 seconds, an event source mapping with its maximum concurrency to 5, and an SQS queue with a 20 second visibility timeout. The SQS queue is configured so that the event source mapping/function gets one attempt to process each message before its sent to a dead-letter queue.
I’ve deployed this demo app myself and can confirm it works as advertised.
Setting maximum concurrency to 5 seemed a little suspicious because event source mappings start polling with 5 concurrent threads and maximum concurrency can be set below that at 2.
The question is, will the event source mapping still take 5 messages off the queue before considering a maximum concurrency of less than 5?
I changed the demo app around and found that maximum concurrency works as expected even when set to less than 5.
But wait, watch out!
In my previous posts, I described the SQS overpull issue as being related to setting reserved concurrency on a Lambda function as the most common cause. That is because event source mappings have no idea about reserved concurrency so they will happily read more messages off your queue than they can send to your function.
However, to generalise the cause, the issue happens when an event source mapping is unable to invoke your Lambda function, for any reason, after it has already taken a message of the queue. The most common reason for being unable to invoke your function is not enough available concurrency.
Put simply, if maximum concurrency is higher than the available concurrency, you will still experience the SQS overpull issue.
In the rest of this post, I’ll discuss what impacts available concurrency.
Concurrent execututions service quota
Service quotas (colloquially called service limits) are a fact of life when building on AWS.
Lambda has a quota called concurrent executions which controls the “maximum number of events that functions can process simultaneously”.
The default value for concurrent executions is 1,000, but AWS accounts don’t always start with the default value. I have seen applied values of 50 and even 10, which can be a real surprise.
An event source mapping’s maximum concurrency can be set between 2 and 1,000. If your concurrent executions quota is 10 and you set maximum concurrency to 20, you will still experience the overpull issue.
The concurrent executions quota is the starting point. You’ll never have more available concurrency than this and you will usually have less.
Reserved concurrency
Every function you configure with reserved concurrency eats away at your available concurrency.
If you have a concurrent executions quota of 100 and 90 of it is reserved, you have 10 available concurrency left for other event source mappings.
In that situation, setting maximum concurrency to 20 on a new event source mapping is likely to still cause the overpull issue.
Consumed concurrency
Continue imagining your account has a concurrent executions quota of 100. In this scenario, you have 50 concurrency reserved for some critical functions that must be able to run. That leaves 50 concurrency for other functions.
When none of the “other functions” are running, the account’s available concurrency is 50. When one of the “other functions” is invoked once, the available concurrency drops to 49.
At any point in time, the available concurrency in this account will be between 50 and 0.
If you create three functions, each with an event source mapping that has maxomum concurrency set to 20, it is possible that all three will be fully saturated and try to consume 60 concurrency. That is more than the available concurrency, so one or more of the event source mappings will experience the overpull issue.
Multiple event sources
It’s not just multiple functions with their own event source mappings that consume available concurrency. It’s also not just event source mappings.
If your account has a quota of 10 concurrent executions, you could cause the overpull issue by creating a single function and two event source mappings that invoke the same function. If the maximum concurrency across both functions adds up to a value higher than 10, you will experience the overpull issue. Event source mappings don’t intelligently communicate with each other just because they invoke the same function.
Moreover, imagine you have a 10 concurrent executions quota, one function, and one event source mapping with maximum concurrency set to 10.
The works as expected, but be aware that not only can you not add another event source, such as an API Gateway or SNS topic subscription, but you also can’t safely invoke the function with the CLI, API, or AWS Console.
Any of these actions will lower the available concurrency below 10 and will cause the overpull issue if your event source mapping is fully saturated at the same time.
Conclusion
The new maximum concurrency feature is a real solution and is exactly what the AWS community has been wanting for years.
Now that it’s here, you should definitely use it, but just be aware that you can still experience the SQS overpull issue if your available concurrency reaches zero.
At any point in time, available concurrency is (concurrent executions quota) - (total reserved concurrency) - (number of running invocations).
