Why Serverless Apps Fail and How to Design Resilient Architectures?

Sign up to our upcoming webinar with AWS to learn more about how to use observability to build resilient Well-Architected serverless applications.

RSVP and more info here:

serverless webinar with AWS

We’ve been monitoring 100,000’s of serverless backend components for 3+ years at Dashbird. In our experience, Serverless infrastructure failures boil down to:

  • Throughput and concurrency limitations;
  • Increased latency;
  • Timeout errors;

These isolated faults become causes of failure due to dependencies in our cloud architectures (ref. Difference of Fault vs. Failure). If a serverless Lambda function relies on a database that is under stress, the entire API may start returning 5XX errors.

You may think this is just a fact of life, but we can dodge or at least mitigate these failures in many cases.

Serverless is not a magical silver bullet. These services have their limitations, especially to scalability capacities. AWS Lambda, for example, can increase concurrency level up to a certain level per minute. Throw in 10,000 concurrent requests out of thin air and it will throttle.

A typical architecture looks like this:

It usually works well under a low scale. Put in more load a single component’s fault can bring the whole implementation to its knees.

Consider this scenario: due to market reasons, API Endpoint 1 starts receiving an unusual amount of requests. Your clients are generating more data and your backend needs to store it in the RDS instance. Relational databases usually don’t scale linearly to I/O level, so we can expect an increase in query latency during this peak demand. API Endpoint 1 or Lambda function 1 will start timing out at some point due to the database delays.

Another possible fault scenario is throttling from Lambda function 1 due to a rapid increase in concurrency.

Not only API Endpoint 1 will become unavailable to clients, but also the second endpoint. In the first scenario, Endpoint 2 also relies on the same RDS instance. In the second scenario, Lambda function 1 will consume the entire concurrency limits for your AWS account, causing Lambda function 2 to throttle requests as well.

AWS Lambda monitoring

We can avoid this by decoupling the API Endpoint 1and Lambda function 1. In the example, our clients are only sending information that needs to be stored, but no processing and customized response are needed. Here is an alternative architecture:

Instead of sending requests directly from API Endpoint 1 to the Lambda function 1, we first store all requests in a highly-scalable SQS queue. The API can immediately return a 200 message to clients. The Lambda function 1 will later pull messages from the queue in a rate that is manageable for its own concurrency limits and the RDS instance capabilities.

With this modification, the potential for widespread failure is minimized by having a queue absorbing peaks in demand. SQS standard queues can handle nearly unlimited throughput. At the same time, all components serving Endpoint 2 can continue to work normally, since data consumption by the Lambda function 1 is smoothed out.

Further reading:

Solving the Challenges of Serverless at Scale

Serverless scalability and concurrency

Log-based monitoring for AWS Lambda

Read our blog

8 Must-Know Tricks to Use S3 More Effectively in Python

The simplicity and scalability of S3 has made it a go-to platform not only for storing objects, but also to host them as static websites, serve ML models, provide backup functionality, and so much more. In this article, we’ll look at various ways to leverage the power of S3 in Python.

Monitoring serverless applications with AWS CloudWatch alarms

Running any application in production assumes reliable monitoring to be in place and serverless applications are no exception. Here’s how to do that.

10x development speed with local serverless debugging

In the development world, the ultimate goal is to increase the business value that you can add to your product. In this article, you’ll find out how to 10x your development speed with local serverless debugging and reach that goal faster.

Made by developers for developers

Dashbird was born out of our own need for an enhanced serverless debugging and monitoring tool, and we take pride in being developers.

What our customers say

Dashbird gives us a simple and easy to use tool to have peace of mind and know that all of our Serverless functions are running correctly. We are instantly aware now if there’s a problem. We love the fact that we have enough information in the Slack notification itself to take appropriate action immediately and know exactly where the issue occurred.

Thanks to Dashbird the time to discover the occurrence of an issue reduced from 2-4 hours to a matter of seconds or minutes. It also means that hundreds of dollars are saved every month.

Great onboarding: it takes just a couple of minutes to connect an AWS account to an organization in Dashbird. The UI is clean and gives a good overview of what is happening with the Lambdas and API Gateways in the account.

I mean, it is just extremely time-saving. It’s so efficient! I don’t think it’s an exaggeration or dramatic to say that Dashbird has been a lifesaver for us.

Dashbird provides an easier interface to monitor and debug problems with our Lambdas. Relevant logs are simple to find and view. Dashbird’s support has been good, and they take product suggestions with grace.

Great UI. Easy to navigate through CloudWatch logs. Simple setup.

Dashbird helped us refine the size of our Lambdas, resulting in significantly reduced costs. We have Dashbird alert us in seconds via email when any of our functions behaves abnormally. Their app immediately makes the cause and severity of errors obvious.