Best Practices For Logging In AWS Lambda

Taavi Rehemägi

April 6th, 2021

Share

Today, we’ll cover some of the ways you might find quite useful in your everyday work. We’ll go through some of the logging best practices in AWS Lambda, and we will explain how and why these ways will simplify your AWS Lambda logging. For more information about similar topics, be sure to visit our blog.

Let’s start with the basics (and if you have the basics covered, feel free to skip ahead): How does logging work with AWS Lambda?

Logging In AWS Lambda

The AWS Lambda service will automatically monitor Lambda functions for you and report the metrics to Amazon CloudWatch. To help you resolve the function failures, Lambda logs will record your function’s requests as well.

It will also store logs generated by your code through Amazon CloudWatch Logs. That’s right; you can put logging statements in your code to validate that your code is working correctly and as expected. The Lambda service integrates with CloudWatch Logs automatically, and it pushes all the logs from your code to a CloudWatch Logs group associated with a Lambda function.

This is a beneficial feature when debugging, but logging can also be a security risk. You need to be very careful about what you log and not forget to remove debugging log statements from your code when deploying to production. Otherwise, sensitive data could end up in CloudWatch.

Tip: Don’t forget to refactor your log statements! The idea is to have a seamless feedback loop between production logs and the modification of log statements. It may be an arduous task but this is crucial for CI/CD.

When logging from a Lambda function to CloudWatch, you don’t pay any additional charges, only the standard CloudWatch rates.

Centralize The Logging In AWS Lambda

There are various log aggregation services — why do we need them? 

When you deploy a new Lambda function, you’ll automatically create a CloudWatch log group with it. With hundred Lambda functions, you’ll end up with a hundred log groups. This can make keeping track of what’s happening a bit cumbersome.

Tip: Observability services help you move the Lambda logs from our existing CloudWatch Logs to a more centralized and user-friendly log storage.

But keep AWS’ egress traffic costs in mind. While it’s free to get data into AWS, you have to pay to get it out again. Also, to get your logs out of AWS, you need a Lambda function that ships your logs to the third-party service. This Lambda function isn’t free, so you should keep parallel invocations in mind.

Let’s say that you have a hundred functions that run simultaneously. Every one of them will push logs to CloudWatch Logs. This can trigger a hundred simultaneous executions of the log shipping function.

If you call the log shipping function for every other Lambda function invocation, this can double the number of simultaneously running functions in your region. There is a soft, regional limit of thousand simultaneous executions for all functions.

You can also buy reserved concurrency for the log shipping function, which will limit its maximum number of simultaneous executions. Still, there is a risk of losing logs when the log shipping functions are suppressed.

Streaming Logs

Take a look at the CloudWatch console, and you’ll see under the “Actions” tab an option to select a log group, one for every Lambda function. Then you can decide whether you should stream the data directly to Amazon’s hosted Elasticsearch service. In case you’re already a user of Elasticsearch service, this will come in quite handy, but if you’re still thinking about the possible options, there are alternatives.

A more economical way to get your logs out of AWS would be to stream them from CloudWatch Logs to a Kinesis stream first because, from there, a Lambda function can process logs in batches and forward them toward a log aggregation service of your choosing.

With Kinesis, you have complete control over the log shipping function’s concurrency. If your log event rate rises, you can increase the shard number inside the Kinesis stream, leading to an increased number of simultaneous log shipping function executions.

When creating a new function from a Lambda console, you’re able to choose from several blueprints to push CloudWatch Logs to some other log aggregation service. This option enables you to write a Lambda function that will ship your CloudWatch Logs to a preferred log aggregation service, but there are additional things you should be careful about.

New Log Groups Auto-Subscription

Every time you create a new Lambda function, the Lambda service automatically forms a new log group in CloudWatch Logs. What you want to avoid is a manual process of subscribing to log groups from your shipping function.

Enabling CloudTrail and setting up an event pattern in CloudWatch Events will allow you to invoke different Lambda functions every time a log group is created. This way, you can create a Lambda function that automatically subscribes your shipping function to new log groups created. You can do this setup manually from the CloudWatch console. In case you’re working with several different AWS accounts, it’s advisable to avoid making a manual configuration.

The Serverless Framework allows you to set up the event source for this function in the serverless.yml file. Avoid subscribing to the log group for the log shipping function since it will create an infinite invocation loop.

Log Retention Best Practices to Save Costs

When Lambda creates a new log group for your function, the retention policy keeps them forever. This is unnecessary because log storage can be quite expensive since the logs add up over time. Luckily, you can add another Lambda function that can automatically update the retention policy to something much more adequate.

You can also dig into log retention best practices since it’s a topic which you will need to master.

Dashbird — Free Serverless Logging and Monitoring

With Dashbird, you’ll be able to do a lot with little effort. Dashbird collects log data directly via the CloudWatch APIs. This means Dashbird doesn’t need to integrate with your Lambda function on a code level — it can simply read out every Lambda function CloudWatch logs right from the start.

Try Dashbird for free

This means there is no overhead when using Dashbird, which is another way of saying that you won’t incur extra costs, nor will your app be slowed down by extra requests.

Another perk offered by Dashbird is Log Search, which can be set up to receive your function’s log streams in real-time. X-Ray is another benefit provided by Dashbird, whose seamless integration gives you the best tracing possibilities. X-Ray tracing when X-Ray is enabled is yet another benefit provided by Dashbird where our seamless integration allows you to keep a close eye on what your invocation is doing.

AWS X-Ray is a potent tool that enables you to trace the data throughout the invocation lifecycle while it also serves as a tool to track the full extent of an invocation flow. With custom traces, X-Ray allows you to track data manually.

It’s also worth mentioning that Dashbird can have an automated system of detecting if the X-Ray option is enabled. In case you start an invocation that belongs to a Lambda function with active X-Ray monitoring, it can get X-Ray data on-demand while it will also show you the results. With X-Ray integration, Dashbird can tell you how and when a Lambda function wastes time so that you can improve it in the future.

Where does that leave us?

Depending on your needs, you should either ship your logs to a log aggregation service directly or stream your logs using Kinesis log streaming. Be careful when choosing a log aggregate service since they all have their pros and cons.

Read users’ comments and their experiences of third-party services to get a feeling of their ups and downs, so you’ll know upfront:

• What you can expect from every single aggregation service; and
• Generally how to avoid situations you don’t want to end up in.

Want to try Dashbird’s logging for AWS Lambda? It’s forever free to use for up to 1M monthly Lambda executions. 2-minute set-up, no credit card required and no code changes.