Debugging with Dashbird: Lambda not logging to CloudWatch

Lambda not logging to CloudWatch? It’s actually one of the most common issues that come up. Let’s briefly go over why this problem needs to be solved.

CloudWatch is the central logging and monitoring service of the AWS cloud platform. It gives you insights into all the AWS services. Even if you can’t deploy and test serverless systems locally, CloudWatch tells you what’s happening to them.

Dashbird is built on top of CloudWatch; it lets Dashbird monitor Lambda functions without any code changes. You just install Dashbird’s CloudFormation stack to your AWS account, and the insights will pour right in.

Regardless of whether you use Dashbird or just CloudWatch on its own, a Lambda function that isn’t logging is a critical problem. So, in this article, we will try to solve this problem once and for all.

Why is Lambda not logging to CloudWatch?

Like with any bug, this one can have multiple causes.

The obvious one is a bug in your code that prevents the log function from being executed in the first place. The less obvious, but still likely, cause is that your Lambda function doesn’t have permissions to write log data to CloudWatch. This usually happens when you created your own custom IAM role and forgot to add CloudWatch permissions.

Lambda, like every other AWS service, is governed by IAM roles and policies. If you don’t give a Lambda function permission to access other services, the only thing it can do is working on the event data it received.

This is basically like writing a program without any side effects: not helpful at all.

For a Lambda function to be interesting, it needs to write data somewhere. This includes, but is not limited to, creating an object in an S3 bucket or writing a record to DynamoDB and logging strings to CloudWatch.

AWS Lambda monitoring

How do I fix it?

First, you should check your code. IaC frameworks like AWS SAM and AWS CDK are usually pretty good in keeping your Lambda functions supplied with sane permissions. If you don’t know what you’re doing, you can create security vulnerabilities. Also, if you create a custom role for CloudWatch logging, it won’t allow access to services that your Lambda function might need to do its real work, like DynamoDB or S3. This means you need to include permissions for those services manually too.

If you have some logic errors in your code, it’s possible that your log statements simply aren’t ever reached. So review your code thoroughly and try a simple “Hello, world!” log output function that doesn’t do anything else before diving into custom IAM roles.

If you already went the way of custom IAM roles for CloudWatch unrelated reasons, it’s possible that you simply forgot to add the right permissions for logging.

In your “infrastructure as code” (IaC) tool, you need to create a custom IAM role for your Lambda function. The function will assume this role then it can access the services defined in its policies.

How to define a custom IAM Role for Lambda?

Let’s look at an AWS SAM example:

Resources:
  CustomRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: my-lambda-role
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: 'sts:AssumeRole'
            Principal:
              Service: lambda.amazonaws.com
      Policies:
        - PolicyName: WriteLogs
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: 'arn:aws:logs:*:*:*'
  SomeFunction:
    Type: AWS::Serverless::Functionn
    Properties:
      Runtime: nodejs14.x
      CodeUri: path/to/lambda/code
      Handler: index.handler
      Role: !GetAtt CustomRole.Arn

Let’s look at the important parts. First, we create a CustomRole resource of the type AWS::IAM::Role. Only Lambda functions can assume this role. Next, we give it a WriteLogs policy that includes all the actions we need to write to CloudWatch Logs. The function needs to create log groups and log streams and then be able to put the actual log events into them.

After the role is defined, we connect it with SomeFunction; this gives all the role’s permissions to this specific Lambda function.

The implementation details may differ between the different IaC tools out there, but they all have some way to define IAM roles and assign them to Lambda functions.

After you defined the role and assigned it to the Lambda function, you redeploy everything, and your logs should now show up in CloudWatch.

Again, keep in mind that you now have to add permissions to other services manually to the custom role; otherwise, the only thing your Lambda function is allowed to do is logging.

Conclusion

CloudWatch is a crucial service for all your cloud resources. If you want to know what’s happening, you need to send your log data to CloudWatch first.

Dashbird, too, uses CloudWatch as a data source for all the insights it creates for you. If no data ends up in CloudWatch, your Lambda functions will be black boxes to you.

Check your code for logic errors, know that it really executes log statements, and have custom IAM roles already; add CloudWatch policies to them.

If you fixed your logging issue and want to be safe from problems in the future, you should try out Dashbird for serverless monitoring. It doesn’t require code changes to work with your existing infrastructure, and even gives you insights into problems before they occur.


Further reading:

The Best Practices for Logging AWS Lambdas

AWS Lambda Logging: Error Types

AWS Lambda configuration error

Read our blog

Introducing easy custom event monitoring for serverless applications.

Today we are excited to announce scheduled searches – a new feature on Dashbird that allows you to track any log event across your stack, turn it into time-series metric and also configure alert notifications based on it.

Why and How to Monitor Amazon OpenSearch Service

One of the most vital aspects to monitor is the metrics. You should know how your cluster performs and if it can keep up with the traffic. Learn more about monitoring Amazon OpenSearch Service.

Why and How to Monitor AWS Elastic Load Balancing

Dashbird recently added support for ELB, so now you can keep track of your load balancers in one central place. It comes with all the information you expect from AWS monitoring services and more!

More articles

Made by developers for developers

Dashbird was born out of our own need for an enhanced serverless debugging and monitoring tool, and we take pride in being developers.

What our customers say

Dashbird gives us a simple and easy to use tool to have peace of mind and know that all of our Serverless functions are running correctly. We are instantly aware now if there’s a problem. We love the fact that we have enough information in the Slack notification itself to take appropriate action immediately and know exactly where the issue occurred.

Thanks to Dashbird the time to discover the occurrence of an issue reduced from 2-4 hours to a matter of seconds or minutes. It also means that hundreds of dollars are saved every month.

Great onboarding: it takes just a couple of minutes to connect an AWS account to an organization in Dashbird. The UI is clean and gives a good overview of what is happening with the Lambdas and API Gateways in the account.

I mean, it is just extremely time-saving. It’s so efficient! I don’t think it’s an exaggeration or dramatic to say that Dashbird has been a lifesaver for us.

Dashbird provides an easier interface to monitor and debug problems with our Lambdas. Relevant logs are simple to find and view. Dashbird’s support has been good, and they take product suggestions with grace.

Great UI. Easy to navigate through CloudWatch logs. Simple setup.

Dashbird helped us refine the size of our Lambdas, resulting in significantly reduced costs. We have Dashbird alert us in seconds via email when any of our functions behaves abnormally. Their app immediately makes the cause and severity of errors obvious.