What is the ideal retention period for application logs

What is the Ideal Retention Period for Application Logs?

That is a common question I see among developers. Most of the time, nobody cares about system logs. But when things go south, we absolutely need them. Like water in the desert, sometimes!

At Dashbird, we have a list of criteria compiled to determine a reasonable retention policy for application logs. There is no one-size-fits-all, though. The analytical dimensions below will give a relative notion of how long the retention period should be.

Criticality

Different retention policies for each part of the system can be set depending on how important they are. Mission-critical components can have an extended period to provide extra peace of mind. Services with a marginal value may have logs dumped in two days, for example.

Security

Applications that deal with sensitive or personal data, as well as high-risk processing, should also have an extended retention policy. Services taking care of credit card authorization and user authentication are some examples.

Each system requires its own analysis. A ride-hailing app, for instance, may need to keep trip logs for a certain period to act on any issues reported later. Services that track customer behavior on the app for improvement purposes, on the other hand, might not need a longer retention policy.

Law-Enforcement

Internet services that can be used for illegal activity, such as payment processors or online gaming, would need an extended or even perpetual retention policy to comply with law enforcement.

That could also be true for companies processing personal data under privacy regulations, as they might need to produce evidence of consent.

Accounting and tax compliance software is also likely to be subject to longer retention policies – possibly for years – as government authorities and corporate auditors may be entitled to conduct retroactive analysis.

Maturity

For systems that have been around for a while and are not under constant development of features, new issues will rarely happen. Therefore, in mature software systems, a shorter retention policy might be acceptable to reduce storage costs.

Frequency

Applications that run infrequently (i.e. once every month) should have extended retention policies, as developers might need to verify multiple executions to track down the source of an issue. The low execution frequency would require going back further in time to conduct the debugging.

Cost-effectiveness

Before settling on a retention policy, make sure the cost will be compatible with the project. Project how much data should be generated and how expensive storage is for the intended period. Even though we might want to be safe and store for a long time, it might be reasonable to settle on a more cost-effective option.

Time to Discovery and Resolution

Track how long it usually takes for your development team to discover and resolve an issue. Make sure the log retention policy gives enough room for debugging.

One way to reduce the average time to discovery and resolution of application issues is by employing automated alerting systems and having an efficient way for developers to quickly browse logs and narrow down the root causes of errors.

Read our blog

4 Tips for AWS Lambda Performance Optimization

In this article, we’re covering 4 tips for AWS Lambda optimization for production. Covering error handling, memory provisioning, monitoring, performance, and more.

AWS Lambda Free Tier: Where Are The Limits?

In this article we’ll go through the ins and outs of AWS Lambda pricing model, how it works, what additional charges you might be looking at and what’s in the fine print.

AWS Lambda vs EC2: A Comparative Guide

More articles

Made by developers for developers

Dashbird was born out of our own need for an enhanced serverless debugging and monitoring tool, and we take pride in being developers.

What our customers say

Dashbird gives us a simple and easy to use tool to have peace of mind and know that all of our Serverless functions are running correctly. We are instantly aware now if there’s a problem. We love the fact that we have enough information in the Slack notification itself to take appropriate action immediately and know exactly where the issue occurred.

Thanks to Dashbird the time to discover the occurrence of an issue reduced from 2-4 hours to a matter of seconds or minutes. It also means that hundreds of dollars are saved every month.

Great onboarding: it takes just a couple of minutes to connect an AWS account to an organization in Dashbird. The UI is clean and gives a good overview of what is happening with the Lambdas and API Gateways in the account.

I mean, it is just extremely time-saving. It’s so efficient! I don’t think it’s an exaggeration or dramatic to say that Dashbird has been a lifesaver for us.

Dashbird provides an easier interface to monitor and debug problems with our Lambdas. Relevant logs are simple to find and view. Dashbird’s support has been good, and they take product suggestions with grace.

Great UI. Easy to navigate through CloudWatch logs. Simple setup.

Dashbird helped us refine the size of our Lambdas, resulting in significantly reduced costs. We have Dashbird alert us in seconds via email when any of our functions behaves abnormally. Their app immediately makes the cause and severity of errors obvious.