What is the Ideal Retention Period for Application Logs?
That is a common question I see among developers. Most of the time, nobody cares about system logs. But when things go south, we absolutely need them. Like water in the desert, sometimes!
At Dashbird, we have a list of criteria compiled to determine a reasonable retention policy for application logs. There is no one-size-fits-all, though. The analytical dimensions below will give a relative notion of how long the retention period should be.
Different retention policies for each part of the system can be set depending on how important they are. Mission-critical components can have an extended period to provide extra peace of mind. Services with a marginal value may have logs dumped in two days, for example.
Applications that deal with sensitive or personal data, as well as high-risk processing, should also have an extended retention policy. Services taking care of credit card authorization and user authentication are some examples.
Each system requires its own analysis. A ride-hailing app, for instance, may need to keep trip logs for a certain period to act on any issues reported later. Services that track customer behavior on the app for improvement purposes, on the other hand, might not need a longer retention policy.
Internet services that can be used for illegal activity, such as payment processors or online gaming, would need an extended or even perpetual retention policy to comply with law enforcement.
That could also be true for companies processing personal data under privacy regulations, as they might need to produce evidence of consent.
Accounting and tax compliance software is also likely to be subject to longer retention policies - possibly for years - as government authorities and corporate auditors may be entitled to conduct retroactive analysis.
For systems that have been around for a while and are not under constant development of features, new issues will rarely happen. Therefore, in mature software systems, a shorter retention policy might be acceptable to reduce storage costs.
Applications that run infrequently (i.e. once every month) should have extended retention policies, as developers might need to verify multiple executions to track down the source of an issue. The low execution frequency would require going back further in time to conduct the debugging.
Before settling on a retention policy, make sure the cost will be compatible with the project. Project how much data should be generated and how expensive storage is for the intended period. Even though we might want to be safe and store for a long time, it might be reasonable to settle on a more cost-effective option.
Time to Discovery and Resolution
Track how long it usually takes for your development team to discover and resolve an issue. Make sure the log retention policy gives enough room for debugging.
One way to reduce the average time to discovery and resolution of application issues is by employing automated alerting systems and having an efficient way for developers to quickly browse logs and narrow down the root causes of errors.