All-in-one serverless DevOps platform.
Full-stack visibility across the entire stack.
Detect and resolve incidents in record time.
Conform to industry best practices.
That is a common question I see among developers. Most of the time, nobody cares about system logs. But when things go south, we absolutely need them. Like water in the desert, sometimes!
At Dashbird, we have a list of criteria compiled to determine a reasonable retention policy for application logs. There is no one-size-fits-all, though. The analytical dimensions below will give a relative notion of how long the retention period should be.
Different retention policies for each part of the system can be set depending on how important they are. Mission-critical components can have an extended period to provide extra peace of mind. Services with a marginal value may have logs dumped in two days, for example.
Applications that deal with sensitive or personal data, as well as high-risk processing, should also have an extended retention policy. Services taking care of credit card authorization and user authentication are some examples.
Each system requires its own analysis. A ride-hailing app, for instance, may need to keep trip logs for a certain period to act on any issues reported later. Services that track customer behavior on the app for improvement purposes, on the other hand, might not need a longer retention policy.
Internet services that can be used for illegal activity, such as payment processors or online gaming, would need an extended or even perpetual retention policy to comply with law enforcement.
That could also be true for companies processing personal data under privacy regulations, as they might need to produce evidence of consent.
Accounting and tax compliance software is also likely to be subject to longer retention policies – possibly for years – as government authorities and corporate auditors may be entitled to conduct retroactive analysis.
For systems that have been around for a while and are not under constant development of features, new issues will rarely happen. Therefore, in mature software systems, a shorter retention policy might be acceptable to reduce storage costs.
Applications that run infrequently (i.e. once every month) should have extended retention policies, as developers might need to verify multiple executions to track down the source of an issue. The low execution frequency would require going back further in time to conduct the debugging.
Before settling on a retention policy, make sure the cost will be compatible with the project. Project how much data should be generated and how expensive storage is for the intended period. Even though we might want to be safe and store for a long time, it might be reasonable to settle on a more cost-effective option.
Track how long it usually takes for your development team to discover and resolve an issue. Make sure the log retention policy gives enough room for debugging.
One way to reduce the average time to discovery and resolution of application issues is by employing automated alerting systems and having an efficient way for developers to quickly browse logs and narrow down the root causes of errors.
Today we are excited to announce scheduled searches – a new feature on Dashbird that allows you to track any log event across your stack, turn it into time-series metric and also configure alert notifications based on it.
One of the most vital aspects to monitor is the metrics. You should know how your cluster performs and if it can keep up with the traffic. Learn more about monitoring Amazon OpenSearch Service.
Dashbird recently added support for ELB, so now you can keep track of your load balancers in one central place. It comes with all the information you expect from AWS monitoring services and more!
Dashbird was born out of our own need for an enhanced serverless debugging and monitoring tool, and we take pride in being developers.
Dashbird gives us a simple and easy to use tool to have peace of mind and know that all of our Serverless functions are running correctly. We are instantly aware now if there’s a problem. We love the fact that we have enough information in the Slack notification itself to take appropriate action immediately and know exactly where the issue occurred.
Thanks to Dashbird the time to discover the occurrence of an issue reduced from 2-4 hours to a matter of seconds or minutes. It also means that hundreds of dollars are saved every month.
Great onboarding: it takes just a couple of minutes to connect an AWS account to an organization in Dashbird. The UI is clean and gives a good overview of what is happening with the Lambdas and API Gateways in the account.
I mean, it is just extremely time-saving. It’s so efficient! I don’t think it’s an exaggeration or dramatic to say that Dashbird has been a lifesaver for us.
Dashbird provides an easier interface to monitor and debug problems with our Lambdas. Relevant logs are simple to find and view. Dashbird’s support has been good, and they take product suggestions with grace.
Great UI. Easy to navigate through CloudWatch logs. Simple setup.
Dashbird helped us refine the size of our Lambdas, resulting in significantly reduced costs. We have Dashbird alert us in seconds via email when any of our functions behaves abnormally. Their app immediately makes the cause and severity of errors obvious.
End-to-end observability and real-time error tracking for AWS applications.