Monitoring platform for keeping systems up and running at all times.
Full stack visibility across the entire stack.
Detect and resolve any incident in record time.
Conform to industry best practices.
Dashbird got an update, and you can now monitor the OpenSearch clusters you set up with Amazon OpenSearch Service. But what does this even mean? Many people don’t even know what OpenSearch is! Wasn’t there an Elasticsearch service before? So, let’s have some explanation before we check out Dashbirds OpenSearch support.
After a heated argument between Elastic, the main contributor to the Elasticsearch (ES) project, and AWS. Elastic then changed the ES license to prohibit AWS from offering their Amazon Elasticsearch Service. Since the ES versions previous to that change were still under the old Apache 2.0 license, AWS created a fork from these and called it OpenSearch (OS).
To put it simply: Two companies disagreed on some software they implemented, and each one went on with their vision for that project. So, OS is merely a fork of ES, and these two open-source projects will probably diverge in the future in terms of compatibility and features.
In terms of managed search services, AWS deprecated Amazon Elasticsearch Service, and Amazon OpenSearch Service (AOS) is the way to go.
OS is a search engine, just like Google or Bing, but for your own data. It will index your data and make it searchable via an HTTP interface.
You can download OS and set it up on whatever computer you like. Install it on your local machine, set up an EC2 instance on AWS and install it there, or let AWS help you run it with the AOS.
They are pretty similar, but each one is on a different trajectory. Some developers aren’t happy with AWS treating Elastic the way they did. Others say switching to a less permissive license isn’t the open-source way, and while they don’t like how AWS might have treated Elastic, they think the Apache 2.0 license is the way to go if things should stay open.
If you want to use a managed, ES-like system on AWS, you either have to use older versions of ES or OS.
First, while AOS is a managed service, it’s not serverless; you have to pay for idle servers.
But what is AOS actually?
AOS is the service on AWS that manages OS deployments for you. Like RDS is for relational databases, EKS is for Kubernetes clusters, and Elasticsearch Service is for ES, so is AOS for OS.
While AOS is an excellent help to get OS running in the cloud, ES and, in turn, OS is notoriously hard to set up and scale. So hard that no serverless versions of them exist.
While AOS isn’t that easy to use, it’s still simpler than managing your OS instances manually. AOS helps with scaling clusters, deploying VMs, and managing the underlying network and storage for you. It will also install bug fixes.
If you need a cloud-based search engine for your project, you might not get around messing with OS and AOS.
When you deploy your infrastructure to the cloud, chances are most of the data you generate is operational logs. If you want to ease the access of these logs for the ops team, you might be inclined to pump them into AOS. But this can get costly.
As mentioned above, AOS isn’t a serverless system, so you pay for every hour it’s deployed, even if you don’t use it. This unused capacity could lead to a massive bill for nothing regarding operational monitoring data.
Cloudwatch Log Insights is a managed service on top of Cloudwatch logs and, in turn, optimized for this exact use-case. It scales automatically, and you only have to pay for the data you analyze, so it can be considered a serverless service.
AOS is a more low-level service you use if you need maximum flexibility. It costs more and isn’t easy to scale, but it gives you all the freedom you need for a customized search engine.
As explained before, AOS isn’t a serverless service. It can help with scaling, but it doesn’t go down to zero, so you will pay for the underlying EC2 instances idling.
This makes integrating AOS into your architecture a hard choice. But a hard choice between which options?
If money is tight, or you believe that a system should scale to zero for whatever reason, you have to ask yourself what your search requirements are.
AOS is the nuclear option here; it’s a Lucene-based distributed search engine that finds everything you indexed with Google-like features like fuzzy matching and such.
But if your search requirements aren’t that strong, you might get away with DynamoDB’s query features. Your results may vary and won’t be on OpenSearch levels, but it might be just enough for a simple search feature. But most importantly, you don’t pay for idle EC2 instances.
As said before, there are different methods of setting up OS on AWS, but we will go with the easiest here. Using the CDK, we will set up an AOS domain in just a few lines of code.
Keep in mind this example taken from the CDK docs will deploy a production-ready OS cluster containing 25 m3.medium instances, which can get expensive quickly.
25 m3.medium
const aosDomain = new opensearch.Domain(this, 'AosDomain', { version: opensearch.EngineVersion.OPENSEARCH_1_0, capacity: {masterNodes: 5, dataNodes: 20}, ebs: {volumeSize: 20}, zoneAwareness: {availabilityZoneCount: 3}, logging: { slowSearchLogEnabled: true, appLogEnabled: true, slowIndexLogEnabled: true, }, });
Coming from ES, you might have heard about a cluster and how it’s made up of one or more nodes. An AOS domain is such a cluster, plus all the things around running such a cluster that AOS takes care of for you. Like networking, updates, resource allocation, etc.
We must use the correct engine version to get an OS cluster because AOS also supports legacy ES clusters.
The nodes are EC2 instances of the default instance type m3.medium.search.
m3.medium.search
We use five master nodes, which means there will be one master and five master-eligible nodes which can take the place of the master if something goes wrong.
We also use twenty data nodes, which will be used to store all the index shards (i.e., the actual data) of our cluster. Each of these data nodes will get a 20 GiB elastic block storage.
The zone awareness makes sure AOS will deploy our 25 EC2 instances across three availability zones. This way, the cluster won’t go down if one zone becomes unavailable.
The logging config makes sure that queries and indices that perform poorly are logged so that when the cluster runs for some time, you get a feeling for the bottlenecks.
With the newest update to Dashbird, all AOS domains will be automatically picked up in your Dashbird inventory.
This update gives you a dashboard out of the box, with all the health metrics related to your domains. Including, but not limited to, the utilization of the different node types and the query latency.
While this is an excellent start to assess the status of your existing clusters in the context of their history, more interesting are the well-architected Dashbird insights.
The insights are periodical checks based on the AWS Well-Architected Framework. This means they test your AOS domains for best practices and notify you if you’re not following them.
The insights also include some recurring checks for the availability and usage of your AOS domains. After all, you have to pay for them if they’re deployed, and if they are never used, you might as well decommission them to save a few dollars.
OS is the search engine software that AWS will use from now on. If you want to stay with ES, you either have to use an older version or deploy your cluster manually with EC2, ECS, or EKS.
But rest assured that Dashbird got your back with that new ES fork. So, you will stay on top of things with monitoring metrics and well-architected insights for your AOS domains.
Further reading:
AWS Elastic Load Balancing from a Serverless perspective
Dashbird now integrates with 5 new AWS services
Triggering AWS Lambda with SNS
Today we are announcing a new, updated pricing model and the end of free tier for Dashbird.
In this article, we’re covering 4 tips for AWS Lambda optimization for production. Covering error handling, memory provisioning, monitoring, performance, and more.
In this article we’ll go through the ins and outs of AWS Lambda pricing model, how it works, what additional charges you might be looking at and what’s in the fine print.
Dashbird was born out of our own need for an enhanced serverless debugging and monitoring tool, and we take pride in being developers.
Dashbird gives us a simple and easy to use tool to have peace of mind and know that all of our Serverless functions are running correctly. We are instantly aware now if there’s a problem. We love the fact that we have enough information in the Slack notification itself to take appropriate action immediately and know exactly where the issue occurred.
Thanks to Dashbird the time to discover the occurrence of an issue reduced from 2-4 hours to a matter of seconds or minutes. It also means that hundreds of dollars are saved every month.
Great onboarding: it takes just a couple of minutes to connect an AWS account to an organization in Dashbird. The UI is clean and gives a good overview of what is happening with the Lambdas and API Gateways in the account.
I mean, it is just extremely time-saving. It’s so efficient! I don’t think it’s an exaggeration or dramatic to say that Dashbird has been a lifesaver for us.
Dashbird provides an easier interface to monitor and debug problems with our Lambdas. Relevant logs are simple to find and view. Dashbird’s support has been good, and they take product suggestions with grace.
Great UI. Easy to navigate through CloudWatch logs. Simple setup.
Dashbird helped us refine the size of our Lambdas, resulting in significantly reduced costs. We have Dashbird alert us in seconds via email when any of our functions behaves abnormally. Their app immediately makes the cause and severity of errors obvious.