Monitoring platform for keeping systems up and running at all times.
Full stack visibility across the entire stack.
Detect and resolve any incident in record time.
Conform to industry best practices.
The simple query service (SQS) was one of the first services AWS offered. It’s a managed queuing service that lets you take pressure from your downstream services. You put your items on the queue, and other services can pull them whenever they have the capacity to work on them.
It’s a managed service, so you don’t have to install or maintain the software yourself; you just configure a queue and start pushing to and pulling from it. So SQS is very simple to get started with.
As with all services on AWS, issues can crop up while using SQS because it’s not always obvious what every service can and cannot do. But fear not, for this article aims to help you solve the most common ones as quickly as possible. Ready to fix your queues? Then let’s dive in!
You tried to get messages with attribute without telling the SDK the attribute name. To fix this you need to call receiveMessage with the following parameter:
This way, the SDK knows that it should include the attributes you care about.
For background, AWS services in general, and SQS in particular, are optimized to lower traffic because you pay for all data you want to get out of AWS. If you used an SQS queue from outside of AWS, you would like to make sure you only get the data you use to save money and increase throughput.
The AWS SDK lets you define which attributes of the message you need to get this optimization going. But later, when you access attributes, you have to keep in mind which you have fetched; otherwise, you will find something akin to a null-pointer exception: accessing data that isn’t there.
You tried to publish to SQS from an SNS topic that doesn’t have permission. To fix this, you need to allow SQS to the SendMessage action from the SNS service principal.
Here is an example policy:
This problem is based on the misconception that SNS topics are deployed in your account. You might expect your account to act as the principal here, but the SNS service itself acts as the principal. So, you end up with your account getting permission it can’t use and the actual principal being locked out.
AWS preconfigures all services with minimal access rights, usually none at all. This helps with security because you don’t accidentally deploy something the whole world can access.
In day-to-day development work, this is often a bit cumbersome. You add a new feature, and suddenly you need to configure new permissions, so a simple one-liner requires you to write JSON documents in a completely different place. Well, that’s the price we pay for safety.
You tried to access SQS with a role or user that hasn’t any permissions. To solve this, initialize the SDK with an account set up with the required SQS permissions. You can do so in the IAM console.
Again, AWS is all about keeping your data safe. So it will lock down all your services by default so that nobody can access anything. If you want to give access to a service, you have to assign a role to it and permit this role to do its work.
You can simply use admin permissions for development purposes inside a development AWS account, so you don’t have to change your roles when you need different permissions.
But in a production environment, you should give every service minimal permission. This way, a compromised service won’t expose your whole infrastructure.
You’re adding messages to a queue and don’t know if you already added a message. This issue can’t be solved with SQS; you would have to either use another service or filter messages before adding them to the queue.
SQS isn’t made for such a use case, so you can’t prevent a queue from accepting items it already holds. If you have this constraint, you need to choose a more complex service like DynamoDB, which allows you to mark documents as unique. Step Functions might also be an option here, but this depends mainly on your use case.
Your consumers pulled messages without deleting them or increasing the timeout. To fix this, you need to call deleteMessage with your receipt handle.
The idea here is that your consumer can delete a message when it is finished processing it. If a consumer crashes before completing their work, the message remains in the queue, and eventually, another consumer can process it.
So, the consumer pulls a message, processes it, deletes it, and pulls the following message.
You configured a queue as FIFO, and a message blocks the whole queue. To mitigate this issue, group your messages with message group IDs; messages will only block those in the same group.
If you have a queue with many different types of messages and all of them have the same group ID, this can become a maintenance nightmare quickly. Make sure you group messages into smaller batches to lower the likelihood of blocking the whole queue when things go wrong.
SQS is a straightforward service. You can see it as a temporary database. If you need to put data somewhere to be processed later, SQS is the way to go.
Every service on AWS, an SQS queue, and all the other services in your stack using that queue will be configured with minimal permissions, leading to access issues. So, make sure you get your IAM policies set up correctly before deploying to production.
Admin roles might help when developing, but they’re a security nightmare when used in production, so you need to watch IAM policies when deploying!
If you need more control about what gets in or out of your queue, Step Functions or DynamoDB might be a better bet.
Dashbird automatically monitors SQS queues in your AWS account; no extra config is needed. Everything will be evaluated according to the Well-Architected Framework, so you see right away if you’re following best practices.
You can try Dashbird now; no credit card is required. See our product tour here.
At Dashbird, we understand that serverless’s core idea and value is to focus on the customer and the ability to avoid heavy lifting. That’s exactly what we provide. We give the focus back to developers to only think about the end customer, and not be distracted by debugging and alarm management or to worry about whether something is working or not.
AWS Kinesis vs SNS vs SQS (with Python examples)
Dashbird explained: the why, what and how
Dashbird now integrates with 5 new AWS services
In this article, we’re covering 4 tips for AWS Lambda optimization for production. Covering error handling, memory provisioning, monitoring, performance, and more.
In this article we’ll go through the ins and outs of AWS Lambda pricing model, how it works, what additional charges you might be looking at and what’s in the fine print.
Dashbird was born out of our own need for an enhanced serverless debugging and monitoring tool, and we take pride in being developers.
Dashbird gives us a simple and easy to use tool to have peace of mind and know that all of our Serverless functions are running correctly. We are instantly aware now if there’s a problem. We love the fact that we have enough information in the Slack notification itself to take appropriate action immediately and know exactly where the issue occurred.
Thanks to Dashbird the time to discover the occurrence of an issue reduced from 2-4 hours to a matter of seconds or minutes. It also means that hundreds of dollars are saved every month.
Great onboarding: it takes just a couple of minutes to connect an AWS account to an organization in Dashbird. The UI is clean and gives a good overview of what is happening with the Lambdas and API Gateways in the account.
I mean, it is just extremely time-saving. It’s so efficient! I don’t think it’s an exaggeration or dramatic to say that Dashbird has been a lifesaver for us.
Dashbird provides an easier interface to monitor and debug problems with our Lambdas. Relevant logs are simple to find and view. Dashbird’s support has been good, and they take product suggestions with grace.
Great UI. Easy to navigate through CloudWatch logs. Simple setup.
Dashbird helped us refine the size of our Lambdas, resulting in significantly reduced costs. We have Dashbird alert us in seconds via email when any of our functions behaves abnormally. Their app immediately makes the cause and severity of errors obvious.
End-to-end observability and real-time error tracking for AWS applications.