Asynchronous message and task processing with Pub/Sub
In a previous topic we covered the Asynchronous Messaging architectural pattern, its advantages and some high-level examples.
Pub/Sub is one way of implementating this type of architecture, and now we are going to cover it more in depth.
Pub/Sub (or Publish/Subscribe) pattern is a way for multiple services to communicate with eachother by publishing messages to a topic, which are then distributed to subscribers of that topic.
Differently than a Message Queue, each topic can have multiple subscribers (or consumers, as it would be called in a Queue model).
Another difference is that messages published to a topic are delivered immediately to all subscribers. As a result, Pub/Sub does not offer many options in terms of message delivery.
The Pub/Sub model allows for one message to be delivered to multiple subscribers. Each of them can perform a different task in parallel.
This pattern can also prove to be a good solution for controlling data processing workflows, as well as for implementing a Fan-Out strategy to conquer larger tasks.
Since topics will notify subscribers when new messages come in, there is no need to continuouslly poll the topic and check for new data. This can save resources and speed up processing of tasks.
It is easy to add new subscribers to an existing topic. New jobs can be performed for the same types of messages without Publishers having to control a list of destinations.
Most Pub/Sub systems will provide multiple protocols to deliver messages. These are some of the protocols commonly available:
Pub/Sub systems usually provide a way for subscribers to filter messages that are relevant to them. This can save resources and also increases flexibility for how messages are distributed across a set of compute services.
Pub/Sub can be easily be deployed in any project by leveraging modern cloud services. AWS, for example, offers SNS, a serverless Pub/Sub system, fully managed by their team.
This releases teams to work on aspects of the software that are really going to differentiate it in the market. Another benefit of using a managed service is faster time-to-market.
Nonetheless, for larger teams with access to advanced DevOps skills and enough financial resources, there are also great open source projects, so that development teams don’t have to reinvent the wheel.
Some options are:
It is possible to combine a Pub/Sub with a Message Queue. If one Service needs to process messages but may not be able to scale as fast as the volume of messagees coming from the Pub/Sub topic at some point, having a Message Queue in front of it will increase the overall reliability of the architecture. ↩︎
Although Apache Kafka is not originally designed for Pub/Sub, it can be used as such. ↩︎
Save time spent on debugging applications.
Increase development velocity and quality.
Get actionable insights to your infrastructure.