11/27/2023 0 Comments Aws dlqJoin!ĭeveloper-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. So what’s the issue with using your database as a queue? None until you actually need a message broker. When you have failures, how are you handling that with your queue if its in your database? Are you implementing retries? Creating another table as a dead letter queue? Do you need to implement different queue tables for different levels of priority of messages? The list goes on and on that out of the box you can do with your typical queue based broker. This can add a lot of load but also can decrease total throughput and increase latency because you’re polling your database. If you’re going to have a lot of consumers, they need to be polling (querying) the table periodically. You also need to understand your use case. Once you get out of the simplest scenario that I showed above, you will end up implementing a lot of functionality on top of your database that out-of-the-box queue-based brokers support (dead letter queues as an example). You need to know what you’re trying to accomplish and the limitations. In some situations, you maybe simplify your infrastructure by using a database you already have. So what’s the issue with using your database as a queue rather than a queue-based message broker? As you can see, it’s feasible to use a database. You’re still in the same situation where you might still be processing the message and now will end up re-processing it. You can only have a transaction or connection open for so long before the timeout occurs and it automatically does a rollback of the transaction or closes the connection. While you don’t have an “invisibility timeout” you do have the same type of mechanism with a database, often called a wait timeout. Then once the message has been processed, you’d delete the message from the queue and commit the transaction. So how would you implement a queue using Postgres or a similar relational database? First, it simply starts with a table where you’d be persisting your messages. But the consumer would still process it since it fetched it, but the since the invisibility timeout expired, the message would get processed again as well.īecause of all these troubles, rightfully or wrongfully, they decided to ditch RabbitMQ and use Postgres as their queue, which they were already in their system. As well as, the invisibility timeout would kick in since the second message that was pre-fetch would often exceed the invisibility timeout because the first message took so long to process. This means that if there were two messages available, a single consumer would pre-fetch 2 messages, not one.Īnd since consuming a single message could take hours, this would then reduce throughput because the other consumers available don’t see the message. The issue they described in the blog post was that they were prefetching multiple messages together from RabbitMQ.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |