I found that non-persistent messages are lost sometimes even though the my pulsar client is up and running.
Those non-persistent messages are lost when the throughput is high (more than 1000 messages within a very short period of time. I personally think that this is not high).
If I increase the parameter receiverQueueSize or change the message type to persistent message, the problem is gone.
I check the Pulsar source code (I am not sure this is the latest one)
https://github.com/apache/pulsar/blob/35f0e13fc3385b54e88ddd8e62e44146cf3b060d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherMultipleConsumers.java#L185
and I think that Pulsar simply ignore those non-persistent messages if no consumer is available to handle the newly arrived non-persistent messages.
"No consumer" here means
no consumer subscribe the topic
OR all consumers are busy on processing messages received before
Is my understanding correct?
The Pulsar broker does not do any buffering of messages for the non-persistent topics, so if consumers are not connected or are connected but not keeping up with the producers, the messages are simply discarded.
This is done because any in-memory buffering would be anyway very limited and not sufficient to change any of the semantics.
Non-persistent topics are really designed for use cases where data loss is an acceptable situation (eg: sensors data which gets updates every 1sec and you just care about last value). For all the other cases, a persistent topic is the way to go.
Related
When we have multiple consumers of Event Hub (or any messaging service, for that matter), how to make sure that no message is processed twice especially in a situation when consumer auto-scales out to multiple instances?
I know we could keep track of last message processed but then again, between the check if message was processed and actuall, processing it,other instance could process it already (race condition?.
so, how to solve that in a scalable way?
[UPDATE]
i am aware there is a recommendation to have at least as many partitions as there are consumers but what to do in case when a single consumer cannot process messages directed to it but needs to scale out to multiple instances?
Each processor takes a lease on a partition, see the docs
An event processor instance typically owns and processes events from one or more partitions. Ownership of partitions is evenly distributed among all the active event processor instances associated with an event hub and consumer group combination.
So scaling out doesn't result in duplicate message processing because a new processor cannot take a lease on a partition that is already being handled by another processor.
Then, regarding your comment:
i am aware there is a recommendation to have at least as many partitions as there are consumers
It is the other way around: it is recommended to have as many consumers as you have partitions. If you have more consumers than partitions the consumers will compete with each other to obtain a lock on a partition.
Now, regarding duplicate messages, since Event Hub guarantees at-least-once delivery there isn't much you can do to prevent this. There aren't that many scalable services that offer at-most-once deliveries, I know that Azure Service Bus Queues do offer this if you really need it.
The question may arise what can cause duplicate message processing. Well, when processing message the processor does some checkpointing: once in a while it will store its position within a partition event sequence (remember, a partition is bound to a single processor). Now when the processer instance crashes between two checkpoint events a new instance will resume processing messages from the position of the last checkpoint. That may very well lead to older messages being processed again.
If a reader disconnects from a partition, when it reconnects it begins reading at the checkpoint that was previously submitted by the last reader of that partition in that consumer group.
So, that means you need to make sure your processing logic is idempotent. How, that is up to you as I don't know your use case.
One option is to track each individual message to see whether it is already processed or not. If you do not have a unique ID to check on maybe you can generate a hash of the whole message and compare with that.
I am implementing a consumer which does processing of messages from a queue where order of messages is of importance. I would like to implement a mechanism using NodeJS where:
the consumer function is consuming messages m1, m2, ..., mN from the queue
doing an IO intensive operation and process the messages. m -> m'
Storing the result m' in a redis cache.
acknowledging the queue after each message process (2)
In a different function, I am listening to the message from the cache
sending the processed messages m' to an external system
if the external system was able to process the external system, then delete the processed message from the cache
If the external system rejects the processed message, then stop sending messages, discard the unsent processed messages in the cache and reset the offset to the last accepted message in the queue. For example if m12' was the last message accepted by the system, and I have acknowledged m23 from the queue, then I have to discard m13' to m23' and reset the offset so that the consumer can read and start processing from m13 again.
Few assumptions:
The processing m to m' is intensive and I am processing them optimistically, knowing that most of the times there won't be a failure
With the current assumptions and goals, is there any way I can achieve this with RabbitMQ or any Azure equivalent? My client doesn't prefer Kafka or any Azure equivalent of Kafka (Azure Event Hub).
In scenarios where the messages will always be generated in sequence then a simple queue is probably all you need.
Azure Queues are pretty simple to get into, but the general mode of operation for queues is to remove the messages as they are processed successfully.
If you can avoid the scenario where you must "roll back" or re-process from an earlier time, so if you can avoid the orchestration aspect then this would be a much simpler option.
It's the "go back and replay" that you will struggle with. If you can implement two queues in a sequential pattern, where processing messages from one queue successfully pushes the message into the next queue, then we never need to go back, because the secondary consumer can never process ahead of the primary.
With Azure Event Hubs it is much easier to reset the offset for processing, because the messages stay in the bucket regardless of their read state, (in fact any given message does not have such a state) and the consumer maintains the offset pointer itself. It also has support for multiple consumer groups, which will make a copy of the message available to each consumer.
You can up your plan to maintain the data for up to 7 days without blowing the budget.
There are two problems with Large scale telemetry ingestion services like Azure Event Hubs for your use case
The order of receipt of the message is less reliable for messages that are extremely close together, the Hub is designed to receive many messages from many sources concurrently, so its internal architecture cares a lot less about trying to preserve the precise order, it records the precise receipt timestamp on the message, but it does not guarantee that the overall sequence of records will match exactly to a scenario where you were to sort by the receipt timestamp. (its a subtle but important distinction)
Event Hubs (and many client processing code examples) are designed to guarantee Exactly Once delivery across multiple concurrent consuming threads. Again the Consumers are encouraged to be asynchronous and the serice will try to ensure that failed processing attempts are retried by the next available thread.
So you could use Event Hubs, but you would have to bypass or disable a lot of its features which is generally a strong message that it is not the correct fit for your purpose, if you want to explore it though, you would want to limit the concurrency aspects:
minimise the partition count
You probably want 1 partition for each message producer, or atleast for each sequential set, maintaining sequence is simpler inside a single partition
make sure your message sender (producer) only sends to a specific partition
Each producer MUST use a unique partition key
create a consumer group for each of your consumers
process messages one at a time, not in batches
process with a single thread
I have a lot of experience in designing MS Azure based solutions for Industrial IoT (Telemetry from PLCs) and Agricultural IoT (Raspberry Pi) device implementations. In almost all cases we think that the order of messaging is important, but unless you are maintaining real-time 2 way command and control, you can usually get away with an optimisitic approach where each message and any derivatives are or were correct at the time of transmission.
If there is the remote possibility that a device can be offline for any period of time, then dealing with the stale data flushing through the system when a device comes back online can really play havok with sequential logic programming.
Take a step back to analyse your solution, EventHubs does offer a convient way to rollback the processing to a previous offset, as long as that record is still in the bucket, but can you re-design your logic flow so that you do not have to re-process old data?
What is the requirement that drives this sequence? If it is so important to maintain the sequence, then you should probably process the data with a single consumer that does everything, or look at chaining the queues in a sequential manner.
I'm missing one piece of understanding of how Event Bus / Hazelcast works.
Imagine a case with a consumer and a producer verticles communicating over the clustered EB. The consuming part is doing CPU / memory / IO-intensive calculations.
When at some point due to the load the consumer is not able to handle the messages immediately, what is going to happen?
Would the messages be queueed inside the ring-buffer and eventually be processed later (considering Netty's SingleThreadEventLoop limits of 2 billion as per Size of event bus in vert.x)? Will they be dropped in case of reaching the limit?
In general, can the messages in EB be considered persistent and with delivery guarantee, as soon as no component in the cluster crashes?
If the consumers cannot cope with the messages, Vert.x will accumulate messages in a queue in memory.
When the queue reaches its limit, the messages will be dropped. The number of elements in the queue can be configured with MessageConsumer.html#setMaxBufferedMessages. It does not depend on message size.
If you need delivery guarantees, don't use the EventBus, use a messaging system like ActiveMQ (Vert.x has clients for such messaging systems).
In general, Vert.x does its best not to lose messages but the EventBus is simply not a full-featured messaging system.
TL;DR
Is there any (Node.JS) pattern to stop consuming a queue after a time (10 seconds), and closing the connection, without interrupting the processing of messages were not yet acked?
Longer version
We are sending out millions of push notifications daily for our customers. After generating the messages, we insert them into RabbitMQ, and then send them out with hundreds of consumers. Often customers would like to send out a huge amount of messages almost the same time, and if these messages are going to the same queue, then one customer's messages will have the processing time of the other customer's messages.
We would like to create a dynamic multiqueue setup, where a campaign's messages are going to a messages. queue, and the customers are dynamically subscribing, processing thousands of messages then switching to an other queue. The idea is that we are sharing our resources between campaigns, so they will be sent at the same time in parallel.
Our current solution is generating the messages slower than we are sending them out, so they are inserted into the queue in mixed order. It's not the ideal solution, because it's hard find out what is slow enough, but not too slow.
I've tried creating a simple setTimeout, and forcing the consumer to finish consuming (Channel#cancel) after a few seconds, then subscribing to a queue has the least consumers. The problem is that this interrupts the processing of some messages. I've tried using Channel#get to process a bunch of messages (then choosing a different queue), but the documentation says it is ideal only for a few messages, but not for millions of messages, using Channel#consume is much better in performance.
Any ideas how to do it well?
I assume that the trade off of using partitioned queues and topics, is that the message ordering is no longer guaranteed.
As the messages are by default sent round-robin to each of the fragments/partitions, then it would mean the message ordering is no longer guaranteed. Can anybody confirm if this is the case?
How can you guarantee message ordering when receiving trades from a partitioned queue.
Is the only way to support FIFO message ordering with partitioned queues/topics, to use sessions? I would assume that all messages for the same session/partition key would at least be delivered FIFO?
I found this documented in a blog post.
Hope it helps!
Partitioned Service Bus Queues and Topics
SessionId. If a message has the SessionId property set, then Service Bus uses the SessionId property as the partition key. This way, all messages that belong to the same session are assigned to the same fragment and handled by the same message broker. This allows Service Bus to guarantee message ordering as well as the consistency of session states.
Just because you're not using a partitioned queue or topic does not mean you will get FIFO. If you have more than one reader or do async then no, you won't get FIFO UNLESS you use Sessions as was pointed out above. Please use sessions.
There is an important point missing about FIFO in the answers above.
When a message comes in to a topic/queue which doesn't have partitioning enabled, then FIFO is observed for message delivery*.
When you enable partitioning on a topic/queue and the SessionId is used for the partitioning key, then messages are no longer guaranteed to be FIFO in relation to each other, they are only guaranteed to be FIFO in relation to the partition they were divvied in to.
Fun fact, partitioning in general can have some interesting side-effects if you have a low number of subscribers to the same subscription/queue, as the partition reader assignments are done round-robin style, and if you have more partitions than subscribers, you can see messages being starved (needs verification from SB team, this is empirical from tests I have done on my own because my messages were being starved).
* As #Dan Rosanova pointed out above, if you have async processing or or multiple readers, then your message processing can't be guaranteed to be FIFO, but the order in which the messages were distributed to the processors is going to be FIFO.
When you use a Session message handler (which requires SessionId to be populated), you're taking it another step further and you're guaranteeing that the messages are processed in order, as the Session message handler takes a lock on the SessionId+MessageId rather than just the MessageId, thus ensuring no other messages within the same session are received by another processor.