I'm new to Pulsar and after reading some document I am a little bit confusing about message acknowledge.
Say, I have one topic, and two subscriptions: SubA and SubB. Now a message is consumed by SubA and SubB, but only SubA acknowledged that message. Now my question is, say after 2 days, our retention policy kicks in and it want to delete all acknowledged message older than 2 days, then in this case, is the message considered acknowledged or not? (because only SubA acknowledged it, SubB didn't)
The message is considered not acknowledged and will not be deleted. It is still being held for delivery in subscription SubB. Messages need to be acknowledged in all subscriptions before they are removed.
Like a traditional message broker, Pulsar keeps unacknowledged messages indefinitely. If this is not what you want, you can configure message TTL, which acknowledges a message after a configurable amount of time.
Related
Azure Service Bus entities (queues/topics) support a Time to Live (TTL). When the TTL passes the message expires. On expiry, the system deletes the message OR moves it to the Dead-Letter Queue (DLQ). Does Service Bus have another setting to delete messages from the DLQ after a specified period? For instance, to avoid passing size quotas, we might like to delete messages from the DLQ after six months.
See also:
Do messages in dead letter queues in Azure Service Bus expire?
https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-expiration?WT.mc_id=Portal-Microsoft_Azure_ServiceBus
Azure Service Bus doesn't have an expiration option on the dead-letter queues. This is likely intentional, as the system shouldn't just lose those messages but rather do something about them.
Sometimes, monitoring all dead-letter queues for total size and whatnot is inconvenient. One option is to create a centralized DLQ. That will allow the following:
Monitoring a single "dead-letter" queue.
Receive messages from a single entity for processing.
Keep the size under control by specifying a TTL on the queue.
For example, let's say you've got two queues, test-dlq and test-dlq2. You'd configure those to auto-forward dead-lettered messages to a 3rd queue, test-dlq-all. With that, when you have messages that are received by test-dlq or test-dlq2 and dead-lettering,
Those messages will end up in the centralized "DLQ" queue (test-dlq-all).
The nice part is whenever you have messages auto-forwarded, you'll always know where they originally dead-lettered.
For example, let's say you've got two messages, each from a different queue, ending up in test-dlq-all, the centralized "DLQ".
Inspecting its messages will reveal a system property, DeadLetterSource, stamped with the name of the queue it was dead-lettered initially in.
This solution lets you set TTL on the test-dlq-all queue and have messages auto-perged.
Also, worth mentioning that it's possible to either set up dead-lettering with the centralized "DLQ" or get messaged dead-lettered as a result of failing processing that exceeds MaxDeliveryCount. For that reason, it is worth wither monitoring test-dlq-alls DLQ.
I need help interpreting this graph:
It looks like messages are coming in, then being processed, and going back to 0. The graph is not continually rising.
However, the outgoing messages has "--". Does this mean 0?
Does an outgoing message represent a message being read by a service?
If the messages are not being read, then what is happening to them? The dead letter queue has 0 messages.
Yes, an outgoing message represents a message being read by a service.
In your case, I see you've disabled the dead letter queue(in your screenshot, the Dead lettering option is disabled.), so there is no messages in the DLQ. If DLQ is disabled, the messages will be got deleted after expired.
The docs describes them like this:
The number of events or messages received from Service Bus over a specified period.
So, the incoming messages are message that are sent to the service bus. The outgoing messages are messages that are picked up by message processors (your application). So the answer to your question
Does an outgoing message represent a message being read by a service?
is: Yes!
From the doc: https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#get-the-ttl-configuration-for-a-namespace, it is a little bit confusing about the difference between backlog quotas and TTL.
As I understand so far, a message arrives broker, and broker will find out all subscriptions on that topic, and retrieve their backlog, and put the message to those backlog. If this message is acknowledged by one subscription, it will be removed from its backlog (backlog is per subscription). If the message is not in any backlog (means all subscription acknowledged it), then this message is considered as acknowledged, and then retention policy kicks in, to decide if it needs to be deleted or keep for some time.
If a message is not acknowledged in one backlog for some time, and the backlog quota reaches a size limitation, then backlog retention policy kicks in. So this is more about size than time. And if we use consumer_backlog_eviction, this message will be discarded from the backlog, but question, is that considered acknowledged or not? so the first retention policy kicks in?
And the TTL, if a message is not acknowledged for some time, will it be removed from all backlogs? and then considered as acknowledged and then let the first retention policy handle it?
UPDATE:
to be more precise of this question:
In backlog quotas document, it says:
consumer_backlog_eviction: The broker will begin discarding backlog
messages
Discarding means, making it acknowledged? So that the global retention policy can kick in?
producer_request_hold: The broker will hold and not persist produce request payload
Is it saying, that, it will not put new messages into the backlog, but for those new coming messages, are they automatically acknowledged or not (say there is just one subscription at that moment)? And does this block the real producer (I guess not, it is just that the broker won't put new messages into the backlog anymore)
(for TTL) If disk space is a concern, you can set a time to live (TTL) that determines how long unacknowledged messages will be retained.
Again, if TTL is exceeded, it will not "retain" it, means, make it acknowledged? or just throw it away?
And if we use consumer_backlog_eviction, this message will be
discarded from the backlog, but question, is that considered
acknowledged or not? so the first retention policy kicks in?
The message will be acknowledged and marked for deletion. Then the retention policy for acknowledged messages will kick in at some point depending on the configuration.
And the TTL, if a message is not acknowledged for some time, will it
be removed from all backlogs? and then considered as acknowledged and
then let the first retention policy handle it?
The TTL should be applied to all backlogs and outdated unconsumed messages will be automatically acknowledged. And again the retention policy for acknowledged messages will kick in.
On Azure (or from VS) the dialog for creating new messages inside a queue says the message expires in 7 days, yet it is gone in less than a few seconds. Why? (I created a continuous running WebJob as described in this article)
The message disapear because it has been consummed by your Web job.
The retention delay means you have X days to consume the message (in your case, 7 days). After the delay expired, the message is automatically deleted.
If you want multiple consumer for your messages, instead of a queue, you can use Service Bus with subscription or topics, or Event Hub with consumer groups.
The messages stay in the Service Bus Queue or Topic subscription until they are processed i.e received in receive and delete mode by the receiver.
The message will not be removed from the queue if it is received in peek lock mode.
In your case as the message is processed by the webjob it was not available in the queue.
The messages also have default time to live property which can be set after which the message will be moved to the dead letter path of the same messaging entity(queue or topic subscription).The messages after the given time duration in time to live after scheduled enwueued time utc will be moved to the dead letter path with dead letter reason TTLExpiration
I have huge number of messages in azure service bus dead letter queue. When I see the messages, I see that most of the messages are expired.
I want to know what happens when we try to re-submit the expired deadletter queue message back to its original queue?
Can anyone help me out in explaining this ?
Thank you !
I am trying to answer two of your questions below,
when you receive an expired message from the dead letter queue to process/resubmit to main queue(Using ReceiveAsync() to receive a message), the state of the message will be changed to deferred state. So, the message won't be available for receiving in the Dead-Letter queue anymore.
For your question on, what happens to the message when you resubmit, it would be submitted as a new message into the target queue.
We could use FormatDeadLetterPath() method to build the format name for the specified dead letter queue path and create a receiver and retrieve messages from a DLQ. If you’d like to resubmit message back into the main queue, you could create and send a new message based on retrieved message in DLQ. And you could investigate why the message has been dead-lettered via checking DeadLetterReason and DeadLetterErrorDescription properties.
This link explained Dead-Letter Queues with a sample, please refer to it.