For my app I need to organize a circular (ring) queue. It means that any processed message immediately goes to the end of the queue for continuous processing.
For example:
Queue: A, B, C.
Receiver processes A.
Queue: B, C, A.
2 and 3 should be performed atomically. So we never lose A or any other message.
Another requirement is to ignore duplicates. So there should be always a single A in the queue. Even if a sender pushes another A item. A refers to some unique (primary) key of the message here.
I looked for using Azure Service Bus, but I cannot find how to meet both requirements with it. Is it possible to implement the scenario with Service Bus? If not, what are best alternatives?
This kind of queue can be implemented with Service Bus sessions. Sessions provide "group by" mechanics, so we can assign our unique key to SessionId of the message and then receive messages in groups ignoring all messages in a group except the first one.
Implementation
1) Create a queue with RequiresSession set to true:
var queueDescription = new QueueDescription("CircularQueue")
{
RequiresSession = true,
};
await namespaceManager.CreateQueueAsync(queueDescription);
2) When sending message to the queue, set SessionId to your unique key value:
var message = new BrokeredMessage($"Message body")
{
MessageId = "MESSAGE_UNIQUE_KEY",
SessionId = "MESSAGE_UNIQUE_KEY"
};
await queueClient.SendAsync(message);
3) Receive messages using sessions:
while (true)
{
var session = await queueClient.AcceptMessageSessionAsync(TimeSpan.FromSeconds(10));
if (session == null)
continue;
try
{
var messages = (await session.ReceiveBatchAsync(100)).ToList();
if (messages.Count == 0)
continue;
var message = messages[0];
ProcessMessage(message);
await queueClient.SendAsync(message.Clone());
await session.CompleteBatchAsync(messages.Select(msg => msg.LockToken));
}
finally
{
await session.CloseAsync();
}
}
Based on the little I know about Azure Service Bus, I believe both of these requirements can be fulfilled with it individually, though I am not sure how both of them can be fulfilled together.
Message Cycling
It is my understanding that Azure Service Bus supports First-In-First-Out (FIFO) behavior. What you could do is fetch the message (say A) from the top of the queue in Receive and Delete mode and then reinsert the message back in the queue. Since you're creating a new message, it will be posted to the end of the queue.
Avoid Duplicate Messages
Service Bus Queues has a boolean property called RequiresDuplicateDetection and setting this value accordingly will prevent duplicate messages being inserted. A simple search for Azure Service Bus Duplicate Detection will lead you to many examples.
Related
I have an API that will call Azure Topic to schedule a message. Is there a way to receive that message before the schedule time? For example in my code below, I schedule a message to azure topic and it will be queue after 60mins/1hr. Is there a way to received that message before 1hr?
string queueName = "topic";
var client = new ServiceBusClient("", new ServiceBusClientOptions()
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage($"Hello world cancel 13 {DateTime.Now}");
// add 5 minutes delay
long seq = await sender.ScheduleMessageAsync(message,
DateTimeOffset.Now.AddMinutes(60)
);
The message sequence number you get back when scheduling is suitable for cancelling but doesn't allow receiving that message earlier. The service doesn't allow early receiving, as anything that gets the messages to get the active messages (not in the future). For this scenario, I would suggest keeping the data in a database and not leveraging the queue as the database.
I am working with an azure service bus queue configured to be FIFO (First input first output). I work on an order application with the following states "Pending", "Received" and "Sent". therefore I have grouped the messages by the "SessionId" service bus option, setting the orderId as sessionId so that it processes the messages in order in case of horizontal scaling.
So far it works perfectly, the problem I have found is when a message in "pending" or "Received" status fails due to a timeout and goes to the dead letter queue. The message in "sent" status is processed correctly and then when the support team re-sends the "Pending" or "Received" status message to the queue it is processed correctly marking the order in a previous status instead of "sent" ".
I can think of several ways to control this, for example that the support team looks at the status of the order before reprocessing the message from the dead letter queue :) but I would like to know if service bus offers the possibility that if there is a message in the dead letter queu all the messages in the session queue that have the same sessionId go to the dead letter queu. Finallly, my question is:
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Thank you very much!!!
I would like to know if service bus offers the possibility that if there is a message in the dead letter queue all the messages in the session queue that have the same sessionId go to the dead letter queue.
No, there is no such offering by Service Bus by default.
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Yes, you can do that. You can first peek the messages in your dead-letter queue to fetch all the session ids. Then you can receive the messages in your main queue whose session id is in the DLQ, and then move those messages to DLQ. Here's one such logic I've implemented in dot net using the latest version of Service Bus SDK.
var queueName = "<queue>";
var connectionString = "<connection-string>";
var client = new ServiceBusClient(connectionString);
var sessionIdInDLQList = new List<string>();
var receiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
var message = await receiver.PeekMessageAsync();
while (message != null)
{
if (!sessionIdInDLQList.Contains(message.SessionId))
sessionIdInDLQList.Add(message.SessionId);
message = await receiver.PeekMessageAsync();
}
foreach (var sessionId in sessionIdInDLQList)
{
var session = await client.AcceptSessionAsync(queueName, sessionId);
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
while (message != null)
{
await session.DeadLetterMessageAsync(message, "Message with this session is to be dead-lettered!");
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
}
}
In your case, you need to do this before your consumers start reading the messages, probably you can write this in your consumer application or any trigger application like Azure Function or worker role. That’s upto your method of handling.
You can try this code to read Dead Letter from Queue.
public static async Task GetMessage()
{
string topic = "myqueue1";
string connectionString = "Endpoint = sb://xxx.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxx";
var servicebusclient = new ServiceBusClient(connectionString);
var reciveroptions = new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter };
var reciver = servicebusclient.CreateReceiver(topic, reciveroptions);
// 10 number of message read from Queue
await receiver.PeekMessageAsync(10);
}
after receiving message from Dead Letter you can send to queue.
As per Microsoft official documents
There's no automatic cleanup of the DLQ. Messages remain in the DLQ
until you explicitly retrieve them from the DLQ and call Complete() on
the dead-letter message.
These following document help you.
Thanks Casually Coding for posting post on Read Message from the Dead Letter Queue
Microsoft Documents Using Dead-Letter Queues to Handle Message Transfer Failures , Receive Message from Dead letter queue
I hope someone can clarify this for me:
I have 2 consumers in the same ConsumerGroup, it is my understanding that they should coordinate between them, but I am having the issue that both consumers are getting all the messages. My code is pretty simple:
const connectionString =...";
const eventHubName = "my-hub-dev";
const consumerGroup = "processor";
async function main() {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log(`Received event...`, event)
}
},
}
);
If I run two instances of this consumer code and publish an event, both instances will receive the event.
So my questions are:
Am I correct in my understanding that only 1 consumer should receive the message?
Is there anything I am missing here?
The EventHubConsumerClient requires a CheckpointStore that facilitates coordination between multiple clients. You can pass this to the EventHubConsumerClient constructor when you instantiate it.
The #azure/eventhubs-checkpointstore-blob uses Azure Storage Blob to store the metadata and required to coordinate multiple consumers using the same consumer group. It also stores checkpoint data: you can call context.updateCheckpoint with an event and if you stop and start a new receiver, it will continue from the last checkpointed event in the partition that event was associated with.
There's a full sample using the #azure/eventhubs-checkpointstore-blob here: https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/eventhub/eventhubs-checkpointstore-blob/samples/javascript/receiveEventsUsingCheckpointStore.js
Clarification: The Event Hubs service doesn't enforce a single owner for a partition when reading from a consumer group unless the client has specified an ownerLevel. The highest ownerLevel "wins". You can set this in the options bag you pass to subscribe, but if you want the CheckpointStore to handle coordination for you it's best not to set it.
I have a azure function that is triggered every 1 minute. The function creates a MessageReceiver like this
var messageReceiver = new MessageReceiver(serviceBusConnectionString, entityPath, ReceiveMode.PeekLock, null, 0);
The subscription has a lock duration of 1 minute.
It then tries to fetch up to 100 messages from the subscription like this:
var allMessages = new List<Message>();
Message message;
do {
message = await messageReceiver.ReceiveAsync(TimeSpan.FromSeconds(2));
if (message != null) allMessages.Add(message);
} while (message != null && allMessages.Count() < 100);
After processing the messages are completed using messageReceiver.CompleteAsync and the messageReceiver is closed using messageReceiver.CloseAsync().
The first time the function runs it fetches up to 100 messages from the subscription, but on the next runs it only fetches 0 or 1 message no matter the number of messages available in the subscription (Service Bus Explorer shows that there > 10 messages in the subscription). So it seems that ReceiveAsync returns null even when there is messages available.
Increasing the timeout for ReceiveAsync doesn't seem to help.
Why does ReceiveAsync return null when there is messages available?
I found a solution or workaround. If I change the the service bus topic to not allow partitioning (it requires deleting the topic and creating it again), ReceiveAsync always returns a message when there is a message available.
The problems seems related to this issue Odd Behavior of Azure Service Bus ReceiveBatch()
I have created the code to get the Queue Client data using QueueClient.Receive() with Broken Message
BrokeredMessage deadmessage = client.Receive();
byte[] dataRaw = deadmessage.GetBody<byte[]>();
Due to some corrupted data, I got the exception on second line, while get the body of the broken message. So i was try to get the body of the message on catch block with SteamReader.
Stream stream = deadmessage.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
I experienced with below exception, Could anyone help me with appropriate fixes?
Exception details :
The message body cannot be read multiple times. To reuse it store the value after reading.
To take multiple attempts to read message body you need to read it as a stream first
serviceBusClient.GetBody<Stream>()
Then you can try to interpret it by different ways. For example it can be serialized directly by following way:
var brokeredMessage = new BrokeredMessage(message);
serviceBusClient.Send(brokeredMessage);
but it's better to serialize it to json first.
var brokeredMessage = new BrokeredMessage(JsonConvert.SerializeObject(message));
serviceBusClient.Send(brokeredMessage);
it's more safe in my view because json serialization ignores namespaces of message type, so you will not break your process when you move class of message to another namespace.
Suppose you are starting to send and read messages serialized in json but some old messages can be still binary serialized. In this case you can use the following logic:
public static T DeserializeMessage<T>(BrokeredMessage brokeredMessage)
{
using (var stream = brokeredMessage.GetBody<Stream>())
using (var streamReader = new StreamReader(stream))
{
string bodyText = streamReader.ReadToEnd();
try
{
return JsonConvert.DeserializeObject<T>(bodyText);
}
catch (JsonReaderException)
{
stream.Position = 0;
var reader = XmlDictionaryReader.CreateBinaryReader(stream, XmlDictionaryReaderQuotas.Max);
var serializer = new DataContractSerializer(typeof(T));
var msgBody = (T)serializer.ReadObject(reader);
return msgBody;
}
}
}
If you need to try to deserialize the message as another type, catch System.Runtime.Serialization.SerializationException on serializer.ReadObject(reader).
As Sean Feldman mentioned that if message is corrupted, then it will be handled by dead-letter queue.
Service Bus queues and topic subscriptions provide a secondary sub-queue, called a dead-letter queue (DLQ). The dead-letter queue does not need to be explicitly created and cannot be deleted or otherwise managed independent of the main entity.
The purpose of the dead-letter queue is to hold messages that cannot be delivered to any receiver, or simply messages that could not be processed.
If you need to know how to create and user Service Bus Queue we can refer to get start with Service Bus queues
To reuse it store the value after reading.
If it be can be read correctly then we can store it messageid and vaule for reuse.
The DLQ is mostly similar to any other queue.
If it is corrupted data, we can get it from the dead-letter queue as mormal queue.
string connectionString = CloudConfigurationManager.GetSetting('Microsoft.ServiceBus.ConnectionString');
QueueClient Client = QueueClient.CreateFromConnectionString(connectionString, deletLetterQueueName);
var message = Client.Receive(TimeSpan.FromSeconds(3));
if (message != null)
{
var ret = message.GetBody<stream>();
message.Complete();
}
I found another reason of the exception. The thing is that when I debugged different problem I got this exception. After some experiments I realized that Visual Studio reads the message behind the scene to show it for example in Watch panel, and when my code tried to get the message it was already read by Visual Studio.
To avoid this it needs to wrap the message to a property with backing field, which will store the value. And then I realized that the exception message abstracly says to make this.
So you should consider that it can be read behind the scene