I have specific business case where I need to read an azure service bus queue – but reading this queue should not be real time.
This is my setup
I have an Azure function that is doing some stuff and part of that processing is to read some messages from a Service bus Queue at the end of the day. This function is Timmer trigered.
I have a service Bus Topic which auto forwards the message to a Service Bus Queue. This is done in real time. So over a 7 hours working period the messages will pile up in this queue (average about 20 messages per day)
At the end of the day the function will then read the messages (average 20) in the service the Service Bus Queue (not real time) and produce a report.
All the code snippets I've seen online are all triggered in real time. As they are all registering an event that get fired as soon as a message get sent to the queue.
I had this code simppet in my application but noticed that as soon as a message is added to the queue the message is pulled imediately from the queue, which i dont want. I want the message to remain in the queue until the end of the day
public async Task<IEnumerable<ChangeNotification>> ReadChangeNotificationMessagesAsync()
{
processor = client.CreateProcessor(serviceBusOptions.TopicName, serviceBusOptions.SubscriptionName, serviceBusProcessorOptions);
processor.ProcessMessageAsync += AddNotificationToQueueEventAsync;
processor.ProcessErrorAsync += ProcessErrorEventAsync;
await processor.StartProcessingAsync();
}
private async Task AddNotificationToQueueEventAsync(ProcessMessageEventArgs args)
{
var changeNotification = args.Message.Body.ToObjectFromJson<ChangeNotification>(
new JsonSerializerOptions { PropertyNameCaseInsensitive = true });
//do some stuff
}
private Task ProcessErrorEventAsync(ProcessErrorEventArgs arg)
{
//log error
}
serviceBusProcessorOptions = new ServiceBusProcessorOptions
{
MaxConcurrentCalls = serviceBusOptions.Value.MaxConcurrentCalls,
AutoCompleteMessages = serviceBusOptions.Value.AutoCompleteMessages
};
Can someone provided a bit of code snippet that will allow me to read the queue but not in real time
You can use a Timer Triggered Azure Function and schedule it to run once a day. In your Function code you can make use of Service Bus SDK to read messages from the Service Bus and process them.
UPDATE
I noticed that you are using Service Bus Processor to process the messages which basically provides an event based model for processing your messages.
Instead of using that, you can simply use ServiceBusReceiver and read messages manually using ReceiveMessagesAsync(Int32, Nullable<TimeSpan>, CancellationToken).
Related
I have an Azure Message Bus Topic.
I have one "Session enabled" Azure Message Bus Consumer for this Topic.
I have around 3 Worker Services that are using the same Consumer. So the work is shared between these 3 Workers.
The Messages which are sent to the consumer need to be ordered, thats why I am using the "Session Feature" on the Consumer.
I believe that on a first Message, the Session of the Message gets bind to a Worker Service.
For certain Reasons I want to abandon not only a Message but also the session so that it can be picked up by another of the 3 Worker Services.
My questions:
Is this possible?
If yes how can I do this in the code?
Is there something like "Accept Session Or Not" Handler which kicks in when Message received?
See code below:
private void SetupServiceBusSessionProcessors2()
{
var busProcessorOptions = new ServiceBusSessionProcessorOptions();
var busProcessor = _busClient.CreateSessionProcessor("fooTopic", "fooSubscription", busProcessorOptions);
busProcessor.ProcessMessageAsync += args => ProcessSessionMessageHandler2(args);
}
private async Task ProcessSessionMessageHandler2(ProcessSessionMessageEventArgs args)
{
if (false) // Condition here which Abandons Message AND Session
{
// the following line of code seems only to abandon the Message
// but it seems like the session is locked to this service
// i want that other services which are listening via the same consumer can try to handle the session
await args.AbandonMessageAsync(args.Message);
}
}
This is possible in version 7.3.0-beta.1 using the ReleaseSession method on the event args. Note that this is a beta version so the API is subject to change before there is a stable release.
What is the correct way to continuously receive messages from the Azure Service Bus? I do the following by polling the queue infinitely.
const receiver = queueClient.createReceiver(ReceiveMode.peekLock);
while (true) {
const maxMessageCount = 1;
try {
if (!receiver.isReceivingMessages()) {
const message = await receiver.receiveMessages(maxMessageCount);
if (message.length > 0) {
/**
* Note: we are always receiving one message per call
*/
processMessage(message[0]);
}
}
} catch (err) {
processError(err);
}
}
But I am hitting a very weird situation. After some time, half of the messages get lost and only half are received by the above receiver. I wonder what is happening. Is there some bug in this code? I usually start the nodejs script and stop it using ctrl+c. Is there a possibility of some deamon listner running in the background with the above logic even when I have stopped the nodejs script?
Note: I always start a single nodejs script.
Messages are never lost with Azure Service Bus. As your code is using PeekLock mode, most likely the messages get retried and eventually end up in the dead-letter queue. Check the dead letter queue to see if they are there. Alternatively, verify that time to live (TTL) is not set to something short and causes messages to be purged.
In Azure WebJobs, in the OnMessageOptions class, I'm calling the QueueClient.Complete(Guid) method by setting the AutoComplete flag to true and messages seem to dequeue just fine when running the ProcessQueue function. Active messages count goes down by 1 after successful processing of each message. However, when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working. Scheduled messages count seems to go up initially. I go back to the queue after a few hours and see active messages in the thousands. The copies are of the same message. What is happening? I'd expect the message to be taken off the queue because of QueueClient.Complete(Guid) and the new scheduled message to be its replacement.
Some detail:
To send the message I do the following:
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
Inside the WebJob I created a ServiceBusConfiguration object which requires a onMessageOptions object where I set the AutoComplete=true. I pass the ServiceBusConfiguration object to the JobHostConfiguration.UserServiceBus
method.
Inside the WebJob service bus queue triggered function I again do the following to requeue, by first creating a new instance of the brokered message again.
//if not available yet for processing please requeue...
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
I don't do the following/use callbacks which is may be why it isn't working?
var options = new OnMessageOptions();
options.AutoComplete = false; // to call complete ourselves
Callback to handle received messages
client.OnMessage(m =>
{
var clone = m.Clone();
clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(60);
client.Send(clone);
m.Complete();
}, options);
when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working
If you fail to process your message, do not re-queue it. Instead, abandon (with a reason) and it will be picked up again.
BrokeredMessage.ScheduledEnqueueTimeUtc is intended to be used for messages added to the queue. When you receive a message, you can complete, dead-letter, defer, or abandon. If you abandon a message, it will be retried, but you can't control when that will happen. If you have no other messages in the queue, it will be retried almost immediately.
Note: when you see a behaviour that you suspect is not right, having a simple repro to share would be very helpful.
Background
I have several clients sending messages to an azure service bus queue. To match it, I need several machines reading from that queue and consuming the messages as they arrive, using Node.js.
Research
I have read the azure service bus queues tutorial and I am aware I can use receiveQueueMessage to read a message from the queue.
However, the tutorial does not mention how one can listen to a queue and read messages as soon as they arrive.
I know I can simply poll the queue for messages, but this spams the servers with requests for no real benefit.
After searching in SO, I found a discussion where someone had a similar issue:
Listen to Queue (Event Driven no polling) Service-Bus / Storage Queue
And I know they ended up using the C# async method ReceiveAsync, but it is not clear to me if:
That method is available for Node.js
If that method reads messages from the queue as soon as they arrive, like I need.
Problem
The documentation for Node.js is close to non-existant, with that one tutorial being the only major document I found.
Question
How can my workers be notified of an incoming message in azure bus service queues ?
Answer
According to Azure support, it is not possible to be notified when a queue receives a message. This is valid for every language.
Work arounds
There are 2 main work arounds for this issue:
Use Azure topics and subscriptions. This way you can have all clients subscribed to an event new-message and have them check the queue once they receive the notification. This has several problems though: first you have to pay yet another Azure service and second you can have multiple clients trying to read the same message.
Continuous Polling. Have the clients check the queue every X seconds. This solution is horrible, as you end up paying the network traffic you generate and you spam the service with useless requests. To help minimize this there is a concept called long polling which is so poorly documented it might as well not exist. I did find this NPM module though: https://www.npmjs.com/package/azure-awesome-queue
Alternatives
Honestly, at this point, you may be wondering why you should be using this service. I agree...
As an alternative there is RabbitMQ which is free, has a community, good documentation and a ton more features.
The downside here is that maintaining a RabbitMQ fault tolerant cluster is not exactly trivial.
Another alternative is Apache Kafka which is also very reliable.
You can receive messages from the service bus queue via subscribe method which listens to a stream of values. Example from Azure documentation below
const { delay, ServiceBusClient, ServiceBusMessage } = require("#azure/service-bus");
// connection string to your Service Bus namespace
const connectionString = "<CONNECTION STRING TO SERVICE BUS NAMESPACE>"
// name of the queue
const queueName = "<QUEUE NAME>"
async function main() {
// create a Service Bus client using the connection string to the Service Bus namespace
const sbClient = new ServiceBusClient(connectionString);
// createReceiver() can also be used to create a receiver for a subscription.
const receiver = sbClient.createReceiver(queueName);
// function to handle messages
const myMessageHandler = async (messageReceived) => {
console.log(`Received message: ${messageReceived.body}`);
};
// function to handle any errors
const myErrorHandler = async (error) => {
console.log(error);
};
// subscribe and specify the message and error handlers
receiver.subscribe({
processMessage: myMessageHandler,
processError: myErrorHandler
});
// Waiting long enough before closing the sender to send messages
await delay(20000);
await receiver.close();
await sbClient.close();
}
// call the main function
main().catch((err) => {
console.log("Error occurred: ", err);
process.exit(1);
});
source :
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-nodejs-how-to-use-queues
I asked myslef the same question, here is what I found.
Use Google PubSub, it does exactly what you are looking for.
If you want to stay with Azure, the following ist possible:
cloud functions can be triggered from SBS messages
trigger an event-hub event with that cloud function
receive the event and fetch the message from SBS
You can make use of serverless functions which are "ServiceBusQueueTrigger",
they are invoked as soon as message arrives in queue,
Its pretty straight forward doing in nodejs, you need bindings defined in function.json which have type as
"type": "serviceBusTrigger",
This article (https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus#trigger---javascript-example) probably would help in more detail.
I understand that MS Azure Queue service document http://msdn.microsoft.com/en-us/library/windowsazure/dd179363.aspx says first out (FIFO) behavior is not guaranteed.
However, our application is such that ALL the messages have to be read and processed in FIFO order. Could anyone please suggest how to achieve a guaranteed FIFO using Azure Queue Service?
Thank you.
The docs say for Azure Storage queues that:
Messages in Storage queues are typically first-in-first-out, but sometimes they can be out of order; for example, when a message's
visibility timeout duration expires (for example, as a result of a
client application crashing during processing). When the visibility
timeout expires, the message becomes visible again on the queue for
another worker to dequeue it. At that point, the newly visible message
might be placed in the queue (to be dequeued again) after a message
that was originally enqueued after it.
Maybe that is good enough for you? Else use Service bus.
The latest Service Bus release offers reliable messaging queuing: Queues, topics and subscriptions
Adding to #RichBower answer... check out this... Azure Storage Queues vs. Azure Service Bus Queues
MSDN (link retired)
http://msdn.microsoft.com/en-us/library/windowsazure/hh767287.aspx
learn.microsoft.com
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
Unfortunately, many answers misleads to Service Bus Queues but I assume the question is about Storage Queues from the tags mentioned. In Azure Storage Queues, FIFO is not guranteed, whereas in Service Bus, FIFO message ordering is guaranteed and that too, only with the use of a concept called Sessions.
A simple scenario could be, if any consumer receives a message from the queue, it is not visible to you when you are the second receiver. So you assume the second message you received is actually the first message (Where FIFO failed :P)
Consider using Service Bus if this is not your requirement.
I don't know how fast do you want to process the messages, but if you need to have a real FIFO, don't allow Azure's queue to get more than one message at a time.
Use this at your "program.cs" at the top of the function.
static void Main()
{
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
config.Queues.BatchSize = 1; //Number of messages to dequeue at the same time.
config.Queues.MaxPollingInterval = TimeSpan.FromMilliseconds(100); //Pooling request to the queue.
JobHost host = new JobHost(config);
....your initial information...
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
This will get one message at a time with a wait period of 100 miliseconds.
This is working perfectly with a logger webjob to write to files the traze information.
As mentioned here https://www.jayway.com/2013/12/20/message-ordering-on-windows-azure-service-bus-queues/ ordering is not guaranteed also in service bus, except of using recieve and delete mode which is risky
You just need to follow below steps to ensure Message ordering.:
1) Create a Queue with session enabled=false.
2) While saving message in the queue, provide the session id like below:-
var message = new BrokeredMessage(item);
message.SessionId = "LB";
Console.WriteLine("Response from Central Scoring System : " + item);
client.Send(message);
3) While creating receiver for reviving message:-
queueClient.OnMessage(s =>
{
var body = s.GetBody<string>();
var messageId = s.MessageId;
Console.WriteLine("Message Body:" + body);
Console.WriteLine("Message Id:" + messageId);
});
4) While having the same session id, it would automatically ensure order and give the ordered message.
Thanks!!