First service adds messages to queue if user does not exist in DB, second service gets message from queue and create user. Possible situation, when first service adds 2 messages for create users before second gets it. How to resolve it? As I understand, no way to review queue...
I use Azure Storage queues
Azure Queue message doesn't support peek-lock to be processed. Once it is read, it becomes invisible. You need to look into Azure Service Bus as it allows you to control message one by one and in order if required.
Related
I am sending messages from my custom application to servicebus. And recently I have added some messages for testing purposes. But later I realized that I have picked the wrong Queue. Now I don't want the Test messages to be in this queue. Is there any possible way to delete those recently added test messages from my queue?
As mentioned by Pedro in his comment you can use service bus explorer: https://github.com/paolosalvatori/ServiceBusExplorer to peek and delete the message from the queue. Alternatively, you can call the service bus REST API or write your own code to consume that message.
As an alternative you can use Service Bus Cloud Explorer which runs in the browser with logged-in user Azure account by default and have a delete and a purge functionality.
Keep in mind that it will take time to clear up queues or subscriptions. More messages it needs to delete; the more time it would take.
In Azure, we have two separate messaging technologies and it's not very well documented when to use what? While EventGrid is really cool, I did not come across when to use EventGrid(scenarios) vs the Storage/ServiceBus queue? Can someone help?
E.g. if I have the following scenario :
A status of a flag changes and based on that, I want to trigger an algorithm that would do recalculations, few inserts/updates etc. in the database.
For implementing this - I can either use EventGrid or Storage Queue. How do we figure what to use in such scenario? I was looking for some kind of guidance.
Basically, Azure Event Grid handles events and Azure ServiceBus handles messages.A message is raw data produced by a service to be consumed or stored. Events are also messages (lightweigth), but they don’t generally convey a publisher intent, other than to inform.
1) If the purpose is to just to store the information ServiceBus can be used.
2) If the information received is used to trigger another service Azure Event Grid can be used.
Find more info here
https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services
https://azure.microsoft.com/en-us/blog/events-data-points-and-messages-choosing-the-right-azure-messaging-service-for-your-data/
Events are like notifications from a service to inform the world that something happened in the domain of the publisher (similar to an email notification). There is no expectations from the publisher to have any actions taken. A message is a command you send to a specific receiver with the expectation of the message to be processed (like an asynchronous post request).
Events will work in pub/sub pattern and multiple subscribers could be configured to the events. The service that needs to react to an event will get notified by the event grid when an event occurs (http call from event grid to the receiver). The event will remain in the event grid until deletion (cleanup) and there is no garantie of keeping the original order (no FIFO).
In the other hand, messages will be added to a queue and will be deleted once the “message processor” is done with it. The messages in the queue will keep the original order (FIFO). The message processor has to pull messages from the queue.
In your scenario, you could use a combination of both. Service A sends an event “StatusChanged”, then you can configure a subscription to that event and send a message to a queue, then have your logic to process that message. This will end up with a fully async communication pattern. This is ideal to support scenarios where you processor is down or too busy. The incoming messages will simply get accumulated in the queue and eventually being processed once the service is back up and running. And without affecting the original service that sent the “StatusChanged” event..
My scenario: I have an Azure Storage Queue where messages can come in at any time. If I have 10 items in that queue, it's imperative that they be processed in order. I'm using c# and the windows azure storage SDK.
If the first item fails after, say, 2 seconds it remains invisible on the queue for another 28 seconds (30 second invisibility by default).
Now, my worker will just continue to check a queue for messages and process them as and when. If a queue message fails, it remains invisible and so the next queue item will be processed before the first message is retried.
This seems like really basic functionality for anyone needing a queue where the items are processed in order.
No, I can't set the timeout to a smaller amount because tasks can take varying lengths of time.
George, if you are looking for a messaging queue solution that processes items in order, you should consider using Azure Service Bus Queues:
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud).
For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
There is a good article comparing both Storage Queues and Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted , you may find the latter better suitable for your case.
Our application consist of 7 microservices that have some intercommunication. Currently we're using simple storage queues that a microservice publish events to (the number of events is relative low). Then we have a azurefunction for each queue that might call another microservices. This is working fine for us right now the services uses about 20 queues with a corresponding function.
Now we need to handle an blobstorage event, and I did some googling and a started to get really confused. Suddenly there was a lot of questions:
Should we switch to Azure Event Grid
It handles blobstorage without any limitations (functions blobstorage trigger has some)
It allows for multiple subscribers (storage queues does not)
It has a lot of fuz - maybe this is the new recommended way
I like the idea of one central thing, but it reminds me a bit about biztalk...
Should I switch to Azure Service Bus
It has a nice tool (ServiceBusExplorer) for monitoring the queues and listners, and I could to a repost of any failed events
It visulizes my azure functions subscribers nicely
Should I continue with only storage queues
A bit difficult to monitor, but it works nice
I'll be really thankful for any advice or insights to this question.
Thanks
EventGrid is great when you have notifications floating to multiple subscribers. Is that the case for you?
An example would be deferring messages. With queues you can defer a message, not with EventGrid. Whenever to choose Storage Queues or Service Bus depends on the specific requirement that you have. Do you need de-duplication? Or ordered delivery? If you do, Service Bus is the way. Otherwise Storage Queues is enough.
First of All, I would like to recommend these two articles, it will clarify most of your doubts about these services:
Choose between Azure services that deliver messages
Storage queues and Service Bus queues compared
Regarding Event Grid, it acts like a bridge between the publisher and the subscriber, where publisher will send messages and forget whether it has been processed or not, and the Event Grid will handle the retry if the receiver\subscriber does not acknowledge that it was processed successfully.
As you mentioned, storage queues has limitations, as such blob triggered functions, and maybe Service Bus, but it will depend on your design requirements. I would like to point out some things you might consider before moving to Event Grid.
Storage queues & Service Bus does not care about your message schema, in Event Grid you have to create a custom event based on their schema to wrap your event, so the publisher and subscriber has to understand Event Grid for that, not that is a big deal, but now you have both sides coupled to Event Grid.
If you want to send the event straight to your micro-service, you have to implement the subscription validation in your service, otherwise the service won't be able to receive the events
Event Grid only retry the delivery of your messages for 24 hours, if your service is down or not process the message correctly for longer than 24h, it will make the event dead. Currently, there is no way to query dead messages. Storage Queues and Service Bus are configurable how long you keep the message and it can be kept for many days.
Your service web-hook must acknowledge the receipt(http 200 or 202) of an event within 60 seconds, otherwise it will consider failed. If your operation is longer that that, you should send it to a queue and handle the locking from your service.
Probably there are more limitations, but these are the ones I remember right now that might change anytime soon, I think Event Grid is a great technology still on early days, and there is much to improve, I would recommencement only as a hub for Azure management events, I don't think it is ready for use as an application integrator.
Regarding your comment for queue manager, for Service Bus your have the Service Bus Explorer, and for Azure Storage you have the Azure Storage Explorer, where you can check the messages in the queue, is not the same as Service Bus, but helps.
It very much depends on how are you consuming the queue messages, you can take a look at this comparison: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
If you don't need ordering and if you don't have a strong limit on message volume, size or TTL, you can stick to storage queues.
I need to keep track of how many failed attempts have been made to process a message in an azure storage queue and delete the message after N unsuccesful attempts.
I have searched, but have not found any particular property that does this automaticaly and was wondering if there was a way other than using a counter in a storage table.
Each cloud queue message has a DequeueCount property. Does this help?
REST API reference here.
As for how to delete messages automatically after n attempts: There's nothing that automatically does this. You'll need to implement your own poison-message handling in Windows Azure queues, based on DequeueCount.
Alternatively, Azure Service Bus queues have a dead-letter queue for undeliverable messages (or ones that can't be processed). More info here.