We have a Queue created in storage account. an external system is creating messages in the Queue.
So I need to process the messages for 2 environments (say PROD and Stage). So I have a Queue trigger available in both environments but the messages available in only one Queue.
If we copy the messages to another Queue also, the Queue trigger will be executed in both environments.
Is it possible to copy messages from one Queue to another Queue or how it can be done?
Thanks
Is it possible to copy messages from one Queue to another Queue or how
it can be done?
Natively Azure Storage does not support copying messages from one queue to another. What you will have to do is create a new message based on the contents/properties of an existing message.
You can create the message in another queue when you are processing the message in your queue trigger.
However a better way to handle this is to make use of Azure Service Bus Topics and Subscriptions. In this case you will send a single message to a Service Bus Topic and create 2 Subscriptions in that Topic. The message will be sent to both Subscriptions. Then you can have two Functions each of which is listening to one of the Subscriptions.
That way you do not have to worry about manually copying the message to another Queue.
Related
I am trying to design a logic app where when a message arrives or is received by poison queue(located in azure storage account) it posts to slack channel.
I know how to do the slack part and I can get the trigger to fire on queues located in my service bus resource because azure logic apps includes triggers for that.
However, I don't see azure logic app triggers for queues in the storage account for when a message arrives or is received in a queue.
My question is: Is there a trigger or other process that I can tap into when messages arrives in those storage account queues? If not, what is the best way to achieve being able to get this data (Message and Message Content) when it arrives in my poison queue in the storage account then take that info and throw that info into the slack channel?
Is there a trigger (for logic app) on the “storage account” queues that can be fired when a message is received?(I see length, and scan all messages in queue)
I have found triggers for this on the “service bus” queues.
These are the trigger options I have found for "service bus queues": https://i.stack.imgur.com/nS0nu.png
These are the trigger options I have found for "storage account queues": https://i.stack.imgur.com/AqxB4.png
Ideally, I just want to configure a simple logic app for when queue messages arrive in poison queue in storage account then take info to slack, i know how to put info in slack via logic app in service bus queue but haven't been able to figure out how to set up the action to pass the message to slack because I can't figure out how to do it when a message is recieved, i can set it as a job and grab length of queue and all items in queue, but not fire on arrival because I can't find a trigger for that.
If I can't figure this out or if at can't be done, I am probably just going to initiate the logic app with and Azure Queue storage trigger on an Azure function, this seems to be how MS wants us to do it anyway.
https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-how-to
ctrl+f "Binding reference information" then see sub heading "Usage: The types you can bind to and information about how the binding works. For example: polling algorithm, poison queue processing."
thanks again for the input folks, appreciate ya!
According to some test, it works fine in my side with "When a specified number of messages are in a given queue" trigger. When I add a message to my queue which named myqueue-items-poison, the logic app was triggered success.
So please check if you have configured the trigger correctly.
Please check if you set the interval of the trigger too long, I set it with 1 minute.
Can I trigger multiple azure function on a single queue trigger.
Use case is that i am storing auth token in a queue, and multiple functions take up the token to call the different endpoints respectively.
Or will the first one grab the message (token) and remove it from the queue.
In this scenario, I would use ServiceBusTrigger function.
You can create a topic in a Service Bus, then for this one topic you can have multiple subscribers.
So, even if 1 subscriber has finished going trough the messages, you can still have a subscriber which has not even started once and don't worry about this subscriber missing any message.
EDIT
Useful links
ServiceBusTrigger Example
Topics and Subscribers
#dhruv Yes you can create multiple functions with the same queue, all works in parallel as likewise event grid subscribers, which queue you're using, is it storage queue or service bus etc?
You could use a durable function for this.
First service adds messages to queue if user does not exist in DB, second service gets message from queue and create user. Possible situation, when first service adds 2 messages for create users before second gets it. How to resolve it? As I understand, no way to review queue...
I use Azure Storage queues
Azure Queue message doesn't support peek-lock to be processed. Once it is read, it becomes invisible. You need to look into Azure Service Bus as it allows you to control message one by one and in order if required.
I have following requirement
Message published to the Topic/Queue
Multiple consumers subscribed to the Topic/Queue. So our requirement is to only one consumer should listen to the message. That means no other consumer can get the same message.
I feel queue would be the best fit. But I have advise from our architect to check whether we can achieve it from Topics?.
So any body please let me know whether we can achieve it through Topics and also pros and constrains?
Thanks.
Azure Service Bus Queue is a single message queue. You send it a message and the message receiver will get the message and be able to process it accordingly. Each message will only be handled once.
Azure Service Bus Topic is a more robust message queue than Azure Service Bus Queue. With Topics there can be multiple Subscriptions configured to catch messages based on a Filter. If multiple Subscriptions have a Filter that matches an incoming message, then each of those Subscriptions will get a copy of the messages. With Topics it's up to you to configure the Subscription Filters according to your projects needs.
If you know a message only needs to be handled once in your system and the message queue is being used by a single message receiver application (single or multiple hosted instances) then Azure Service Bus Queue is likely the tool for the job.
I need to keep track of how many failed attempts have been made to process a message in an azure storage queue and delete the message after N unsuccesful attempts.
I have searched, but have not found any particular property that does this automaticaly and was wondering if there was a way other than using a counter in a storage table.
Each cloud queue message has a DequeueCount property. Does this help?
REST API reference here.
As for how to delete messages automatically after n attempts: There's nothing that automatically does this. You'll need to implement your own poison-message handling in Windows Azure queues, based on DequeueCount.
Alternatively, Azure Service Bus queues have a dead-letter queue for undeliverable messages (or ones that can't be processed). More info here.