Similar past question of mine: Azure Service Bus Queue: Can I manage/cancel scheduled messages? - accepted answer here details how to cancel scheduled messages.
I'm wondering now if there's a way to view scheduled messages using the AMQP APIs. The Azure UI (and this method https://learn.microsoft.com/en-gb/java/api/com.microsoft.azure.management.servicebus._message_count_details?view=azure-java-stable) offer message counts but I can't see anything that lets me actually view those messages.
Is this a limitation of the service bus/Azure architecture? Or is there an API that will allow it? My goal would be to build a nice UI that displays scheduled tasks, and ideally I'd like to do that without maintaining that list of tasks myself outside of Azure if possible.
As of today, all messages can be peeked, but there's no way to peek those based on message status. There's an open issue on GitHub for the broker to add this feature. You can leave use-case scenario to help product team to have an idea why a feature as such would help customers.
Adding some information I found useful while reading about this.
Azure Service Bus Queues allows browsing/peeking scheduled messages.
Azure Service Bus Subscriptions does not support this feature yet.
Read More
View Example
Related
I have multiple WebApi that are hosted in Azure but some of the API taking a long time to process.
Instead of a calling from the scheduler function in Azure, I was advised to use the Messaging Queue.
Is this a good approach?
Also, let say my API URL.. https://testwebapi.net/api1 and https://testwebapi.net/api2, how can we communicate these API using Message queue and also how to call the WebAPI individually from Messaging Queue?
Is this Azure Bus service is the same as Messaging Queue for Azure.
I would strongly recommend you look at the respective docs on Azure, and start doing some quick tutorials and post more specific questions.
Now, to answer, what I believe is your main question, Azure Bus Service and Messaging Queue.
The thing is, there is no one Messaging Queue, rather three options for Azure messaging - Service Bus, Event Grid, and Event Hubs. Also, there is something called 'Azure Queue Storage', but I dont think you are thinking of that, considering your context.
There is a very cool comparison chart available here - https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services
But, I think, my guess, is you are looking for ultimately looking to use Azure Service Bus.
Also, please understand that one does not invalidate the other. As your project develops, you may end up using Azure Service Bus with other options like Event Hubs or Grids or all of them in different places.
Also, you keep using the incorrect terms. for example, you say 'Azure Bus service' but its really 'Azure Service Bus'. Also, there is no thing called 'Messaging Queue for Azure.' as there are so many different messaging services for Azure. This is what I want to add here. Spent some time with the docs and post more specific questions as you continue your project.
Using Azure Service Bus - Topics, I want to implement a solution wherein messages are sent/notified to end consumers once the producer sends the message to Topic (like Queues).
I understand that Topics work as Pub/Sub model wherein subscribers need to read messages from subscriptions. But I'm looking for a workaround that works some what similar to Queue (where it triggers a web job / service when any message is received).
I have few thoughts like
1. Using Auto-Forwarding in subscriptions to forward messages to Queues but again I think if this kills the purpose of Topics
2. Schedule a job to process these requests but again I think if I'm delaying the process
First, I want to know if Service Bus Topic is right option to go with? Next, If possible to implement a workaround what is the best/better way?
PS: I have to send messages which has information - I guess I can't use Relays
Just to be clear, Queues and Topics in Service Bus are different. As you noted, Topics are useful in publish/subscribe scenarios.
Since you are looking for something that gets triggered, Azure functions might be what you need.
Azure Functions supports trigger and output bindings for Service Bus
queues and topics
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus
I think that #William is right, you can use/attach other process to the subscription to make what you trying to do.
he mentioned Azure Functions which is a good tool and I want to suggest Azure Logic Apps as well in case you want to take some decisions based in the message that you received.
With Azure Logic Apps you can create a logic Workflow and integrate many services using connectors provided by this tool.
You will find more in:
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-servicebus
And for answer your question
First, I want to know if Service Bus Topic is right option to go with?
The quick answer is yes, using messaging patterns is the best way to create reliable solutions.
In your case you want as well notify another system after receiving a message.
The only thing that you need to be aware is, whenever you did not receive the notification what you'll do? you need to think about this scenario.
From discussion above.
Azure functions with Queues/Topics
Regardless of queues or topics. you can trigger azure function with both. This function will process the message . Now you can create two methods in same function SendEmail(), sendPhoneNotifcation() and parrellize the tasks using C# task parallel library. So same function will do both tasks in parallel.
Every time you get a message , your function is triggered. Process the message and notify user. The benefit is this function will scale automatically if you have large number of message coming through queue.
I have created subscription(s) for some of the existing Azure Service Bus topic(s).
Now i need to create Azure Web Jobs that will process for any new Message arriving at the Topic end. This whole process will be used as part of incremental load.
Please provide some input/reference for implementing the above work flow.
Thanks! in advance
You can have Azure Functions instead of Web Jobs to process the messages, which comes with ServiceBusTrigger. You could also go for Web Jobs to process the messages. Refer the following links for more details
https://code.msdn.microsoft.com/Processing-Service-Bus-84db27b4
https://social.technet.microsoft.com/wiki/contents/articles/31981.azure-webjobs-servicebustrigger.aspx
You should provide the Topic Subscription path in place of Queue path to receive message from Subscriptions.
For my new project every component is going to be deployed in Azure. I have a 3rd party application that processes events using RabbitMQ and I want to subscribe to these events and process them to store the data in the events in my own database.
What would be the best way to go? Using webjobs and Writing my own Custom Trigger/ Binder for RabbitMQ?
Thanks for the advice in advance
Based on your requirement, I assume that Azure WebJob is an ideal approach to achieve your purpose. In that case, you could use a WebJob as a consumer client to subscribe the events and process the data. Please try to create a WebJob and following the link provided by Mitra to subscribe the event and implement your logic processes in the WebJob.
Please pay attention that WebJob run as background processes in the context of an Azure Web App. In order to keep your WebJob running continuously, you need to be running in standard mode or highly and enable the "Always On" setting.
Consideration of scaling, you could use the Azure Websites scale feature to scale extra WebJobs instances. For scaling, you could refer to this tutorial.
For having subscription based routing, you can use Topics in Rabbitmq. Using topics you can push events to specific queues and then consumers at those queues can do processing to write data into the database. The only thing to take care of is to have a correct routing key for each queue.
That way you can have subscription based mechanism. The only thing with this approach will be that for each event there will be one queue.
The benefit of having one queue per event is it will be easy to keep track of events and so easy debugging.
If the number of events are very large then you can have only one queue but after consuming the message you have to trigger the event.
Here is the link for the reference:
https://www.rabbitmq.com/tutorials/tutorial-five-python.html
I have been researching on Microsoft Azure Event Hubs. My goal is to figure out a way to provide automatic scalability. This is a experimental work and I am really only trying to know what can I do with Azure event hubs. I do not have access to the Azure platform to test test anything :( .
Well, so far, I found that through REST API and Service Bus Powershell I can add Throughput Units (to increase performance - I am relying on this: Scale Azure Service Bus through Powershell or API) and increase or decrease Event's Expiration time (which might influence capacity - https://msdn.microsoft.com/en-us/library/azure/dn790675.aspx).
The problem is that, presuming that the previous techniques work and I am able to scale event hubs' performance automatically, I still need a way to know when to trigger scalability mechanisms. To know when and how to trigger scalability, I need to work on some functions that rely upon the event hub's metrics (or a way to monitoring it). The problem is that I can't really find any metrics. The only thing that I find is this: https://azure.microsoft.com/en-us/documentation/articles/cloud-services-how-to-monitor/ - Which actually does not solve my problem because although it may present some interesting metrics, it does not serve the purposes of my "application" (which will come if I can prove that I can successfully scale Azure automatically); and this Azure service bus statistics/Monitoring - which's links are not working.
Surely I can find more information about Service Bus Explorer, and surely it may provide some interesting insights over the event hub metrics, I am just wondering if there is something like this: https://github.com/HBOCodeLabs/incubator-storm/blob/master/STORM-UI-REST-API.md that allow me to access some kind of metrics, rather than creating my own metrics
Thanks in advance
Best regards
You can retrieve metrics about Event Hubs (an Event Hub is a Service Bus Entity) using the Service Bus Entity Metrics REST APIs(https://msdn.microsoft.com/library/azure/dn163589.aspx). Using this you can retrieve the same metrics displayed in the portal such as:
Number of incoming messages
Incoming throughput
Outgoing throughput
These should help you determine when you need to scale your application up or down.
This video is useful for getting started https://channel9.msdn.com/Blogs/Subscribe/Service-Bus-Namespace-Management-and-Analytics
If 3rd party services are an option, look into CloudMonix # http://cloudmonix.com
It can monitor Event Hubs (among gazillion other Azure-related things) and execute Azure Automation runbooks (among gazillion other actions) as a reaction to load conditions/throughout of a whole hub or individual partitions and optionally based on any other metrics in your environment.
Your Azure Automation runbooks could have the logic to execute increases in your EH's throughout, etc.
Disclaimer: I'm affiliated with the product.
HTH
Service Bus Explorer is great. I actually use this.
ServiceBus Explorer