I have multiple WebApi that are hosted in Azure but some of the API taking a long time to process.
Instead of a calling from the scheduler function in Azure, I was advised to use the Messaging Queue.
Is this a good approach?
Also, let say my API URL.. https://testwebapi.net/api1 and https://testwebapi.net/api2, how can we communicate these API using Message queue and also how to call the WebAPI individually from Messaging Queue?
Is this Azure Bus service is the same as Messaging Queue for Azure.
I would strongly recommend you look at the respective docs on Azure, and start doing some quick tutorials and post more specific questions.
Now, to answer, what I believe is your main question, Azure Bus Service and Messaging Queue.
The thing is, there is no one Messaging Queue, rather three options for Azure messaging - Service Bus, Event Grid, and Event Hubs. Also, there is something called 'Azure Queue Storage', but I dont think you are thinking of that, considering your context.
There is a very cool comparison chart available here - https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services
But, I think, my guess, is you are looking for ultimately looking to use Azure Service Bus.
Also, please understand that one does not invalidate the other. As your project develops, you may end up using Azure Service Bus with other options like Event Hubs or Grids or all of them in different places.
Also, you keep using the incorrect terms. for example, you say 'Azure Bus service' but its really 'Azure Service Bus'. Also, there is no thing called 'Messaging Queue for Azure.' as there are so many different messaging services for Azure. This is what I want to add here. Spent some time with the docs and post more specific questions as you continue your project.
Related
I'm new to azure cloud and my assignments requires me to implement service bus queues.
Question: Implement the Service Bus queues in your application. By using the queues and Service Bus, you will be able to manage the communication between the N-tier applications in Azure.
I have made a library management system using .net C# and not in MVC. and deployed it in the cloud. I don't know how to implement it.
Do i need to make new project for service bus or can i implement it on the existing system.
I have made a form page with Fullname, Email, Subject and Message Box. there is submit Button and onclick event can i implement service bus queue? help please. Thank you for time. Sorry for asking question in unclear manner. New to stackoverflow.
I believe you have already create the service bus queue, still you may check https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-quickstart-portal for your reference.
Then, you need to leverage the nuget pacakges in your project so that you can send messages to and receive messages from a Service Bus queue :
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues
For more related details : https://github.com/toddkitta/azure-content/blob/master/articles/service-bus/service-bus-dotnet-how-to-use-queues.md
https://www.c-sharpcorner.com/article/azure-service-bus-queues/
https://www.c-sharpcorner.com/article/azure-service-bus-and-queue-implementation-using-c-sharp-small-intro-and-sample-app/
https://azuresdkdocs.blob.core.windows.net/$web/dotnet/Azure.Messaging.ServiceBus/7.0.0-preview.1/api/index.html
i want to develop a Game using Azure Cloud but i am not sure which ressources i could use for it. Folowing is my problem:
There is one publisher which is providing some messages. I need to find a way to add subscriber on demand. That means whenever i need more subscriber i need to add them without changing any settings on the portal.
All subscriber needs to get the published message at the same time.
Received message from the publisher needs to be deleted for all subscriber at the same time.
Is there a ressource or a way which fits theese needs?
Thanks.
Azure SignalR Service is WebSocket Gateway service that covers all your requirements.
While you could always host your own server, the service supports scaling out.
Using Azure Service Bus - Topics, I want to implement a solution wherein messages are sent/notified to end consumers once the producer sends the message to Topic (like Queues).
I understand that Topics work as Pub/Sub model wherein subscribers need to read messages from subscriptions. But I'm looking for a workaround that works some what similar to Queue (where it triggers a web job / service when any message is received).
I have few thoughts like
1. Using Auto-Forwarding in subscriptions to forward messages to Queues but again I think if this kills the purpose of Topics
2. Schedule a job to process these requests but again I think if I'm delaying the process
First, I want to know if Service Bus Topic is right option to go with? Next, If possible to implement a workaround what is the best/better way?
PS: I have to send messages which has information - I guess I can't use Relays
Just to be clear, Queues and Topics in Service Bus are different. As you noted, Topics are useful in publish/subscribe scenarios.
Since you are looking for something that gets triggered, Azure functions might be what you need.
Azure Functions supports trigger and output bindings for Service Bus
queues and topics
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus
I think that #William is right, you can use/attach other process to the subscription to make what you trying to do.
he mentioned Azure Functions which is a good tool and I want to suggest Azure Logic Apps as well in case you want to take some decisions based in the message that you received.
With Azure Logic Apps you can create a logic Workflow and integrate many services using connectors provided by this tool.
You will find more in:
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-servicebus
And for answer your question
First, I want to know if Service Bus Topic is right option to go with?
The quick answer is yes, using messaging patterns is the best way to create reliable solutions.
In your case you want as well notify another system after receiving a message.
The only thing that you need to be aware is, whenever you did not receive the notification what you'll do? you need to think about this scenario.
From discussion above.
Azure functions with Queues/Topics
Regardless of queues or topics. you can trigger azure function with both. This function will process the message . Now you can create two methods in same function SendEmail(), sendPhoneNotifcation() and parrellize the tasks using C# task parallel library. So same function will do both tasks in parallel.
Every time you get a message , your function is triggered. Process the message and notify user. The benefit is this function will scale automatically if you have large number of message coming through queue.
I would appreciate some help picking out the best suited Azure services for my scenario - I am just beginning with Azure services and my knowledge is pretty limited.
I have data from multiple sources, and of different shapes, coming into an Event Hub. I need to subscribe to the events from the Event Hub and, based on their format, process them and ultimately save them into an SQL Database. All components - events consumers, the SQL Database - need to be hosted in the cloud.
How would I implement this in an "Azure Orientated Architecture"?
In an off cloud application, I would have competing consumers subscribing to the Event Hub. They would be some console applications or Windows services, and each would be processing the events asynchronously (this is further simplified by the event processing being idempotent).
Ideally, the Azure equivalent of the above consumers would scale up and down automatically, so I would like to not have to use VMs that host console applications (where I would need to keep an eye on the VM's resources myself). Scaling and deployment wise they would have to behave like App Services, however I'm under the impression that those are just for web applications. I've also briefly looked at Web Jobs, but those seem to be polling data at various intervals, whereas I need a proper event subscriber that the Event Hub pushes data into.
Any help will be greatly appreciated!
Thank you.
Later Edit:
I've looked into Web Jobs and they do allow continuous
processing, hence looks like they can be used as automatically
scaling subscribers.
Ideally I would like to write the code for
the subscribers in F#. C# is the other option if that is not
available.
You can see my post regarding IoT Hub. Its basically the same for Event Hub.
(each of the examples in the post can be used on Event Hubs).
https://stackoverflow.com/a/38682324/6659347
In addition, For Event Hub you can also use Azure Function which has an Event Hub trigger - a function that will run whenever an event hub receive a new event. And it will also answer your requirement of scaling.
Make sure that if you are working with multiple consumers make use of the Event Hub Consumer Groups so each consumer can read the stream independently.
I'd say use a WebJob in combination with an EventProcessor. I wrote some demo code that can easily be transferred to a WebJob: https://github.com/DeHeerSoftware/SemanticLogging.EventHub/tree/master/SemanticLogging.EventHub.Processor
See https://azure.microsoft.com/en-us/documentation/articles/event-hubs-csharp-ephcs-getstarted/#receive-messages-with-eventprocessorhost for official documentation.
I've created a WebJob myself using this approach. Works like a charm.
I'm simply trying to work out how best to retrieve messages as quickly as possible from an Azure Service Bus Queue.
I was shocked that there wasn't some way to properly subscribe to the queue for notifications and that I'm going to have to poll. (unless I'm wrong in which case the documentation is terrible).
I got long polling working, but checking a single message every 60 seconds looks like it'll cost around £900 per month (again, unless I've misunderstood that). And if I add a redundant/second service to poll it'll double.
So I'm wondering what the best/most cost efficient way of doing it is.
Essentially I just want to take a message from the queue, perform an API lookup on some internally held data (perhaps using hybrid services?) and then perhaps post a message back to a different queue with some additional information .
I looked at worker roles(?) -- is that something that could do it?
I should mention that I've been looking at doing this with node.js.
Check out these videos from Scott Hanselman and Mark Simms on Azure Queues.
It's C# but you get the idea.
https://channel9.msdn.com/Search?term=azure%20queues%20simms#ch9Search
Touches on:
Storage Queues vs. Service Bus Queues
Grabbing messages in bulk vs. one by one (chunky vs. chatty)
Dealing with poison messages (bad actors)
Misc implementation details
Much more stuff i can't remember now
As for your compute, you can either do a VM, a Worker Role (Cloud Services), App Service Webjobs, or Azure Functions.
The Webjobs SDK and Azure Functions bot have a way to subscribe to Queue events (notify on message).
(Listed from IaaS to PaaS to FaaS - Azure Functions - if such a thing exists).
Azure Functions already has sample code provided as templates to do all that with Node. Just make a new Function and follow the wizard.
If you need to touch data on-prem you either need to look at integrating with a VNET that has site-to-site connectivity back to your prem, or Hybrid Connections (App Service only!). Azure Functions can't do that yet, but every other compute is a go.
https://azure.microsoft.com/en-us/documentation/articles/web-sites-hybrid-connection-get-started/
(That tutorial is Windows only but you can pull data from any OS. The Hybrid Connection Manager has to live on a Windows box, but then it acts as a reverse proxy to any host on your network).
To deal with Azure ServiceBus Queue easily, the best option seems to be Azure Webjob.
There is a ServiceBusTrigger that allows you to get messages from an Azure ServiceBus queue.
For node.js integration, you should have a look at Azure Function. It is built on top of the webjob SDK and have node.js integration :
Azure Functions NodeJS developer reference
Azure Functions Service Bus triggers and bindings for queues and topics
In the second article, there is an example on how get messages from a queue using Azure Function and nodejs :
module.exports = function(context, myQueueItem) {
context.log('Node.js ServiceBus queue trigger function processed message', myQueueItem);
context.done();
};