I have been struggling with this concept for a while. I am attempting to come up with a loosely coupled Azure component design that is completely scalable using Queues and worker roles, which dequeue and process the items. I can scale the worker roles at will, and publishing to the queue is never an issue. So far so good, but, it seems that the only real world model this could work in is fire and forget. It would work fantastic for logging and other one way operations, but let's say I want to up load a file using queues/worker roles, save it to blob, then get a response back once it is complete. Or should this type of model not be used for online apps? What is the best way to send a notification back once an operation is completed? Do I create a response Q, then (somehow) retrieve the associated response? Any help is greatly appreciated!!!!!
I usually do a polling model.
Client (usually a browser) sends a request to do some work.
Front-end (web role) enqueues the work and replies with an ID.
Back-end (worker role) processes the queue and stores the result in a blob or table entity named .
Client polls ("Is done yet?") at some interval.
Front-end checks to see if the blob or table entity is there and replies accordingly.
See http://blog.smarx.com/posts/web-page-image-capture-in-windows-azure for one example of this pattern.
you could also look into the servicebus appfabric instead of using queues. with the servicebus you can send messages, use queues etc all from the servicebus appfabric. you could go to publish and subscribe instead of polling then!
Related
I’m in need of some second opinions and guidance on how to use Azure Functions in combination with Azure Service Bus in the scenario described below. Coding is not an issue its about selecting the most appropriate method. Sadly, I have not found any good example of this online so now I’m reaching out for some help.
Scenario
I have an ecommerce customer that is sending a few thousand orders a day to an ERP system. The normal day operations are not an issue, but we would like to make the solution more robust to handle for example “Black Friday” surges. Currently the website can hold x amount of orders before that database is full and is forced to close or send order downstream. Currently the website sends order directly into the ERP system and it is this part I want to decouple with Azure Server Bus Queues. With this decoupling we can continue pushing new orders to the queue and consuming these at our own pace in the ERP without flooding any system.
My thoughts about how to set this up
The website can send messages directly to the Service Bus Queue. An Azure Function is bound to trigger on every new message in the queue and will send that message to the ERP system.
Same as above however the website first sends a message to an Azure Function that puts it into the queue.
The website sends messages to the queue like in point 1 or 2. Instead of binding a function to the queue we setup a scheduled function. The function will run frequently and send 1 message to the ERP system per run.
The website sends messages to the queue like in point 1 or 2. Here we do not send messages to the ERP system but instead the ERP system is the one who reads the queue. Do not like this approach but its possible to do and easy to administrate by ERP users.
Questions
If I go with point 1 or 2 above should the function responsible for delivering the message to the ERP system send 1 or multiple orders per trigger?
If I go with point 1 or 2 it should still be possible to flood the ERP system since they most likely trigger at the same time they get put in?
If the ERP system is down and the queue grows, do I need a separate scheduled function to handle the queue until it is empty?
We do not have to discuss the dead letter queue here, that is another topic.
How would you approach this or if you have done a similar solution what method did you use?
Thank you for your guidance much appriciated!
We've learned a lot in the past couple of years working with Azure Functions and Service Bus to solve similar scenarios you mentioned above. You're definitely on the right track regards to wanting to decouple in case of a surge. To give you some peace of mind with your choice to use Azure Service Bus, we normally push hundreds of events a minute through our topics and subscriptions and it holds up pretty well.
Let me just share some of the lessons we learned:
The concurrent number of incoming requests within the same second
was one of our breaking points. The website when written properly
will easily accommodate multiple incoming requests but we learned
about "port exhaustion" related to outbound web requests to our
Azure Function. Review the scope and lifetime of your web client and
the limits of your app service plan / web server.
If you choose to use a consumption plan for your Azure Function, be
aware that it sometimes takes a long time to start. Whatever is
hitting the function will have to implement retries (probably a good
practice anyway).
A Service Bus Message has a size limit (which can be increased, but
there's still a limit). We randomly hit it with one of our payloads
that contains a bulk of information. Know the worst case scenario
payload size you may encounter.
In the event something goes wrong and there are tens of thousands of
messages in the queue, there is no easy way to query what's in
there. Make sure you're fine with that otherwise consider
doing fast writes into a database that can be queried.
The Azure Function can be triggered by Service Bus and can spawn
multiple concurrent executions of the code (which is desired) and
with a limit. Be aware of any limitations with code to update your
ERP. You will have no control over Service Bus triggers.
Be conscious about the function's storage account, functions with
same name will have their trigger settings and locks stepping on
each other (dev vs. prod environment).
Connections to Azure Service Bus will sometimes fail, just the
nature of services hosted in the cloud. It only happens a few times
and recovers after a few seconds.
Consider doing this:
Website -> Azure API Management Gateway -> Azure Function A -> Service Bus -> Azure Function B -> ERP
Azure API Management with AppInsights enabled is a nice extra layer allowing you to secure, monitor, and route to your Function A. In cases where you need to route incoming requests to some emergency bucket it's a life saver.
Consider allowing function 1 to accept an array of your items. Enable AppInsights, add code for telemetry, providing preview of throughput in terms of orders.
Function B with a configurable timer trigger and some app configuration for number of messages to process from the queue. Allows you to throttle flow of data to your ERP. This may be debatable as you won't be able to scale this function out with multiple instances, but I'm assuming the original concern was to control the pace. Also enable same AppInsights, telemetry, logging, etc.
I'm hoping I don't draw too much criticism from this. We learned the hard way and eventually received some really good guidance from Azure architects and engineers later.
I have an online service hosted on Azure, that asynchronously sends data to on-premise clients.
Each client is identified by an unique code.
Actually there is a single topic, with a subscription for each client which has a filter on the unique code, that is sent as a parameter in the message. No message will ever be broadcasted to all the clients.
I feel that using topic this way is wrong.
The alternative that comes to my mind is to use a dedicated queue for each client, that is created on first contact
Could this be a better approach?
Thanks
In my opinion using Topics and Subscriptions is the right way to go. Here's the reason why:
Currently the routing logic (which message needs to go to which subscription) is handled by Azure Service Bus based on the rules you have configured. If you go with queues, the routing logic will need to come to your hosted service. You'll need to ensure that the queue exists before sending each message. I think it will increase the complexity at your service level somehow.
Furthermore, topics and subscriptions would enable you to do build an audit trail kind of functionality (not sure if you're looking for this kind of functionality). You can create a separate subscription that has a rule to deliver all messages (True SQL Rule) to that subscription along with client specific subscription.
Creating a separate Queue for each client is not advisable. This is the problem solved by Topics.
If you have separate Queue for each client, then you need to send messages to multiple Queues from Server. This will become tedious when the number of clients increases.
Having a single Topic and multiple Subscriptions is easy to manage as the message will be sent only to a single Topic from Server.
We are evaluating "Azure service bus" to use between web server and app server for request response pattern. We are planning to have two queues:
Request Queue
Response Queue
Webserver will push a message to request queue and subscribe to response queue.
By comparing the MessageID and CorrelationId, it can receive the response back, which can be sent back to browser.
But over cloud, using elastic scaling, we can increase/decrease web server (and app server) instances.
We are wondering if this pattern will work here optimally.
To make this work, we will have to have one Request queue and multiple topics (one for each web server instance).
This will have two down sides:
Along with increasing/decreasing web server instance, we will have
to create/delete topic as well.
All the message will be pushed to
all the topics. So, every message will be processed by all the web
servers. And this is not an efficient way.
Please share your thoughts.
Thanks In Advance
When you scale out your endpoint, you don't want to have an instance affinity. You want to rely on the competing consumers and not care which instance of your endpoint processes messages.
For example, if you receive a response and write that to a database, most likely you don't care what instance of an endpoint has written the data. But if you have some in-memory state or anything other info only available to the endpoint that has originated the request and processing reply messages requires that information, then you have instance affinity and need to either remove it or use technology that allows to address that. For example, something like a SignalR with a backplane to communicate a reply message to all your web endpoint instances.
Note that ideally you should avoid instance affinity as much as you can.
I know this is old, but thought I should comment to complete this thread.
I agree with Sean.
In principle, Do not design with instance affinity in mind.
Any design should work irrespective of number of instances and whichever instance runs the code.
Microsoft does recommend the same when designing application architecture for running in the cloud.
In your case, I do not think you should plan to have one topic for each instance.
You should just put the request messages into one topic, with a subscription to allow your receiving app service to process those request messages.
When your receiving app service scales out, that's where your design needs to allow reading messages from the subscription from multiple receivers (multiple instances), which is described in the Competing consumers pattern.
https://learn.microsoft.com/en-us/azure/architecture/patterns/competing-consumers
Please post what you have finally implemented.
For my new project every component is going to be deployed in Azure. I have a 3rd party application that processes events using RabbitMQ and I want to subscribe to these events and process them to store the data in the events in my own database.
What would be the best way to go? Using webjobs and Writing my own Custom Trigger/ Binder for RabbitMQ?
Thanks for the advice in advance
Based on your requirement, I assume that Azure WebJob is an ideal approach to achieve your purpose. In that case, you could use a WebJob as a consumer client to subscribe the events and process the data. Please try to create a WebJob and following the link provided by Mitra to subscribe the event and implement your logic processes in the WebJob.
Please pay attention that WebJob run as background processes in the context of an Azure Web App. In order to keep your WebJob running continuously, you need to be running in standard mode or highly and enable the "Always On" setting.
Consideration of scaling, you could use the Azure Websites scale feature to scale extra WebJobs instances. For scaling, you could refer to this tutorial.
For having subscription based routing, you can use Topics in Rabbitmq. Using topics you can push events to specific queues and then consumers at those queues can do processing to write data into the database. The only thing to take care of is to have a correct routing key for each queue.
That way you can have subscription based mechanism. The only thing with this approach will be that for each event there will be one queue.
The benefit of having one queue per event is it will be easy to keep track of events and so easy debugging.
If the number of events are very large then you can have only one queue but after consuming the message you have to trigger the event.
Here is the link for the reference:
https://www.rabbitmq.com/tutorials/tutorial-five-python.html
I have a web api application which performs different type of actions an a Domain entity, called Application. Actions like "Copy", "Update", "Recycle", "Restore" etc.
This actions needs to be executed, per Application, in First In First Out order, not randomly or simultaneous. However, it can process simultaneously two Actions as long as they are for two separate Applications.
Is some kind of a queue, but not a big queue for all the requests, but a queue of actions for each Application in database.
Knowing this, i think that azure service bus queue is a good solution for this scenario.
However, the solution i can think of right now is to programmatically create a queue for each Application i have in database, and start listening to that queue.
Is possible to get messages from the queue based on a filter? (using FIFO principle) So i have to subscribe only to one queue? (instead of having a queue for each Application - which is very hard to maintain)
What you want is Azure Service Bus Topics/Subscriptions.
Subscriptions allow you to filter messages that are published to a topic using a SqlFilter on the message headers.
The article linked above should provide enough examples to meet your needs.
I think u can solve this by using Sessions.
I just came across this very clear article: https://dev.to/azure/ordered-queue-processing-in-azure-functions-4h6c which explains in to detail how Azure Service Bus Queue sessions work.
In short: by defining a SessionId on the messages you can force the ordering of the processing within a session, the downside is that there will be no parallelization for all messages in a session between multiple consumers of the queue.