I do have a AzureWorker that receives SMTP messages from TCP ports and pushes them to queues. Other threads pick up these messages from the queues and process them. Currently, process threads have their queue polling logic. Simply they check the queues and increase wait interval if the queues are empty.
I want to simplify the queue logic and make use of other Webjobs functionalities in this AzureWorker.
Is it possible to start a WebJobs thread in this AzureWorker and let that thread handle the details? Are there any limitations that I need to know?
Azure Worker Roles are a feature of Azure Cloud Services. Azure Web Jobs are a feature of Azure App Service. They are both built to provide similar ability to run background process tasks within the context of your application. Although, since they are features of different Azure services they can't be run together like you are asking in a nested fashion.
Is it possible to start a WebJobs thread in this AzureWorker and let that thread handle the details?
I agree with Chris Pietschmann, it does not enable us to start WebJobs thread directly in Azure Worker Role.
Other threads pick up these messages from the queues and process them. Currently, process threads have their queue polling logic. Simply they check the queues and increase wait interval if the queues are empty.
I want to simplify the queue logic and make use of other Webjobs functionalities in this AzureWorker.
If you’d like to complete this task by using WebJobs, you could write a program and run as a WebJobs in your Azure App Service. And WebJobs API provides a way to dynamically start/stop WebJobs via REST API, you could use it to manage your WebJobs in your Worker Role.
Related
We are currently using Azure more like IAAS, rather than cloud services.
As a start, I would like to utilise Azure Messaging Queues to process some database actions and Web API calls.
I am assuming I would need to write another piece of code that subscribes to the queues, so when messages arrive, it knows to process the transaction?
Is that piece of code, a console app? runs on a scheduled task? a windows service? or a function app within azure?
What is the Best Practice for this architecture?
You can write console app and schedule web job to monitor the queue. However better way is to use Azure Functions. You don't have to monitor the queue then. Whenever the message arrives in the queue it will trigger Azure Function and you can process the message. The benefit is it's server-less.
I currently have a webjob that runs when user does specific action.
When the webjob is running it cannot accept a new request until its done with the current process.
Is it possible to run webjob simultaneously, meaning be able to process a job for multiple clients at a time?
Absolutely. But how are you triggering your WebJob when a user performs a specific action? If you trigger a method within your WebJob via a QueueTrigger that listens for a message on an Azure Storage Queue, you can have multiple threads processing messages at once. You can do something similar using Service Bus queues as well.
Check out the docs in the parallel execution section here.
I'm using Azure webjobs with queue-triggered functions (which rely on the Azure webjobs sdk) to perform some background processing work. Within the webjobs I make various connects to a SQL Azure database (using PetaPoco which uses System.Data.SqlClient).
I want to be purposeful in my database connection strategy - specifically because there are some concurrency issues inherent to the environment.
One concurrency scenario is with the SDK's BatchSize property that you can set for queue-triggered webjobs. It's my understanding that setting BatchSize > 1 results in multiple instances of the queue-triggered function running within the same webjob process.
The second concurrency scenario is the website scale-out scenario where you're running multiple instances of the webjob itself. These of course are in different processes.
In my website I have a database connection per request (the machine handles connection pooling by default). No problems there.
How should I treat connections in the webjob scenario, accounting for the concurrency scenarios described above? Webjobs are of course just long-lived console processes (these are continuous webjobs). Should I create a database connection when my webjob starts and simply re-use that connection through the webjob's lifetime? Should I instantiate and close connections per function when I need them?
These are the types of things I'm trying to understand.
Webjobs are of course just long-lived console processes (these are continuous webjobs).
The main process is the long-lived processes , but for trigged sub- process will be released after the triggered function is executed. It means that connection will also be released automatically in the sub-process. For best program practices that we 'd better close it manually before exit function.
The second concurrency scenario is the website scale-out scenario where you're running multiple instances of the webjob itself. These of course are in different processes.
WebJob SDK queue trigger will automatically prevents a queue triggered by multiple instances.
If your web app runs on multiple instances, a continuous WebJob runs on each machine, and each machine will wait for triggers and attempt to run functions. The WebJobs SDK queue trigger automatically prevents a function from processing a queue message multiple times; functions do not have to be written to be idempotent. However, if you want to ensure that only one instance of a function runs even when there are multiple instances of the host web app, you can use the Singleton attribute.
It's my understanding that setting BatchSize > 1 results in multiple instances of the queue-triggered function running within the same webjob process
BatchSize it means that how many queue messages that can be picked up simutaneouly to be executed in Parallel in a WebJob.
How to use Azure queue storage with the WebJobs SDK induling parallel execution and multiple instances, we could get more info from the doucment.
For my new project every component is going to be deployed in Azure. I have a 3rd party application that processes events using RabbitMQ and I want to subscribe to these events and process them to store the data in the events in my own database.
What would be the best way to go? Using webjobs and Writing my own Custom Trigger/ Binder for RabbitMQ?
Thanks for the advice in advance
Based on your requirement, I assume that Azure WebJob is an ideal approach to achieve your purpose. In that case, you could use a WebJob as a consumer client to subscribe the events and process the data. Please try to create a WebJob and following the link provided by Mitra to subscribe the event and implement your logic processes in the WebJob.
Please pay attention that WebJob run as background processes in the context of an Azure Web App. In order to keep your WebJob running continuously, you need to be running in standard mode or highly and enable the "Always On" setting.
Consideration of scaling, you could use the Azure Websites scale feature to scale extra WebJobs instances. For scaling, you could refer to this tutorial.
For having subscription based routing, you can use Topics in Rabbitmq. Using topics you can push events to specific queues and then consumers at those queues can do processing to write data into the database. The only thing to take care of is to have a correct routing key for each queue.
That way you can have subscription based mechanism. The only thing with this approach will be that for each event there will be one queue.
The benefit of having one queue per event is it will be easy to keep track of events and so easy debugging.
If the number of events are very large then you can have only one queue but after consuming the message you have to trigger the event.
Here is the link for the reference:
https://www.rabbitmq.com/tutorials/tutorial-five-python.html
I want to create an Azure application which does the following:
User is presented with a MVC 4 website (web role) which shows a list of commands.
When the user selects a command, it is broadcast to all worker roles.
Worker roles process the task, store the results and notify web role
Web role displays the combined results of the worker roles
From what I've been reading there seem to be two ways of doing this: the Windows Azure Service Bus or using Queues. Each worker role also stores the results in the database.
The Service Bus seems more appropriate with its publish/subscribe model, so all worker roles would get the same command and roughly the same time. Queues seem easier to use though.
Can the service bus be used locally with the emulator when developing? I am using a free trial and cannot keep the application constantly whilst still developing. Also, when using queues how can you notify the web role that processing is complete?
I agree. ServiceBus is a better choice for this messaging requirement. You could, with some effort, do the same with queues. But, you'll be writing a lot of code to implement things that the ServiceBus already gives you.
There is not a local emulator for ServiceBus like there is for the Azure Strorage service (queues/tables/blobs). However, you could still use the ServiceBus for messaging between roles while they are running locally in your development environment.
As for your last question about notifying the web role that processing is complete, there are a several ways to go here. Just a few thoughts (not exhaustive list)...
Table storage where the web role can periodically check the status of the unit of work.
Another ServiceBus Queue/topic for completed work.
Internal endpoints. You'll have to have logic to know if it's just an update from worker role N or if it is indicating a completed unit of work for all worker roles.
I agree with Rick's answer, but would also add the following things to think about:
If you choose the Service Bus Topic approach then as each worker role comes online it would need to generate a subscription to the topic. You'll need to think about subscription maintenance of when one of the workers has a failure and is recycled, or any number of reasons why a subscription may be out there.
Telling the web role that all the workers are complete is interesting. The options Rick provides are good ones, but you'll need to think about some things here. It means that the web role needs to know just how many workers are out there or some other mechanism to decide when all have reported done. You could have the situation of five worker roles receieving a message and start working, then one of them starts to repeatedly fail processing. The other four report their completion but now the web role is waiting on the fifth. How long do you wait for a reply? Can you continue? What if you just told the system to scale down and while the web role thinks there are 5 there is now only 4. These are things you'll need to to think about and they all depend on your requirements.
Based on your question, you could use either queue service and get good results. But each of them are going to have different challenges to overcome as well as advantages.
Some advantages of service bus queues is that it provides blocking receipt with a persistent connection (up to 100 connections), it can monitor messages for completion, and it can send larger messages (256KB).
Some advantages of storage queues over the service bus solution is that it's slightly faster (if 15 ms matters to you), you can use a single storage system (since you'll probably be using Storage for blob and table services anyways), and simple auto-scaling. If you need to auto-scale your worker roles based on the load, passing the the requests through a storage queue makes auto-scaling trivial -- you just setup auto-scaling in the Azure Cloud Service UI under the scale tab.
A more in-depth comparison of the two azure queue services can be found here: http://msdn.microsoft.com/en-us/library/hh767287.aspx
Also, when using queues how can you notify the web role that processing is complete?
For the Azure Storage Queues solution, I've written a library that can help: https://github.com/brentrossen/AzureDistributedService.
It provides a proxy layer that facilitates RPC style communication from web roles to worker roles and back through Storage Queues.