I am writing an Azure hosted MVC website for a gym booking system. I need to be able to maintain membership expiry, suspensions as well as gym class attendence (i.e. logging to the database if a session has been missed). Each of these tasks requires a "c# service function" to be run that will go through the database, perform some checks and update records as and when required.
I need this to run pretty regularly to ensure that missed sessions are logged asap. Should I be developing this as an Azure WebJob and running it continuously? Should i be doing it in another manner? If I could get some suggestions on routes to take that would be massively appreciated.
Thanks
You have a few options: Web Jobs, Scheduler, and Worker Roles.
Web Jobs are a nice addon to an existing azure web app and have the benefit of no additional cost. Web Jobs use Scheduler under the covers if you choose to schedule the Web Job to run at an interval other than continuously. Here is a nice answer that describes the differences between the two.
Worker Roles would be the next logical step up from a Web Job. Worker Roles are dedicated Cloud Service VMs that can provide more dedicated power and offer greater scaling capabilities. Worker Roles can also do much more than just run jobs.
For the application you have described, if you are already running on Azure App Services (Web App) it sounds like a continuously running Web Job would be the correct choice.
Related
We are on Azure since 2010 and had a great benefit from a performance and reliability in our application. Azure offers a lot of enterprise-level services and I think that the new "Azure Service Fabric" is great.
What I cannot understand by reading the documentation is the approach on migrating an "old" Cloud Service to the new Service Fabric. Why do we want to migrate? For horizontal scaling and more reliability.
Currently we have a single-instance cloud service, that spins up a lot of subservices. Those subservices are great candidates for microservices. The only problem is that some of these subservices are "runners", i.e. they just cycle on our users database and decide whether an operation (service) has to be run for a particular user or not.
How would you migrate a service like this considering that more than one instance may run this service?
Thanks
First thing to keep in mind is that once a service is started it keeps running, and his lifecycle and uptime is controlled by Service Fabric (ex: it will restart it automatically if it crashes). Second thing to keep in mind is that you will end-up with multiple instances of the service running at the same time (on different nodes), so they will end-up doing the exact same thing on different nodes of your cluster.
Your first reflex could be to have one stateless service kind/instance per runner "subservice" that keeps running and leverage the RunAsync (https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-advanced-usage). Personally, I wouldn't take that approach, since this could then require some kind of synchronization between services to prevent useless concurrency, since they do the exact same thing independently.
A better approach would be to have your runner services need to run only once in a while when requested by the "main" service acting as an orchestrator, you could have a Queue based approach where the "main" service submit tasks (messages) to be processed by the runners, who are listening concurrently on the same Queue, making sure that maximum one service instance would complete the task.
For the Queue, think Service Bus or Reliable Concurrent Queue (https://learn.microsoft.com/enus/dotnet/api/microsoft.servicefabric.data.collections.preview.ireliableconcurrentqueue-1).
We currently have a window service which send some notification emails to users after doing some processing on database(SQL database). Runs once in day.
We want to move this on azure cloud. One alternate is to put it on Azure VM as is. but I am finding some other best possible solution for that.
I study about recurring and on demand Web jobs but I am not sure is this is best solution.
Also is there any possibility to update configuration of service code in App.config without re-deploy the code of service on cloud. I means we can manage configuration from Azure portal.
Thanks in advance.
Update 11/4/2016
Since this was written, there are 2 additional features available in Azure that are both excellent choices depending on what functionality you need:
Azure Functions (which was based on the WebJobs described below): Serverless code that can be trigger/invoked in various ways, and has scaling support.
Azure Service Fabric: Microservice platform, with support for actor model, stateful and stateless services.
You've got 3 basic options:
Windows service running on VM
WebJob
Cloud service
There's a lot of information out there on the tradeoffs between these choices, but here's a brief summary.
VM - Advantages: you can move your service basically as it is without having to change much or any of your code. They also have the easiest connectivity with other resources in Azure (blob storage, virtual networks, etc). The disadvantage is you're giving up all the of PaaS advantages and are still stuck managing your own VM infrastructure
WebJob - Advantages: Multiple invocation options (queues, blobs, manually, queue receive loops, continuous while-loop style, etc), scheduled (would cover your case). Easy to deploy (can go with website, as a console app, automatically through Kudu), has some built in logging in Azure portal - and yes, to answer your question, you can alter the configuration in the portal itself for connection strings and app settings.
Disadvantages - you'll need to update code, you don't have access to underlying resources (if you need that), and more of something to keep in mind than a disadvantage - it uses the same resources as the webapp it's deployed with.
Web Jobs are the newest of the options, but at the same time appear to have active development going on to increase the functionality and usefulness.
Cloud Service - like a managed VM, has some deployment options, access to underlying VM if needed. Would require some code changes from your existing service.
There's nothing you've mentioned in your use case that makes me think a Web Job shouldn't be first thing you try.
(Edit: Troy Hunt has a great and relatively recent blog post illustrating most of the points I've mentioned about Web Jobs above: http://www.troyhunt.com/2015/01/azure-webjobs-are-awesome-and-you.html)
I'm trying to figure out a solution for recurring data aggregation of several thousand remote XML and JSON data files, by using Azure queues and WebJobs to fetch the data.
Basically, an input endpoint URL of some sort would be called (with a data URL as parameter) on an Azure website/app. It should trigger a WebJobs background job (or can it continuously running and checking the queue periodically for new work), fetch the data URL and then callback an external endpoint URL on completion.
Now the main concern is the volume and its performance/scaling/pricing overhead. There will be around 10,000 URLs to be fetched every 10-60 minutes (most URLs will be fetched once every 60 minutes). With regards to this scenario of recurring high-volume background jobs, I have a couple of questions:
Is Azure WebJobs (or Workers?) the right option for background processing at this volume, and be able to scale accordingly?
For this sort of volume, which Azure website tier will be most suitable (comparison at http://azure.microsoft.com/en-us/pricing/details/app-service/)? Or would only a Cloud or VM(s) work at this scale?
Any suggestions or tips are appreciated.
Yes, Azure WebJobs is an ideal solution to this. Azure WebJobs will scale with your Web App (formerly Websites). So, if you increase your web app instances, you will also increase your web job instances. There are ways to prevent this but that's the default behavior. You could also setup autoscale to automatically scale your web app based on CPU or other performance rules you specify.
It is also possible to scale your web job independently of your web front end (WFE) by deploying the web job to a web app separate from the web app where your WFE is deployed. This has the benefit of not taking up machine resources (CPU, RAM) that your WFE is using while giving you flexibility to scale your web job instances to the appropriate level. Not saying this is what you should do. You will have to do some load testing to determine if this strategy is right (or necessary) for your situation.
You should consider at least the Basic tier for your web app. That would allow you to scale out to 3 instances if you needed to and also removes the CPU and Network I/O limits that the Free and Shared plans have.
As for the queue, I would definitely suggest using the WebJobs SDK and let the JobHost (from the SDK) invoke your web job function for you instead of polling the queue. This is a really slick solution and frees you from having to write the infrastructure code to retrieve messages from the queue, manage message visibility, delete the message, etc. For a working example of this and a quick start on building your web job like this, take a look at the sample code the Azure WebJobs SDK Queues template punches out for you.
From what I understand both run small repeatable tasks in the cloud.
What reasons and in what situations might I want to choose one over the other?
Some basic information:
WebJobs are good for lightweight work items that don't need any customization of the environment they run in and don't consume very much resources. They are also really good for tasks that only need to be run periodically, scheduled, or triggered. They are cheap and easy to setup/run. They run in the context of your Website which means you get the same environment that your Website runs in, and any resources they use are resources that your Website can't use.
Worker Roles are good for more resource intensive workloads or if you need to modify the environment where they are running (ie. a particular .NET framework version or something installed into the OS). Worker Roles are more expensive and slightly more difficult to setup and run, but they offer significantly more power.
In general I would start with WebJobs and then move to Worker Roles if you find that your workload requires more than WebJobs can offer.
If we are to measure "power" as computational power, then in a virtual environment, this translates to how many layers are on top of the physical machine (the metal). The user code on a virtual machine runs on top of a hypervisor, which manages the physical machine. This is the thickest layer. Whenever possible the hypervisor tries to simply serve as a pass-through to the metal.
There is fundamentally little overhead for WebJobs. It is sandboxed, OS is maintained, and there are services & modules to make sure it runs. But the application code is essentially as close to the metal as in Worker Roles, since they use the same hypervisor.
If what you want to measure is "flexibility", then use Worker Roles, since it is not managed or sandboxed, it is more flexible. You are able to use more sockets, define your own environment, install more packages, etc.
If what you want is "features", then WebJobs has a full array of features. Including, virtual-networking to on-prem resources, staging environments, remote debugging, triggering, scheduling, easy connection to storage and service bus, etc...
Most people want to focus on solving their problem, and not invest time in infrastructure. For that, you use WebJobs. If you do find that you need more flexibility, or the security sandbox is preventing you from doing something that can't be accomplished any other way, then move to Worker Roles.
It is even possible to build hybrid solutions where some parts are done in WebJobs and others are done in Worker Roles, but that's out of the scope of this question. (hint: WebJobs SDK)
Somethings to remember when choosing to use a Web Job or a Worker Role:
A Worker Role is self hosted on a dedicated VM, a Web Job is hosted in a Web App container.
A Worker Role will scale independently, a Web Job will scale along with the Web App container.
Web Jobs are perfect for polling RSS feeds, checking for and processing messages and for sending notifications, they are lightweight and cheaper than Worker Roles but are less powerful.
A very simple question:
Why would someone use the Azure Scheduler if Azure WebJobs are free?
I couldnt find any topic regarding "azure webjobs vs azure scheduler"
The main difference is that the webjob contains everything that the scheduler can do:
Scheduler can make HTTP calls
WebJob can do that and more (run SQL commands, etc)
The actual scheduling bits of WebJobs are built on top of the scheduler. When you set up a Web Job on a schedule under the hood it uses the scheduler to kick it off. WebJobs provides a nice little location to host the code that gets executed. In fact, if you create WebJobs for a web site look in the Scheduler on the portal and you'll see them listed there as well.
Also note that the scheduler could call out to other systems not running Azure. If you have something running in a Cloud Service that needs to be called regularly, or even if something was hosted elsewhere (another provider or on premises) the scheduler is where you can set that up.
Regarding the cost aspect, there is a free tier to the scheduler as well: http://www.windowsazure.com/en-us/pricing/details/scheduler/.
It's 2016. The below answers are no longer accurate.
WebJobs now also has a built-in scheduler and the schedule can be defined by a cron expression.
When publishing to Azure, you can choose if you want to have the WebJob sparked off by the Scheduler or by the WebJob internal scheduler.
Important Note: The Azure Scheduler has limits to frequency of either 1hr or 1min depending on if paid or not. For the internal scheduler however, your App Service requires Always On to keep on running and firing off the job. This Always On status may affect your pricing.
Continuous jobs are monitored, and if they exit they are re-executed. In this way they act more like "services" in your local machine. There is a module that monitors and keeps your app working. Always-ON is a feature that will help your site stay alive and hence, your webjobs to continuously run.
Scheduler is used to trigger the webjobs. It uses the scheduler user account (not the back-end account). This way you can move out of the free tier for scheduler, sign up to higher tiers to suit your needs. But essentially, all the scheduler is doing is hitting an https endpoint (which is public, but required your auth).
Triggered jobs (scheduled and on demand) are invoked by an https call. These calls are load balanced - much in the same way that a web app with many instances is load balanced. Continuous jobs run concurrently by default, but can be set to be a singleton.
For Webjobs starting from version 2, there is no reason to use an Azure Scheduler anymore. As a matter of fact, the Azure Portal already flags this functionality as (Legacy).
From WebJob SDK v2 additional triggers have been introduced and one of them is TimerTrigger, which works with CRON expressions to schedule executions. This execution mode does not need any additional Azure construct, you just need the webapp to be set as AlwaysOn to guarantee the webjob to run.
Another azure service that works with TimerTriggers is Azure Functions, which is built on top of the WebJob SDK that allows a serverless execution.