I have a web api application deployed to App Service.
I need to be able to set arbitrary number of scheduled jobs (http calls to my web api) with arbitrary due date times (from a few hours to a few months).
The point is that i need to be able to set/edit/cancel them on the fly programatically based on different state of my app and i need to maintain thousands of them.
Is there some recommended way to do it?
I would persist the jobs into either SQL database or Table Storage (I would call it 'ScheduledJobs').
Then have an Azure Function that will query the ScheduledJobs storage at some interval, say every hour to pickup jobs that should be processed at that point in time.
The jobs that are due for processing can then be written to a queue (I would name it 'jobs-queue').
Then have another Azure Function that will pickup jobs from 'jobs-queue'. This Azure Function has the business logic of how to process each job.
From my experience, an Azure Function can only run up to 10 minutes. The time it will take to process each job should not be longer than this period.
Hope this gives you some idea.
Related
I have this .NET long running API process/function that usually runs 30 mins in one execution that is hosted in AKS. This API is usually executed from the users coming from the front end of the app.
Due to concurrent executions from users, this is causing exhaustion of the app so I'm planning to implement a some sort of a queueing mechanism with the help of a scheduler(s).
What possibly is applicable Azure service that can execute my API in AKS on a scheduled basis (let's say every minute) and possibly check the database for some flagging values.
I need a way to check the table for some flagging value if there a currently running process or its been completed so it can process the next one, otherwise ignore the call until current on is complete.
I was looking into Azure Web Apps, Web Jobs or Batch Jobs but kinda confused which is applicable with my case.
Please advise thank you in advance.
There are a couple of options here.
Hangfire
Hangfire is an open-source library that can run background jobs in queues. In your case, you can enqueue each request from the client in a queue. Then Hangfire server will process them one by one (even with retry if the job fails). Hangfire supports SQL Server or Redis. You can query the storage to see the status of the queued jobs.
Hangfire can also run scheduled jobs, which will take care of that only one job run at a time.
Azure Service Bus
A more expensive option is to use Azure Service Bus for your queueing capability. For scheduled jobs, you can use AKS CronJobs but you will
implement the check yourself to see if there is a job already running.
Overall, I would recommend Hangfire, which can meet your requirements and is cheaper.
What is the best way of using Azure technology (azure function apps, durable function app, azure web jobs, ?) to get a schedule from a database depending on the specific task that needs to be performed.
The schedule that would be read from the database, would then be converted to a CRON expression. I have hourly, daily, and weekly triggers to run specific tasks.
The schedule within the database could possibly change, hence why it would need to be dynamic.
I could create individual specific function apps (over 20) but wanting it to be more flexible and dynamic.
How would i go about this? what azure tech would i use?
Thanks
Two options I can think of, depending on the details of what you're after:
Use timer-triggered Azure Functions with the %appsetting% format for the timer expression. Then have a separate function that's timed to run regularly or triggered when the schedules update, get the current schedule and update the function app settings with any changes to the CRON expressions (using a managed identity that gives it the necessary permissions).
If you're just running on the hour/day/week, you could have a function that runs every hour, reads the current schedule and triggers any functions that are scheduled for that time (basically your own simple timer).
I have created an azure service which is responsible for below task:
(1) Access the blob containers and download the files from there.
(2) Extract some data from downloaded files
(3) Stored the extracted data to an Azure SQL Server
I want to run this processing after every 7 days. Is there a way to achieve this? or can I use any other option than cloud service to achieve the above goal?
I would recommend you to use Azure Function as its Timer-based processing (Timer trigger) feature is able to fulfill your requirements.
Timer triggers call functions based on a schedule, one time or
recurring.
Reference: Azure Functions timer trigger, Azure Functions Pricing
Another great advantage of using Azure Function for your scenario is its pricing model.
Azure Functions consumption plan is billed based on resource
consumption and executions.
Consumption plan pricing includes a
monthly free grant of 1 million requests and 400,000 GB-s of resource
consumption per month.
Certainly not natively with the Cloud Service itself. I mean, you can obviously code it so it performs some task(s) and sleeps for 7 days, but you will pay for all of that time, that makes no sense
You can use Azure WebJobs, Functions and Scheduler for this purpose, or you can create a PowerShell\Cli or something else cron task\task scheduler to turn on your Azure Cloud Service, wait for it to finish processing and turn it off. But that seems like a lot of extra effort, I'd rather go with Scheduler or Functions.
I want to do a certain task periodically (per day) on our Web/Worker role. I have multiple instances in my Cloud Service, and I want only one of these instances to do this task per day (for example Instance0 can do it one day, next day it could be Instance1 doing the work, but 0 and 1 will not try to do the same work during the same day/period)
Azure queues seem to be a great way to achieve this because by design only one instance will dequeue the message (assuming it deletes it after doing the work).
What I am having trouble with is figuring out a way to put only one copy of this message in the queue per day. The only way I have figured to do this is by enqueuing a message every day from Azure scheduler jobs.
My problem with Azure scheduler is the fact that I need to create a job for every single storage account I have across all of my deployments.
Is there a way to do this from within the cloud service, without taking the scheduler dependency?
If you don't want to have Scheduler dependency, consider using Blob leases as a semaphore of sorts. http://justazure.com/azure-blob-storage-part-8-blob-leases/
At a certain time of the day, have your worker instances compete to get a lease on some central storage blob. Whoever gets the lease, prevents other instances from getting that lease and can queue up messages into the queue.
Having said that, why are you afraid of Scheduler dependency? Have it kick off a single job that will queue up a "start work" message. Have your instances monitor that queue. Whoever picks up that message, can then run thru all of your storage accounts and queue up individual storage-work messages for all instances to pickup.
If you know you need to perform one job a day I don't understand why you need to use a queue - you could just have a scheduled job run once per day. If the job only needs to run if a certain condition is met I would just build this logic into your scheduled task - maybe by setting a property in a Table or something like that.
Can anybody explain the difference between Azure Web Jobs and Azure Scheduler
Azure Web Jobs
Only available on Azure Websites
It is used to run code at particular intervals. E.g. a console application every day
Used to trigger and run workloads.
Mainly recommended for workloads that either scale with the website or are relatively small.
Can be persistently running if "Always On" selected, otherwise you will get the 20 min timeout.
The code that needs to be run and schedule are defined together.
Azure Scheduler
Is not tied to Websites or Cloud Services
It allows you to call a website or add a message to a storage queue
Used for triggering events or triggering small workloads (e.g. add to queue), usually to trigger larger workloads
Mainly recommended for triggering more complex workloads.
This is only a trigger, and a separate function listening to trigger events (e.g. queue's) needs to be coded separately.
For many instances I prefer to use the scheduler to push to a storage queue and a worker role on each instance takes off the queue. This keeps tasks controlled granularly and can also move up or down in scale outside of your website.
With WebJobs they scale up and down with your site and hence your background tasks can become over taxed if your website is experiencing low traffic and scaled down.
Azure Scheduler - Provides a way to easily schedule http calls in a well-defined schedule, like every hour, every Friday at 9:00 am, Once a day, ...
Azure WebJobs - Provides a way to run small to medium work load (in the form of a script: .exe, .cmd, .sh, .js, ...) at the same context of an Azure Website (but can be hosted even with an empty website).
While a WebJob can run continuously (with a process that has a while loop) and Azure will make sure this WebJob is always running (with "Always On" set).
There is also an integration between Azure scheduler and Azure WebJobs where you have a WebJob that is running some finite work and the schduler is responsible for scheduling this work (invoking the WebJob).
So in summary, the scheduler is about scheduling work and WebJobs is about running work load.