In order to process cleanup jobs, that will run for every 8 hours, currently we have implemented as :
Create scheduled Job using Azure Scheduler that puts message in storage queue when it is triggered.
Implement client in such a way that it will poll continuously and process whenever it receives message. Sample implementation of client is :
while (!CancellationToken.Value.IsCancellationRequested)
{
var message = await client.GetMessageAsync();
if (message != null)
{
// process the message
}
}
But the problem is we are waiting indefinetly even we know that we will get messages only after 8hours and also as per documentation, every attempt to read message from queue will incur cost.
How to optimize this in such a way that listeners will be spawned up on the fly for every configurable time instead of continuous loop?
You did not mention how is your client deployed, so I'm hoping one of the below strategies can help you optimize.
If you client is deployed as Cloud Resource: You can use Azure Automation service to schedule start/stop of the cloud resource, for example: start the cloud service just before message appears in the queue and trigger shutdown once it is done.
If your client is deployed on premise: You can of course use Thread.Sleep to reduce the number of hits
Also consider Azure Service Bus which allows you to subscribe for messages/topics.
Related
We have a usecase where we need to schedule jobs which will be sent as a message from Azure web api service to Azure Service Bus Queue. As we need to schedule it at later point in time one solution is to use Scheduled Delivery ScheduledEnqueueTimeUtc.
What i understand is message gets engqueued only after the time specified expires . My concern is what happens if Web API crashes or undergoes upgrade meanwhile.
1.Will the messages be lost as its not enqueued yet?
2.Where does this messages are stored in the intermediate time ?
Second solution is to use visibilityTimeOut of storage queue where messages are enqueued and will not be impacted by Web API.
From stability and scalability perspective which would be a better option ?
The message is sent to Service Bus, which is enqueued (available to receive) according to the schedule. So, to answer your queries
Nope
In the queue, just not available to receive
visibilityTimeOut is for storage queues. Refer the comparison doc for making the decision.
Note that while you cannot receive scheduled messages, you can peek them.
We are currently using Azure more like IAAS, rather than cloud services.
As a start, I would like to utilise Azure Messaging Queues to process some database actions and Web API calls.
I am assuming I would need to write another piece of code that subscribes to the queues, so when messages arrive, it knows to process the transaction?
Is that piece of code, a console app? runs on a scheduled task? a windows service? or a function app within azure?
What is the Best Practice for this architecture?
You can write console app and schedule web job to monitor the queue. However better way is to use Azure Functions. You don't have to monitor the queue then. Whenever the message arrives in the queue it will trigger Azure Function and you can process the message. The benefit is it's server-less.
I have looked through documentation for WebJobs, Functions and Logic Apps in Azure but I cannot find a way to schedule a one-time execution of a process through code. My users need to be able to schedule notifications to go out at a specific time in the future (usually within a few hours or a day from being scheduled). Everything I am reading on those processes is using CRON expressions which is not designed for one-time executions. I realize that I could schedule the job to run on intervals and check the database to see if the rest of the job needs to run, but I would like to avoid running the jobs unnecessarily if possible. Any help is appreciated.
If it is relevant, I am using C#, ASP.NET MVC Core, App Services and a SQL database all hosted in Azure. My plan was to use Logic apps to check the database for a scheduled event and send notifications through Twilio, SendGrid, and iOS/Android push notifications.
One option is to create Azure Service Bus Messages in your App using the ScheduledEnqueueTimeUtc property. This will create the message in the queue, but will only be consumable at that time.
Then a Logic App could be listening to that Service Bus Queue and doing the further processing, e.g. SendGrid, Twilio, etc...
HTH
You could use Azure Queue trigger with deferred visibility. This will keep the message invisible for a specified timeout. This conveniently acts as a timer.
CloudQueue queueOutput; // same queue as trigger listens on
var strjson = JsonConvert.SerializeObject(message); // message is your payload
var cloudMsg = new CloudQueueMessage(strjson);
var delay = TimeSpan.FromHours(1);
queueOutput.AddMessage(cloudMsg, initialVisibilityDelay: delay);
See https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.storage.queue.cloudqueue.addmessage?view=azure-dotnet for more details on this overload of AddMessage.
You can use Azure Automation to schedule tasks programmatically using REST API. Learn about it here.
You can use Azure Event Grid also. Based on this article you can “Extend existing workflows by triggering a Logic App once there is a new record in your database".
Hope this helps.
The other answers are all valid options, but there are some others as well.
For Logic Apps you can build this behavior into the app as described in the Scheduler migration guide. The solution described there is to create a logic app with a http trigger, and pass the desired execution time to that trigger (in post data or query parameters). The 'Delay Until' block can then be used to postpone the execution of the following steps to the time passed to the trigger.
You'd have to change the logic app to support this, but depending on the use case that may not be an issue.
For Azure functions a similar pattern could be achieved using Durable Functions which has support for Timers.
I have a web job that is supposed to be a triggered web job. I have it been deployed to azure fine and it is listed as a triggered webjob. However when I add things to the azure service bus that it has a function for. When i trigger it from the UI it works and will respond to my messages.
My host is configured like so
var config = new JobHostConfiguration
{
JobActivator = new MyActivator(container)
};
config.UseServiceBus();
var host = new JobHost(config);
host.RunAndBlock();
My Function looks something like this
public void ProcessQueueMessage([ServiceBusTrigger("recipetest")] ProductName message, TextWriter log)
{
//code
}
I have been looking for a while now but all google searches have given me back continuous web jobs with function triggers. Can anyone tell me how to get the web job to wake up and handle messages. I have found most other answers talk about always on but triggered jobs should work without always on.
In my opinion, the differences between continuous webJob and trigger webjob is as below:
Continuous webJob: Always run a backend exe in the web application.
In webjob SDK ServiceBusTrigger scenarios, even though your individual functions are 'triggered', the WebJob as a while runs continuously (i.e. your exe keeps running and does its own internal triggering).
Triggered webjob: Triggered by schedule or manually.
Notice: Webjobs are all run by the web app's process and web apps are unloaded if they are idle for some period of time. This lets the system conserve resources.
So both Continuous and Triggered (scheduled CRON) webjobs require 'Always on'.
To use ServiceBusTrigger, your WebJob has to be continuous.
To add a little more detail, the ServiceBusTriggerAttribute actually does poll the Message Queue that it monitors using a backoff if no messages are found. So you can see that messages added to the queue do not "wake up" a WebJob and tell it to do something - instead the ServiceBusTriggerAttribute polls and will find when new messages are waiting to be processed. This is why the WebJob can't be Triggered if you're using ServiceBusTriggerAttribute.
I currently have a webjob that runs when user does specific action.
When the webjob is running it cannot accept a new request until its done with the current process.
Is it possible to run webjob simultaneously, meaning be able to process a job for multiple clients at a time?
Absolutely. But how are you triggering your WebJob when a user performs a specific action? If you trigger a method within your WebJob via a QueueTrigger that listens for a message on an Azure Storage Queue, you can have multiple threads processing messages at once. You can do something similar using Service Bus queues as well.
Check out the docs in the parallel execution section here.