I am wondering if a Semaphore (lock) would work in Azure functions.
I do not want two separate webjobs running at the same time. The webjobs live on the same app service plan.
Is this something I can guarantee with a semaphore? (as this enables cross process locking?)
First question: you're talking about Functions and WebJobs. Which one is it?
If your App Service Plan does any scaling, the semaphore will not work since two instances might be started on two different machines. The good thing: (for WebJobs) there's a simple solution for that.
[Singleton]
public static async Task ProcessImage([BlobTrigger("images")] Stream image)
{
// Process the image
}
In this example, only a single instance of the ProcessImage function will run at any given time. When the function is triggered by a new image being added to the images container, the runtime will first attempt to acquire the lock (blob lease). Once acquired, the lock is held (and the blob lease is renewed) for the duration of the function execution, ensuring no other instances will run. If another function instance is triggered while this function is running it will wait for the lock, periodically polling for it.
You can find more information here: Azure WebJobs SDK - Singleton
Edit:
If you're using Azure Functions: those running based on a TimerTrigger seem to run as Singletons.
The timer trigger supports multi-instance scale-out. A single instance of a particular timer function is run across all instances.
Related
I have an Azure Function, with a queue trigger. The function must proces one queue message each time, one by one, sequentially. This is important, since the function uses an external OAuth API, with various limitions on requesting new access and refresh tokens.
In order to process the queue sequentially, I've the following settings:
host.json
"queues": {
"batchSize": 1,
"newBatchThreshold": 0
}
Application settings
FUNCTIONS_WORKER_PROCESS_COUNT = 1
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
Despite these settings, it happens sometimes that the function gets triggered multiple times. Not in my tests, where I dump a lot of messages in the queue, but when the functions is live.
I've dived into the Application Insights Log, and I found that the function gets restarted a lot during the day, and when it restarts, it sometimes start the functions twice.
For example:
Two instances are started. I'm guessing this causes my double triggers from the queue.
How can I make sure the function is triggered once?
I believe the only way to make it even more secure that only one message will be executed at the same time is to use sessions. Give every message the same sessionId to guarantee your desired behavior. Be aware that some things like deadlettering will behave differently if you switch to sessions.
You can also use the [Singleton] attribute if that fits your function app. It should also work, but can incur additional costs when you use the consumption plan. So you need to validate how it affects costs given your workload.
See also my other answer here: azure servicebus maxConcurrentCalls totally ignored
I'm experiencing TimerTrigger not actually triggering in multiple of my Azure Functions. The flow always looks similar to this:
As shown in the log statements, this timer is configured to trigger every 5 minutes (0 */5 * * * *). It triggers 5:10, 5:15, ... 5:40. But then on 5:45 no trigger. The same goes for 5:50. Then at 5:51 it "wakes up". I have RunOnStartup = true on my trigger, so this is probably caused by the function app being started.
My function app is consumption based, why I would expect the app to simply run on another machine if the current machine is shot down or in other ways unavailable. The app is running on Azure Functions version 3.
Am I missing something here, or does anyone experience similar issues?
AFAIK, There is no specific reason for the timer trigger not firing properly within the given time.
Few of the workaround we can follow,
I have not faced the similar issue yet, would suggest you to please try to restart/refresh your function app.
or, It may be due to of the sync issue with the function which is not happening properly.
As suggested by #Anand Sowmithiran the SO THREAD, #MayankBargali-MSFT suggest about singleton lock i,e;
TimerTrigger uses the Singleton feature of the WebJobs SDK to ensure
that only a single instance of your triggered function is running at
any given time. When the JobHost starts up, for each of your
TimerTrigger functions a blob lease (the Singleton Lock) is taken.
This distributed lock ensures that only a single instance of your
scheduled function is running at any time. If the blob for that
function is not currently leased, the function will acquire the lease
and start running on schedule immediately. If the blob lease cannot be
acquired, it generally means that another instance of that function is
running, so the function is not started in the current host.
Also please try to set runonstartup to false to check whether its behaving the same or not as provided the MS DOC in comment.
For more information Please refer the below links :-
MS Q&A| Azure Timer Trigger not firing
GitHub | azure timer function not executing all of sudden
I need to develop a process (e.g. Azure fucntion app) that will load a file from FTP once every week, and perform ETL and update to other service for a long time (100mins).
My question is that will Timer Trigger Azure Function app with COMSUMPTION plan works in this scenario, given that the max running time of Azure function app is 10 mins.
Update
My theory of using Timer trigger function with Comumption plan is that if the timer is set to wake up every 4 mins from certain period (e.g. 5am - 10am Monday only), and within the function, a status tells whether or not an existing processing is in progress. If it is, the process continues its on-going job, otherwise, the function exits.
Is it doable or any flaw?
I'm not sure what is your exact scenario, but I would consider one of the following options:
Option 1
Use durable functions. (Here is a C# example)
It will allow you to start your process and while you wait for different tasks to complete, your function won't actually be running.
Option2
In case durable functions doesn't suit your needs, you can try to use a combination of a timer triggered function and ACI with your logic.
In a nutshell, your flow should looks something like this:
Timer function is triggered
Call an API to create the ACI
End of timer function.
The service in the ACI starts his job
After the service is done, it calls an API to remove it's own ACI.
But in anyway, durable functions usually do the trick.
Let me know if something is unclear.
Good luck. :)
With Consumptions plan, the azure function can run for max 10 minutes, still, you need to configure in host.json
You can go for the App Service Plan which has no time limit. Again you need to configure function timeout property in host.json
for more seed the following tutorial
https://sps-cloud-architect.blogspot.com/2019/12/azure-data-load-etl-process-using-azure.html
I have an Azure worker role whose job is to periodically run some code against a SQL Azure database. Here's my current code:
const int oneHour = 216000000; // milliseconds
while (true)
{
var numConversions = SaveSeedsToSQL.ConvertRemainingPotentialQueryURLsToSeeds();
SaveLogEntryToSQL.Save(new LogEntry { Count = numConversions });
Thread.Sleep(oneHour);
}
Is Thread.Sleep(216000000) the best way of programming such regular but infrequent events or is there some kind of wake-up-and-run-again mechanism for Azure worker roles that I should be utilizing?
This code works of course, but there are some problems:
You can fail somewhere and this schedule gets all thrown off. That
is important if you must actually do it at a specific time.
There is no concurrency control here. If you want something only done once,
you need a mechanism such that a single instance will perform the
work and the other instances won't.
There are a few solutions to this problem:
Run the Windows Scheduler on the role (built in). That solves problem 1, but not 2.
Run Quartz.NET and schedule things. That solves #1 and depending on how you do it, also #2.
Use future scheduled queue messages in either Service Bus or Windows Azure queues. That solves both.
The first two options work with caveats, so I think the last option deserves more attention. You can simply create a message(s) that your role(s) will understand and post it to the queue. Once the time comes, it becomes visible and your normally polling roles will see it and can work on it. The benefit here is that it is both time accurate as well as a single instance operates on it since it is a queue message. When completed with the work, you can have the instance schedule the next one and post it to the queue. We use this technique all the time. You only have to be careful that if for some reason your role fails before scheduling the next one, the whole system kinda fails. You should have some sanity checks and safeguards there.
I have two instances of a worker role.
I want to run a sub-task (on a Thread Pool thread) only on one of the Worker Role instances.
My initial idea was to do something like this:
ThreadPool.QueueUserWorkItem((o) =>
{
if (RoleEnvironment.CurrentRoleInstance.Id == RoleEnvironment.Roles[RoleEnvironment.CurrentRoleInstance.Role.Name].Instances.First().Id)
{
emailWorker.Start();
}
});
However, the above code relies on Role.Instances collection always returning the instances in the same order. Is this the case? or can the items be returned in any order?
Is there another approved way of running a task on one role instance only?
Joe, the solution you are looking for typically rely on:
either acquiring on lease (similar to a lock, but with an expiration) on a specific blob using the Blob Storage as a synchronization point between your role instances.
or queuing / dequeuing a message from the Queue Storage, which is usually the suggested pattern to delay long running operations such as sending an email.
Either ways, you need to go through the Azure Storage to make it work. I suggest to have a look at Lokad.Cloud, as we have designed this open-source framework precisely to handle this sort of situations.
If they need to be doing different things, then it sounds to me like you don't have 2 instances of a single worker role. In reality you have 2 different worker roles.
Especially when looking at the scalability of your application, processes need to be able to run on more than one instance. What happens when that task that you only want to run on one role gets large enough that it needs to scale to 2 or more role instances?
One of the benefits of developing for Azure is that you get scalability automatically if you design your app properly. If makes you work extra to get something that's not scalable, which is what you're trying to do.
What triggers this task to be started? If you use a message on Queue Storage (as suggested by Joannes) then only one worker role will pick up the message and process it and it doesn't matter which instance of your worker role does that.
So for now if you've got one worker role that's doing the sub task and another worker role that's doing everything else, just add 2 worker roles to your Azure solution. However, even if you do that, the worker role that processes the sub task should be written in such a way that if you ever scale it to run more than a single instance that it will run properly. In that case, you might as well stick with the single worker role and code for processing messages off the Queue to start your sub task.