I have a timer triggered Azure Function whitch executes every morning at 04:30 AM
public static void Run([TimerTrigger("0 30 4 * * *")] TimerInfo myTimer)
Now I noticed that the invocation log shows me quite different execution times:
This function worked like a charm. Since one week I notice the below issues:
How can this be that there are executions 2 minutes before the defined time? And why there are executions up to 8 minutes (!!) after the defined time?
Another strange behaviour is that in a different environment I see that the exact same Azure Function is triggered multiple times within the same minute:
Could this be an issue with the display in the invocation log or does somebody know more about this strange effect?
Any hint is highly appreciated!
How can this be that there are executions 2 minutes before the defined time?
An error of about two minutes may be normal, which may be related to the design.
And why there are executions up to 8 minutes (!!) after the defined time?
Processing time depends on the code you write and the size of the data processed.
Another strange behaviour is that in a different environment I see that the exact same Azure Function is triggered multiple times within the same minute
Please check if runOnStartup is set to True. I think this is caused by multiple instances running at the same time. You can refer to this official documentation.
Related
I have Azure Function app and have function inside:
[Microsoft.Azure.WebJobs.Singleton]
[Function(nameof(FunctionDataUsageParseFromBlobFile))]
public async Task Run([TimerTrigger("0 */1 * * * *")] MyInfo myTimer)
{
_logger.LogInformation($"{functionName} function processed: {DateTime.UtcNow} UTC.");
so, after starting it logs string about started (and UTC time)
it works fine and look like following:
so, it's ok
But sometimes I got timeout exception of execution (not only for this function, for other functions inside this app too, this is only as example):
Timeout value of 00:09:59 exceeded by function
'Functions.FunctionDataUsageParseFromBlobFile' (Id:
'91921e6e-8ddf-48c0-b846-6c69dd61a369'). Initiating cancellation.
and the following full-cycle of function execution look like:
As we can see, only 3 messages in traces for the call, 2 from them are technical (Executing/Executed) and third - about timeout.
So, it's not an error inside function, because even first line was not logged, where an error is not possible. It looks like problem is not at function at all, function had not been started at all.
Why it happened and how to localize this problem? My idea to increase scale app service plan, but I don't have to observe and need to solve problem now.
Any other place to see detailed error and why it happened?
Thank you!
In the Consumption plan, the maximum time for each invocation of your function is 10 minutes. If the work your function is doing goes beyond it, it will be terminated by the Azure Function runtime host. You need to upgrade your service tier [to Premium] to get more execution time.
If your logic of parsing the blob file can be broken down to set of activities, then you can use the 'Durable function' approach, refer here.
I have a firebase function that runs every 2 minutes. The problem is that sometimes it takes over 540sec. to finish. Hence two executions of the function occur which messes up things.
Is there a way to ensure that the function does not fire till a previous instance finishes?
I tried to handle it using a flag stored in firestore which was set to true when function would start running, and false when function would finish. However sometimes function execution times out hence the flag is never set to false, thereby stopping all future executions.
So how do I make sure that only one execution of the function is running at a time?
You can limit the number of instances using the runWith method and using the maxInstances parameter. Read more here.
By the way, why are your functions taking too long to execute? are you terminating them correctly? You can post relevant part of you code so we can see why or you can learn about how to terminate your function here
Is there a way to ensure that the function does not fire till a previous instance finishes?
No. You'll have to store some value in a database as you are doing now and terminate the function if an instance is active.
However sometimes function execution times out hence the flag is never set to false, thereby stopping all future executions.
Checkout Cloud Functions V2 (beta) or Cloud Run itself that can run up to 1 hour.
Also, if you know a function execution is going to take more than 540 seconds every time, it might be best to increase the interval between 2 invocations.
Solved : It's a node bug. Happens after ~25 days (2^32 milliseconds), See answer for details.
Is there a maximum number of iterations of this cycle?
function do_every_x_seconds() {
//Do some basic things to get x
setTimeout( do_every_x_seconds, 1000 * x );
};
As I understand, this is considered a "best practice" way of getting things to run periodically, so I very much doubt it.
I'm running an express server on Ubuntu with a number of timeout loops .
One loop that runs every second and basically prints the timestamp.
One that calls an external http request every 5 seconds
and one that runs every X 30 to 300 seconds.
It all seemes to work well enough. However after 25 days without any usage, and several million iterations later, the node instance is still up, but all three of the setTimout loops have stopped. No error messages are reported at all.
Even stranger is that the Express server is still up, and I can load http sites which prints to the same console as where the periodic timestamp was being printed.
I'm not sure if its related, but I also run nodejs with the --expose-gc flag and perform periodic garbage collection and to monitor that memory is in acceptable ranges.
It is a development server, so I have left the instance up in case there is some advice on what I can do to look further into the issue.
Could it be that somehow the event-loop dropped all it's timers?
I have a similar problem with setInterval().
I think it may be caused by the following bug in Node.js, which seem to have been fixed recently: setInterval callback function unexpected halt #22149
Update: it seems the fix has been released in Node.js 10.9.0.
I think the problem is that you are relying on setTimeout to be active over days. setTimeout is great for periodic running of functions, but I don't think you should trust it over extended time periods. Consider this question: can setInterval drift over time? and one of its linked issues: setInterval interval includes duration of callback #7346.
If you need to have things happen intermittently at particular times, a better way to attack this would be to schedule cron tasks that perform the tasks instead. They are more resilient and failures are recorded at a system level in the journal rather than from within the node process.
A good related answer/question is Node.js setTimeout for 24 hours - any caveats? which mentions using the npm package cron to do task scheduling.
I would like to have a function called on a timer (every X minutes) but I want to ensure that only one instance of this function is running at a time. The work that is happening in the function shouldn't take long, but if for some reason it takes longer than the scheduled timer (X minutes) I don't want another instance to start and the processes to step on each other.
The simplest way that I can think of would be to set a maximum execution time on the function to also be X minutes. I would want to know how to accomplish this in both the App Service and Consumption plans, even if they are different approaches. I also want to be able to set this on an individual function level.
This type of feature is normally built-in to a FaaS environment, but I am having the hardest time google-binging it. Is this possible in the function.json? Or also are there different ways to make sure that this runs only once?
(PS. I know I could this in my own code by wrapping the work in a thread with a timeout. But I was hoping for something more idiomatic.)
Timer functions already have this behavior - they take out a blob lease from the AzureWebJobsStorage storage account to ensure that only one instance is executing the timer function. Also, the timer will not execute while a previous scheduled execution is in flight.
Another roll-your-own possibility is to handle this with storage queues and visibility timeout - when the queue has finished processing, push a new queue message with visibility timeout to match the desired schedule.
I want to mention that the functionTimeout host.json property will add a timeout to all of your functions, but has the side effect that your function will fail with a timeout error and that function instance will restart, so I wouldn't rely on it in this case.
You can specify 'functionTimeout' property in host.json
https://github.com/Azure/azure-webjobs-sdk-script/wiki/host.json
// Value indicating the timeout duration for all functions.
// In Dynamic SKUs, the valid range is from 1 second to 10 minutes and the default value is 5 minutes.
// In Paid SKUs there is no limit and the default value is null (indicating no timeut).
"functionTimeout": "00:05:00"
There is a new Azure Functions plan called Premium (in public preview as of May 2019) that allows for unlimited execution duration:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale
It will probably end up the goto plan for most Enterprise scenarios.
I have 3 different jobs set up in crontab (call them jobA, jobB, jobC) that run at different intervals and start at different times during the day. For example, jobA runs once per hour at 5 mins past the hour, jobB runs every 30 mins at 9 and 39 mins past the hour, and jobC runs every 15 mins. They are not dependent on each other, but for various reasons they can NOT be running at the same time.
The problem is that sometimes one of the jobs takes a long time to run and another one starts before the first one is done, causing issues.
Is there some way to queue or spool these jobs so that one will not start until the current running one has finished? I tried using this solution but this does not guarantee that the pending jobs will resume in the same order they were supposed to start. A queue would be best, but I cannot find anything about how to do this.
You can't do that using cron. Cron is used to run a specific command at specific time. You can do it by the solution you proposed, but that adds a lot more complexity.
I suggest, writing/coding the requirement in high level language like java and use a mutil-thread program to achieve what you need.
Control-m is another scheduling software, with a lot of other features as well. You would be able to integrate the above use-case in it.