I have run into a situation similar to
Azure Function timer is running twice and when I log onto the Azure portal
but I the solution there hasn't worked for me so far.
My function (written in Node) runs on a timer (every 15 minutes). In Application Insights and in the Azure portal Monitor logs, I see a single invocation every 15 minutes, as expected (presumably due to sampling). However, the output is occurring twice (I'm writing a record to a Cosmos collection). When I watch the live logs for my function (in the portal, navigate to the function code and pull up Logs from the bottom of the screen) I can see that it is running twice, very close together. Sample logs below. The intervals overlap, so I don't think it's a CRON issue (but I'm no CRON expert).
I did consider runOnStartup. I'm not sure if that defaults to false so I set it to false and restarted. Same problem after restart. (I'm still suspicious of this because a similar function app on a one minute timer runs as expected).
I have also tried running the function from the portal manually. When I do this, it only runs once.
Providing the information requested on the Azure functions github wiki:
I am using Consumption Plan
Not sure how much info is needed - we use Azure DevOps pipelines and Kudu.
v2 (Preview) runtime
Not setting WEBSITE_TIME_ZONE
My CRON expression: 0 */15 * * * * My expectation about what it means: function should run every 15 minutes.
N/A
Providing two from the close-together starts -
Invocation id: 4e142315-60e3-420d-b71a-9990683ba5aa
Invocation datetime: 2019-04-18T17:45:00.0044464+00:00
Region: East US
Invocation id: 4c6f4e7a-1e9d-4278-b3c6-0a2b5310199c
Invocation datetime: 2019-04-18T17:45:00.0131739+00:00
Region: East US
Sample logging showing two overlapping invocations (actual log text edited slightly for simplicity):
2019-04-18T17:45:00.004 [Information] Executing 'Functions.unassignDriverPermits' (Reason='Timer fired at 2019-04-18T17:45:00.0044464+00:00', Id=4e142315-60e3-420d-b71a-9990683ba5aa)
2019-04-18T17:45:00.013 [Information] Executing 'Functions.unassignDriverPermits' (Reason='Timer fired at 2019-04-18T17:45:00.0131739+00:00', Id=4c6f4e7a-1e9d-4278-b3c6-0a2b5310199c)
2019-04-18T17:45:33.577 [Information] Log a thing about something
2019-04-18T17:45:33.577 [Information] Log a thing about id 0
2019-04-18T17:45:33.586 [Information] Executed 'Functions.unassignDriverPermits' (Succeeded, Id=4e142315-60e3-420d-b71a-9990683ba5aa)
2019-04-18T17:45:33.281 [Information] Log a thing about something
2019-04-18T17:45:33.282 [Information] Log a thing about id 0
2019-04-18T17:45:33.294 [Information] Executed 'Functions.unassignDriverPermits' (Succeeded, Id=4c6f4e7a-1e9d-4278-b3c6-0a2b5310199c)
I once had a similar issue with my function that ran daily at 12:00. RunOnStartup was set to true. The function was running on a consumption plan, and by running only daily the function deallocates and is restarted when triggered (in cold start). Perhaps this was the reason for the double invocation: once for the trigger and once because it was started). It was fixed by setting RunOnStartup to false.
Make sure the RunOnStartup is set to false.
Looks like the 2 invocation ids are from 2 different apps (prod/dev may be?)
Are you able to filter your logs based on App name to verify if it is the same trigger firing twice?
Related
I'm trying to run Azure Orchestrations and I sunddenly I started getting the error from the statusQueryGetUri:
Orchestrator function 'UploadDocumentOrchestrator' failed: The function 'UploadDocumentOrchestrator' doesn't exist, is disabled, or is not an orchestrator function.
I've run this functions dozens of times today without any issue. VSCode doesn't flag any issues. I've tried restarting and that hasn't helped either.
This has happened once before. I got the error, troubleshot for like 3 hours, got frustrated, left, and came back to my computer an hour later and the problem was resolved without changing anything. I suspect the issue is related to caching somehow, but I'm not sure where to fix this.
There is a similar issue noted at:
https://github.com/Azure/azure-functions-durable-extension/issues/577
After I run func start, my "missing" orchestration appears in the launch screen indicating that it is valid as shown below.
I start the UploadDocumentOrchestrator with the HttpTrigger StartUploadDocuments. I see the trigger complete its execution UploadDocumentOrchestrator never starts in the logs.
UploadDocumentOrchestrator: orchestrationTrigger
For detailed output, run func with --verbose flag.
[2023-01-11T19:58:10.490Z] Executing 'StartUploadDocuments' (Reason='This function was programmatically called via the host APIs.', Id=f158ac3e-fa39-403c-9671-307ea54d5948)
[2023-01-11T19:58:10.599Z] Started orchestration with ID = '29b9ab7fcffe4f59bd47032ef21c19e8'.
[2023-01-11T19:58:10.620Z] Executed 'StartUploadDocuments' (Succeeded, Id=f158ac3e-fa39-403c-9671-307ea54d5948, Duration=154ms)
[2023-01-11T19:58:13.952Z] Host lock lease acquired by instance ID '0000000000000000000000004906C298'.
As it mentioned in the Same GitHub Ticket, it is solved by clearing the Azure storage emulator data, check here.
It might be due to the Caching issue and also #ConnorMcMahon given another scenario in GitHub Issue #1381 of the same error - renaming the function caused the same issue for a few users:
The function ‘XYZ’ doesn’t exist, is disabled, or is not an orchestrator function.
I have a function app, that was running once a month on 26th for a few months, but for a business reason we disabled the run. When we turned it back today 14th, it got triggered automatically stating the previous run was missed
Message Trigger Details: UnscheduledInvocationReason: IsPastDue, OriginalSchedule: 2022-05-26T14:05:00.0000000-05:00
Is there any way not to run the previous missed days?
Error "UnscheduledInvocationReason" occurs when either RunOnStartup or isPastDue property is set to true. See code
isPastDue property is set to true when the current function invocation is later than scheduled(for example if a function was restarted) See code
You can set UseMonitor = false to disable "UnscheduledInvocationReason" due to isPastDue.
Azure Function TimerTrigger uses the Singleton feature of WebJobs SDK to ensure that a single instance of a function app is running by acquiring a Blob lease, the log you are observing here is regarding this singleton behavior. See documentation
https://learn.microsoft.com/en-us/answers/questions/110043/unscheduledinvocationreason-ispastdue-originalsche.html
I have a v2 function app written in C# that is deployed to azure. I have application insights monitoring set up to monitor it. I'm looking at the logs to try and diagnose some performance issues and I'm noticing a bunch of messages like this:
Host started (xyz ms)
I see one of these messages every hour + 4 minutes.
7/9/2019, 8:27:04 AM - TRACE
7/9/2019, 7:23:03 AM - TRACE
7/9/2019, 6:19:02 AM - TRACE
7/9/2019, 5:15:03 AM - TRACE
etc.
I have a function that runs on a trigger that I'm using to keep the function alive so I can avoid cold starts, which end up in really slow function calls when it first starts.
[FunctionName("KeepAlive")]
public void Run([TimerTrigger("30 */4 * * * *", RunOnStartup=true)]TimerInfo myTimer, ILogger log)
{
log.LogInformation("Keep Alive");
}
I thought that with this function running every 4 minutes it would prevent my function app from shutting down, but for some reason it is restarting every hour + four minutes. What am I doing wrong?
From the back-end logs of 9th and 10th July, there were no restarts.
All these functions and rest of the function executed successfully without a single failure.
Sta*****Function
Mo*****st
Physical*******List
We have detected that you are running with the default setting of logging sampling enabled for Application Insights. This could cause missing execution logs from your monitor logs.
Enable the application insights logging sampling might lead to:
Timer Trigger executions missing from your monitor logs
Other data log missing
You may just need to adjust the sampling settings to fit your particular monitoring scenario.
Please review this guidance to configure sampling.
Also runOnStartup is enabled. We recommend against setting runOnStartup to true in production.
The function will be invoked when the runtime starts. This might lead to unscheduled executions in the execution list below.
Please check here to disable runOnStartup configuration.
I have long running web job in AppService (around 1h).
AppService has "Always on" turned on.
It is initialized with:
var host = new JobHost(config);
host.Call(typeof(Functions).GetMethod("SyncUsers"));
host.Start();
Actual methods SyncUsers wrapped with attributes:
[Timeout("00:59:00", ThrowOnTimeout = true)]
[NoAutomaticTrigger]
Schedule is set with settings file settings.job:
{
"is_singleton": true,
"schedule": "0 0 */4 * * *"
}
Main issue is that in WebJobs Dashboard I see status "Never finished" in 90% of cases (or failed with exception - OK situation). Running time for such jobs is different: from 5 min to 30 mins. Logs just stopped at some moment without any exception or detailed message.
Another thing is that I can see that multiple jobs are running in the same time. So looks like singleton and schedule don't work (since job should run every 4h).
Also some jobs that have this status, displayed without running time, like this: "1 hour ago ( running time)" and I am not able to see logs or download them.
Anybody had such experience?
Thank you
Looks like you're trying to run an executable that never ends as a triggered WebJob, which has no chance of working. You need to either:
Use a continuous WebJob and rely on the WebJobs SDK for your timer
Use a plain console app deployed as a Scheduled WebJob. No need to use the SDK here. Just do what you need to do from your Main() and let it end.
I'd suggest #2 unless you have a specific need to use the WebJobs SDK.
Like the title describes - I have an Azure Function on the App Service Plan, configured for Always On and no functionTimeout set in my host.json, and it appears to timeout / not finish anytime after 30 minutes to 1 hour.(...but I feel this may be a false positive...)
The HTTP Triggered function can sometimes take over 1-2 hours to complete. I understand that this probably isn't the best design and according to the Azure Function Best Practices I should break this out into smaller / more manageable pieces - I get that. However, I expect the Function on the App Service plan to work as advertised - no hard limit on execution time. Perhaps this is the same question as Unexpected azure-function timeouts on app-service-plan, but that has no answer and I am using an HTTP Trigger instead.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
According to the Kudu Function Invocation Logs, this case reports "Never Finished", and when I click on the Toggle Output button to view the logs, they never come in.
When I viewed this function's run in the Logs section of that trigger, it seems like the function just stopped, and the log stream just reports no new trace:
2017-07-26T16:36:43.116 [INFO] [Class1] Update operation started processing 790 sales records ...
2017-07-26T16:36:43.116 [DBUG] [Class2] Matching and updating ids from the map...
2017-07-26T16:38:07 No new trace in the past 1 min(s).
2017-07-26T16:39:07 No new trace in the past 2 min(s).
2017-07-26T16:40:07 No new trace in the past 3 min(s).
2017-07-26T16:41:07 No new trace in the past 4 min(s).
So not sure why this function just seemed to stop - or perhaps it stopped collecting log statements (there are many), and for some reason, the function never completed.
Any ideas?
Approx time: 2017-07-26T16:00:00 UTC
InvocationID: d856c107-f1ee-455a-892b-ed970dcad128 (I think?)
If it is indeed being timed out, is there any way for us to know, (Exception? App Insights? etc.)
Based on my test, I found azure function will not stop your function if you don't set the timeout.
Here is my test, I create a ManualTrigger function which will log the message every 10 minutes.
The codes like below:
public static void Run(string input, TraceWriter log)
{
for (int i = 0; i < 100; i++)
{
log.Info( "Worked " + i*10 + " minutes ");
Thread.Sleep(600000);
}
}
The log details:
In the log, you could find my function executed 70 minutes.It still works well.
The no trace means there are no new requests send to the azure function.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
As Jesse Carter says, you couldn't execute long time function when you used HTTP Triggered method.
Since your client-side(send request) will have a timeout value. It will wait for the function's response.
Normally, if we want to execute long time function, I suggest you could use http trigger to get the request. In the http trigger function you could add a queue message to the azure storage queue.
Then you could write a queue trigger function which will execute the long time work.
If your HTTP method takes more than a minute, you should be offloading it to a Queue. Period. (I know the other answers have said this, but it's worth repeating).
Http connections are a limited resource.
While Azure Functions as an execution engine can handle long running
operations (as demonstrated by queue / service bus support), the
http pipeline may cut off / timeout long running requests.
Queue triggers can easily run for 30+ minutes. If your job is longer than that, you really should split it into multiple queue messages.
Also check out Durable Function support: https://github.com/Azure/azure-functions-durable-extension/
Regardless of the function app timeout setting, 230 seconds is the maximum amount of time that an HTTP triggered function can take to respond to a request. This is because of the default idle timeout of Azure Load Balancer. For longer processing times, consider using the Durable Functions async pattern or defer the actual work and return an immediate response.
Function app timeout duration: Check Notes