I have a v2 function app written in C# that is deployed to azure. I have application insights monitoring set up to monitor it. I'm looking at the logs to try and diagnose some performance issues and I'm noticing a bunch of messages like this:
Host started (xyz ms)
I see one of these messages every hour + 4 minutes.
7/9/2019, 8:27:04 AM - TRACE
7/9/2019, 7:23:03 AM - TRACE
7/9/2019, 6:19:02 AM - TRACE
7/9/2019, 5:15:03 AM - TRACE
etc.
I have a function that runs on a trigger that I'm using to keep the function alive so I can avoid cold starts, which end up in really slow function calls when it first starts.
[FunctionName("KeepAlive")]
public void Run([TimerTrigger("30 */4 * * * *", RunOnStartup=true)]TimerInfo myTimer, ILogger log)
{
log.LogInformation("Keep Alive");
}
I thought that with this function running every 4 minutes it would prevent my function app from shutting down, but for some reason it is restarting every hour + four minutes. What am I doing wrong?
From the back-end logs of 9th and 10th July, there were no restarts.
All these functions and rest of the function executed successfully without a single failure.
Sta*****Function
Mo*****st
Physical*******List
We have detected that you are running with the default setting of logging sampling enabled for Application Insights. This could cause missing execution logs from your monitor logs.
Enable the application insights logging sampling might lead to:
Timer Trigger executions missing from your monitor logs
Other data log missing
You may just need to adjust the sampling settings to fit your particular monitoring scenario.
Please review this guidance to configure sampling.
Also runOnStartup is enabled. We recommend against setting runOnStartup to true in production.
The function will be invoked when the runtime starts. This might lead to unscheduled executions in the execution list below.
Please check here to disable runOnStartup configuration.
Related
Our function app has TimerTrigger-function with CRON expression */10 * * * * * (every 10 seconds), and periodically it seems like trigger stops to fire, only manual app restarting helps. We're sure that function wasn't fired because it should insert records into DB.
We're using .net6 (In-Process) and runtime v4, App Service Plan with 'Always On' enabled and disabled logging sampling.
Sample InvocationId: 74089037-5f8a-4f30-beca-0261aba58400
You can see gaps in a timeline chart from Application Insights (all executions for last 48 hours).
I have run into a situation similar to
Azure Function timer is running twice and when I log onto the Azure portal
but I the solution there hasn't worked for me so far.
My function (written in Node) runs on a timer (every 15 minutes). In Application Insights and in the Azure portal Monitor logs, I see a single invocation every 15 minutes, as expected (presumably due to sampling). However, the output is occurring twice (I'm writing a record to a Cosmos collection). When I watch the live logs for my function (in the portal, navigate to the function code and pull up Logs from the bottom of the screen) I can see that it is running twice, very close together. Sample logs below. The intervals overlap, so I don't think it's a CRON issue (but I'm no CRON expert).
I did consider runOnStartup. I'm not sure if that defaults to false so I set it to false and restarted. Same problem after restart. (I'm still suspicious of this because a similar function app on a one minute timer runs as expected).
I have also tried running the function from the portal manually. When I do this, it only runs once.
Providing the information requested on the Azure functions github wiki:
I am using Consumption Plan
Not sure how much info is needed - we use Azure DevOps pipelines and Kudu.
v2 (Preview) runtime
Not setting WEBSITE_TIME_ZONE
My CRON expression: 0 */15 * * * * My expectation about what it means: function should run every 15 minutes.
N/A
Providing two from the close-together starts -
Invocation id: 4e142315-60e3-420d-b71a-9990683ba5aa
Invocation datetime: 2019-04-18T17:45:00.0044464+00:00
Region: East US
Invocation id: 4c6f4e7a-1e9d-4278-b3c6-0a2b5310199c
Invocation datetime: 2019-04-18T17:45:00.0131739+00:00
Region: East US
Sample logging showing two overlapping invocations (actual log text edited slightly for simplicity):
2019-04-18T17:45:00.004 [Information] Executing 'Functions.unassignDriverPermits' (Reason='Timer fired at 2019-04-18T17:45:00.0044464+00:00', Id=4e142315-60e3-420d-b71a-9990683ba5aa)
2019-04-18T17:45:00.013 [Information] Executing 'Functions.unassignDriverPermits' (Reason='Timer fired at 2019-04-18T17:45:00.0131739+00:00', Id=4c6f4e7a-1e9d-4278-b3c6-0a2b5310199c)
2019-04-18T17:45:33.577 [Information] Log a thing about something
2019-04-18T17:45:33.577 [Information] Log a thing about id 0
2019-04-18T17:45:33.586 [Information] Executed 'Functions.unassignDriverPermits' (Succeeded, Id=4e142315-60e3-420d-b71a-9990683ba5aa)
2019-04-18T17:45:33.281 [Information] Log a thing about something
2019-04-18T17:45:33.282 [Information] Log a thing about id 0
2019-04-18T17:45:33.294 [Information] Executed 'Functions.unassignDriverPermits' (Succeeded, Id=4c6f4e7a-1e9d-4278-b3c6-0a2b5310199c)
I once had a similar issue with my function that ran daily at 12:00. RunOnStartup was set to true. The function was running on a consumption plan, and by running only daily the function deallocates and is restarted when triggered (in cold start). Perhaps this was the reason for the double invocation: once for the trigger and once because it was started). It was fixed by setting RunOnStartup to false.
Make sure the RunOnStartup is set to false.
Looks like the 2 invocation ids are from 2 different apps (prod/dev may be?)
Are you able to filter your logs based on App name to verify if it is the same trigger firing twice?
I have long running web job in AppService (around 1h).
AppService has "Always on" turned on.
It is initialized with:
var host = new JobHost(config);
host.Call(typeof(Functions).GetMethod("SyncUsers"));
host.Start();
Actual methods SyncUsers wrapped with attributes:
[Timeout("00:59:00", ThrowOnTimeout = true)]
[NoAutomaticTrigger]
Schedule is set with settings file settings.job:
{
"is_singleton": true,
"schedule": "0 0 */4 * * *"
}
Main issue is that in WebJobs Dashboard I see status "Never finished" in 90% of cases (or failed with exception - OK situation). Running time for such jobs is different: from 5 min to 30 mins. Logs just stopped at some moment without any exception or detailed message.
Another thing is that I can see that multiple jobs are running in the same time. So looks like singleton and schedule don't work (since job should run every 4h).
Also some jobs that have this status, displayed without running time, like this: "1 hour ago ( running time)" and I am not able to see logs or download them.
Anybody had such experience?
Thank you
Looks like you're trying to run an executable that never ends as a triggered WebJob, which has no chance of working. You need to either:
Use a continuous WebJob and rely on the WebJobs SDK for your timer
Use a plain console app deployed as a Scheduled WebJob. No need to use the SDK here. Just do what you need to do from your Main() and let it end.
I'd suggest #2 unless you have a specific need to use the WebJobs SDK.
Like the title describes - I have an Azure Function on the App Service Plan, configured for Always On and no functionTimeout set in my host.json, and it appears to timeout / not finish anytime after 30 minutes to 1 hour.(...but I feel this may be a false positive...)
The HTTP Triggered function can sometimes take over 1-2 hours to complete. I understand that this probably isn't the best design and according to the Azure Function Best Practices I should break this out into smaller / more manageable pieces - I get that. However, I expect the Function on the App Service plan to work as advertised - no hard limit on execution time. Perhaps this is the same question as Unexpected azure-function timeouts on app-service-plan, but that has no answer and I am using an HTTP Trigger instead.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
According to the Kudu Function Invocation Logs, this case reports "Never Finished", and when I click on the Toggle Output button to view the logs, they never come in.
When I viewed this function's run in the Logs section of that trigger, it seems like the function just stopped, and the log stream just reports no new trace:
2017-07-26T16:36:43.116 [INFO] [Class1] Update operation started processing 790 sales records ...
2017-07-26T16:36:43.116 [DBUG] [Class2] Matching and updating ids from the map...
2017-07-26T16:38:07 No new trace in the past 1 min(s).
2017-07-26T16:39:07 No new trace in the past 2 min(s).
2017-07-26T16:40:07 No new trace in the past 3 min(s).
2017-07-26T16:41:07 No new trace in the past 4 min(s).
So not sure why this function just seemed to stop - or perhaps it stopped collecting log statements (there are many), and for some reason, the function never completed.
Any ideas?
Approx time: 2017-07-26T16:00:00 UTC
InvocationID: d856c107-f1ee-455a-892b-ed970dcad128 (I think?)
If it is indeed being timed out, is there any way for us to know, (Exception? App Insights? etc.)
Based on my test, I found azure function will not stop your function if you don't set the timeout.
Here is my test, I create a ManualTrigger function which will log the message every 10 minutes.
The codes like below:
public static void Run(string input, TraceWriter log)
{
for (int i = 0; i < 100; i++)
{
log.Info( "Worked " + i*10 + " minutes ");
Thread.Sleep(600000);
}
}
The log details:
In the log, you could find my function executed 70 minutes.It still works well.
The no trace means there are no new requests send to the azure function.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
As Jesse Carter says, you couldn't execute long time function when you used HTTP Triggered method.
Since your client-side(send request) will have a timeout value. It will wait for the function's response.
Normally, if we want to execute long time function, I suggest you could use http trigger to get the request. In the http trigger function you could add a queue message to the azure storage queue.
Then you could write a queue trigger function which will execute the long time work.
If your HTTP method takes more than a minute, you should be offloading it to a Queue. Period. (I know the other answers have said this, but it's worth repeating).
Http connections are a limited resource.
While Azure Functions as an execution engine can handle long running
operations (as demonstrated by queue / service bus support), the
http pipeline may cut off / timeout long running requests.
Queue triggers can easily run for 30+ minutes. If your job is longer than that, you really should split it into multiple queue messages.
Also check out Durable Function support: https://github.com/Azure/azure-functions-durable-extension/
Regardless of the function app timeout setting, 230 seconds is the maximum amount of time that an HTTP triggered function can take to respond to a request. This is because of the default idle timeout of Azure Load Balancer. For longer processing times, consider using the Durable Functions async pattern or defer the actual work and return an immediate response.
Function app timeout duration: Check Notes
So i'm trying to get a small project of mine going that I want to host on azure, it's a web app which works fine and I've recently found webjobs which I now want to use to have a task run which does data gathering and updating, which I have a Console App for.
My problem is that I can't set a schedule, since it is published to the web app which dosen't support scheduling, so I tried using the Azure Webjobs SDK and using a timer but it wont run without a AzureWebJobsStorage connection string which I cannot get since my Azure account is a Dreamspark account and I cannot make a Azure Storage Account with it.
So I was wondering if there is some way to get this webjob to run on a time somehow (every hour or so). Otherwise if I just upgraded my account to "Pay-As-You-Go"? would I still retain my free features? namely SQL Server.
Im not sure if this is the right palce to ask but I tried googling for it without success.
Update: Decided to just make the console app run oin a infinate loop and ill just monitor it through the portal, the code below is what I am using to made that loop.
class Program
{
static void Main()
{
var time = 1000 * 60 * 30;
Timer myTimer = new Timer(time);
myTimer.Start();
myTimer.Elapsed += new ElapsedEventHandler(myTimer_Elapsed);
Console.ReadLine();
}
public static void myTimer_Elapsed(object sender, ElapsedEventArgs e)
{
Functions.PullAndUpdateDatabase();
}
}
The simplest way to get your Web Job on a schedule is detailed in Amit Apple's blog titled "How to add a schedule to a triggered WebJob".
It's as simple as adding a JSON file called settings.job to your console application and in it describing the schedule you want as a cron expression like so:
{"schedule": "the schedule as a cron expression"}
For example, to run your job every 30 minutes you'd have this in your settings.job file:
{"schedule": "0 0,30 * * * *"}
Amit's blog also goes into details on how to write a cron expression.
Caveat: The scheduling mechanism used in this method is hosted on the instance where your web application is running. If your web application is not configured as Always On and is not in constant use it might be unloaded and the scheduler will then stop running.
To prevent this you will either need to set your web application to Always On or choose an alternative scheduling option - based on the Azure Scheduler service, as described in a blog post titled "Hooking up a scheduler job to a WebJob" written by David Ebbo.