Azure function is not triggering on scheduled time - azure

Note: Even though it may seem duplicate, My issue is different and I request you to read complete description before hastily marking the question down just by reading Question title.
I opened "Azure function is not triggering on scheduled time " issue on official "azure-webjobs-sdk-script" git repository on May 24, 2017-but there is no reply yet. So I am re-asking this here.
I am using azure function in consumption plan, and have scheduled it to execute at every 4.00 am utc by setting following cron expression in function.json:
{
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 0 4 * * *"
}
],
"disabled": false
}
Azure function does get execute on time only if I am logged into portal or clicks on function blade. It does not invoke when I am logged out from portal (suggesting either system listener or function go to sleep after some interval).
Their official documentation states that functions in consumption plan do not require any other settings (like Always on) to keep the function alive, instances of the Azure Functions host are dynamically added and removed based on the number of incoming events.. So as per documentation, no other settings I have to configure to execute function in consumption plan.
What I have tried?
From "Timer triggered azure function not getting triggered" question on SO, I re-checked and ensured my plan (consumption plan) and time zone. (I want it to run in utc, so no explicit setting is required)
From "#1445: Azure function timer trigger not firing" git issue, I checked whether it was just logs that are not appearing. But I am certainly sure, that its not the logs but the actual function does not get triggered unless I am having my portal on or trigger it manually.
I tried to check whether this behavior exists if I change the schedule to a more closer recursive invocations--I scheduled function to get executed at every 2 hours, and this schedule perfectly worked even when I logged out or did not awake function manually. This means, there is some issue when schedule is set to run on larger set of intervals (in my case each day)

As discussed here in #1534, deleting and redeploying the function completely in new function app did not reproduce the issue. So deleting the function and/or function app-and redeploying the same should make things working. Meanwhile, Azure team has announced to add internal logging which will help future diagnosis.

Related

Azure WebJob has status "Never finished" in WebJobs Dashboard

I have long running web job in AppService (around 1h).
AppService has "Always on" turned on.
It is initialized with:
var host = new JobHost(config);
host.Call(typeof(Functions).GetMethod("SyncUsers"));
host.Start();
Actual methods SyncUsers wrapped with attributes:
[Timeout("00:59:00", ThrowOnTimeout = true)]
[NoAutomaticTrigger]
Schedule is set with settings file settings.job:
{
"is_singleton": true,
"schedule": "0 0 */4 * * *"
}
Main issue is that in WebJobs Dashboard I see status "Never finished" in 90% of cases (or failed with exception - OK situation). Running time for such jobs is different: from 5 min to 30 mins. Logs just stopped at some moment without any exception or detailed message.
Another thing is that I can see that multiple jobs are running in the same time. So looks like singleton and schedule don't work (since job should run every 4h).
Also some jobs that have this status, displayed without running time, like this: "1 hour ago ( running time)" and I am not able to see logs or download them.
Anybody had such experience?
Thank you
Looks like you're trying to run an executable that never ends as a triggered WebJob, which has no chance of working. You need to either:
Use a continuous WebJob and rely on the WebJobs SDK for your timer
Use a plain console app deployed as a Scheduled WebJob. No need to use the SDK here. Just do what you need to do from your Main() and let it end.
I'd suggest #2 unless you have a specific need to use the WebJobs SDK.

Azure function goes idle when running in Consumption Plan with ServiceBus Queue trigger

I have also asked this question in the MSDN Azure forums, but have not received any guidance as to why my function goes idle.
I have an Azure function running on a Consumption plan that goes idle (i.e. does not respond to new messages on the ServiceBus trigger queue) despite following the instructions outlined in this GitHub issue:
The configuration for the function is the following json:
{
"ConnectionStrings": {
"MyConnectionString": "Server=tcp:project.database.windows.net,1433;Database=myDB;User ID=user#project;Password=password;Encrypt=True;Connection Timeout=30;"
},
"Values": {
"serviceBusConnection": "Endpoint=sb://project.servicebus.windows.net/;SharedAccessKeyName=SharedAccessKeyName;SharedAccessKey=KEY_HERE",
}
}
And the function signature is:
public static void ProcessQueue([ServiceBusTrigger("queueName", AccessRights.Listen, Connection = "serviceBusConnection")] ...)
Based on the discussion in the GitHub issue, I believed that having either a serviceBusConnection entry OR an AzureWebJobServiceBus entry should be enough to ensure that the central listener triggers the function when a new message is added to the ServiceBusQueue, but that is proving to not be the case.
Can anyone clarify the difference between how those two settings are used, or notice anything else with the settings I provided that might be causing the function to not properly be triggered after a period of inactivity?
I suggest there are several possible causes for this behavior. I have several Azure subs and only one of them had issues with Storage/Service Bus-based triggers only popping up when app is not idle. So far I have observed that actions listed below will prevent triggers from working correctly:
Creating any Storage-based trigger, deleting (for any reason) the triggering object and re-creating it.
Corrupting azure function input parameters by deleting/altering associated objects without recompiling a function
Restarting functions app when one of the functions fails to compile/bind to trigger OR input parameter and hangs may cause same problems.
It has also been observed that using legacy Connection Strings setting for trigger binding will not work.
Clean deploy of an affected function app will most likely solve the problem if it was caused by any of the actions described above.
EDIT:
It looks like this is also caused by setting Authorization/Authentication on the functions app, but I have not yet figured out if it happens in general or when Auth has specific configuration. Tested on affected Azure sub by disabling auth at all - function going idle after 30-40 mins, queue trigger still initiates an execution, though with a delay as expected. I have found an old bug related to this, but it says issue resolved.

Disabled Azure Functions still running

I am just starting to test with Microsoft Azure Functions. I have my VS2017 publishing and my function is working nicely. I currently have one function that I am working with. It is set on a timer of every 5 minutes.
However, it appears that that function is executing even when I have it "disabled". This can be seen in the Monitor and in one of the systems that it is interacting with. The only way that I am able to stop it is to stop the overall function group. When I then start the function group, it starts the disabled function running every 5 minutes again.
Am I missing something? Does the disabling of an individual function have some other purpose?
How do I get an individual function within a function group to not execute on its defined schedule?
Thanks.
What you are experiencing is an expected behavior though not an ideal one. It is a bug in the portal experience.
The Function runtime directly consumes metadata in the binary files of the pre-compiled functions. Here is sample of annotation for the disabled function.
[TimerTrigger("0 */5 * * * *"), Disable()]
This is the function.json generated by visual studio the above annotations.
{
"generatedBy": "Microsoft.NET.Sdk.Functions.MSBuild-1.0.2",
"configurationSource": "attributes",
"bindings": [
{
"type": "timerTrigger",
"schedule": "0 */5 * * * *",
"useMonitor": true,
"runOnStartup": false,
"name": "myTimer"
}
],
"disabled": true,
"scriptFile": "..\\bin\\FunctionApp3.dll",
"entryPoint": "FunctionApp3.Function1.Run"
}
The function.json generated by the precompiled functions is consumed by the portal and that is what is shown in the portal. When you change the disabled state of the function in the portal the disabled property is changed in the function.json but it is not consumed by the functions runtime. Hence it continues to execute.
When you deploy it in disabled state, runtime is aware of it and honors it as expected.
I have opened this bug to fix the portal experience.
https://github.com/Azure/azure-functions-ux/issues/1857
Today, I got the same problem, and after to disable the function in Azure, I recommend you to restart the Function service. Because the Azure needs to refresh the metadata, and the restart is one of the solutions to accomplish it.

Azure Function Multiple Job Hosts

I am experiencing a strange problem with Azure Functions that is starting up multiple job hosts. The initial host seems to startup and subsequent hosts error trying to acquire singleton lock. It is really noticeable when I disable one of the jobs and the error message appears that the "function runtime is unable to start". I noticed that my timer triggers were executing multiple times per their configuration "0 */30 * * * *" which caused me to dig deeper into this situation.
Pid 1 2017-04-25T13:30:06.680 Staging table updated successfully.
Pid 1 2017-04-25T13:30:06.680 Updating the base table from the staging table.
Pid 2 2017-04-25T13:30:06.680 Staging table updated successfully.
Pid 2 2017-04-25T13:30:06.680 Updating the base table from the staging table.
Details about the Function App:
- Azure Function running under the Dynamic/Consumption plan
- 5 functions running from class libraries (followed this guide - https://blogs.msdn.microsoft.com/appserviceteam/2017/03/16/publishing-a-net-class-library-as-a-function-app/)
- 2 functions are executed from a timer, every 30 minutes "0 */30 * * * *"
- 1 timer trigger disabled while waiting for development time
- 1 blob trigger watching a container for uploads from IoT Hub
- 1 EventHub trigger receiving events from IoT Hub (sparse events so no heavy load here)
Steps to reproduce:
- Stand up Azure function with Dynamic plan
- Create the Azure functions in the portal (ran into issues not doing this prior)
- Deploy the functions from VSTS, using WebDeploy from the guide above
- Make sure the functions tried to start
- Disable one of the functions to force a restart
- Error messages start displaying
Log pulled from the Function:
Link to log file
I have stopped the Azure Function App Service, removed the lock folder to see if that helps acquire singleton locks which it does, but as soon as a function is enabled/disabled or pushed from VSTS using the web deploy the errors return. I have rebuilt the Azure Function a couple of times and the outcome is still the same.
We are in the process of trying to understand how to troubleshoot this issue so we can create a monitoring process around this scenario.
Edit
The function that executed twice is setup with the following (all of the functions look very similar to this):
function.json
{
"scriptFile": "..\\bin\\IngestionFunctionClassLibrary.dll",
"entryPoint": "IngestionFunctionClassLibrary.Functions.AnalyticsUpdate.Run",
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 */30 * * * *"
}
],
"disabled": true`
}
project.json
{
"frameworks": {
"net46":{
"dependencies": {
}
}
}
}
Messages that look like Unable to acquire Singleton lock are actually not errors, but simply informational messages. What it means that your Function App was scaled out to multiple instances (in your case about 5). There are some lease resources that can intrinsically only be held by one instance (to support singleton behavior). So once an instance gets the lease, all others will display this message.

Error deploying webjob schedule - Response status code does not indicate success: 409 (Conflict)

I have a project with three scheduled webjobs. They all deploy correctly from Visual Studio, but it can't create a schedule for the third one. I get the following error:
webjobs.console.targets(110,5): Error : An error occurred while
creating the WebJob schedule: Response status code does not indicate
success: 409 (Conflict).
There's nothing special about my schedule in webjob-publish-settings.json:
{
"$schema": "http://schemastore.org/schemas/json/webjob-publish-settings.json",
"webJobName": "...",
"startTime": "2015-12-07T00:00:00-05:00",
"endTime": null,
"jobRecurrenceFrequency": "Day",
"interval": 1,
"runMode": "Scheduled"
}
I tried adding the schedule manually from the Azure portal and got a bit more information.
Job collection 'WebJobs-EastUS' reaches maximum number of jobs
allowed.
It turns out that you can only have 5 jobs per collection. This project has 3 jobs and two environments, so there are 6 in total. I created a new job schedule in a new collection, then deleted the job, and tried redeploying to see if it used the new empty collection. It did not, and I got the same error.
Next, I deleted a job in the original collection and redeployed. That time it worked fine. This isn't an ideal solution, since I'm still limited to 5 jobs when I need 6.
Is there a way to specify the job collection to use for the scheduler? Or is there something else I'm missing?
You can manage the scale of the Scheduler JobCollection used by your WebJobs in the old portal. Navigate to Scheduler/JobCollections and increase the scale on your Scheduler JobCollection to increase your job limit. This blog post shows where to find this stuff in the portal, and also details how WebJobs + Azure Scheduler work behind the scenes.
However, we highly recommend using the new inbuilt scheduling mechanism detailed in this blog post. This mechanism keeps the schedule with your job and involves no outside dependencies.

Resources