I am experiencing a strange problem with Azure Functions that is starting up multiple job hosts. The initial host seems to startup and subsequent hosts error trying to acquire singleton lock. It is really noticeable when I disable one of the jobs and the error message appears that the "function runtime is unable to start". I noticed that my timer triggers were executing multiple times per their configuration "0 */30 * * * *" which caused me to dig deeper into this situation.
Pid 1 2017-04-25T13:30:06.680 Staging table updated successfully.
Pid 1 2017-04-25T13:30:06.680 Updating the base table from the staging table.
Pid 2 2017-04-25T13:30:06.680 Staging table updated successfully.
Pid 2 2017-04-25T13:30:06.680 Updating the base table from the staging table.
Details about the Function App:
- Azure Function running under the Dynamic/Consumption plan
- 5 functions running from class libraries (followed this guide - https://blogs.msdn.microsoft.com/appserviceteam/2017/03/16/publishing-a-net-class-library-as-a-function-app/)
- 2 functions are executed from a timer, every 30 minutes "0 */30 * * * *"
- 1 timer trigger disabled while waiting for development time
- 1 blob trigger watching a container for uploads from IoT Hub
- 1 EventHub trigger receiving events from IoT Hub (sparse events so no heavy load here)
Steps to reproduce:
- Stand up Azure function with Dynamic plan
- Create the Azure functions in the portal (ran into issues not doing this prior)
- Deploy the functions from VSTS, using WebDeploy from the guide above
- Make sure the functions tried to start
- Disable one of the functions to force a restart
- Error messages start displaying
Log pulled from the Function:
Link to log file
I have stopped the Azure Function App Service, removed the lock folder to see if that helps acquire singleton locks which it does, but as soon as a function is enabled/disabled or pushed from VSTS using the web deploy the errors return. I have rebuilt the Azure Function a couple of times and the outcome is still the same.
We are in the process of trying to understand how to troubleshoot this issue so we can create a monitoring process around this scenario.
Edit
The function that executed twice is setup with the following (all of the functions look very similar to this):
function.json
{
"scriptFile": "..\\bin\\IngestionFunctionClassLibrary.dll",
"entryPoint": "IngestionFunctionClassLibrary.Functions.AnalyticsUpdate.Run",
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 */30 * * * *"
}
],
"disabled": true`
}
project.json
{
"frameworks": {
"net46":{
"dependencies": {
}
}
}
}
Messages that look like Unable to acquire Singleton lock are actually not errors, but simply informational messages. What it means that your Function App was scaled out to multiple instances (in your case about 5). There are some lease resources that can intrinsically only be held by one instance (to support singleton behavior). So once an instance gets the lease, all others will display this message.
Related
There are many tutorials using the following code to create a Webjobs via the WebJob SDK 3.0 library. Specifically 'TimerTrigger'
public void DoSomethingUseful([TimerTrigger("0 */1 * * * *", RunOnStartup = false)] TimerInfo timerInfo, TextWriter log)
{
// Act on the DI-ed class:
string thing = _usefulRepository.GetFoo();
Console.WriteLine($"{DateTime.Now} - {thing}");
}
The above example should run this method as a webjob every 1 minute. However this doesn't work.
I have managed to get the webjob to work when including a setting.job file.
setting.job: { "schedule": "0 */1 * * * *" }
My question is what is the different between these two?
Update:
Please go to the azure webjobs log, then you can see it actually runs as per the timerTrigger defined by SDK(even though the Schedule is n/a, and settings.job is blank, it does not matter):
In short, When using webjob sdk 3.x, you can use TimerTrigger attribute to run the function as per the time you defined. Without using webjobs SDK(like use .zip file or publish a console project from visual studio), you can use setting.job to defined timer instead of TimerTrigger attribute.
1.When you're using webjobs SDK 3.x for timer trigger, you should add this line of code: config.AddTimers(); .
Here are my code using webjobs SDK 3.x(it's a .net core 2.2 console project created in visual studio):
The packages with latest version: Microsoft.Azure.WebJobs / Microsoft.Azure.WebJobs.Extensions / Microsoft.Extensions.Logging.Console
The code in Program.cs:
class Program
{
static void Main(string[] args)
{
var builder = new HostBuilder()
.ConfigureWebJobs(config =>
{
config.AddTimers();
config.AddAzureStorageCoreServices();
})
.ConfigureLogging((context, b) =>
{
b.AddConsole();
}
)
.Build();
builder.Run();
}
}
Then create a new file, like SayHelloWebJob.cs, and code in it:
public class SayHelloWebJob
{
public void ProcessCollateFiles([TimerTrigger("0 */1 * * * *", RunOnStartup = false)]TimerInfo timerInfo,TextWriter writer)
{
writer.WriteLine("hi, it is a testing running");
Console.WriteLine("test");
}
}
Note that in the appsettings.json file, add your storage connection string, like below:
{
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=xxx;AccountKey=xxx;EndpointSuffix=core.windows.net"
}
Then run the project, you can see the function is triggered as per 1 minute:
2.For settings.job, eg. if you're just creating a console project, and does not use the webjobs sdk. Since you're not using webjobs sdk, you cannot use the timerTrigger attribute. At this moment, you can include the settings.job file(in it's property, set "Copy to Output Directory" as "copy if newer") in this project and configure the scheduled timer like you did in your post. After publish as webjob(from visual studio, when publish, select "Webjob run mode" as "run on demand"), it can run as per the schedule you defined in settings.job.
I have been struggling with the same problem. Here is my understanding after longer research.
There are two kind of Webjobs:
triggered
continuous
Triggered one has to be triggered manually or can be triggered by App Service as per CRON expression schedule provided in setting.job. These jobs are not present in memory once not running.
Continuous one runs always, so the process exists in memory all the time. You can schedule it using Webjobs SKD TimeTrigger attribute.
You will also notice difference between these two Webjob types in the Dashboard.
For triggered Webjobs you will see on top level jobs runs then functions invoked and eventually invocation details.
For continuous Webjobs this will be functions invoked and eventually invocation details. Job runs are missing as this is just one long running job.
Check App Service / Process explorer under Kudu w3wp process to see Webjobs processes running.
Note that continuous and triggered Webjobs have to be started in different way in Main method where you provide the configuration. All comes configured when Webjob of specific type is added via Visual Studio.
This is based on WebJobs 2.x.
My recommendation is
for periodical (e.g. once per few hours, days) jobs use triggered
ones, job when not running will not consume resources,
for more frequent jobs use continuous ones with TimeTrigger
attribute, it will consume resources all the time but will not need
extra time for start-up.
I have long running web job in AppService (around 1h).
AppService has "Always on" turned on.
It is initialized with:
var host = new JobHost(config);
host.Call(typeof(Functions).GetMethod("SyncUsers"));
host.Start();
Actual methods SyncUsers wrapped with attributes:
[Timeout("00:59:00", ThrowOnTimeout = true)]
[NoAutomaticTrigger]
Schedule is set with settings file settings.job:
{
"is_singleton": true,
"schedule": "0 0 */4 * * *"
}
Main issue is that in WebJobs Dashboard I see status "Never finished" in 90% of cases (or failed with exception - OK situation). Running time for such jobs is different: from 5 min to 30 mins. Logs just stopped at some moment without any exception or detailed message.
Another thing is that I can see that multiple jobs are running in the same time. So looks like singleton and schedule don't work (since job should run every 4h).
Also some jobs that have this status, displayed without running time, like this: "1 hour ago ( running time)" and I am not able to see logs or download them.
Anybody had such experience?
Thank you
Looks like you're trying to run an executable that never ends as a triggered WebJob, which has no chance of working. You need to either:
Use a continuous WebJob and rely on the WebJobs SDK for your timer
Use a plain console app deployed as a Scheduled WebJob. No need to use the SDK here. Just do what you need to do from your Main() and let it end.
I'd suggest #2 unless you have a specific need to use the WebJobs SDK.
I am just starting to test with Microsoft Azure Functions. I have my VS2017 publishing and my function is working nicely. I currently have one function that I am working with. It is set on a timer of every 5 minutes.
However, it appears that that function is executing even when I have it "disabled". This can be seen in the Monitor and in one of the systems that it is interacting with. The only way that I am able to stop it is to stop the overall function group. When I then start the function group, it starts the disabled function running every 5 minutes again.
Am I missing something? Does the disabling of an individual function have some other purpose?
How do I get an individual function within a function group to not execute on its defined schedule?
Thanks.
What you are experiencing is an expected behavior though not an ideal one. It is a bug in the portal experience.
The Function runtime directly consumes metadata in the binary files of the pre-compiled functions. Here is sample of annotation for the disabled function.
[TimerTrigger("0 */5 * * * *"), Disable()]
This is the function.json generated by visual studio the above annotations.
{
"generatedBy": "Microsoft.NET.Sdk.Functions.MSBuild-1.0.2",
"configurationSource": "attributes",
"bindings": [
{
"type": "timerTrigger",
"schedule": "0 */5 * * * *",
"useMonitor": true,
"runOnStartup": false,
"name": "myTimer"
}
],
"disabled": true,
"scriptFile": "..\\bin\\FunctionApp3.dll",
"entryPoint": "FunctionApp3.Function1.Run"
}
The function.json generated by the precompiled functions is consumed by the portal and that is what is shown in the portal. When you change the disabled state of the function in the portal the disabled property is changed in the function.json but it is not consumed by the functions runtime. Hence it continues to execute.
When you deploy it in disabled state, runtime is aware of it and honors it as expected.
I have opened this bug to fix the portal experience.
https://github.com/Azure/azure-functions-ux/issues/1857
Today, I got the same problem, and after to disable the function in Azure, I recommend you to restart the Function service. Because the Azure needs to refresh the metadata, and the restart is one of the solutions to accomplish it.
Note: Even though it may seem duplicate, My issue is different and I request you to read complete description before hastily marking the question down just by reading Question title.
I opened "Azure function is not triggering on scheduled time " issue on official "azure-webjobs-sdk-script" git repository on May 24, 2017-but there is no reply yet. So I am re-asking this here.
I am using azure function in consumption plan, and have scheduled it to execute at every 4.00 am utc by setting following cron expression in function.json:
{
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 0 4 * * *"
}
],
"disabled": false
}
Azure function does get execute on time only if I am logged into portal or clicks on function blade. It does not invoke when I am logged out from portal (suggesting either system listener or function go to sleep after some interval).
Their official documentation states that functions in consumption plan do not require any other settings (like Always on) to keep the function alive, instances of the Azure Functions host are dynamically added and removed based on the number of incoming events.. So as per documentation, no other settings I have to configure to execute function in consumption plan.
What I have tried?
From "Timer triggered azure function not getting triggered" question on SO, I re-checked and ensured my plan (consumption plan) and time zone. (I want it to run in utc, so no explicit setting is required)
From "#1445: Azure function timer trigger not firing" git issue, I checked whether it was just logs that are not appearing. But I am certainly sure, that its not the logs but the actual function does not get triggered unless I am having my portal on or trigger it manually.
I tried to check whether this behavior exists if I change the schedule to a more closer recursive invocations--I scheduled function to get executed at every 2 hours, and this schedule perfectly worked even when I logged out or did not awake function manually. This means, there is some issue when schedule is set to run on larger set of intervals (in my case each day)
As discussed here in #1534, deleting and redeploying the function completely in new function app did not reproduce the issue. So deleting the function and/or function app-and redeploying the same should make things working. Meanwhile, Azure team has announced to add internal logging which will help future diagnosis.
I'm creating a new Azure WebJob project -- which appears to be a polished version of a console app that can run as a web job.
I want this job to run based on a schedule but in the Main() method -- see below -- Microsoft gives you the host.RunAndBlock() for the job to run continuously.
Do I need to change that if I want the job to run at regularly scheduled intervals?
static void Main()
{
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
When using the Azure WebJobs SDK you can use TimerTrigger to declare job functions that run on a schedule. For example here's a function that runs immediately on startup, then every two hours thereafter:
public static void StartupJob(
[TimerTrigger("0 0 */2 * * *", RunOnStartup = true)] TimerInfo timerInfo)
{
Console.WriteLine("Timer job fired!");
}
You can get TimerTrigger and other extensions by installing the Microsoft.Azure.WebJobs.Extensions nuget package. More information on TimerTrigger and the other extensions in that package and how to use them can be found in the azure-webjobs-sdk-extensions repo. When using the TimerTrigger, be sure to add a call to config.UseTimers() to your startup code to register the extension.
When using the Azure WebJobs SDK, you deploy your code to a Continuous WebJob, with AlwaysOn enabled. You can then add however many scheduled functions you desire in that WebJob.
An easy way to trigger a WebJob on a schedule would be to code it as a regular console application, and just add a 'settings.job' with the cron based scheduling configuration to the project.
For example, the following definition would trigger it every 5 minutes:
{
"schedule": "0 */5 * * * *"
}
No need to use JobHost, just make sure your WebApp is configured as 'Always On'.
You should then deploy the job as a triggered WebJob.
There are 2 ways that I know of for scheduling a web job instead of making it run continiously:
Create a scheduled WebJob using a CRON expression
Create a scheduled WebJob using the Azure Scheduler
You can find the documentation for both ways on azure.microsoft.com
I think you need RunAndBlock in case of Scheduled or Continuous but you can remove it if you have your job as on-demand