For one of my two Azure scheduler service job collections, the one that is hosted in the South Central US region is having some real problems. Absolutely nothing shows up now on the Dashboard, Scale, Jobs or History "tabs" (see screen shot). Is anyone else having similar issues with any of their job collections in this (or any other) Azure region?
(sorry, can't attach an image)
This was due to an issue that was impacting the portal display for jobs with more complex schedules (e.g. on a particular day of the week).
In the meantime, the jobs themselves do not have any issue and will continue to execute as expected.
You can also see the jobs via the powershell cmdlets (can be installed from here):
PS C:\> Get-AzureSchedulerJob -JobCollectionName northcentraljc -Location "North Central US"
JobCollectionName : northcentraljc
JobName : northcentralbing
Lastrun : 8/4/2014 9:30:01 AM
Nextrun : 8/11/2014 9:30:00 AM
StartTime : 5/26/2014 9:30:00 AM
Status : Enabled
Recurrence : 1 per Week
Failures : 0
Faults : 0
Executions : 11
EndSchedule : None
As an update: the issue was fixed on ~10 PM (PST) on 8/5/2014.
Related
since 5 days every pipeline run is canceled (not failed) automatically after round about 30 minutes.
The pipeline stage shows the error message:
Response status code does not indicate success: 403 (Identity does not have permissions for Microsoft.MachineLearningServices/workspaces/experiments/runs).
Microsoft.RelInfra.Common.Exceptions.ErrorResponseException: Identity does not have permissions for Microsoft.MachineLearningServices/workspaces/experiments/runs/read actions.
runs
I verified that my user has the rights of the role (owner and contributor). So every write/ read access should be there.
access rights
I created a completly new machine learning ressource and tried with different users (role contributor).
The error message does not make sense, because the pipeline starts and runs for 30 minutes (every user right is there?!) and is canceled automatically after some time. If I re-run the pipeline step from the ML studio, the step succeed.
I am working with Azure Machine Learning for 4 months and everything went fine.
The pipeline is created with a local python SDK.
The abonnement contains the payed version.
EDIT:
The 70_driver_log log does not show any error message - it stops after printing some own messages.
70_driver_log.txt
7%|█ | 10388/139445 [12:55<2:00:20, 17.87it/s]
7%|█ | 10390/139445 [12:55<3:04:23, 11.67it/s]
executionlogs.txt:
[2021-05-25 12:00:47Z] Job is running, job runstatus is Running
[2021-05-25 12:02:49Z] Cancelling the job
I have long running web job in AppService (around 1h).
AppService has "Always on" turned on.
It is initialized with:
var host = new JobHost(config);
host.Call(typeof(Functions).GetMethod("SyncUsers"));
host.Start();
Actual methods SyncUsers wrapped with attributes:
[Timeout("00:59:00", ThrowOnTimeout = true)]
[NoAutomaticTrigger]
Schedule is set with settings file settings.job:
{
"is_singleton": true,
"schedule": "0 0 */4 * * *"
}
Main issue is that in WebJobs Dashboard I see status "Never finished" in 90% of cases (or failed with exception - OK situation). Running time for such jobs is different: from 5 min to 30 mins. Logs just stopped at some moment without any exception or detailed message.
Another thing is that I can see that multiple jobs are running in the same time. So looks like singleton and schedule don't work (since job should run every 4h).
Also some jobs that have this status, displayed without running time, like this: "1 hour ago ( running time)" and I am not able to see logs or download them.
Anybody had such experience?
Thank you
Looks like you're trying to run an executable that never ends as a triggered WebJob, which has no chance of working. You need to either:
Use a continuous WebJob and rely on the WebJobs SDK for your timer
Use a plain console app deployed as a Scheduled WebJob. No need to use the SDK here. Just do what you need to do from your Main() and let it end.
I'd suggest #2 unless you have a specific need to use the WebJobs SDK.
Note: Even though it may seem duplicate, My issue is different and I request you to read complete description before hastily marking the question down just by reading Question title.
I opened "Azure function is not triggering on scheduled time " issue on official "azure-webjobs-sdk-script" git repository on May 24, 2017-but there is no reply yet. So I am re-asking this here.
I am using azure function in consumption plan, and have scheduled it to execute at every 4.00 am utc by setting following cron expression in function.json:
{
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 0 4 * * *"
}
],
"disabled": false
}
Azure function does get execute on time only if I am logged into portal or clicks on function blade. It does not invoke when I am logged out from portal (suggesting either system listener or function go to sleep after some interval).
Their official documentation states that functions in consumption plan do not require any other settings (like Always on) to keep the function alive, instances of the Azure Functions host are dynamically added and removed based on the number of incoming events.. So as per documentation, no other settings I have to configure to execute function in consumption plan.
What I have tried?
From "Timer triggered azure function not getting triggered" question on SO, I re-checked and ensured my plan (consumption plan) and time zone. (I want it to run in utc, so no explicit setting is required)
From "#1445: Azure function timer trigger not firing" git issue, I checked whether it was just logs that are not appearing. But I am certainly sure, that its not the logs but the actual function does not get triggered unless I am having my portal on or trigger it manually.
I tried to check whether this behavior exists if I change the schedule to a more closer recursive invocations--I scheduled function to get executed at every 2 hours, and this schedule perfectly worked even when I logged out or did not awake function manually. This means, there is some issue when schedule is set to run on larger set of intervals (in my case each day)
As discussed here in #1534, deleting and redeploying the function completely in new function app did not reproduce the issue. So deleting the function and/or function app-and redeploying the same should make things working. Meanwhile, Azure team has announced to add internal logging which will help future diagnosis.
I have a project with three scheduled webjobs. They all deploy correctly from Visual Studio, but it can't create a schedule for the third one. I get the following error:
webjobs.console.targets(110,5): Error : An error occurred while
creating the WebJob schedule: Response status code does not indicate
success: 409 (Conflict).
There's nothing special about my schedule in webjob-publish-settings.json:
{
"$schema": "http://schemastore.org/schemas/json/webjob-publish-settings.json",
"webJobName": "...",
"startTime": "2015-12-07T00:00:00-05:00",
"endTime": null,
"jobRecurrenceFrequency": "Day",
"interval": 1,
"runMode": "Scheduled"
}
I tried adding the schedule manually from the Azure portal and got a bit more information.
Job collection 'WebJobs-EastUS' reaches maximum number of jobs
allowed.
It turns out that you can only have 5 jobs per collection. This project has 3 jobs and two environments, so there are 6 in total. I created a new job schedule in a new collection, then deleted the job, and tried redeploying to see if it used the new empty collection. It did not, and I got the same error.
Next, I deleted a job in the original collection and redeployed. That time it worked fine. This isn't an ideal solution, since I'm still limited to 5 jobs when I need 6.
Is there a way to specify the job collection to use for the scheduler? Or is there something else I'm missing?
You can manage the scale of the Scheduler JobCollection used by your WebJobs in the old portal. Navigate to Scheduler/JobCollections and increase the scale on your Scheduler JobCollection to increase your job limit. This blog post shows where to find this stuff in the portal, and also details how WebJobs + Azure Scheduler work behind the scenes.
However, we highly recommend using the new inbuilt scheduling mechanism detailed in this blog post. This mechanism keeps the schedule with your job and involves no outside dependencies.
I'm using Quartz.Net on Azure worker role (hosted in west europe zone) and for some reason my schedulder is not working throwing this error:
An error occurred while scanning for the next trigger to fire.
Couldn't acquire next trigger: Couldn't retrieve trigger: The time
zone ID 'Coordinated Universal Time' was not found on the local
computer.
The same scheduler with the same configuration is working when hosted in my azure website application. any idea ?
I actually went ahead and exported the key/value for "UTC" as a template and edited it using the name "Coordinated Universal Time" which satisfies Quartz, but still preserves the original key. Then I just added this command to my deployment.cmd:
regedit /s addtutctz.reg
Now, I am assured that, even if my instance is reprovisioned or cloud service recreated, the registry key will be merged automatically.
Ok this is not the best solution but what I did is to create new Virtual Machine on Azure, logged remotely to the VM, deploy my worker and changed the registry keys for UTC. I basically changed the key from 'UTC' to ''Coordinated Universal Time' and that sorted my problem. Now Quartz.Net is running correctly.
I had the same problem. I fixed it by creating the triggers on a specific TimeZone:
private static ITrigger BuildTriggerForEveryDayAtSpecifiedTime(int hour, int minutes)
{
var timeZoneInfo = TimeZoneInfo.FindSystemTimeZoneById("GMT Standard Time");
return TriggerBuilder
.Create()
.WithIdentity(string.Format("Triggers every Day at {0}:{1}", hour, minutes), "MyJobs")
.StartNow()
.WithCronSchedule(string.Format("0 {0} {1} ? * *", minutes, hour), csb => csb.InTimeZone(timeZoneInfo))
.Build();
}