I have 2 Azure web jobs. One is triggered and other one is Continuous. Whenever there is some change in server configuration or app configuration of the web job, Triggered web job goes into aborted state. It goes because because of the change, application need to be restarted and somehow it is not able to restart properly. Need help on this
Here is the source code of setting the status:
if (triggeredJobStatus.Status == JobStatus.Running)
{
if (isLatest)
{
// If it is the latest run, make sure it's actually running
string triggeredJobDataPath = Path.Combine(JobsDataPath, jobName);
LockFile triggeredJobRunLockFile = TriggeredJobRunner.BuildTriggeredJobRunnerLockFile(triggeredJobDataPath, TraceFactory);
if (!triggeredJobRunLockFile.IsHeld)
{
triggeredJobStatus.Status = JobStatus.Aborted;
}
}
else
{
// If it's not latest run it cannot be running
triggeredJobStatus.Status = JobStatus.Aborted;
}
}
Additionally have you checked the logs for the WebJob and all of its triggered methods? Please read through the logs and check if anything abnormal is going on.
Job status is shown as aborted if its status file shows it as running, but it is not actually running.
Additional Reference:
https://github.com/MicrosoftDocs/azure-docs/issues/19686
Hope it helps.
Related
I have a few Azure WebJobs that run to completion, once my business logic is done I call await StopAsync(stoppingToken);
However, Azure Portal continues to show their status as "Running" until eventually the jobs terminated after the default 120 second timeout.
How can I correctly tell Azure Portal/Kudu that the job is in fact finished?
Here is an example that shows the issue:
namespace MyService
{
public class MyService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// same issue whether or not I call this:
await StopAsync(stoppingToken);
}
}
}
A week after posting this question my jobs are now mysteriously completing successfully even though no changes were made, I assume someone at Microsoft fixed something on their end.
I have long running web job in AppService (around 1h).
AppService has "Always on" turned on.
It is initialized with:
var host = new JobHost(config);
host.Call(typeof(Functions).GetMethod("SyncUsers"));
host.Start();
Actual methods SyncUsers wrapped with attributes:
[Timeout("00:59:00", ThrowOnTimeout = true)]
[NoAutomaticTrigger]
Schedule is set with settings file settings.job:
{
"is_singleton": true,
"schedule": "0 0 */4 * * *"
}
Main issue is that in WebJobs Dashboard I see status "Never finished" in 90% of cases (or failed with exception - OK situation). Running time for such jobs is different: from 5 min to 30 mins. Logs just stopped at some moment without any exception or detailed message.
Another thing is that I can see that multiple jobs are running in the same time. So looks like singleton and schedule don't work (since job should run every 4h).
Also some jobs that have this status, displayed without running time, like this: "1 hour ago ( running time)" and I am not able to see logs or download them.
Anybody had such experience?
Thank you
Looks like you're trying to run an executable that never ends as a triggered WebJob, which has no chance of working. You need to either:
Use a continuous WebJob and rely on the WebJobs SDK for your timer
Use a plain console app deployed as a Scheduled WebJob. No need to use the SDK here. Just do what you need to do from your Main() and let it end.
I'd suggest #2 unless you have a specific need to use the WebJobs SDK.
I have a Webjob that I want to be time triggered:
public class ArchiveFunctions
{
private readonly IOrderArchiver _orderArchiver;
public ArchiveFunctions(IOrderArchiver orderArchiver)
{
_orderArchiver = orderArchiver;
}
public async Task Archive([TimerTrigger("0 */5 * * * *")] TimerInfo timer, TextWriter log)
{
log.WriteLine("Hello world");
}
}
My program.cs:
public static void Main()
{
var config = new JobHostConfiguration
{
JobActivator = new AutofacJobActivator(RegisterComponents())
};
config.UseTimers();
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
my publish-setting.json:
{
"$schema": "http://schemastore.org/schemas/json/webjob-publish-settings.json",
"webJobName": "OrdersArchiving",
"runMode": "OnDemand"
}
Here is what it looks like on azure portal:
My problem is that the job runs, I have the hello world, but the job keeps in run state and it get to a time out error message:
[02/05/2018 15:34:05 > f0ea5f: ERR ] Command 'cmd /c ""Ores.Contr ...' was aborted due to no output nor CPU activity for 121 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.
What can I do to fix this?
I have a wild guess RunAndBlock could be a problem.. but I do not see a solution..
Thanks!
Edit:
I have tested Rob Reagan answer, it does help with the error, thank you!
On my same service, I have one other time triggerd job (was done in core, while mine is not).
You can see the Webjob.Missions is 'triggered', and status update on last time it ran. You can see as well the schedule on it.
I would like to have the same for mine 'OrdersArchiving'.
How can I achieve that?
Thanks!
Change your run mode to continuous and not triggered. The TimerTrigger will handle executing the method you've placed it on.
Also, make sure that you're not using a Free tier for hosting your WebJob. After twenty minutes of inactivity, the app will be paused and will await a new HTTP request to wake it up.
Also, make sure you've enabled Always On on your Web App settings to prevent the same thing from happening to a higher service tier web app.
Edit
Tom asked how to invoke methods on a schedule for a Triggered WebJob. There are two options to do so:
Set the job up as triggered and use a settings.json file to set up the schedule. You can read about it here.
Invoke a method via HTTP using an Azure Scheduler. The Azure Scheduler is a separate Azure service that you can provision. It has a free tier which may be sufficient for your use. Please see David Ebbo's post on this here.
I have an Azure Web Job, and currently the function iself will always complete with a success. However the job itself fails with this error:
[11/26/2016 07:12:11 > 462baa: ERR ] Command 'cmd /c ""MyJob.Webjob ...' was aborted due to no output nor CPU activity for 121 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.
cmd /c ""MyJob.Webjob.StuffJob.exe""
[11/26/2016 07:12:11 > 462baa: SYS INFO] Status changed to Failed
What can I do once my function is complete to let the job know it's done instead of sitting idle and timing out?
Here is my Functions' class method. Everything runs fine, but it doesn't properly stop and I don't know how to tell the job when I'm done.
[NoAutomaticTrigger()]
public void ProcessLoanOriginationDocuments(TextWriter log)
{
try
{
log.WriteLine("Starting Job");
_service.DoSomething();
log.WriteLine("Ending Job");
}
catch (Exception ex)
{
log.WriteLine(ex.Message);
}
}
What can I do once my function is complete to let the job know it's done instead of sitting idle and timing out?
From my experience, we need to set the WEBJOBS_IDLE_TIMEOUT value enough large to let the WebJob be executed completely.
From your description, the azure WebJob stopped may be caused by a website in idle. Please have a try to troubleshoot as following ways:
Set the website “Always On”
"Always On" is not supported if WebApp is in a free Service Plan tier. If the WebApp is in a free tier, please scale up the App Service Plan. Please refer to the document for more info about how to scale up the App Service Plan.
Add the WEBJOBS_IDLE_TIMEOUT in the Azure Portal. More Info about WebApp setting please refer to document.
If it is not resolved with above ways, please have a try keep writing out to console to keep the WebJob running.
I have a continous webjob associated with a website and I am running that website on the Shared mode. I don't want to go to Always On option as there is no real need for my application. I only want to process the message when the calls are made to my website.
My issue is that the job keeps stopping after few minutes even though I am continuously calling a dummy keep alive method on my website at every 5 minute that posts a message to the queue that is monitored by that webjob.
My webjob is a simple console application built using the WebJob SDK that has a code like this
JobHost host = new JobHost(new JobHostConfiguration(storageConnictionSttring));
host.RunAndBlock();
and the message processing function looks like below:
public static void ProcessKeepAliveMessages([QueueTrigger("keepalive")] KeepAliveTrigger message)
{
Console.WriteLine("Keep Alive message called on :{0}", message.MessageTime);
}
The message log for the job basically says says
[03/05/2015 18:51:02 > 4660f6: SYS INFO] WebJob is stopping due to website shutting down
I don't mind if that happen this way, but when the website starts with the next call to keep alive, the webjob is not started. All the messages are queued till I go to the management dashboard or the SCM portal as shown below
https://mysite.scm.azurewebsites.net/api/continuouswebjobs
I can see the status like this:
[{"status":"Starting","detailed_status":"4660f6 - Starting\r\n","log_url":"https://mysite.scm.azurewebsites.net/vfs/data/jobs/continuous/WebJobs/job_log.txt","name":"WebJobs","run_command":"mysite.WebJobs.exe","url":"https://mysite.scm.azurewebsites.net/api/continuouswebjobs/WebJobs","extra_info_url":"https://mysite.scm.azurewebsites.net/azurejobs/#/jobs/continuous/WebJobs","type":"continuous","error":null,"using_sdk":true,"settings":{}}]
I would really appreciate if someone can help me understand what is going wrong here.
I've run into a similar problem. I have a website (shared mode) and an associated webjob (continuous type). Looking at webjob logs, I found that the job enters stopped state after about 15 min. of inactivity and stops reacting to trigger messages. It seems contradictory to the concept of continuous job concept but, apparently, to get it running truly continuously you have to subscribe to a paid website. You get what you pay for...
That said, my website needs to be used only about every few days and running in a shared mode makes perfect sense. I don't mind that the site needs a bit extra time to get started - as long as it restarts automatically. The problem with the webjob is that once stopped it won't restart by itself. So, my goal was to restart it with the website.
I have noticed that a mere look at the webjob from Azure Management Portal starts it. Following this line of thinking, I have found that fetching webjob properties is enough to switch it to the running state. The only trick is how to fetch the properties programmatically, so that restarting the website will also restart the webjob.
Because the call to fetch webjob properties must be authenticated, the first step is to go to Azure Management Portal and download the website publishing profile. In the publishing profile you can find the authentication credentials: username (usually $<website_name>) and userPWD (hash of the password). Copy them down.
Here is a function that will get webjob properties and wake it up (if not yet running):
class Program
{
static void Main(string[] args)
{
string websiteName = "<website_name>";
string webjobName = "<webjob_name>";
string userName = "<from_publishing_profile>";
string userPWD = "<from_publishing_profile>";
string webjobUrl = string.Format("https://{0}.scm.azurewebsites.net/api/continuouswebjobs/{1}", websiteName, webjobName);
var result = GetWebjobState(webjobUrl, userName, userPWD);
Console.WriteLine(result);
Console.ReadKey(true);
}
private static JObject GetWebjobState(string webjobUrl, string userName, string userPWD)
{
HttpClient client = new HttpClient();
string auth = "Basic " + Convert.ToBase64String(Encoding.UTF8.GetBytes(userName + ':' + userPWD));
client.DefaultRequestHeaders.Add("authorization", auth);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var data = client.GetStringAsync(webjobUrl).Result;
var result = JsonConvert.DeserializeObject(data) as JObject;
return result;
}
}
You can use a similar function to get all webjobs in your website (use endpoint https://<website_name>.scm.azurewebsites.net/api/webjobs). You may also look at the returned JObject to verify the actual state of the webjob and other properties.
If you want the WebJob to not stop you need to make sure your scm site is alive.
So the keep-alive requests should go to https://sitename.scm.azurewebsites.net and these requests need to be authenticated (basic auth using your deployment credentials).