We have a webjob that is invoked with a queutrigger that we trigger every time an order is submitted.
public async Task OrderJobAsync([QueueTrigger("newshipment")] string filename, TextWriter log)
{
doSomething()
}
Over the past week we had some very high order activity and our webjob didn't run for a handful of these orders. We have email and slack notifications set up to send when a webjob fails but those were not triggered at all because the job simply did not run. Does anyone know what could have caused this? Could our order activity have triggered this webjob too many times?
I believe this was fixed by adding the WEB_JOB_IDLE_TIMEOUT setting in our azure configuration.
What I think was happening was that our webjob queue was getting backed up during periods of high order activity and some jobs did not run within the 2:00 minute window that is the default idle timeout. I extended this to be a half an hour and we haven't had this problem since.
Related
I have a Queue trigger Azure function which is triggered whenever a queue msg appears in Azure Queue Storage.
My work flow is:
A user may schedule a task which needs to run after a few days at a particular time (execute at time)
So I put a message in azure queue with visibility timeout as the time difference b/w current time and the execute at time of that task.
So when the msg is visible in the queue it gets picked up by the azure Function and gets executed.
I'm facing an intermittent issue when the queue message is supposed to be visible after a few days(<7 days). But somehow it got dropped/removed from the queue. So it was never picked up by the function, and that task still shows pending.
I've gone through all the articles I have found on the internet and didn't find solution to my problem.
The worst part is that it works fine for a few weeks but every now and then the queue messages (invisible ones)
Suddenly disappears. (I use azure storage explorer to check number of invisible messages)
So My webjob runs on 10 instances, grabs 10 messages of the queue and processes them from what I can tell in my personal logs, but the webjob log never shows it finishing and the status continues to be "running" even though it should be finished. This job does run for awhile, about 45-60 minutes, since I'm syncing a ton of data for each call. I checked the process explorer and the thread says "Running" but when I look in the details I see below:
Process Explorer Example Here
Not sure what to do to make the job change its status to "Success" and continue on with the next item in the queue.
Another related issue, I'm using a ServiceBusTrigger but since the call is taking more than 5 minutes to complete, the next instance of the job picks up the same item from the queue again, so then I have 2 processes running the same message off the queue. It keeps doing this every 5 minutes until I maxed out my instance count available which is 10. Is there a way to stop this from happening? This may be related to issue above.
In order to fix this, I had to add the following:
public async Task SyncTest([ServiceBusTrigger("syncqueue")] BrokeredMessage message, TextWriter log)
{
message.Complete();
}
I had developed a time triggered Azure Web Job and published into Azure with Triggered type. Everything working fine but sometimes the web job goes into shutdown state without logging any exception information in web job dashboard logs and kudu logs.
Before posting question here, I read this blog about Graceful Shutdown.
So, can anyone suggest me how to resolve the above issue.
For Continuous jobs, there is a default period of 5 seconds waiting for the job process to shutdown before getting killed.
For Triggered jobs, when a shutdown request is detected there is a 30 seconds default waiting period for the job process to stop.
You can change the grace period of a job by specifying it (in seconds) in the settings.job file where the name of the setting is stopping_wait_time like so:
{ "stopping_wait_time": 60 }
Here is a similar issue you could refer to.
A strange this is happening in one of my WebJobs and I cannot find what's wrong.
This WebJob is configured to use ServiceBusTrigger as soon as a message is received on a particular Service Bus Queue. This function has an execution time of minutes / hours, so I configured the JobHostConfiguration like this:
config.UseServiceBus(new ServiceBusConfiguration()
{
MessageOptions = new Microsoft.ServiceBus.Messaging.OnMessageOptions()
{
MaxConcurrentCalls = 3,
AutoRenewTimeout = new TimeSpan(4, 0, 0),
AutoComplete = true
}
});
If I understood correctly, this would grant that each function:
Is executed maximum three times per instance in parallel (one per each message)
The lock, on the message, is kept for a maximum of 4 hours. This would avoid the messages to be requeued if the processing time is less than 4 hours.
As soon as the function completes, the message is removed from the queue (even if it should be the default behavior).
These messages contains jobs that must be executed for different users. In order to be executed in parallalel for different users, but as a singleton for each user, inside the function I take a lease (different from each user) and I keep it for 60 seconds, renewing it every 30 seconds with a Task.
This seems to work: jobs are executed in parallel for different customer and same requests for same users are serialized.
Now, sometimes it happens that the WebJob stops triggering the function, even if the queue is full of messages. I don't understand why.
As soon as I restart that webjob, everything runs as before.
Any ideas on why this happens?
What I noticed is that the App was (wrongly) stopped, even if the webjobs were running. Maybe that was the problem?
Subject says it all really :) Say I've got a pretty busy Azure continuous webjob that is processing from an azure Queue:
public static void ProcessQueue([QueueTrigger("trigger")] Info info)
{ .... }
If I re-deploy the webjob, I can see that any currently executing job seems to be aborted (I see a "Never Finished" status). Is that job replayed after I release or is it lost forever?
Also, is there a nice way to make sure that no jobs are running when we deploy webjobs, or is it up to the developer to code a solution to that (such as a config flag that is checked every run).
Thanks
When a WebJob that uses the WebJobs SDK picks up a message from a queue, it acquires it with a 10 minutes lease. If the job process dies while processing the message, the lease expires after 10 minutes and the message goes back in the queue. If the WebJob is restarted, it will pick that message again. The message is only deleted if the function completes successfully.
Therefore, if the job dies and restarts immediately, like in the case of a redeploy, it might take up to 10 minutes to pick again the message. Also, because of this, it is recommended to either save state yourself or make the function idempotent.
In the WebJobs Dashboard you will see two invocations for the same message. One of them will be marked as Never Finished because the function execution never completed.
Unfortunately, there is no out of the box solution to prevent jobs from running during deploy. You would have to create your own logic that notifies (through a queue message?) that a deploy is about the start and then aborts the host. The host abort will wait for any existing function to stop and will prevent new ones from starting. However, this is a very tricky situation if you have multiple instances of the webjob because only one of them will get the notification.