A strange this is happening in one of my WebJobs and I cannot find what's wrong.
This WebJob is configured to use ServiceBusTrigger as soon as a message is received on a particular Service Bus Queue. This function has an execution time of minutes / hours, so I configured the JobHostConfiguration like this:
config.UseServiceBus(new ServiceBusConfiguration()
{
MessageOptions = new Microsoft.ServiceBus.Messaging.OnMessageOptions()
{
MaxConcurrentCalls = 3,
AutoRenewTimeout = new TimeSpan(4, 0, 0),
AutoComplete = true
}
});
If I understood correctly, this would grant that each function:
Is executed maximum three times per instance in parallel (one per each message)
The lock, on the message, is kept for a maximum of 4 hours. This would avoid the messages to be requeued if the processing time is less than 4 hours.
As soon as the function completes, the message is removed from the queue (even if it should be the default behavior).
These messages contains jobs that must be executed for different users. In order to be executed in parallalel for different users, but as a singleton for each user, inside the function I take a lease (different from each user) and I keep it for 60 seconds, renewing it every 30 seconds with a Task.
This seems to work: jobs are executed in parallel for different customer and same requests for same users are serialized.
Now, sometimes it happens that the WebJob stops triggering the function, even if the queue is full of messages. I don't understand why.
As soon as I restart that webjob, everything runs as before.
Any ideas on why this happens?
What I noticed is that the App was (wrongly) stopped, even if the webjobs were running. Maybe that was the problem?
Related
I have an Azure Function, with a queue trigger. The function must proces one queue message each time, one by one, sequentially. This is important, since the function uses an external OAuth API, with various limitions on requesting new access and refresh tokens.
In order to process the queue sequentially, I've the following settings:
host.json
"queues": {
"batchSize": 1,
"newBatchThreshold": 0
}
Application settings
FUNCTIONS_WORKER_PROCESS_COUNT = 1
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
Despite these settings, it happens sometimes that the function gets triggered multiple times. Not in my tests, where I dump a lot of messages in the queue, but when the functions is live.
I've dived into the Application Insights Log, and I found that the function gets restarted a lot during the day, and when it restarts, it sometimes start the functions twice.
For example:
Two instances are started. I'm guessing this causes my double triggers from the queue.
How can I make sure the function is triggered once?
I believe the only way to make it even more secure that only one message will be executed at the same time is to use sessions. Give every message the same sessionId to guarantee your desired behavior. Be aware that some things like deadlettering will behave differently if you switch to sessions.
You can also use the [Singleton] attribute if that fits your function app. It should also work, but can incur additional costs when you use the consumption plan. So you need to validate how it affects costs given your workload.
See also my other answer here: azure servicebus maxConcurrentCalls totally ignored
I have a Queue trigger Azure function which is triggered whenever a queue msg appears in Azure Queue Storage.
My work flow is:
A user may schedule a task which needs to run after a few days at a particular time (execute at time)
So I put a message in azure queue with visibility timeout as the time difference b/w current time and the execute at time of that task.
So when the msg is visible in the queue it gets picked up by the azure Function and gets executed.
I'm facing an intermittent issue when the queue message is supposed to be visible after a few days(<7 days). But somehow it got dropped/removed from the queue. So it was never picked up by the function, and that task still shows pending.
I've gone through all the articles I have found on the internet and didn't find solution to my problem.
The worst part is that it works fine for a few weeks but every now and then the queue messages (invisible ones)
Suddenly disappears. (I use azure storage explorer to check number of invisible messages)
So My webjob runs on 10 instances, grabs 10 messages of the queue and processes them from what I can tell in my personal logs, but the webjob log never shows it finishing and the status continues to be "running" even though it should be finished. This job does run for awhile, about 45-60 minutes, since I'm syncing a ton of data for each call. I checked the process explorer and the thread says "Running" but when I look in the details I see below:
Process Explorer Example Here
Not sure what to do to make the job change its status to "Success" and continue on with the next item in the queue.
Another related issue, I'm using a ServiceBusTrigger but since the call is taking more than 5 minutes to complete, the next instance of the job picks up the same item from the queue again, so then I have 2 processes running the same message off the queue. It keeps doing this every 5 minutes until I maxed out my instance count available which is 10. Is there a way to stop this from happening? This may be related to issue above.
In order to fix this, I had to add the following:
public async Task SyncTest([ServiceBusTrigger("syncqueue")] BrokeredMessage message, TextWriter log)
{
message.Complete();
}
I have a console app that pushes a lot of messages in parallel to an Azure Storage Queue. There's a continuous triggered Azure WebJob that invokes whenever a message gets added into that queue. In one of the scenarios, I had added 300 items to queue , 100 each in three different threads. Since there are only 300 messages in the queue, the WebJob should ideally be invoked only 300 times. But I could see it has invoked 308 times. What could be the reason for it?
Also note that the additional trigger count is not predictable. Sometimes it may be 306 or 310 etc.
I have tried posting messages sequentially by removing Parallel.Invoke to check if its related to parallel processing, but the issue is still there. I am debugging the issue by running the WebJob project in local.
It is possible that exceptions occurred during processing of some messages. If a message fails to process successfully, it will be retried. Each message has a dequeueCount. If memory serves, the default for a Queue Trigger is five attempts.
Try running locally and watching the output window. Or, you can check the webjob logs on your live Azure instance to see if a message was retried.
I have azure function triggered off of a storage queue. The behavior of the function is to create out around 10,000 additional messages to another storage queue for another function (within the same function app) to execute. I'm seeing some strange behavior whenever it executes, where the first function appears to be executed multiple times. I observed this by watching the queue it's publishing to receive significantly more messages than expected.
I understand that the function should be coded defensively (i.e. expect to be executed multiple times), but this is happening consistently each time the first function executes. I don't think the repeated executions re due to it timing out or failing (according to app insights).
Could it be that when the 10,000 messages get queued up the function is scaling out and that is somehow causing the original message to be executed multiple times?
The lock on the original message that triggers the first Azure Function to execute is likely expiring. This will cause the Queue to assume that processing the message failed, and it will then use that message to trigger the Function to execute again. You'll want to look into renewing the message lock while you're sending out 10,000 messages to the next Queue.
Also, since you're sending out 10,000 messages, you may need to redesign that to be more efficient at scaling out what ever massively parallel processing you're attempting to implement. 10,000 is a really high amount of messages to send from a single triggered event.