I have an azure function which contain PnP SDK Core code to integrate with SharePoint. the code loop through list items newly created >> add a folder structure to them. i configure the Azure Function to run each 10 minutes. but the azure function execution might exceed 10 minutes if the user added many list items. so is there a way to prevent a new instance of the azure function from running if there is a already a running instance of the azure function that has not finished yet?
Thanks
AFAIK that is the default behavior, see this doc:
Behind the scenes, TimerTrigger uses the Singleton feature of the WebJobs SDK to ensure that only a single instance of your triggered function is running at any given time.
If any process runs longer than the scheduled timer, the new incoming process waits for the older process to finish and then uses the same instance.
Related
I have a long running (node.js) orchestrator in Azure Function App that calls a couple hundred activity functions. Sometimes with a group of 5 or so running in parallel with context.df.Task.all. I find that it will run steadily for about two hours then the function app itself seems to abruptly stop. The logs stop displaying in the log stream. And the records in my database that the activity functions are supposed to be writing stop writing. There are no exceptions in the logs. It will remain paused or stalled like this indefinitely... until I restart the function app. Then it will come back to to life and resume where it stopped before for a time and then stop again.
Does this behavior sound familiar to anyone?
Should I update the extension bundle to [4.0.0, 5.0.0)
Could my storage account be the problem? Should I create a new one?
We are using the "Premium Plan", Could I be running up against a limit of some kind? If so what and what should I tell the IT team to increase.
As far as I know,
Should I update the extension bundle to [4.0.0, 5.0.0)
I believe this issue is not related to extension bundles because this is regarding on the usage compatible extensions, libraries, packages used in the Function App and extension bundle is versioned where each version comprises of Rich set of supported binding extensions to be installed based on the version of the Function App.
If any timeout value is defined in the host.json, make it as unbounded (-1) as the function project is deployed/hosting in the premium plan for the longer timeout duration of function executions.
Could my storage account be the problem? Should I create a new one?
Instead of creating a new account, you can increase the quota of the Storage account to 5 PiB.
If Storage account is in consideration, then make sure that both the function app and storage account are in same region to reduce latency issues.
Also, in production environment - it is better to allocate a separate storage account for each azure function app.
We are using the "Premium Plan", Could I be running up against a limit of some kind? If so what and what should I tell the IT team to increase.
Also, you mentioned in the question that the function app stalls, with no executions after stalling and works by restart from where it has paused. I have seen some points mentioned by Microsoft even the long running functions hosted in premium plan will stops with no executions like your scenario:
Refer to the MS Doc for more information.
Is there a specific way to have Azure WebJobs trigger a function once on startup without using external messages/triggers? I know there is the RunOnStartup annotation that can be added to the TimerTrigger trigger but that still requires a time interval to run the function on. In my case, I am simply looking to run the function once on the startup.
WebJobs have two types: Continuous and Triggered.
Here is a table describes the differences between continuous and triggered WebJobs.
If you just want your task run once, you could set it triggered manually. Follow this turorial: Create a manually triggered WebJob
Thanks for asking question! Azure WebJob is known for its ‘Event-driven background processing’ method. Further, WebJobs is a feature of Azure App Service that enables you to run a program or script in the same instance as a web app. To elaborate on this there are continuous and triggered types of Web Jobs.
The continuous type; Starts immediately when the WebJob is created. To keep the job from ending, the program or script typically does its work inside an endless loop. If the job does end, you can restart it. But triggered type Starts only when triggered manually or on a schedule.
Check this link on creating scheduled web job: https://learn.microsoft.com/en-us/azure/app-service/webjobs-create#CreateScheduledCRON
Background
I have a set of logic apps that each call a set function apps which are run in parallel.
Each logic app is triggered to start at a certain time during the night with all staggered an hour apart.
The Azure functions are written using the async pattern and call external APIs.
Problem
Sometimes the logic apps will run fine and complete their execution in a normal time period, and can do so for two or three days in a row.
However sometimes they will take hours or days forcing me to cancel their run.
Can any body shed any light on this might be happening?
Notes
I'm using the latest nuget packages of the durable functions extension
When debugging the functions always complete in a timely fashion
I have noticed that the functions sometimes get stuck at pending.
It appears you have at least two function apps that are configured with the same storage account and task hub name:
AzureConsumptionXXX
AzureComputeXXX
This causes the two function apps to steal messages from each other. If functions in one app do not exist in the other app, then it's very possible for orchestrations to get stuck in a Pending state like this.
The simplest way to mitigate this is to give each function app a unique task hub name. Please see the Task Hubs documentation for more information: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-task-hubs.
I am struggling to answer this question for myself so can anyone answer it for me. How, exactly, do webjobs get run? By this I mean does the Azure framework manage access to the WebJob and are multiple instances run in separate processes?
I am under the impression that I get 12 or 16 parrallel instances by default. Is this correct? So if three messages are placed on a queue, that my webjob is triggered by, they will all run in parrallel.
AFAIK, when you scale out multiple instances, the webjobs will run on the instances in parallel in separate processes. But there are prerequisites of it,
The webjob should be continous and not Manual/Scheduled.
For this to happen correctly, you need to be running in Standard mode, and have the Always On setting enabled.
Note: If you use TimerTrigger in your webjob, it will not scale out. Refer to this article.
Behind the scenes, TimerTrigger uses the Singleton feature of the WebJobs SDK to ensure that only a single instance of your triggered function is running at any given time. When the JobHost starts up, for each of your TimerTrigger functions a blob lease (the Singleton Lock) is taken. This distrubuted lock ensures that only a single instance of your scheduled function is running at any time.
For more details about the issue, here are two similar posts for you to refer, 1 and 2.
I have a C# console application which extracts 15GB FireBird database file on a server location to multiple files and loads the data from files to SQLServer database. The console application uses System.Threading.Tasks.Parallel class to perform parallel execution of the dataload from files to sqlserver database.
It is a weekly process and it takes 6 hours to complete.
What is best option to move this (console application) process to azure cloud - WebJob or WorkerRole or Any other cloud service ?
How to reduce the execution time (6 hrs) after moving to cloud ?
How to implement the suggested option ? Please provide pointers or code samples etc.
Your help in detail comments is very much appreciated.
Thanks
Bhanu.
let me give some thought on this question of yours
"What is best option to move this (console application) process to
azure cloud - WebJob or WorkerRole or Any other cloud service ?"
First you can achieve the task with both WebJob and WorkerRole, but i would suggest you to go with WebJob.
PROS about WebJob is:
Deployment time is quicker, you can turn your console app without any change into a continues running webjob within mintues (https://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/)
Build in timer support, where WorkerRole you will need to handle on your own
Fault tolerant, when your WebJob fail, there is built-in resume logic
You might want to check out Azure Functions. You pay only for the processing time you use and there doesn't appear to be a maximum run time (unlike AWS Lambda).
They can be set up on a schedule or kicked off from other events.
If you are already doing work in parallel you could break out some of the parallel tasks into separate azure functions. Aside from that, how to speed things up would require specific knowledge of what you are trying to accomplish.
In the past when I've tried to speed up work like this, I would start by spitting out log messages during the processing that contain the current time or that calculate the duration (using the StopWatch class). Then find out which areas can be improved. The slowness may also be due to slowdown on the SQL Server side. More investigation would be needed on your part. But the first step is always capturing metrics.
Since Azure Functions can scale out horizontally, you might want to first break out the data from the files into smaller chunks and let the functions handle each chunk. Then spin up multiple parallel processing of those chunks. Be sure not to spin up more than your SQL Server can handle.