I have a scenario where an azure function(http trigger) calls an orchestrator function, which calls multiple entities/activies.
The azure function then waits for the orchestrator to finish all its subtasks.
The following image shows a simplified sequence diagram.
The problem:
Calls to the orchestrator or entity function often(>20% of the time) take 15-20s to start executing(marked red in the diagram).
What I tried so far:
Switched the service plan from consumption(serverless) to premium with a minimum of 5 instances(=more than the number of activities/entities being called).
=> No cold start of new instances should occur
Called the http trigger many times in succession (but one after the other)
=> Some calls work, some don't, seems random
Gave it a few minutes between calls to the http trigger
When I run it locally with azurite, it works flawless every time
Why do the orchestator or entity functions often take so much time to start executing when hosted in Azure?
Update:
Changed maxQueuePollingInterval down to 3 seconds. No change in behavior.
Related
I need to develop a process (e.g. Azure fucntion app) that will load a file from FTP once every week, and perform ETL and update to other service for a long time (100mins).
My question is that will Timer Trigger Azure Function app with COMSUMPTION plan works in this scenario, given that the max running time of Azure function app is 10 mins.
Update
My theory of using Timer trigger function with Comumption plan is that if the timer is set to wake up every 4 mins from certain period (e.g. 5am - 10am Monday only), and within the function, a status tells whether or not an existing processing is in progress. If it is, the process continues its on-going job, otherwise, the function exits.
Is it doable or any flaw?
I'm not sure what is your exact scenario, but I would consider one of the following options:
Option 1
Use durable functions. (Here is a C# example)
It will allow you to start your process and while you wait for different tasks to complete, your function won't actually be running.
Option2
In case durable functions doesn't suit your needs, you can try to use a combination of a timer triggered function and ACI with your logic.
In a nutshell, your flow should looks something like this:
Timer function is triggered
Call an API to create the ACI
End of timer function.
The service in the ACI starts his job
After the service is done, it calls an API to remove it's own ACI.
But in anyway, durable functions usually do the trick.
Let me know if something is unclear.
Good luck. :)
With Consumptions plan, the azure function can run for max 10 minutes, still, you need to configure in host.json
You can go for the App Service Plan which has no time limit. Again you need to configure function timeout property in host.json
for more seed the following tutorial
https://sps-cloud-architect.blogspot.com/2019/12/azure-data-load-etl-process-using-azure.html
I want to process millions of records on-demand, which takes approximate 2-3 hours to process. I want to go serverless that is why tried durable function (first-time). I want to check, how long I can run durable function so I created 3 functions
Http function to kick start Orchestrator function
Orchestrator function
Activity function
My DurableFunction is running and emitting logs in Application Insights from last 5 days and based on my code it would take 15 more days to complete.
I want to know that how to stop Orchestrator function manually?
I can see thousands of entry in ApplicationInsights requests table for single execution, Is there any way to check how many DurableFunction running in backend? and how much time taken by single execution?
I can see some information regarding orchestrator function in "DurableFunctionHubInstance" table but as MS recommended not rely on table.
Since Durable Functions does a lot of checkpointing and replays the orchestration, normal logging might not always be very insightful.
Getting the status
There are several ways to query for the status of orchestrations. One of them is through the Azure Functions Core tools as George Chen mentioned.
Another way to query the status is by using the HTTP API of Durable Functions directly:
GET <rooturl>/runtime/webhooks/durableTask/instances?
taskHub={taskHub}
&connection={connectionName}
&code={systemKey}
&createdTimeFrom={timestamp}
&createdTimeTo={timestamp}
&runtimeStatus={runtimeStatus1,runtimeStatus2,...}
&showInput=[true|false]
&top={integer}
More info in the docs.
The HTTP API also has methods to purge orchestrations. Either a single one by ID or multiple by datetime/status.
DELETE <rooturl>/runtime/webhooks/durabletask/instances/{instanceId}
?taskHub={taskHub}
&connection={connection}
&code={systemKey}
Finally you can also manage your instances using the DurableOrchestrationClient API in C#. Here's a sample on GitHub: HttpGetStatusForMany.cs
I have written & vlogged about using the DurableOrchestrationClient API in case you want to know more about how to use this in C#.
Custom status
Small addition: it's possible to add a custom status object to the orchestration so you can add enriched information about the progress of the orchestration.
Getting the duration
When you query the status of an orchestration instance you get back a DurableOrchestrationStatus object. This contains two properties:
CreatedTime
LastUpdatedTime
I'm guessing you can subtract those and get a reasonable indication of the time it has taken.
You could manage the Durable Functions orchestration instances with Azure Functions Core Tools.
Terminate instances:
func durable terminate --id 0ab8c55a66644d68a3a8b220b12d209c --reason "It was time to be done."
Query instances with filters: you could add the parameter(runtime-status) to filter the running instances.
func durable get-instances --created-after 2018-03-10T13:57:31Z --created-before 2018-03-10T23:59Z --top 15
As for the time functions took, looks like it doesn't support. The similar parameter is the get-history.
I have an Azure Function app with 4 functions
one triggered on a timer every 24 hours
one triggered on events from IoT Hub
two others triggered on events from Service Bus as a result of the previous function
All functions work as expected when first deployed but after a period of time the functions stop running and the app appears to be scaled down with no servers online. At this point the functions are never triggered again unless I either restart the app, or drill into a function and view details of it (supposedly, forcing the function to start up).
I have the exact same code deployed to a different environment and it runs perfectly and has never encountered this issue. I've checked all the settings and configuration and can't see any material differences between the two.
This is really frustrating and is becoming a big issue. Any help would be much appreciated.
Function App is hosted in Australia Southeast.
This is the last execution (as of now)
10:45 PM UTC - Function started (Id=4d29555b-d3af-43d7-95e9-1a4a2d43dc46)
The event triggered function should run every few minutes as the IoT Hub it's triggering from has a steady stream of events coming in. When I prod the function (or restart it) and it comes to life it quickly churns through a backlog of messages queued in the IoT Hub.
I see the problem: you have comments in your host.json, which makes it invalid and throws off the parser at the scale controller level.
Admittedly, the error handling is quite poor here. But anyway, remove the commented out logger, and it should all work.
I am very new to Azure so I am not sure if my question is stated correctly but I will do my best:
I have an App that sends data in the form (1.bin, 2.bin, 3.bin...) always in consecutive order to a blob input container, when this happens it triggers an Azure function via QueueTrigger and the output of the function (1output.bin, 2output.bin, 3output.bin...) is stored in a blog output container.
When azure crashes the program tries 5 times before giving up. When azure succeeds it will run just once and that's it.
I am not sure what happened last week but since last week after each successful run, functions is idle like for 7 minutes and then it starts the process again as if it was the first time. So for example the blob receives 22.bin and functions process 22.bin and generates 22output.bin, it is supossed to stop after that but after seven minutes is processing 22.bin again.
I don't think is the app because each time the app sends data, even if it is the same one it will name the data with the next number (in my example 23.bin) but this is not the case it is just doing 22.bin again as if the trigger queue was not clear after the successful azure run, and it keeps doing it over and over again until I have to stop functions and make it crash i order to stop it.
Any idea in why is this happening and what can I try to correct it is greatly appreciated. I am just starting to learn about all this stuff.
One thing that could be possibly happening is that, the function execution time is exceeding 5 mins. Since this is a hard limit, function runtime would terminate the current execution and restart the function host.
One way to test this would be to create a Function app using Standard App Service plan instead of Consumption plan. Function app created with standard plan does not have execution time limit. You can log function start time and end time to see if it is taking longer than 5 mins to complete processing queue message.
I have an SFTP trigger that send the contents to an Azure Function. When Logic App invokes the Function, in the designer view I observe that it fails after 9 minutes. When I look at the Function monitor, I observe that the function is still running. The function is C#. I When the function completes, it logs the difference in DateTime between when it starts and ends. The time printed is about 300 seconds or five minutes. I know this is the limit for the time for a function to run.
This function runs in only 30 seconds on a VM on my five year old computer. Why is performance in Azure Function so poor? Is there anything that can be done to make it perform better?
What are you trying to do in the function?
What is the processing-time of the function if you test it in Function app itself rather than from the Logic App?