When going through Azure Durable function they mention that we can write stateful functions. What is meant by stateful and what state is maintained? Are we talking about running state of a function?
A stateful function is a function which has a state, that is, some data is asscociated to the function. In our specific case, we are talking about:
managing state of the workflow (at what step we are)
create progress checkpoints (when a checkpoint is reached, the state is changed)
persisting execution history
scheduling activity
From the docs:
Durable Functions is an extension to the Azure Functions runtime that
enables the definition of stateful workflows in code. By breaking down
workflows into activities, the Durable Functions extension can manage
state, create progress checkpoints, and handle the distribution of
function calls across servers. In the background, it makes use of an
Azure Storage account to persist execution history, schedule activity
functions and retrieve responses. Your serverless code should never
interact with persisted information in that storage account, and is
typically not something with which developers need to interact.
Related
I have a .NET isolated function with a queue trigger.
When triggering that queue from the storage explorer or from another function using a QueueServiceClient, a new operationId is made up. Thus I cannot correlate it, it seems.
Is it possible to do distributed tracing using W3C standard for Azure Function Queue trigger? I can not find any source on this.
If so, how?
Currently not supported.
Azure Functions team will evaluate this scenario (at some unmentioned point in time) whether or not it can be/will be supported. This has to do with their dependency on the team creating the Azure.Storage.Queues SDK.
https://github.com/Azure/azure-functions-dotnet-worker/issues/1126
I have a requirement where I need to transfer file (20-150 MB) between two systems. For this requirement , is it better to use Durable function instead of Azure data factory (ADF). As per my understanding , ADF execution will costlier as compared to durable functions. Note : durable function trigger is eventGrid trigger. Any suggestion will be helpful. File transfer will be simple pass through, no transformation is involved.
Also, for my requirement even simple azure function will work right instead of durable function? There is no need of function orchestration as file is not processed in batch. Since, file will be processed based on event trigger.
As of my experience, I would like to recommend using Azure functions over ADF is a good idea because of the following reasons:
Azure Data Factory is too expensive. ADF costs way more than azure functions.
Custom Logic: ADF is not built to perform cleansing logics or any custom code. Its primary goal is for data integration from external systems using its vast connector pool
Latency: ADF has much higher latency due to the large overhead of its job framework
Durable function is just related to the maximum execution time of a single call. For "out of the box" functions, that timeout is 10min, for durable functions this limitation gets removed. In this case, where you simply need to copy the data, there might be timeout issue and therefore you can consider the Durable function. Otherwise, simple function should also work fine. Moreover, Durable functions and normal functions share the same billing pattern.
For more details: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp
We have an Azure Function App (timer triggered) on a Consumption plan for testing purpose. The App fist fires a bunch of Stored Procedures on a SQL Server. We use Task.Run() and inside of it it's just a Synchronous operation to run an SP on the Server. It's a fire and forgets tasks that we require and the Exceptions/Errors from SQL are logged to the table inside of the SQL Server. This particular Azure App is a plan to migrate our SQL Agent Jobs (as we are moving towards a PaaS Database) to the cloud. Moreover, the Function App triggers an SP across multiple databases. So a single Task.Run for each DB.
The thing is the execution of the SP might take around 20 minutes to complete itself. I see that around 19 minutes the Connection is dropped. So I see that an SP was was started let's say at 5:00 AM and with appropriate logging inside of an SP, it went on till 5:19 AM and then it stopped (no success log). So I believe the SQLConnection from C# is dropped. The consumption plan default is 5 minutes. So if it's a timeout issue then why still I can continue till 19 minutes and then only it's dropped. I have observed this behavior for some days now.
I cannot arrive at a feasible explanation of the above behavior.
Maximum timeout for azure functions in consumption plan is 10min:
Change plan to support longer timeout or you can use Durable functions (intended for long-running tasks).
Durable Functions is an extension of Azure Functions that lets you
write stateful functions in a serverless compute environment. The
extension lets you define stateful workflows by writing orchestrator
functions and stateful entities by writing entity functions using the
Azure Functions programming model. Behind the scenes, the extension
manages state, checkpoints, and restarts for you, allowing you to
focus on your business logic.
Refs:
https://learn.microsoft.com/pl-pl/azure/azure-functions/functions-scale#function-app-timeout-duration
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp
https://learn.microsoft.com/en-us/learn/modules/create-long-running-serverless-workflow-with-durable-functions/
When would I prefer Azure Functions to Azure Container Instances, considering they both offer the possibility to perform run-once tasks and they bill on consumption?
Also, reading this Microsoft Learn Module:
Serverless compute can be thought of as a function as a service (FaaS), or a microservice that is hosted on a cloud platform.
Azure Functions is a platform that allows you to run plain code (instead of containers). The strength of Azure Functions is the rich set of bindings (input- and output bindings) it supports. If you want to execute a piece of code when something happen (e. g. a blob was added to a storage Account, a timer gets triggered, ....) then I definitely would go with Azure Functions.
If you want to run some container-based workload for a short period of time and you don't have an orchestrator (like Azure Kubernetes Services) in place - Azure Container Instances makes sense.
Take a look at this from Microsoft doc
Source: https://learn.microsoft.com/en-us/dotnet/architecture/modernize-with-azure-containers/modernize-existing-apps-to-cloud-optimized/choosing-azure-compute-options-for-container-based-applications
If you would like to simplify application development model where your application architecture has microservices that are more granular, such that various functionalities are reduced typically to a single function then, Azure functions can be considered for usage.
In case, the solution needs some extension to existing azure application with event trigger based use cases , the azure functions can be better choice . Here, the specific code (function) shall be invoked only for specific event or trigger as per requirement and the function instances are created and destroyed on demand (compute on demand - function as a service (FaaS) ).
More often, the event driven architecture is seen in IoT where typically you can define a specific trigger that causes execution of Azure function. Accordingly, Azure functions have its place in IoT ecosystem as well.
If the solution has fast bursting and scaling requirement, then container Instances can be used whereas if the requirement is predictable scaling then, VMs can be used.
Azure function avoids allocation of extra resources (VMs) and also the cost is considered only when the function is processing work. Here, we need not take care of infrastructure such as where the code is going to execute, server configuration, memory etc. For ACI, the cost is per-second where it is accounted based on the time the container runs - CaaS(Container As A Service).
ACI enables for quickly spawning a container for performing the operation and deletion of it when done where the cost is only for few hours of usage rather than a dedicated VM which would be of high cost. ACI enables one to run a container by avoiding dependency on orchestrators like Kubernetes in scenarios where we would not need orchestration functions like service discovery, mesh and co-ordination features.
The key difference is that, in case of Azure function, the function is the unit of work whereas in container instance, the entire container contains the unit of work. So, Azure functions start and end based on event triggers whereas the microservices in containers shall get executed the entire time.
The processing / execution time also plays a critical role where if the event handler function consumes processing time of 10 minutes or more to execute, it is better to host in VM because the maximum timeout that is configurable for functions is 10 minutes.
There are typical solutions that utilize both the functionalities such that Azure function shall be triggered for minimal processing / decision making and inturn can invoke container instance for specific burst processing / complete processing.
Also, ACI along with AKS form a powerful deployment model for microservices where AKS can be for typical deployment of microservices and ACIs for handling the burst workloads thereby reducing the challenges in management of scaling and ensuring effective utilization of the per second usage cost model.
I've recently started experimenting with Azure functions and I'd like to get some info on whether the following makes sense and is possible..
How feasible is it to build a normal .NET Core Web API following DDD patterns but replacing the API endpoints with durable azure functions? Is this a possible use case for Azure Functions, to make the web API "serverless"?
And how would the whole thing be structured? Does each Azure function need its own project or can they all be placed in one project?
Thanks
As I wrote in comments why not?
You can define bounded context and deploy to one azure function as microservice for instance service which will be responsible for orders, other azure function of delivery and so on.
Use durable function when you need to orchestrate actions, for instance you have buy flow when in first action you lock products, take payments and unlock so you have kind of dependency on each other.
You can use azure functions with service bus or azure queue storage for event processing.
One thing keep in mind that when you design function you have time limitations is up to 5 minutes on provisioning plan. So when you design newsletter for instance keep in mind that you would need to send email in batches.