I have a durable function that has inputs determined by a previous activity function
For each activity function I have multiple awaitable tasks where each task is dependent on the previous task's output.
This is my structure as follows:
Orchestrator
[FunctionName("MessageController")]
public static async void Run(
[OrchestrationTrigger] DurableOrchestrationContext context,
TraceWriter log)
{
if (!context.IsReplaying) log.Warning("MessageController started");
var Input1= context.CallActivityAsync<ResultMessage>("Function_1", new InputMessage());
var Input2= context.CallActivityAsync<ResultMessage>("Function_2", Input1);
var Input3= context.CallActivityAsync<ResultMessage>("Function_2", Input2);
}
Activity Function
[FunctionName("Function_1")]
public static ResultMessage Run(
[ActivityTrigger] DurableActivityContext activityContext,
TraceWriter log)
{
//Awaitable task
var taskOutput= await DoSomething();
//Awaitable task
var token = await DoAnotherThing(taskOutput);
}
I have tested this and all works fine. But i ma wondering if this is good practice?
Is it normal to have awaitable tasks within an activity function for a durable function?
Yes, that's totally fine. In fact, you can do pretty much anything you like in activity functions, as long as they complete in reasonable time (below 5 minutes on Consumption Plan). You can do async calls, switch threads, and do non-deterministic operations.
Constraints apply only to orchestrator functions.
Related
I want to run durable function from EventHubTrigger azure function.
public async Task<bool> Activities(
[EventHubTrigger(EventHubName, Connection = EventHubConnStrName)] EventData[] events,
[DurableClient] IDurableOrchestrationClient durableClient,
ILogger logger)
{
// chaining pattern durable function
await durableClient.StartNewAsync<string>(
FunctionNames.BatchEvents,
JsonConvertUtil.SerializeObject(events));
}
[FunctionName(FunctionNames.BatchEvents)]
public static async Task<bool> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger logger)
{
try
{
var events =
JsonConvertUtil.DeserializeObject<EventData[]>(context.GetInput<string>()));
....
}
catch (Exception ex)
{
logger.LogError(ex.Message);
throw;
}
}
Let me know how can i pass events to durable function or i can design it better way ?
I believe the way you written the workflow to pass Event data in Azure Durable Functions makes sense after reading this event-based-workflows-with-durable-function.
var events =
JsonConvertUtil.DeserializeObject<EventData[]>(context.GetInput<string>()));
In your code, as you have used context.GetInput<string> Method unpacks the event data you passed and also with deserialization to JSON.
In the above article, there is some information regarding the workflows usage of Event based Triggered Data using in Azure Durable Functions which might help you.
There are couple of durable functions that call each other.
Main orchestration -> Sub orchestration -> Activity -> Helper async method
Each func has ILogger dependency and log on function start and on function end.
Both orchestrators duplicates "on start" message for some reason. (See pic)
Activity does not have this effect. (See pic)
Ran example below many times - same story.
I am also sure that the whole process has been triggered once.
Is this a bug in orchestrators or expected behavior?
using System;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
namespace Issues
{
public static class Log_Issue
{
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
log.LogWarning("Main Start");
await context.CallSubOrchestratorAsync("Sub", null);
log.LogWarning("Main End");
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
[FunctionName("Sub")]
public static async Task RunSubOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log.LogWarning("Sub Start");
var data = await context.CallActivityAsync<string>("Activity", null);
log.LogWarning("Sub End");
}
[FunctionName("Activity")]
public static async Task<string> GetDataActivity([ActivityTrigger] string name, ILogger log)
{
log.LogWarning("Activity Start");
var data = await GetDataAsync("https://www.google.com");
log.LogWarning("Activity End");
return data;
}
[FunctionName("Start")]
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]
HttpRequestMessage req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
var instanceId = await starter.StartNewAsync("Main", null);
log.LogWarning($"Started orchestration with ID = '{instanceId}'.");
return new OkResult();
}
private static async Task<string> GetDataAsync(string url)
{
var httpClient = new HttpClient();
using var request = new HttpRequestMessage
{
RequestUri = new Uri(url),
Method = HttpMethod.Get,
};
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
This is expected. For example on await context.CallActivityAsync("Activity", null); The code pauses itself and may even be loaded out of memory (in order to save on cost).
Then the orchestrator waits for an event to be placed in another Azure Storage Table which the activity creates, this may occur many days later. For activities they are usually very instant but it still waits for this event to occur.
When that happens the code needs to start from where it last stopped but there is no way to do that. Therefor the code reruns from the beginning but instead of waiting for the activity to finish again it first looks in the table and sees that we already have done this activity and can continue running. If the activity function returned some value it would be returned from the await call. During both runs of the orchestrator it would log but since we only go inte the activity ones that would only be logged ones.
This is why orchestrators have to be deterministic since e.g. a random value on the first run would not be the same as during the second run. Instead we would put the random.Next() into an activity funtion so that the value is saved to Azure Table Storage to be used on subsequent reruns. The orchestrator could also be waiting for some external events which normal functions create. E.g someone has to verify their email account which could take some number of days and this is why durable functions can unload themself and restart when they are triggered by the event.
All that #FilipB said is true. It is just missing the actual code to solve it ;)
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log = context.CreateReplaySafeLogger(log); // this is what you should use at the start of every Orchestrator
As a general practice, we have been injecting our own "service" classes to all our function apps, and we want to do the same thing for the Orchestrator.
Example:
public async Task<string> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
// get input
var input = context.GetInput<MyInputType>();
// do some stuff #1
var input1 = new BlahBlahOne();
await context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1);
// do some stuff #2
var input1 = new BlahBlahTwo();
await context.CallActivityWithRetryAsync<string>("activityfn3", retryOptions, input1);
// do some stuff #3
var input1 = new BlahBlahThree();
await context.CallActivityWithRetryAsync<string>("activityfn3", retryOptions, input1);
// do some stuff #4
return "I'm done";
}
catch (Exception ex)
{
log.LogError(ex, "unexpected error");
throw;
}
}
We'd like to do something like this:
public async Task<string> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
string output = await _myOrchestratorService.RunAsync(context); // NOT allowed!
return output
}
catch (Exception ex)
{
log.LogError(ex, "unexpected error");
throw;
}
}
However, Note that we can't use 'await' as per Durable Functions code constraints on multi-threading. So I tried below, but how do I code it? Calling .Result makes the code 'hang' on the Activity function. What am I doing wrong?
public string Run(IDurableOrchestrationContext context)
{
// code like before, but then how do I call this?
// await context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1);
// I tried this, doesn't work, will hang on the first activity function
context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1).Result;
}
To clarify, the restriction on the use of await in Orchestrator functions only applies to tasks that are not generated by IDurableOrchestrationContext APIs. To quote the Durable Functions orchestration code constraint documentation:
Orchestrator code must never start any async operation except by using the IDurableOrchestrationContext API or the context.df object's API. For example, you can't use Task.Run, Task.Delay, and HttpClient.SendAsync in .NET or setTimeout and setInterval in JavaScript. The Durable Task Framework runs orchestrator code on a single thread. It can't interact with any other threads that might be called by other async APIs.
As this answer shows, asynchronous helper methods that only call await on Task objects created by IDurableOrchestrationContext are technically safe to await on as well. That means that your call to await _myOrchestratorService.RunAsync(context); may be alright, as long as that asynchronous method follows all of the normal orchestration code constraints.
All of that being said, I am not entirely sure what you gain by injecting a service that appears to only have a single method that contains all of the logic that would normally live in your orchestration method. That abstraction doesn't appear to improve the testability of the code, and the extra layer of abstraction may confuse our Durable Functions analyzer that is helpful in diagnosing when your code violates orchestration constraints.
Just make it async Task<string>. It should work
public async Task<string> RunAsync(IDurableOrchestrationContext context)
{
var result = await context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1);
return result;
}
I am looking at this example to run a durable function Activity after a set timeout.
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-eternal-orchestrations
This will allow my function activity to perform processing of data, then wait exactly 1 hour before it attempts to load again. This will continue to run forever. Perfect.
However, when publishing the Function to Azure, I don't want to have to manually invoke/start the function via the associated HTTP Trigger. I just want the durable function to kickoff automatically and start processing.
Is this possible? If not, what is a suggested work around?
Thanks!
As discussed in the comments, one way of doing this would be to add a new Task in your Release pipeline.
Here is what I understood of your setup from your question:
[FunctionName("ClientFunction")]
public static async Task<HttpResponseMessage> OnHttpTriggerAsync([HttpTrigger(AuthorizationLevel.Anonymous, "post")]
HttpRequestMessage request, [OrchestrationClient] DurableOrchestrationClient starter, ILogger logger)
{
// Triggers the orchestrator.
string instanceId = await starter.StartNewAsync("OrchestratorFunction", null);
return new HttpResponseMessage(HttpStatusCode.OK);
}
[FunctionName("OrchestratorFunction")]
public static async Task DoOrchestrationThingsAsync([OrchestrationTrigger] DurableOrchestrationContext context, ILogger logger)
{
DateTime deadline = context.CurrentUtcDateTime.Add(TimeSpan.FromHours(1));
await context.CreateTimer(deadline, CancellationToken.None);
// Triggers some yout activity.
await context.CallActivityAsync("ActivityFunction", null);
}
[FunctionName("ActivityFunction")]
public static Task DoAnAwesomeActivity([ActivityTrigger] DurableActivityContext context)
{
}
Now, every time you deploy a new version of the Function App, you need the orchestrator to be run. However, I do not think it can be started by itself.
What I propose is to have a simple bash script (using curl or something else) that would call the ClientFunction at the appropriate URL.
On top of that, one of the nice things of this solution is that you could make the deployment fail if the Azure Function does not respond.
This seems to be working too.
[FunctionName("AutoStart")]
public static async Task Run([TimerTrigger("*/5 * * * * *", RunOnStartup = true, UseMonitor = false)]TimerInfo myStartTimer,
[DurableClient] IDurableClient orchestrationClient, ILogger log)
{
string instanceId = await orchestrationClient.StartNewAsync("Start_Orchestrator", null);
}
I don't know if there are hidden problems with this, but I'm experimenting now with having a TimerTrigger that runs on startup and also once a day at midnight (or whatever schedule you want). That TimerTrigger will search the list of instances for any running instances of this orchestration, terminate them, then start a new one.
private const string MyOrchestrationName = "MyOrchestration";
[FunctionName("MyOrchestration_Trigger")]
public async Task MyOrchestrationr_Trigger(
[TimerTrigger("0 0 0 * * *", RunOnStartup = true)] TimerInfo timer,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log,
CancellationToken cancellationToken)
{
// Get all the instances currently running that have a status of Pending, Running, ContinuedAsNew
var instances = await starter.ListInstancesAsync(new OrchestrationStatusQueryCondition()
{
ShowInput = false,
RuntimeStatus = new List<OrchestrationRuntimeStatus>() { OrchestrationRuntimeStatus.Suspended, OrchestrationRuntimeStatus.Pending, OrchestrationRuntimeStatus.Running, OrchestrationRuntimeStatus.ContinuedAsNew }
}, cancellationToken);
// Find any instances of the current orchestration that are running.
var myInstances = instances.DurableOrchestrationState.Where(inst => inst.Name == MyOrchestrationName);
List<Task> terminateTasks = new List<Task>();
foreach (var instance in myInstances )
{
// Delete any instances that are currently running.
terminateTasks.Add(starter.TerminateAsync(instance.InstanceId, $"Restarting eternal orchestration"));
}
await Task.WhenAll(terminateTasks);
// Start the new task now that other instances have been terminated.
string instanceId = await starter.StartNewAsync(MyOrchestrationName, null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
I think at least for my purposes this will be safe. Any activities that are running when you terminate will still run to completion (which is what I want in my case), so you would just kill it and restart it on a schedule.
I have an azure function with EventHubTrigger:
[FunctionName("TradesDataProcessStarterEh")]
public static async Task TradesDataProcessStarterEh([EventHubTrigger("aeehrobotronapiintegrationdev", Connection = "EventHubConnectionString", ConsumerGroup = "$Default")]
EventData eventData, PartitionContext partitionContext, [OrchestrationClient] DurableOrchestrationClient starter, ILogger log)
{
if (partitionContext.PartitionId != "1")
return;
var orchestrationId = await starter.StartNewAsync("O_ProcessTradesFromEventHub", eventData);
await partitionContext.CheckpointAsync();
}
The orchestrator function is receiving then the eventData:
[FunctionName("O_ProcessTradesFromEventHub")]
public static async Task ProcessTradesFromEventHub([OrchestrationTrigger] DurableOrchestrationContext context,
ILogger log)
{
if (!context.IsReplaying)
Console.WriteLine("O_ProcessTradesFromEventHub is triggered");
var eventData = context.GetInput<EventData>();
//do stuff...
}
But by execution of context.GetInput() I get an exception:
Function 'O_ProcessTradesFromEventHub (Orchestrator)' failed with an error. Reason: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type Microsoft.Azure.EventHubs.EventData. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'Body', line 1, position 81.
I can think of 3 possible solutions that you can try:
1 - Wrap EventData in your own class with a constructor (possibly via inheritance?).
2 - Try casting to object, doubt this will work but, but worth a try as it's a simple fix.
3 - Build your own DTO (Data Transfer Object) to transform EventData to <your class> and then pass <your class> to the orchestration.
I think (3) is the cleanest solution and you have full control over what you pass, unfortunately it is likely the least performant and most tedious.
Good luck!
Use LINQ to JSON - a year later but hopefully it'll save somebody else some time.
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public static async Task Run(
[OrchestrationTrigger] DurableOrchestrationContext context, ILogger log) {
var eventData = context.GetInput<JObject>();
log.LogInformation ($"Executing tasks with eventData = {eventData}");
string step = (string)eventData.SelectToken("Step");
log.LogInformation ($"Step = {step}");
}