I want to run durable function from EventHubTrigger azure function.
public async Task<bool> Activities(
[EventHubTrigger(EventHubName, Connection = EventHubConnStrName)] EventData[] events,
[DurableClient] IDurableOrchestrationClient durableClient,
ILogger logger)
{
// chaining pattern durable function
await durableClient.StartNewAsync<string>(
FunctionNames.BatchEvents,
JsonConvertUtil.SerializeObject(events));
}
[FunctionName(FunctionNames.BatchEvents)]
public static async Task<bool> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger logger)
{
try
{
var events =
JsonConvertUtil.DeserializeObject<EventData[]>(context.GetInput<string>()));
....
}
catch (Exception ex)
{
logger.LogError(ex.Message);
throw;
}
}
Let me know how can i pass events to durable function or i can design it better way ?
I believe the way you written the workflow to pass Event data in Azure Durable Functions makes sense after reading this event-based-workflows-with-durable-function.
var events =
JsonConvertUtil.DeserializeObject<EventData[]>(context.GetInput<string>()));
In your code, as you have used context.GetInput<string> Method unpacks the event data you passed and also with deserialization to JSON.
In the above article, there is some information regarding the workflows usage of Event based Triggered Data using in Azure Durable Functions which might help you.
Related
There are couple of durable functions that call each other.
Main orchestration -> Sub orchestration -> Activity -> Helper async method
Each func has ILogger dependency and log on function start and on function end.
Both orchestrators duplicates "on start" message for some reason. (See pic)
Activity does not have this effect. (See pic)
Ran example below many times - same story.
I am also sure that the whole process has been triggered once.
Is this a bug in orchestrators or expected behavior?
using System;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
namespace Issues
{
public static class Log_Issue
{
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
log.LogWarning("Main Start");
await context.CallSubOrchestratorAsync("Sub", null);
log.LogWarning("Main End");
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
[FunctionName("Sub")]
public static async Task RunSubOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log.LogWarning("Sub Start");
var data = await context.CallActivityAsync<string>("Activity", null);
log.LogWarning("Sub End");
}
[FunctionName("Activity")]
public static async Task<string> GetDataActivity([ActivityTrigger] string name, ILogger log)
{
log.LogWarning("Activity Start");
var data = await GetDataAsync("https://www.google.com");
log.LogWarning("Activity End");
return data;
}
[FunctionName("Start")]
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]
HttpRequestMessage req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
var instanceId = await starter.StartNewAsync("Main", null);
log.LogWarning($"Started orchestration with ID = '{instanceId}'.");
return new OkResult();
}
private static async Task<string> GetDataAsync(string url)
{
var httpClient = new HttpClient();
using var request = new HttpRequestMessage
{
RequestUri = new Uri(url),
Method = HttpMethod.Get,
};
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
This is expected. For example on await context.CallActivityAsync("Activity", null); The code pauses itself and may even be loaded out of memory (in order to save on cost).
Then the orchestrator waits for an event to be placed in another Azure Storage Table which the activity creates, this may occur many days later. For activities they are usually very instant but it still waits for this event to occur.
When that happens the code needs to start from where it last stopped but there is no way to do that. Therefor the code reruns from the beginning but instead of waiting for the activity to finish again it first looks in the table and sees that we already have done this activity and can continue running. If the activity function returned some value it would be returned from the await call. During both runs of the orchestrator it would log but since we only go inte the activity ones that would only be logged ones.
This is why orchestrators have to be deterministic since e.g. a random value on the first run would not be the same as during the second run. Instead we would put the random.Next() into an activity funtion so that the value is saved to Azure Table Storage to be used on subsequent reruns. The orchestrator could also be waiting for some external events which normal functions create. E.g someone has to verify their email account which could take some number of days and this is why durable functions can unload themself and restart when they are triggered by the event.
All that #FilipB said is true. It is just missing the actual code to solve it ;)
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log = context.CreateReplaySafeLogger(log); // this is what you should use at the start of every Orchestrator
I am trying to store the time stamp information in durable entities and retrieve it every time a trigger fired. Here is how I am doing it. I want the timestamp value set by the current execution to be available for the next trigger. But when the control reaches "string prevTS = await context.CallEntityAsync(entityId, "Get");" to goes back to start of the function again. What am I missing here.
I want execution to be sequential between the timer triggers.
'''
***public static class GetOpenDataRealtimeFeed
{
[FunctionName("GetOpenDataOrchestrator")]
public static async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context, Binder binder, ILogger log)
{
var outputs = new List<string>();
var entityId = new EntityId(nameof(GetPrevLastModifiedTimestamp), "entityKey2");
string prevTS = await context.CallEntityAsync<string>(entityId, "Get");
string currentTS = DateTime.Now.ToString();
outputs.Add(currentTS);
outputs.Add(prevTS);
context.SignalEntity(entityId, "Set", currentTS);
return null;
}
//Durable entity function to get & set the last modified timestamp
[FunctionName("GetPrevLastModifiedTimestamp")]
public static void GetPrevLastModifiedTimestamp([EntityTrigger] IDurableEntityContext ctx)
{
switch (ctx.OperationName.ToLowerInvariant())
{
case "set":
ctx.SetState(ctx.GetInput<string>());
break;
case "get":
ctx.Return(ctx.GetState<string>());
break;
}
}
[FunctionName("getOpenDataRealtimeFeed_Trigger")]
public static async Task Run(
[TimerTrigger("%triggerTimer%")] TimerInfo myTimer,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
// Function input comes from the request content.
string instanceId = await starter.StartNewAsync("GetOpenDataOrchestrator", null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
}
}***
'''
I assume you are referring to the current line while debugging. If so, this is expected.
Since Durable Functions replays functions after awaiting a durable client call, execution won't ever go through the first round. Only the final replay will be "sequential" step overs.
As a general practice, we have been injecting our own "service" classes to all our function apps, and we want to do the same thing for the Orchestrator.
Example:
public async Task<string> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
// get input
var input = context.GetInput<MyInputType>();
// do some stuff #1
var input1 = new BlahBlahOne();
await context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1);
// do some stuff #2
var input1 = new BlahBlahTwo();
await context.CallActivityWithRetryAsync<string>("activityfn3", retryOptions, input1);
// do some stuff #3
var input1 = new BlahBlahThree();
await context.CallActivityWithRetryAsync<string>("activityfn3", retryOptions, input1);
// do some stuff #4
return "I'm done";
}
catch (Exception ex)
{
log.LogError(ex, "unexpected error");
throw;
}
}
We'd like to do something like this:
public async Task<string> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
string output = await _myOrchestratorService.RunAsync(context); // NOT allowed!
return output
}
catch (Exception ex)
{
log.LogError(ex, "unexpected error");
throw;
}
}
However, Note that we can't use 'await' as per Durable Functions code constraints on multi-threading. So I tried below, but how do I code it? Calling .Result makes the code 'hang' on the Activity function. What am I doing wrong?
public string Run(IDurableOrchestrationContext context)
{
// code like before, but then how do I call this?
// await context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1);
// I tried this, doesn't work, will hang on the first activity function
context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1).Result;
}
To clarify, the restriction on the use of await in Orchestrator functions only applies to tasks that are not generated by IDurableOrchestrationContext APIs. To quote the Durable Functions orchestration code constraint documentation:
Orchestrator code must never start any async operation except by using the IDurableOrchestrationContext API or the context.df object's API. For example, you can't use Task.Run, Task.Delay, and HttpClient.SendAsync in .NET or setTimeout and setInterval in JavaScript. The Durable Task Framework runs orchestrator code on a single thread. It can't interact with any other threads that might be called by other async APIs.
As this answer shows, asynchronous helper methods that only call await on Task objects created by IDurableOrchestrationContext are technically safe to await on as well. That means that your call to await _myOrchestratorService.RunAsync(context); may be alright, as long as that asynchronous method follows all of the normal orchestration code constraints.
All of that being said, I am not entirely sure what you gain by injecting a service that appears to only have a single method that contains all of the logic that would normally live in your orchestration method. That abstraction doesn't appear to improve the testability of the code, and the extra layer of abstraction may confuse our Durable Functions analyzer that is helpful in diagnosing when your code violates orchestration constraints.
Just make it async Task<string>. It should work
public async Task<string> RunAsync(IDurableOrchestrationContext context)
{
var result = await context.CallActivityWithRetryAsync<string>("activityfn1", retryOptions, input1);
return result;
}
I have an azure function with EventHubTrigger:
[FunctionName("TradesDataProcessStarterEh")]
public static async Task TradesDataProcessStarterEh([EventHubTrigger("aeehrobotronapiintegrationdev", Connection = "EventHubConnectionString", ConsumerGroup = "$Default")]
EventData eventData, PartitionContext partitionContext, [OrchestrationClient] DurableOrchestrationClient starter, ILogger log)
{
if (partitionContext.PartitionId != "1")
return;
var orchestrationId = await starter.StartNewAsync("O_ProcessTradesFromEventHub", eventData);
await partitionContext.CheckpointAsync();
}
The orchestrator function is receiving then the eventData:
[FunctionName("O_ProcessTradesFromEventHub")]
public static async Task ProcessTradesFromEventHub([OrchestrationTrigger] DurableOrchestrationContext context,
ILogger log)
{
if (!context.IsReplaying)
Console.WriteLine("O_ProcessTradesFromEventHub is triggered");
var eventData = context.GetInput<EventData>();
//do stuff...
}
But by execution of context.GetInput() I get an exception:
Function 'O_ProcessTradesFromEventHub (Orchestrator)' failed with an error. Reason: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type Microsoft.Azure.EventHubs.EventData. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'Body', line 1, position 81.
I can think of 3 possible solutions that you can try:
1 - Wrap EventData in your own class with a constructor (possibly via inheritance?).
2 - Try casting to object, doubt this will work but, but worth a try as it's a simple fix.
3 - Build your own DTO (Data Transfer Object) to transform EventData to <your class> and then pass <your class> to the orchestration.
I think (3) is the cleanest solution and you have full control over what you pass, unfortunately it is likely the least performant and most tedious.
Good luck!
Use LINQ to JSON - a year later but hopefully it'll save somebody else some time.
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public static async Task Run(
[OrchestrationTrigger] DurableOrchestrationContext context, ILogger log) {
var eventData = context.GetInput<JObject>();
log.LogInformation ($"Executing tasks with eventData = {eventData}");
string step = (string)eventData.SelectToken("Step");
log.LogInformation ($"Step = {step}");
}
I have a durable function that has inputs determined by a previous activity function
For each activity function I have multiple awaitable tasks where each task is dependent on the previous task's output.
This is my structure as follows:
Orchestrator
[FunctionName("MessageController")]
public static async void Run(
[OrchestrationTrigger] DurableOrchestrationContext context,
TraceWriter log)
{
if (!context.IsReplaying) log.Warning("MessageController started");
var Input1= context.CallActivityAsync<ResultMessage>("Function_1", new InputMessage());
var Input2= context.CallActivityAsync<ResultMessage>("Function_2", Input1);
var Input3= context.CallActivityAsync<ResultMessage>("Function_2", Input2);
}
Activity Function
[FunctionName("Function_1")]
public static ResultMessage Run(
[ActivityTrigger] DurableActivityContext activityContext,
TraceWriter log)
{
//Awaitable task
var taskOutput= await DoSomething();
//Awaitable task
var token = await DoAnotherThing(taskOutput);
}
I have tested this and all works fine. But i ma wondering if this is good practice?
Is it normal to have awaitable tasks within an activity function for a durable function?
Yes, that's totally fine. In fact, you can do pretty much anything you like in activity functions, as long as they complete in reasonable time (below 5 minutes on Consumption Plan). You can do async calls, switch threads, and do non-deterministic operations.
Constraints apply only to orchestrator functions.