Azure Durable Function Invoke without HttpTrigger (Autostart) - azure

I am looking at this example to run a durable function Activity after a set timeout.
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-eternal-orchestrations
This will allow my function activity to perform processing of data, then wait exactly 1 hour before it attempts to load again. This will continue to run forever. Perfect.
However, when publishing the Function to Azure, I don't want to have to manually invoke/start the function via the associated HTTP Trigger. I just want the durable function to kickoff automatically and start processing.
Is this possible? If not, what is a suggested work around?
Thanks!

As discussed in the comments, one way of doing this would be to add a new Task in your Release pipeline.
Here is what I understood of your setup from your question:
[FunctionName("ClientFunction")]
public static async Task<HttpResponseMessage> OnHttpTriggerAsync([HttpTrigger(AuthorizationLevel.Anonymous, "post")]
HttpRequestMessage request, [OrchestrationClient] DurableOrchestrationClient starter, ILogger logger)
{
// Triggers the orchestrator.
string instanceId = await starter.StartNewAsync("OrchestratorFunction", null);
return new HttpResponseMessage(HttpStatusCode.OK);
}
[FunctionName("OrchestratorFunction")]
public static async Task DoOrchestrationThingsAsync([OrchestrationTrigger] DurableOrchestrationContext context, ILogger logger)
{
DateTime deadline = context.CurrentUtcDateTime.Add(TimeSpan.FromHours(1));
await context.CreateTimer(deadline, CancellationToken.None);
// Triggers some yout activity.
await context.CallActivityAsync("ActivityFunction", null);
}
[FunctionName("ActivityFunction")]
public static Task DoAnAwesomeActivity([ActivityTrigger] DurableActivityContext context)
{
}
Now, every time you deploy a new version of the Function App, you need the orchestrator to be run. However, I do not think it can be started by itself.
What I propose is to have a simple bash script (using curl or something else) that would call the ClientFunction at the appropriate URL.
On top of that, one of the nice things of this solution is that you could make the deployment fail if the Azure Function does not respond.

This seems to be working too.
[FunctionName("AutoStart")]
public static async Task Run([TimerTrigger("*/5 * * * * *", RunOnStartup = true, UseMonitor = false)]TimerInfo myStartTimer,
[DurableClient] IDurableClient orchestrationClient, ILogger log)
{
string instanceId = await orchestrationClient.StartNewAsync("Start_Orchestrator", null);
}

I don't know if there are hidden problems with this, but I'm experimenting now with having a TimerTrigger that runs on startup and also once a day at midnight (or whatever schedule you want). That TimerTrigger will search the list of instances for any running instances of this orchestration, terminate them, then start a new one.
private const string MyOrchestrationName = "MyOrchestration";
[FunctionName("MyOrchestration_Trigger")]
public async Task MyOrchestrationr_Trigger(
[TimerTrigger("0 0 0 * * *", RunOnStartup = true)] TimerInfo timer,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log,
CancellationToken cancellationToken)
{
// Get all the instances currently running that have a status of Pending, Running, ContinuedAsNew
var instances = await starter.ListInstancesAsync(new OrchestrationStatusQueryCondition()
{
ShowInput = false,
RuntimeStatus = new List<OrchestrationRuntimeStatus>() { OrchestrationRuntimeStatus.Suspended, OrchestrationRuntimeStatus.Pending, OrchestrationRuntimeStatus.Running, OrchestrationRuntimeStatus.ContinuedAsNew }
}, cancellationToken);
// Find any instances of the current orchestration that are running.
var myInstances = instances.DurableOrchestrationState.Where(inst => inst.Name == MyOrchestrationName);
List<Task> terminateTasks = new List<Task>();
foreach (var instance in myInstances )
{
// Delete any instances that are currently running.
terminateTasks.Add(starter.TerminateAsync(instance.InstanceId, $"Restarting eternal orchestration"));
}
await Task.WhenAll(terminateTasks);
// Start the new task now that other instances have been terminated.
string instanceId = await starter.StartNewAsync(MyOrchestrationName, null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
I think at least for my purposes this will be safe. Any activities that are running when you terminate will still run to completion (which is what I want in my case), so you would just kill it and restart it on a schedule.

Related

Azure Durable orchestration function ILogger outputs logs twice

There are couple of durable functions that call each other.
Main orchestration -> Sub orchestration -> Activity -> Helper async method
Each func has ILogger dependency and log on function start and on function end.
Both orchestrators duplicates "on start" message for some reason. (See pic)
Activity does not have this effect. (See pic)
Ran example below many times - same story.
I am also sure that the whole process has been triggered once.
Is this a bug in orchestrators or expected behavior?
using System;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
namespace Issues
{
public static class Log_Issue
{
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
log.LogWarning("Main Start");
await context.CallSubOrchestratorAsync("Sub", null);
log.LogWarning("Main End");
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
[FunctionName("Sub")]
public static async Task RunSubOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log.LogWarning("Sub Start");
var data = await context.CallActivityAsync<string>("Activity", null);
log.LogWarning("Sub End");
}
[FunctionName("Activity")]
public static async Task<string> GetDataActivity([ActivityTrigger] string name, ILogger log)
{
log.LogWarning("Activity Start");
var data = await GetDataAsync("https://www.google.com");
log.LogWarning("Activity End");
return data;
}
[FunctionName("Start")]
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]
HttpRequestMessage req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
var instanceId = await starter.StartNewAsync("Main", null);
log.LogWarning($"Started orchestration with ID = '{instanceId}'.");
return new OkResult();
}
private static async Task<string> GetDataAsync(string url)
{
var httpClient = new HttpClient();
using var request = new HttpRequestMessage
{
RequestUri = new Uri(url),
Method = HttpMethod.Get,
};
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
This is expected. For example on await context.CallActivityAsync("Activity", null); The code pauses itself and may even be loaded out of memory (in order to save on cost).
Then the orchestrator waits for an event to be placed in another Azure Storage Table which the activity creates, this may occur many days later. For activities they are usually very instant but it still waits for this event to occur.
When that happens the code needs to start from where it last stopped but there is no way to do that. Therefor the code reruns from the beginning but instead of waiting for the activity to finish again it first looks in the table and sees that we already have done this activity and can continue running. If the activity function returned some value it would be returned from the await call. During both runs of the orchestrator it would log but since we only go inte the activity ones that would only be logged ones.
This is why orchestrators have to be deterministic since e.g. a random value on the first run would not be the same as during the second run. Instead we would put the random.Next() into an activity funtion so that the value is saved to Azure Table Storage to be used on subsequent reruns. The orchestrator could also be waiting for some external events which normal functions create. E.g someone has to verify their email account which could take some number of days and this is why durable functions can unload themself and restart when they are triggered by the event.
All that #FilipB said is true. It is just missing the actual code to solve it ;)
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log = context.CreateReplaySafeLogger(log); // this is what you should use at the start of every Orchestrator

Azure Durable entity functions to get & set timestamp info using timer trigger

I am trying to store the time stamp information in durable entities and retrieve it every time a trigger fired. Here is how I am doing it. I want the timestamp value set by the current execution to be available for the next trigger. But when the control reaches "string prevTS = await context.CallEntityAsync(entityId, "Get");" to goes back to start of the function again. What am I missing here.
I want execution to be sequential between the timer triggers.
'''
***public static class GetOpenDataRealtimeFeed
{
[FunctionName("GetOpenDataOrchestrator")]
public static async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context, Binder binder, ILogger log)
{
var outputs = new List<string>();
var entityId = new EntityId(nameof(GetPrevLastModifiedTimestamp), "entityKey2");
string prevTS = await context.CallEntityAsync<string>(entityId, "Get");
string currentTS = DateTime.Now.ToString();
outputs.Add(currentTS);
outputs.Add(prevTS);
context.SignalEntity(entityId, "Set", currentTS);
return null;
}
//Durable entity function to get & set the last modified timestamp
[FunctionName("GetPrevLastModifiedTimestamp")]
public static void GetPrevLastModifiedTimestamp([EntityTrigger] IDurableEntityContext ctx)
{
switch (ctx.OperationName.ToLowerInvariant())
{
case "set":
ctx.SetState(ctx.GetInput<string>());
break;
case "get":
ctx.Return(ctx.GetState<string>());
break;
}
}
[FunctionName("getOpenDataRealtimeFeed_Trigger")]
public static async Task Run(
[TimerTrigger("%triggerTimer%")] TimerInfo myTimer,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
// Function input comes from the request content.
string instanceId = await starter.StartNewAsync("GetOpenDataOrchestrator", null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
}
}***
'''
I assume you are referring to the current line while debugging. If so, this is expected.
Since Durable Functions replays functions after awaiting a durable client call, execution won't ever go through the first round. Only the final replay will be "sequential" step overs.

Azure Durable Function: JsonSerializationException by passing complex object from trigger to orchestrator

I have an azure function with EventHubTrigger:
[FunctionName("TradesDataProcessStarterEh")]
public static async Task TradesDataProcessStarterEh([EventHubTrigger("aeehrobotronapiintegrationdev", Connection = "EventHubConnectionString", ConsumerGroup = "$Default")]
EventData eventData, PartitionContext partitionContext, [OrchestrationClient] DurableOrchestrationClient starter, ILogger log)
{
if (partitionContext.PartitionId != "1")
return;
var orchestrationId = await starter.StartNewAsync("O_ProcessTradesFromEventHub", eventData);
await partitionContext.CheckpointAsync();
}
The orchestrator function is receiving then the eventData:
[FunctionName("O_ProcessTradesFromEventHub")]
public static async Task ProcessTradesFromEventHub([OrchestrationTrigger] DurableOrchestrationContext context,
ILogger log)
{
if (!context.IsReplaying)
Console.WriteLine("O_ProcessTradesFromEventHub is triggered");
var eventData = context.GetInput<EventData>();
//do stuff...
}
But by execution of context.GetInput() I get an exception:
Function 'O_ProcessTradesFromEventHub (Orchestrator)' failed with an error. Reason: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type Microsoft.Azure.EventHubs.EventData. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'Body', line 1, position 81.
I can think of 3 possible solutions that you can try:
1 - Wrap EventData in your own class with a constructor (possibly via inheritance?).
2 - Try casting to object, doubt this will work but, but worth a try as it's a simple fix.
3 - Build your own DTO (Data Transfer Object) to transform EventData to <your class> and then pass <your class> to the orchestration.
I think (3) is the cleanest solution and you have full control over what you pass, unfortunately it is likely the least performant and most tedious.
Good luck!
Use LINQ to JSON - a year later but hopefully it'll save somebody else some time.
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public static async Task Run(
[OrchestrationTrigger] DurableOrchestrationContext context, ILogger log) {
var eventData = context.GetInput<JObject>();
log.LogInformation ($"Executing tasks with eventData = {eventData}");
string step = (string)eventData.SelectToken("Step");
log.LogInformation ($"Step = {step}");
}

Durable Functions - Awaitable Tasks inside Activity Function

I have a durable function that has inputs determined by a previous activity function
For each activity function I have multiple awaitable tasks where each task is dependent on the previous task's output.
This is my structure as follows:
Orchestrator
[FunctionName("MessageController")]
public static async void Run(
[OrchestrationTrigger] DurableOrchestrationContext context,
TraceWriter log)
{
if (!context.IsReplaying) log.Warning("MessageController started");
var Input1= context.CallActivityAsync<ResultMessage>("Function_1", new InputMessage());
var Input2= context.CallActivityAsync<ResultMessage>("Function_2", Input1);
var Input3= context.CallActivityAsync<ResultMessage>("Function_2", Input2);
}
Activity Function
[FunctionName("Function_1")]
public static ResultMessage Run(
[ActivityTrigger] DurableActivityContext activityContext,
TraceWriter log)
{
//Awaitable task
var taskOutput= await DoSomething();
//Awaitable task
var token = await DoAnotherThing(taskOutput);
}
I have tested this and all works fine. But i ma wondering if this is good practice?
Is it normal to have awaitable tasks within an activity function for a durable function?
Yes, that's totally fine. In fact, you can do pretty much anything you like in activity functions, as long as they complete in reasonable time (below 5 minutes on Consumption Plan). You can do async calls, switch threads, and do non-deterministic operations.
Constraints apply only to orchestrator functions.

How to invoke Durable function by timer trigger?

I am new to Durable function(Orchestration function) and seen sample application as per Microsoft documentation.So I have few doubts.
example:
public static async Task<HttpResponseMessage> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, methods: "post",
Route = "orchestrators/{functionName}")] HttpRequestMessage req,
[OrchestrationClient] DurableOrchestrationClient starter,
string functionName,
TraceWriter log)
{
// Function input comes from the request content.
dynamic eventData = await req.Content.ReadAsAsync<object>();
string instanceId = await starter.StartNewAsync(functionName, eventData);
log.Info($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
to invoke it I made HTTP POST request using postman so request processed successfully but when I configured different verb like HTTP GET it was responded with NotFound" error in console as well as request made to it with http request from browser responded with "NotFound" error in console .Why this happened?
Can I invoke any Orchestration function with in timer trigger azure function?
If not why?
UPDATE:
Some additional details about question
[FunctionName("TimerTrigger")]
public static async Task Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, TraceWriter log)
{//this runs for every 5minutes
using (HttpClient client = new HttpClient())
{
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("", "")
});
//making request to above function by http trigger
var result = await client.PostAsync("http://localhost:7071/orchestrators/E1_HelloSequence", content);
}
log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
return;
}
can I make request to http trigger by timer triggred why because my durable function has long running process so if invoke orchestration function in timer trigger itself so there might be possibility of timer triggered timeout so that why I am trying to follow this approach.Is it possible to invoke by above code?
This is general Azure Functions behavior. The reason GET doesn't work is because the function in the sample is only configured to work with POST. See the [HttpTrigger] attribute in the function signature:
[HttpTrigger(AuthorizationLevel.Anonymous, methods: "post",
Route = "orchestrators/{functionName}")]
If you want to support GET, then change the methods parameter accordingly:
[HttpTrigger(AuthorizationLevel.Anonymous, methods: "get",
Route = "orchestrators/{functionName}")]
Note that Visual Studio seems to have a caching bug where making changes to route information is not properly saved when debugging locally. I opened a GitHub issue to track that here: https://github.com/Azure/Azure-Functions/issues/552
For more information on HTTP triggers, see this documentation: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-http-webhook
If you want to trigger a Durable Function using a timer trigger, then just change the trigger type. For example:
[FunctionName("ScheduledStart")]
public static async Task RunScheduled(
[TimerTrigger("0 0 * * * *")] TimerInfo timerInfo,
[OrchestrationClient] DurableOrchestrationClient starter,
TraceWriter log)
{
string functionName = "E1_HelloSequence";
string instanceId = await starter.StartNewAsync(functionName, null);
log.Info($"Started orchestration with ID = '{instanceId}'.");
}
EDIT: If you're using Durable v2.x, then the syntax looks like this:
[FunctionName("ScheduledStart")]
public static async Task RunScheduled(
[TimerTrigger("0 0 * * * *")] TimerInfo timerInfo,
[DurableClient] IDurableClient starter,
ILogger log)
{
string functionName = "E1_HelloSequence";
string instanceId = await starter.StartNewAsync(functionName, null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
For more information on Timer triggers, see this documentation: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer
when I configured different verb like HTTP GET it was responded with NotFound" error in console as well as request made to it with http request from browser responded with "NotFound" error in console .Why this happened?
Because you specified your function to be triggered on POST only:
[HttpTrigger(AuthorizationLevel.Anonymous, methods: "post",
Route = "orchestrators/{functionName}")] HttpRequestMessage req,
To enable GET, add get to methods parameter.
Can I invoke any Orchestration function with in timer trigger azure function?
You can define timer-triggered function with OrchestrationClient input binding similar to your HTTP function. Sample declaration:
public static async Task Run(
[TimerTrigger("0 */1 * * * *")] TimerInfo info,
[OrchestrationClient] DurableOrchestrationClient starter)
The Microsoft Docs provide an example for an "eternal work" orchestrator to orchestrate work that needs to be done periodically but forever: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-eternal-orchestrations?tabs=csharp#periodic-work-example
[FunctionName("EternalWorkOrchestrator")]
public static async Task Run([OrchestrationTrigger] DurableOrchestrationContext context)
{
await context.CallActivityAsync("DoWork", null);
// sleep for one hour between doing work
DateTime nextJobStart = context.CurrentUtcDateTime.AddHours(1);
await context.CreateTimer(nextJobStart, CancellationToken.None);
context.ContinueAsNew(null);
}
More info:
"IDurableClient" is a union of "IDurableEntityClient" and "IDurableOrchastrationClient". It has all the methods of the other two.
Entity Methods:
CleanEntityStorageAsync(Boolean, Boolean, CancellationToken)
ListEntitiesAsync(EntityQuery, CancellationToken)
ReadEntityStateAsync(EntityId, String, String)
SignalEntityAsync(EntityId, DateTime, String, Object, String, String)
SignalEntityAsync(EntityId, String, Object, String, String)
SignalEntityAsync(EntityId, Action)
SignalEntityAsync(EntityId, DateTime, Action)
SignalEntityAsync(String, Action)
SignalEntityAsync(String, DateTime, Action)
Orchestration Methods:
CreateCheckStatusResponse(HttpRequest, String, Boolean)
CreateCheckStatusResponse(HttpRequestMessage, String, Boolean) CreateHttpManagementPayload(String)
GetStatusAsync(String, Boolean, Boolean, Boolean) ListInstancesAsync(OrchestrationStatusQueryCondition, CancellationToken)
PurgeInstanceHistoryAsync(DateTime, Nullable, IEnumerable)
PurgeInstanceHistoryAsync(String)
RaiseEventAsync(String, String, Object)
RaiseEventAsync(String, String, String, Object, String)
RestartAsync(String, Boolean)
StartNewAsync(String, String)
StartNewAsync(String, String, T)
StartNewAsync(String, T)
TerminateAsync(String, String)
WaitForCompletionOrCreateCheckStatusResponseAsync(HttpRequest, String, Nullable, Nullable, Boolean)
WaitForCompletionOrCreateCheckStatusResponseAsync(HttpRequestMessage, String, Nullable, Nullable, Boolean)
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.webjobs.extensions.durabletask?view=azure-dotnet

Resources