Durable function not safe? - azure

Question

I am working in a durable function based as serverless timers, for azure but keep getting the error:
[2022-10-11T03:42:06.874Z] ServerlessTimers.Application: Exception of type 'System.Exception' was thrown.
[2022-10-11T03:42:06.883Z] 0396b0bd-6a87-4490-a2fe-b0b9121a9504: Function 'OrchestrateTimerFunction (Orchestrator)' failed with an error. Reason: System.InvalidOperationException: Multithreaded execution was detected. This can happen if the orchestrator function code awaits on a task that was not created by a DurableOrchestrationContext method. More details can be found in this article https://docs.microsoft.com/en-us/azure/azure-functions/durable-functions-checkpointing-and-replay#orchestrator-code-constraints.
[2022-10-11T03:42:06.886Z] at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableOrchestrationContext.ThrowIfInvalidAccess() in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableOrchestrationContext.cs:line 1163
[2022-10-11T03:42:06.887Z] at Microsoft.Azure.WebJobs.Extensions.DurableTask.TaskOrchestrationShim.InvokeUserCodeAndHandleResults(RegisteredFunctionInfo orchestratorInfo, OrchestrationContext innerContext) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\Listener\TaskOrchestrationShim.cs:line 150. IsReplay: False. State: Failed. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.7.1. SequenceNumber: 4. TaskEventId: -1
From my perspective, I don't see anything wrong in that orchestrator function.
What is funny is that when I set a breakpoint inside that orchestrator function and the function gets called, the error is gone, nowhere to be seen in the logs.
Does that mean it might be an race condition regarding the http-triggerd function that invokes the orchestrator? Seems highly unlikely, but please correct me if I am wrong.
Here the orchestrator function. This "timer" is the same as the one on your phone, but in the cloud.
namespace ServerlessTimers.Application.Functions.Durables;
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Extensions.Logging;
using ServerlessTimers.Application.Exceptions;
using ServerlessTimers.Application.Models.DurableEvents;
using ServerlessTimers.Application.Models.Durables;
using ServerlessTimers.Application.Services.Durables;
using ServerlessTimers.Domain.Aggregators.Timers;
using ServerlessTimers.Domain.Services;
public class OrchestrateTimerFunction
{
private readonly ILogger logger;
private readonly IDurableFacade durableFacade;
private readonly ITimerRepository timerRepository;
private readonly ITimerCalculatorFactory calculatorFactory;
private readonly CancellationTokenSource cts;
public OrchestrateTimerFunction(
IDurableFacade durableFacade,
ITimerRepository timerRepository,
ITimerCalculatorFactory calculatorFactory,
ILogger<OrchestrateTimerFunction> logger)
{
this.logger = logger;
this.durableFacade = durableFacade;
this.timerRepository = timerRepository;
this.calculatorFactory = calculatorFactory;
cts = new CancellationTokenSource();
}
[FunctionName(nameof(OrchestrateTimerFunction))]
public async Task RunOrchestrator(
[OrchestrationTrigger]
IDurableOrchestrationContext context)
{
try
{
// Get timer
var input = context.GetInput<TimerOrchestratorInput>();
var timer = await timerRepository.FindByIdAsync(input.TimerId) ??
throw new TimerNotFoundException(input.TimerId);
// Do not run orchestration if timer's shouldn't be running
if(!timer.State.EqualRunningState())
{
logger.LogError($"Timer {timer.Id}: " +
$"Tried to be orchestrated but has {timer.State} state");
throw new Exception();
}
// Calculate the completion date of the timer
var calculator = calculatorFactory.GetCalculator(timer);
var remainingTime = calculator.CalculateRemainingTime();
logger.LogInformation($"Timer {timer.Id}: " +
$"To complete in {remainingTime}");
if (remainingTime <= TimeSpan.Zero)
{
logger.LogError($"Timer {timer.Id}: " +
$"Remaining time is negative");
throw new Exception();
}
// Set external events
var timerPausedEventTask = context.WaitForExternalEvent<DurableEvent>(
name: nameof(TimerPausedDurableEvent),
defaultValue: new TimerCompletedDurableEvent(),
timeout: remainingTime,
cancelToken: cts.Token);
var timerStoppedEventTask = context.WaitForExternalEvent<DurableEvent>(
name: nameof(TimerStoppedDurableEvent),
defaultValue: new TimerCompletedDurableEvent(),
timeout: remainingTime,
cancelToken: cts.Token);
// Await timer
var durableEvent = await Task.WhenAny<DurableEvent>(
timerPausedEventTask, timerStoppedEventTask);
cts.Cancel();
// Handle events
if(durableEvent.Result is TimerCompletedDurableEvent)
{
logger.LogInformation($"Timer {timer.Id}: Completed");
}
else if (durableEvent.Result is TimerStoppedDurableEvent)
{
logger.LogInformation($"Timer {timer.Id}: Stopped");
}
else if (durableEvent.Result is TimerPausedDurableEvent pausedEvent)
{
logger.LogInformation($"Timer {timer.Id}: Paused ({pausedEvent.Reason})");
}
}
catch(Exception ex)
{
logger.LogError(ex, ex.Message);
}
}
}

Answer 1

Your code is probably going outside of code constraints required for orchestrators to work. You should move such code to activity functions to get this to work.
In your case, I would say the following lines are the cause because on replays, these might yield different results. Try moving them into activity functions.
1.
var timer = await timerRepository.FindByIdAsync(input.TimerId) ??
throw new TimerNotFoundException(input.TimerId);
2.
var calculator = calculatorFactory.GetCalculator(timer);
3.
var remainingTime = calculator.CalculateRemainingTime();