Add Azure Function (Timer Triggered) to Azure Data Factory Pipeline - azure

I would like to add an Azure Function (Timer triggered) to my Azure Data Factory pipeine. When adding the function to the pipeline, there is a mandatory field to select the HTTP trigger and when I try to debug the function predictably fails as there is no HTTP interface/trigger. Is there a mechanism to add a Timer Triggered Function to the Data Factory pipeline?
I could implement a 2nd version of my function with an HTTP interface but I was hoping to avoid this.
Thx

I couldn't see a way to get the "timer trigger" triggered in the Data Factory pipeline so I just did what I was hoping to avoid.
I added an extra HTTPTrigger and the rest of my function code remained the same. I could then reference that HTTP Function Trigger in my Data Factory pipeline
I could then add a "recurring trigger" from the Data Factory/Data Studio/Manage page that triggered the new HTTP interface/trigger
HTTP TRIGGER
...
FunctionController controller=null;
...
#FunctionName("MyFnHTTPTrigger")
public HttpResponseMessage run(
#HttpTrigger(name = "req", methods = {HttpMethod.GET, HttpMethod.POST}, authLevel = AuthorizationLevel.FUNCTION) HttpRequestMessage<Optional<String>> request,
final ExecutionContext context) {
controller = new FunctionController();
try {
controller.execute(context);
} catch (Exception e) {
context.getLogger().logp(Level.WARNING, "MyFnHTTPTrigger", "run", e.getMessage());
}
TIMER TRIGGER
...
FunctionController controller=null;
...
#FunctionName("MyFnTimer")
public void run(
#TimerTrigger(name = "timerInfo", schedule = "0 */1 * * * *") String timerInfo,
final ExecutionContext context
) {
context.getLogger().info("Java Timer trigger function executed at: " + LocalDateTime.now());
controller = new FunctionController();
try {
controller.execute(context);
} catch (Exception e) {
context.getLogger().logp(Level.WARNING, "MyFnTimer", "run", e.getMessage());
}

Related

How to Pass EventData to Durable Functions?

I want to run durable function from EventHubTrigger azure function.
public async Task<bool> Activities(
[EventHubTrigger(EventHubName, Connection = EventHubConnStrName)] EventData[] events,
[DurableClient] IDurableOrchestrationClient durableClient,
ILogger logger)
{
// chaining pattern durable function
await durableClient.StartNewAsync<string>(
FunctionNames.BatchEvents,
JsonConvertUtil.SerializeObject(events));
}
[FunctionName(FunctionNames.BatchEvents)]
public static async Task<bool> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger logger)
{
try
{
var events =
JsonConvertUtil.DeserializeObject<EventData[]>(context.GetInput<string>()));
....
}
catch (Exception ex)
{
logger.LogError(ex.Message);
throw;
}
}
Let me know how can i pass events to durable function or i can design it better way ?
I believe the way you written the workflow to pass Event data in Azure Durable Functions makes sense after reading this event-based-workflows-with-durable-function.
var events =
JsonConvertUtil.DeserializeObject<EventData[]>(context.GetInput<string>()));
In your code, as you have used context.GetInput<string> Method unpacks the event data you passed and also with deserialization to JSON.
In the above article, there is some information regarding the workflows usage of Event based Triggered Data using in Azure Durable Functions which might help you.

Azure Durable orchestration function ILogger outputs logs twice

There are couple of durable functions that call each other.
Main orchestration -> Sub orchestration -> Activity -> Helper async method
Each func has ILogger dependency and log on function start and on function end.
Both orchestrators duplicates "on start" message for some reason. (See pic)
Activity does not have this effect. (See pic)
Ran example below many times - same story.
I am also sure that the whole process has been triggered once.
Is this a bug in orchestrators or expected behavior?
using System;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
namespace Issues
{
public static class Log_Issue
{
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
try
{
log.LogWarning("Main Start");
await context.CallSubOrchestratorAsync("Sub", null);
log.LogWarning("Main End");
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
[FunctionName("Sub")]
public static async Task RunSubOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log.LogWarning("Sub Start");
var data = await context.CallActivityAsync<string>("Activity", null);
log.LogWarning("Sub End");
}
[FunctionName("Activity")]
public static async Task<string> GetDataActivity([ActivityTrigger] string name, ILogger log)
{
log.LogWarning("Activity Start");
var data = await GetDataAsync("https://www.google.com");
log.LogWarning("Activity End");
return data;
}
[FunctionName("Start")]
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]
HttpRequestMessage req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
var instanceId = await starter.StartNewAsync("Main", null);
log.LogWarning($"Started orchestration with ID = '{instanceId}'.");
return new OkResult();
}
private static async Task<string> GetDataAsync(string url)
{
var httpClient = new HttpClient();
using var request = new HttpRequestMessage
{
RequestUri = new Uri(url),
Method = HttpMethod.Get,
};
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
This is expected. For example on await context.CallActivityAsync("Activity", null); The code pauses itself and may even be loaded out of memory (in order to save on cost).
Then the orchestrator waits for an event to be placed in another Azure Storage Table which the activity creates, this may occur many days later. For activities they are usually very instant but it still waits for this event to occur.
When that happens the code needs to start from where it last stopped but there is no way to do that. Therefor the code reruns from the beginning but instead of waiting for the activity to finish again it first looks in the table and sees that we already have done this activity and can continue running. If the activity function returned some value it would be returned from the await call. During both runs of the orchestrator it would log but since we only go inte the activity ones that would only be logged ones.
This is why orchestrators have to be deterministic since e.g. a random value on the first run would not be the same as during the second run. Instead we would put the random.Next() into an activity funtion so that the value is saved to Azure Table Storage to be used on subsequent reruns. The orchestrator could also be waiting for some external events which normal functions create. E.g someone has to verify their email account which could take some number of days and this is why durable functions can unload themself and restart when they are triggered by the event.
All that #FilipB said is true. It is just missing the actual code to solve it ;)
[FunctionName("Main")]
public static async Task RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context,
ILogger log)
{
log = context.CreateReplaySafeLogger(log); // this is what you should use at the start of every Orchestrator

Azure Durable entity functions to get & set timestamp info using timer trigger

I am trying to store the time stamp information in durable entities and retrieve it every time a trigger fired. Here is how I am doing it. I want the timestamp value set by the current execution to be available for the next trigger. But when the control reaches "string prevTS = await context.CallEntityAsync(entityId, "Get");" to goes back to start of the function again. What am I missing here.
I want execution to be sequential between the timer triggers.
'''
***public static class GetOpenDataRealtimeFeed
{
[FunctionName("GetOpenDataOrchestrator")]
public static async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context, Binder binder, ILogger log)
{
var outputs = new List<string>();
var entityId = new EntityId(nameof(GetPrevLastModifiedTimestamp), "entityKey2");
string prevTS = await context.CallEntityAsync<string>(entityId, "Get");
string currentTS = DateTime.Now.ToString();
outputs.Add(currentTS);
outputs.Add(prevTS);
context.SignalEntity(entityId, "Set", currentTS);
return null;
}
//Durable entity function to get & set the last modified timestamp
[FunctionName("GetPrevLastModifiedTimestamp")]
public static void GetPrevLastModifiedTimestamp([EntityTrigger] IDurableEntityContext ctx)
{
switch (ctx.OperationName.ToLowerInvariant())
{
case "set":
ctx.SetState(ctx.GetInput<string>());
break;
case "get":
ctx.Return(ctx.GetState<string>());
break;
}
}
[FunctionName("getOpenDataRealtimeFeed_Trigger")]
public static async Task Run(
[TimerTrigger("%triggerTimer%")] TimerInfo myTimer,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
// Function input comes from the request content.
string instanceId = await starter.StartNewAsync("GetOpenDataOrchestrator", null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
}
}***
'''
I assume you are referring to the current line while debugging. If so, this is expected.
Since Durable Functions replays functions after awaiting a durable client call, execution won't ever go through the first round. Only the final replay will be "sequential" step overs.

Azure Durable Function Invoke without HttpTrigger (Autostart)

I am looking at this example to run a durable function Activity after a set timeout.
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-eternal-orchestrations
This will allow my function activity to perform processing of data, then wait exactly 1 hour before it attempts to load again. This will continue to run forever. Perfect.
However, when publishing the Function to Azure, I don't want to have to manually invoke/start the function via the associated HTTP Trigger. I just want the durable function to kickoff automatically and start processing.
Is this possible? If not, what is a suggested work around?
Thanks!
As discussed in the comments, one way of doing this would be to add a new Task in your Release pipeline.
Here is what I understood of your setup from your question:
[FunctionName("ClientFunction")]
public static async Task<HttpResponseMessage> OnHttpTriggerAsync([HttpTrigger(AuthorizationLevel.Anonymous, "post")]
HttpRequestMessage request, [OrchestrationClient] DurableOrchestrationClient starter, ILogger logger)
{
// Triggers the orchestrator.
string instanceId = await starter.StartNewAsync("OrchestratorFunction", null);
return new HttpResponseMessage(HttpStatusCode.OK);
}
[FunctionName("OrchestratorFunction")]
public static async Task DoOrchestrationThingsAsync([OrchestrationTrigger] DurableOrchestrationContext context, ILogger logger)
{
DateTime deadline = context.CurrentUtcDateTime.Add(TimeSpan.FromHours(1));
await context.CreateTimer(deadline, CancellationToken.None);
// Triggers some yout activity.
await context.CallActivityAsync("ActivityFunction", null);
}
[FunctionName("ActivityFunction")]
public static Task DoAnAwesomeActivity([ActivityTrigger] DurableActivityContext context)
{
}
Now, every time you deploy a new version of the Function App, you need the orchestrator to be run. However, I do not think it can be started by itself.
What I propose is to have a simple bash script (using curl or something else) that would call the ClientFunction at the appropriate URL.
On top of that, one of the nice things of this solution is that you could make the deployment fail if the Azure Function does not respond.
This seems to be working too.
[FunctionName("AutoStart")]
public static async Task Run([TimerTrigger("*/5 * * * * *", RunOnStartup = true, UseMonitor = false)]TimerInfo myStartTimer,
[DurableClient] IDurableClient orchestrationClient, ILogger log)
{
string instanceId = await orchestrationClient.StartNewAsync("Start_Orchestrator", null);
}
I don't know if there are hidden problems with this, but I'm experimenting now with having a TimerTrigger that runs on startup and also once a day at midnight (or whatever schedule you want). That TimerTrigger will search the list of instances for any running instances of this orchestration, terminate them, then start a new one.
private const string MyOrchestrationName = "MyOrchestration";
[FunctionName("MyOrchestration_Trigger")]
public async Task MyOrchestrationr_Trigger(
[TimerTrigger("0 0 0 * * *", RunOnStartup = true)] TimerInfo timer,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log,
CancellationToken cancellationToken)
{
// Get all the instances currently running that have a status of Pending, Running, ContinuedAsNew
var instances = await starter.ListInstancesAsync(new OrchestrationStatusQueryCondition()
{
ShowInput = false,
RuntimeStatus = new List<OrchestrationRuntimeStatus>() { OrchestrationRuntimeStatus.Suspended, OrchestrationRuntimeStatus.Pending, OrchestrationRuntimeStatus.Running, OrchestrationRuntimeStatus.ContinuedAsNew }
}, cancellationToken);
// Find any instances of the current orchestration that are running.
var myInstances = instances.DurableOrchestrationState.Where(inst => inst.Name == MyOrchestrationName);
List<Task> terminateTasks = new List<Task>();
foreach (var instance in myInstances )
{
// Delete any instances that are currently running.
terminateTasks.Add(starter.TerminateAsync(instance.InstanceId, $"Restarting eternal orchestration"));
}
await Task.WhenAll(terminateTasks);
// Start the new task now that other instances have been terminated.
string instanceId = await starter.StartNewAsync(MyOrchestrationName, null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
}
I think at least for my purposes this will be safe. Any activities that are running when you terminate will still run to completion (which is what I want in my case), so you would just kill it and restart it on a schedule.

CancellationToken doesn't get triggered in the Azure Functions

I've got this simple Azure Function:
public static class MyCounter
{
public static int _timerRound = 0;
public static bool _isFirst = true;
[FunctionName("Counter")]
//[TimeoutAttribute("00:00:05")]
public async static Task Run([TimerTrigger("*/10 * * * * *")]TimerInfo myTimer, TraceWriter log, CancellationToken token)
{
try
{
log.Info($"C# Timer trigger function executed at: {DateTime.UtcNow}");
if (_isFirst)
{
log.Info("Cancellation token registered");
token.Register(async () =>
{
log.Info("Cancellation token requested");
return;
});
_isFirst = false;
}
Interlocked.Increment(ref _timerRound);
for (int i = 0; i < 10; i++)
{
log.Info($"Round: {_timerRound}, Step: {i}, cancel request:{token.IsCancellationRequested}");
await Task.Delay(500, token).ConfigureAwait(false);
}
}
catch (Exception ex)
{
log.Error("hold on, exception!", ex);
}
}
}
What I'm trying to do is capturing the CancellationToken request event when the app stops or a code redeploy happened (host shutdown event).
BTW, I've also tried to check the IsCancellationRequested property in the for loop as well. Never turns true.
The main requirement is not to loose any operation/data during the function deployments, I want to know that the app is being stopped so that I persist some data to be processed once host started again after update.
Based on your code, I tested it on my side, here are my test result:
From the above screenshots, we could find that the subsequent rounds could not handle the cancellation callback except the first round. As Fabio Cavalcante commented, I removed the _isFirst logical checking and found it could work for all rounds as follows:
Note: I simulated the shutdown of my host by disabling my function when the TimerTrigger is triggered.

Resources