Azure Data Factory - How can I trigger Scheduled/OneTime Pipelines? - azure

Question

Background : I have scheduled pipelines running for copying data from source to destination. This is scheduled to run daily at a specific time.
Problem : The input dataset to the pipeline is external and not available at specific time intervals. This means the copy activity will have to wait until the Scheduled Start time mentioned in the Pipeline to kickoff. Considering the volume of data, I don't want to waste my time here.
Requirement : At any given time I have access to the time when my input data set is available. With this in hand, I want to know how to trigger a ADF Pipeline from C# though its scheduled to start only at a specific time.

Answer 1

I ran into this same issue, I needed to run my pipeline only when a local job was completed. To do that I modified the local job to kick off the pipeline as its last step. I have a write up here on how to start an ADF pipeline with C#. Here is the link to the ADF developer reference which might also be helpful. I also have an example here on how to trigger ADF pipelines from Azure Functions, if you are interested. This is using the same code from the first example but I get the benefit of running the whole process in the cloud and the ability to use the azure function scheduler.
Here is the relevant method to modify the pipeline. You would need to change the start and end dates based on when you want the slice to run.
public void StartPipeline(string resourceGroup, string dataFactory, string pipelineName, DateTime slice)
{
var pipeline = inner_client.Pipelines.Get(resourceGroup, dataFactory, pipelineName);
pipeline.Pipeline.Properties.Start = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T00:00:00Z");
pipeline.Pipeline.Properties.End = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T23:59:59Z");
pipeline.Pipeline.Properties.IsPaused = false;
inner_client.Pipelines.CreateOrUpdate(resourceGroup, dataFactory, new PipelineCreateOrUpdateParameters()
{
Pipeline = pipeline.Pipeline
});
}

Answer 2

To trigger ADF you need to have input dataset in 'Ready' state. If it is in ready state you can manually go to Monitoring tab to manually 'Re-Run', if input dataset is not ready then you need to make that dataset ready to manually start ADF.

Answer 3

If you want to trigger the job only once then you can set StartDate and EndDate to be the same time:
pipeline.Pipeline.Properties.Start = DateTime.Parse($"{someDate:yyyy-MM-dd}T00:00:00Z");
pipeline.Pipeline.Properties.End = DateTime.Parse($"{someDate:yyyy-MM-dd}T00:00:00Z");
pipeline.Pipeline.Properties.IsPaused = false;

Answer 4

Here is some example from Microsoft Doc...(link for reference)
(Only applies to V2)
{
"properties": {
"name": "MyTrigger",
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Hour",
"interval": 1,
"startTime": "2017-11-01T09:00:00-08:00",
"endTime": "2017-11-02T22:00:00-08:00"
}
},
"pipelines": [{
"pipelineReference": {
"type": "PipelineReference",
"referenceName": "SQLServerToBlobPipeline"
},
"parameters": {}
},
{
"pipelineReference": {
"type": "PipelineReference",
"referenceName": "SQLServerToAzureSQLPipeline"
},
"parameters": {}
}
]
}
}
Save the code with .JSON file in your dir and deploy using following command...
Set-AzureRmDataFactoryV2Trigger -ResourceGroupName resourceGroupName -DataFactoryName dataFactoryName -Name "ScheduleTriggerName" -DefinitionFile ".\ScheduleTriggerName.json"

Answer 5

Check this out: https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers.
As of today, I believe you can use this:
POST
https://management.azure.com/subscriptions/mySubId/resourceGroups/myResourceGroup/providers/Microsoft.DataFactory/factories/myDataFactory/pipelines/copyPipeline/createRun?api-version=2017-03-01-preview