How can I get the value of the following in DataFactory:
Last time the pipeline was triggered
Current starting time of the triggered pipeline
There is no easy way. As far as I know, you cannot do that with just data factory, I'd run an Azure Function to look for that using PowerShell or Python's sdk.
This one is easy, you can get it using:
"#trigger().startTime"
And that will give you the current starting time. Doc here: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-system-variables
Hope this helped!
You could get some of this messages in Data Factory monitor/pipeline run page:
It include: last run time(triggered time) and DURACTION.
But for now, we can not export it.
Hope this helps.
Related
I am trying to create a pipeline where the pipeline needs to trigger only when the file is modified in the FTP Server.
I have used GET METADATA activity to get the lastmodified date and used IF activity to copy the data.
Below is the expression i have used in IF activity
#less(activity('GET_DATA').output.lastModified,formatDateTime(utcnow(),'yyyy-MM-dd HH:mm:ss'))
I would want the lasted updated file to be copied into the Destination.
So Can anyone please suggest on how to model the pipeline for this?
Here is a guide for incremental load. Hope it helps
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-lastmodified-copy-data-tool
Also there is a template for incremental load.
https://learn.microsoft.com/en-us/azure/data-factory/solution-template-copy-new-files-lastmodifieddate
I've created a new onetime pipeline in Azure Data Factory using Copy Data wizard.
The pipeline has 4 activities and it was run just fine - all 4 activities succeeded.
Now I'd like to rerun the pipeline. So I do:
Clone the pipeline.
Change name to [name]_rev2.
Remove start and end properties.
Deploy the cloned pipeline.
Now the status of the new cloned pipeline is Running.
But no activities are executed at all.
What's wrong?
Mmmmm. Where to start!
In short. You can't just clone the pipeline. If its a one time data factory that you've created it won't have a schedule attached and therefore won't have any time slices provisioned that now require execution.
If your unsure how time slices in ADF work a recommend some reading around this concept.
Next, I recommend opening up a Visual Studio 2015 solution and downloading the data factory that did run as a project. Check out the various JSON blocks for scheduling and interval availability in the datasets, activities and the time frame in the pipeline.
Here's a handy guide for scheduling and execution to understand how to control your pipelines.
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-scheduling-and-execution
Once you've made all the required changes (not just the name) publish the project to a new ADF service.
Hope this helps
Is there a way to pick up the latest file to process using Azure Data Factory? The files are landing in a folder structure by year/month/day/[file1]...
In the day folder the files will land every hour and my job will also be scheduled to pick it up every hour.
Thanks in advance.
Currently you could use the Custom-Activity,with the help of C# code you can do this.
Also you can schedule our pipeline to run for every hours basics.Since you are receiving the files in an hourly basics
Hope this Helps!
I am using this - https://msdn.microsoft.com/en-us/pnp_articles/getting-started-with-building-azure-webjobs-for-your-office365-sites approach to create a web job and I am not able to see below screen. Is ‘Run on Schedule’ not an option anymore?
Below is the screen I am getting:
The web job works perfectly on demand but what I really need is to schedule. Any help on this would be highly appreciated.
You can add a settings.job file to your WebJob. This can include a CHRON expression that specifies when the job should run. The appropriate docs are here.
Alternatively, you can use a TimerTriggerAttribute. This is a WebJobs SDK Extension. Docs and examples for doing so are here. Note that if you use TimerTrigger, you'll need to make sure your WebJob is deployed as continuous. The TimerTrigger will wake up and call the designated methods based on the schedule that you provide.
I would like to execute certain rest commands after my ADF Pipeline has finished executing.
Does anyone have any ideas how I can do this?
Thank you!
Have you looked at using a .net custom activity at the end of your pipeline process? The other alternative could be to use PowerShell to look for a completed pipeline and trigger your REST command.
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-use-custom-activities