I have a Copy Data task which is obtaining data from an API
I want to add the Request Method as a parameter to be defined at run time so I want it to be a Drop Down list instead of a String, Can this be done in Azure Data Factory?
Unfortunately, It is not possible to create dropdown list for Request Method.
But you can raise feature request here
Refere - https://learn.microsoft.com/en-us/azure/data-factory/connector-http?tabs=data-factory
Related
I am loading data via pipelines in ADLS gen2 container.
Now I want to create a table that has details that when the pipeline start running and then completed. like below fields
where
startts - start time of job
endts- end time of job
but extractts is the extraction time which is what i want to create.. is there any approch i can create table ?? help will be really appreciated.
It might not be exactly what you need, but you can add columns inside a copy activity with the property #pipeline().TriggerTime to get your "startts" field. Go to the bottom option on the source tab and select new, and on value choose "Add dynamic content".
Here is an example
For your "extractts" you could use the property of the activity called "executionDuration", which will give you the time it took to adf to run the activity in seconds. Again, use dynamic content and get this: #activity('ReadFile').output.executionDuration, then store this value wherever you need.
In this example I'm storing the value in a variable, but you could use it anywhere.
I dont understand your need for a field "endts", I would just simply do startts+extractts to get it.
I would like to retrieve all results from Rest API endpoint.The URL has the below form
https://myapi.com/relativeapi?project=&repo=&prId=&page=&pageSize=&startTime=&endTime
By default when requesting data it is returned only the first page. A sample output is the below
"pageSize":50,
"num":50,
"isLastPage":false,
"data":
{"ABC":{"mock1":[{"Id":18,"Date":"202104T02:04:53.000Z","attr1":0,"attr2":0,"attr3":0,"historyData":[{"Date":"2021-11-03T00:08:13.000Z","attr1":0,"attr2":0,"attr3":0,"attr4":{}}
How can we achieve this in Azure Data Factory and retrieve all results from all pages (last page is till "IsLastPage=TRUE and "data" is empty)?
Also how can we incrementally request API data, so the pipeline does not need to run all results from beginning (page 1), but get results from last updated page
#christi08
Since the next page information is not returned in the output. Unfortunately you will not be able to make utilize of the inbuilt pagination feature.
As alternative/workaround - you could use the below approach.
You could use an iterative approach to achieve your end goal.
STEP 1 :
Your request is going to be in the below format
https://myapi.com/relativeapi?page=1.......
https://myapi.com/relativeapi?page=2.......
https://myapi.com/relativeapi?page=3.......
https://myapi.com/relativeapi?page=n.......
Step 2 :
Create a variable named pageno at the pipeline level.
Step 3:
In the Rest Connector create a Parameter page.
This page parameter would be added as a relative url along with other parameter & path.
In your case, the base url will be different.
Step 4 :
Noe in the copy activity, under the source setting.
You will pass the parameter with the value of the pipeline variable.
This pipeline variable will be incremented.
So for each iteration - the pageno will be incremented and therefore the relative url is also dynamic.
You would need SET VARIABLE activity to increase the pageno pipeline variable.
To loop you could use an Until Activity
End condition
For the until activity to end, you will have to provide an expression.
You could add another web activitity / lookup activity with the dynamic relative url.
You could access the output of the webactivity / lookup activity and access the isLastPage node - Until this is true.
You could access the copyactivity output and see whether the number of rows written is 0. and end the until activity.
I want to call a ZenDesk REST API in Data Factory,
here's how the output looks like
next_page determines the next URL page of the REST API
end_of_stream returns false if I've reached the end of the page
In Data Factory, I've set up a Copy Activity in a pipeline which copies the data from REST JSON into my Azure DB, something like this
The problem is the pagination in ADF doesn't support the type of pagination that Zendesk API provides, from my research it looks like I'll need to create an Until loop in order to make this work, so something like this
I need to know how I can
Set a variable to true/false based on the Copy Activity output from the REST API call
Dynamically set the URL of my Copy Activity, so it can loop through all the pages in the REST API
Thanks!
Set a variable to true/false based on the Copy Activity output from the REST API call
In the value Variables section of the Set Variable Activity, in the variable click on the add dynamic content.
#activity('Copy data1').output.end_of_stream
You can replace the Copy data1 with that of your own Copy Activity Name
Dynamically set the URL of my Copy Activity, so it can loop through all the pages in the REST API
For your REST API dataset,
Configure the below Parameter
Relative URL :
In the Copy Data Activity :
Once the first copy action is done, you should update the variable start_time with the value of the end_time of the output.
Snippet Output of the REST API
The reason being, the next page url is same relative url of the API with start_time parameter with the value of end_time of the REST API (Current Page) Output.
Hello I am trying to create a LogicApp that first:
Extracts data from CosmosDB, using a query
Loops over the results
Pushes the results data into CRM
Sidenote: Once this data is pushed into CRM, CRM automatically generates an ID for each record. How can I then:
My biggest dilemma is figuring out how can I return the newly generated ID back to the original CosmosDB container in which it was pulled from?
I have started on this and these are my questions:
Does this first part look correct in what I am doing? I must use this SQL query for this:
a. I am simply telling the Logic App to extract data from a particular container inside CosmosDB
b. Then I would like to loop over the results I just obtained, and push this to CRM
c. My dilemma is:
Once data is pushed to CRM, CRM then automatically generates an ID for each record that is uploaded. How would I then return the updated ID to cosmos?
Should I create a variable that stores the IDS, then replace the old IDs with the new ones?
I am not sure how to construct/write this logic within LogicApps and have been researching examples of this.
Any help is greatly appreciated.
Thanks
If the call to your CRM system returns the ID you are talking about then I would just add one additional action in your loop in Azure Logic App to update the read record in Azure Cosmos DB. Given that you are doing a SELECT * from the container you should have the whole original document.
Add the 'Create or update document' action as a step with a reference to the THFeature container along with your Database ID and then provide the new values for the document. I pasted an example below.
BTW. Your select query looks strange - you should avoid slow cross partition queries if you can.
I'm trying to set up a simple pipeline in ADF. First step is selecting access tokens and logfilenames from a table on an MSSQL server. This works ok, preview shows a table containing two columns; token and logfilename. After this lookup I have a foreach loop which needs to do http requests to a REST API using the values in the columns token and logfilename. As items, in the foreach block, I set #activity('nameoftheactivity').output. In the foreach loop is a copy block. Source of this copy block is a REST API with its base URL (https://api.domain.com/v2/), relative URL is set as
#concat('logfile/',dataset().ForEachLogfilename,'.',formatDateTime(utcNow(), 'yyyy-MM-dd'),'.log/json?access_token=',dataset().ForEachToken)
The ForEachLogfilename and the ForEachToken are set as Dataset properties with values as
#{item().token} and #{item().logfilename}
When I hit the preview button Azure suggest that I set values for #item().token and #item().logfilename which I do as suggested. A click on finish AND I HAVE DATA from the rest API. But only with the preview data... It just errors when I perform a "trigger now"... Can anyone point me in the right direction?
cheers!
Found it!!
instead of
#concat('logfile/',dataset().ForEachLogfilename,'.',formatDateTime(utcNow(), 'yyyy-MM-dd'),'.log/json?access_token=',dataset().ForEachToken)
I had to use
#concat('logfile/',item().logfilename,'.',formatDateTime(utcNow(), 'yyyy-MM-dd'),'.log/json?access_token=',item().token)
in the "Add dynamic content" field....