I'm using ADF 2 and trying to grapple with the web activity.
The tasks are.
Copy file from blob storage and put the data in an azure SQL database
Iterate through the data and use a PUT call to a REST API to update the data
Okay so I can get the data in the table no problem. I can also make the call using a Web activity to the the API and put some hard coded data there.
But I've been trying to use a For Each to iterate through the table I have and call the web activity to pass that data to the API
This is where I'm stuck. I'm New to data factory and been through all their standard help information but not getting any where.
Any help is appreciated
I think you need to drive the foreach via a SQL lookup task that populates a data set and then call the activity for each row:
here are some posts to get you started:
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity
replace the copy activity with the web call in the tutorial below:
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-bulk-copy-portal
Related
I am looking for a proper solution architecture for a data transfer scenario from SQL Server to an external API and then from the API back to SQL Server. We are thinking of using Azure technologies.
We have a database hosted on an Azure VM. When the value of the author of the book table changes, we would like to get all the data for that book from related table and transfer it an external API. the quantity of the rows to be transferred (the select-join) is huge so it takes a long time to execute the select-join query, After this data is read it is transformed and then it is sent to an external API (over which we have no control) The transfer of the data to the API could take up to an hour. After the data is written into this API, we read some reports from this API and write these reports back into the original database.
We must repeat this process more than 50 per day.
We are thinking of using Logic app to detect the trigger from SQL Server (as it is hosted in Azure VMs) publish this even to an Azure Data grid and then use Azure Durable functions to handle the Read SQL data-Transform it- and Send to the external API.
Does this make sense? Does anybody have any better ideas?
Thanks in advance
At this moment, Logic App SQL connector can't detect when a particular row changes, it will perform a select (which you'll provide), and then it will check for changes every X interval (you'll specify).
In other words, SQL Database doesn't offer a change feed like CosmosDB where you can subscribe to events and trigger an Azure Function.
Things you can do:
1-Add a Trigger on SQL after insert / update which will insert the new/changed row into a separated table, and then you can use Logic App / Azure Functions to query this table and retrieve data.
2-Migrate to Cosmos DB and use the change feed + Azure Functions
3-Change your code to after insert into SQL Database, also add a message with the Identifier for the row you're about to insert / update, then add it to a Queue, which will be consumed by Azure Function.
I've setup a connection from our data factory in azure to a sharepoint site so I can pull some of the lists on the site into blob storage so I can then process into our warehouse. This all works fine and I can see the data I want. I don't however want to pull all the columns contained in the list I'm after. Looking at the connection I can specify a query, however anything I put in here has no affect on the data that comes back. Is there a way to specify the columns from a list in sharepoint through the copy activity into blob storage?
You need to user select query like below -
$select=Title,Number,OrderDate in the Query text field of the Azure Data Factory Source.
You could use Preview Data button to validate the results. Please refer to the documentation for using Custom OData query options.
I have tried this and it works fine for me (see screenshot below)-
Thanks
Saurabh
I am new to Azure Data Factory and wonder if you can please help me in acheiving with the following scenario:
I want to get data from a REST Endpoint using API. The REST Endpoint is stored in a SQL Database table and therefore I fetch the URL using a Lookup activity
Further on, I am storing the URL value in a variable using a "Set Variables" activity
Post that I am fetching the data from the Endpoint using REST API in a Web activity.
Now , I want to store the output data from the Web activity into a Blob storage. For this, i am using Copy activity , but I am not able to get this working at all. Meaning , I am unable to collect the output from the Web Activity into my Copy activity.
In case any of you have come across this situation then it will be of very good help for me indeed.
Seems you are not using correct output from web activity in copy activity. To check what is the output from web activity follow this GIF and pass those values appropriately to copy activity
Is there a method that gives me the list of files copied in azure data lake storage after a copy activity in azure data factory? I have to copy data from a datasource and after i have to skip files based on a particular condition. Condition must check also file path and name with other data from sql database. any idea?
As of now, there's no function to get the files list after a copy activity. You can however use a get Metadata activity or a Lookup Activity and chain a Filter activity to it to get the list of files based on your condition.
There's a workaround that you can check out here.
"The solution was actually quite simple in this case. I just created another pipeline in Azure Data Factory, which was triggered by a Blob Created event, and the folder and filename passed as parameters to my notebook. Seems to work well, and a minimal amount of configuration or code required. Basic filtering can be done with the event, and the rest is up to the notebook.
For anyone else stumbling across this scenario, details below:
https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger"
I have a ADF copy activity copy rows of data from Azure SQL to Azure Cosmos DB.
I have a need to manipulate the document generated. I wrote the logic for the same inside a Pre Create Database Trigger that gets executed whenever a new document is created.
The trigger is not getting executed.
I was not able understand what the problem is, couldn't find any documentation either. The Cosmos DB client API's to create document needs the trigger to execute to be specified explicitly. Not sure if something similar could be done for ADF copy activity as well. Please help.
I am trying to avoid writing a custom activity (so as to leverage built-in scaling and error handling capabilities).
This seems similar to Azure cosmos db trigger, but the answers are not applicable to this question.