How to pass LinkedService parameter value to all dataset in ADF - azure

I have created a parameter in the linked service of the oracle database connection and would like to get the value of the parameter to all datasets, I have created datasets for each table and would like to get oracleSchemeName parameter from the Linked service, In my scenario schema name will change based on environments, SO I passed schema name as parameter but when I'm trying to read the value of linked service, I am getting error table or view doesn't exist.
Can someone please guide me on how to provide expression here? {#linkedService().oracleSchemaName} --Not working.
Below is the screenshot of dataset

{#linkedService().SchemaName} is invalid as schema name is not part of the linked service connection string. You can parameterize the properties which are part of the linked service connection string properties.
Example:
If you want to pass the schema/table name dynamically, you can create a pipeline level parameter and pass the values in run time.
Steps to pass the table schema value dynamically at runtime:
Create a parameter at database level (SchemaName) and do not pass any value to it currently.
Edit the table and pass dataset parameter in add dynamic content.
Create a parameter at pipeline level and add the parameter dynamically in sink properties and do not pass any value to the parameter currently.
While running the pipeline, it will ask to pass value to the pipeline parameter, then pass the value and run the pipeline.
Reference: Parameterize linked services

Related

Is there any way to dynamic source dataset in ADF?

I want to create a dynamic source dataset for copy activity based on parameters or variables.
I tried to edit referenceName on JSON but it's not working.
Creating Linked Service dynamically by passing the parameters.
Parameters were provided while creating the linked service to pass dynamically.
Creating new Dataset with linked service which we created dynamically.
Needed values for the parameters to load the tables while creating the Dataset dynamically.
Result with expected dataset created dynamically.
We can also use dynamic content for tables in the dataset by providing parameters.

Add Parameter Values to Query String of get request in ADF

I have a Copy Data task which is obtaining data from an API
The API is a GET call to a method and requires 2 parameters
_token
Symbols
I have defined these as parameters
What is the syntax that allows me to use the values of my parameters as the values that are in the query string? So in the screenshot above Symbols is hard coded, but I want the value to be the value of the parameters
I need a screen solution rather than code please as I am not comfortable with ADF yet and I dont know how to get to the code/ARM views
Paul
Using a feature called string interpolation where expressions are wrapped in #{ ... }
Click on the Base URL field. Add Parameters. Using Concat expression function,
Example:
#{Concat('https://stackoverflow.com/questions/GetRealTimeRates?',linkedService().Symbols,'=',linkedService()._token)}
Add first parameter:
Add second parameter:
Test connection. If you see any error, it would provide a description as to debug further.

Deploying Azure ARM Templates using Parameter Functions not pulling through correct values

I've been trying to set the value of a parameter, based on the value I set of another parameter. This wasn't working but then I discovered the Parameters with Functions JSON example on the Azure GitHub
This is giving the same behaviour as my own template so is perfect to show the issues I am having.
As you can see from the JSON linked to above, the parameter hostingPlanName should concat the parameter siteName with the string -plan. When I edit the parameter file, I see the function instead of the value it resolves to.
This example pulls other values in to set the value of siteName, I wondered if that was the reasons so hard set siteName to 'TEST' and the result for hostingPlanName was the same.
I haven't deployed this example, but if I deploy my real template, it throws an error with Bad Request for the name of the resource I am deploying.
Is this me or is this not possible anymore?
I am using VS2019 Community.
This is a great place to use variables. Your hostingPlanName is not set directly by user input (as parameters should be), but is a dynamic evaluation of a complex value based on a parameter, so it should be a variable.

Azure Data Factory v2 - having trouble iterating through list of servers in table

I have a pipeline that includes a simple copy task that reads data from an SFTP source and writes to a table within a server. I have successfully parameterized the pipeline to prompt for which server and table I want to use at runtime but I want to specify a list of server/table pairs in a table that is accessed by a lookup task for use as parameters instead of needing to manually enter the server/table each time. For now it's only three combinations of servers and tables but that number should be able to flex as needed.
The issue I'm running into as that when I try to specify the array variable as my parameter in the lookup task within a For Each loop the pipeline fails telling me I need to specify an integer in the value array. I understand what it's telling me but it doesn't seem logical to me that I'd have to specify '0', '1','2' and so on each time.
How do I just let it iterate through the server and table pairs until there aren't any more to process? I'm not sure of the exact syntax but there has to be a way to tell it run the pipeline once with this server and table, again with a different server and table, then again and again until no more pairs are found in the table.
Not sure if it matters but I am on the data flow preview and using ADFv2
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity#iteration-expression-language
I guess you want to access the iterate item, which is item() in adf expression language.
If you append a foreach activity after a look up activity, and put the output of lookup activity in items field in foreach activity, then item() means the iterate item in the lookup output.

Azure Data Factory dynamic output path based on source dataset payload

I have a stream analytics job which constantly dumps data in Cosmos DB. The payload has a property "Type" which determines the payload itself. i.e. which columns are included in the payload. It is an integer value of either 1 or 2.
I'm using Azure Data Factory V2 to copy data from Cosmos DB to Data Lake. I've created a pipeline with an activity that does this job. I'm setting the output path folder name using :
#concat('datafactoryingress/rawdata/',dataset().productFilter,'/',formatDateTime(utcnow(),'yyyy'),'/')
What I want in the datafactory is to identify the payload itself, i.e. determine if the type is 1 or 2 and then determine if the data goes in folder 1 or folder 2. I want to iterate the data from Cosmos DB and determine the message type and segregate based on message Type and set the folder paths dynamically.
Is there a way to do that? Can I check the Cosmos DB document to find out the message type and then how do I set the folder path dynamically based on that?
Is there a way to do that? Can I check the Cosmos DB document to find
out the message type and then how do I set the folder path dynamically
based on that?
Unfortunately, based on the doc, dynamic content from source dataset is not supported by adf so far. You can't grab the fields in the source data as sink output dynamic parameters. Based on your situation, I suggest you setting up two separate pipelines to transfer data according to the Type field respectively.
If the Type field is varied and you do want to differentiate the output path, the ADF may not be the suitable choice for you. You could write logical code to fulfill your needs.

Resources