I am working on a pipeline and we have DEV,QA and UAT env, so we are trying to use a parameter in the linked service in order to change the conection to the different DB (based on the environment)
we also have different triggers to run the pipeline based on the environment so my question is, is there a way to add a parameter in the trigger, execute the pipeline and send the linked service to connect to a specific environment?
You can have parameters to any type of triggers. Assuming you have a custom event trigger and SQL server as source, checkout the below example:
While creating SQL server linked service as a dataset, create string parameter for database name field
Create New parameter in dataset
Assign dataset parameter to Linked service parameter, which we will then use to store the data from trigger.
Create or use existing trigger, I am using a custom event trigger for example
A custom event trigger can parse and send a custom data payload to your pipeline. You create the pipeline parameters, and then fill in the values on the Parameters page. Use the format #triggerBody().event.data._keyName_ to parse the data payload and pass values to the pipeline parameters.
For a detailed explanation, see the following articles:
Reference trigger metadata in pipelines
System variables in custom event trigger
existing pipeline parameter.
Inside pipeline activity, specific your activity when used in source it would prompt for dataset parameter. Here use dynamic content and select the parameter holding trigger data.
Finally.. when the pipeline is triggered, trigger metadata is passed to pipeline parameter, which will be used in dataset property to switch between database dynamically in a server. Use multiple parameters similar to my example for use in different triggers and pipeline as per your environment.
Related
If I define a global variable at the management panel level. Is that variable accessible by other concurrently running data factories?
I am looking into using a global as a validation flag as it will be available as a return value from child pipelines. And I do not want a concurrent data factory invocation to have scope into that variable.
I believe you are referring to Global parameters in Azure Data factory. If that is the case, then the answer is No. Global parameters are constants across a particular Data Factory that can be consumed by a pipeline in any expression. You cannot use that global parameter in other data factories except the one in which it was created. But it can be referenced in other pipelines within the same data factory.
Adding to the #KranthiPakala-MSFT, Yes Global parameters are constant, and we cannot use a Global parameter of one ADF in another ADF.
But if you want to use it, store it in a file in blob using copy activity source additional column.
Then use this file value by lookup activity in another ADF workspace.
Or store it in a set variable.
Run this pipeline and in another ADF get this activity run details by using the below REST API in web activity.
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}/queryActivityruns?api-version=2018-06-01
I have a requirement where I need to point our DEV Azure Data Factory to a Production Azure SQL database and also have the ability to switch the data source back to the Dev database should we need to.
I've been looking at creating parameters against the linked services but unsure of the best approach.
Should I create parameters as follows and choose the relevant parameters depending on the environment I want to pull data from?
DevFullyQualifiedDomainName
ProdFullyQualifiedDomainName
DevDatabaseName
ProdDatabaseName
DevUserName
ProdUserName
Thanks
Any sort of trigger can also have parameters attached to it. Check out the following example, assuming you have a custom event trigger and SQL server as a source:
Create a string parameter for the database name field while establishing a SQL server connected service as a dataset.
Create New parameter in dataset, assign the dataset parameter to that same Linked service parameter, which will be used to store the trigger data.
A custom event trigger has the ability to parse and deliver a custom data payload to your pipeline. You define the pipeline parameters and then populate the values on the Parameters page. To parse the data payload and provide values to the pipeline parameters, use the format #triggerBody().event.data. keyName_.
As per Microsoft Official Documents, which could be referred:
Reference trigger metadata in pipelines
System variables in custom event trigger
When you utilize a pipeline activity in a source, it will request you for a dataset parameter. In this case, utilize dynamic content and choose the parameter containing the trigger data.
I would suggest using Azure Key Vault for that.
Create an Azure Key Vault for each environment (dev, prod, etc.)
Create secrets inside both key vaults with the same name but different values.
For example, for the database server name, create the same secret "database-server" in both dev and prod key vaults but with the correct value representing the connection string of the dev and prod server respectively, in the following format:
integrated security=False;encrypt=True;connection timeout=30;data source=<serverName>.database.windows.net;initial catalog=<databaseName>;user id=<userName>;password=<loginPassword>
In your Azure Data Factory, create a Key Vault linked service pointing to your key vault.
In your Azure Data Factory, create a new Azure SQL Database linked service selecting the Key Vault created in step 1 and the secret created in step 2.
Now you can easily switch between dev and prod by simply adjusting your Key Vault linked service to point to the desired environment.
Have fun ;)
Reference:
https://learn.microsoft.com/en-us/azure/data-factory/store-credentials-in-key-vault
I have Azure Data pipeline where I have to pass a parameter to Databricks activity. I have multiple Event based triggers (Updation of blob folder) added for that activity. When specific trigger gets activated, it should pass a parameter to Databricks activity and based on that notebook should run. Is there any way to pass parameter from Event based trigger to Databricks notebook activity?
Trigger gives out 2 parameters.
#triggerBody().fileName
#triggerBody().folderPath
You will have to add this to JSON code of trigger
"parameters": {
"FPath": "#triggerBody().folderPath"
}
Use this parameter as Pipeline variable #triggerBody().FPath and use that variable with other activities. Please refer to link below for detailed explanation
https://www.mssqltips.com/sqlservertip/6063/create-event-based-trigger-in-azure-data-factory/
I have a REST data source where I need pass in multiple parameters to build out a dataset in Azure Data Factory V2.
I have about 500 parameters that I need to pass in so don’t want to pass these individually. I can manually put these in a list (I don’t have to link to another data source to source these). The parameters would be something like [a123, d345, e678]
I'm working in the UI. I cannot figure out how to pass these into the relative URL (where it says Parameter) to then form the dataset. I could do this in Power BI using functions and parameters but can't figure it out in Azure Data Factory as I'm completely new to it. I'm using the Copy Data functionality in ADF to do this.
The sink would be a json file in an Azure blob that I can then access via Power BI. I'm fine with this part.
Relative URL with Parameter requirement
How to add dynamic content
I'm afraid that your requirement can't be implemented.As you know,ADF REST dataset is used to retrieving data from a REST endpoint by using the GET or POST methods with Http request. No way to configure a list of parameters in the relativeUrl property which ADF would loop it automaticlly for you.
Two ways to retrieve your goal:
1.Loop your parameter array ,pass single item into relativeUrl to execute copy activity individually.Using this way,you could use foreach activity in the ADF.
2.Write a overall api to accept list paramter from the requestBody,execute your business in the api inside with loop.
I want to do some steps in my PowerShell based on a value from an Azure ADF(Azure Data Factory) pipeline. How can I pass a value from an ADF pipeline to the PowerShell, where I invoked this ADF Pipeline? So that, I can do the appropriate steps in the PowerShell based on a value I received from ADF pipeline.
NOTE: I am not looking for the run-status of the pipeline (success, failure etc), but I am looking for some variable-value that we get inside a pipeline - say, a flag-value we obtained from a table using a Lookup activity etc.
Any thoughts?
KPK, the requirements you're talking about definitely can be fulfilled though I do not know where does your Powershell scripts run.
You could write your Powershell scripts in HTTP Trigger Azure Function,please refer to this doc. Then you could get the output of the pipeline in Powershell:
https://learn.microsoft.com/en-us/powershell/module/azurerm.datafactoryv2/invoke-azurermdatafactoryv2pipeline?view=azurermps-4.4.1#outputs.
Then pass the value you want to HTTP Trigger Azure Function as parameters.