I have a requirement where I need to point our DEV Azure Data Factory to a Production Azure SQL database and also have the ability to switch the data source back to the Dev database should we need to.
I've been looking at creating parameters against the linked services but unsure of the best approach.
Should I create parameters as follows and choose the relevant parameters depending on the environment I want to pull data from?
DevFullyQualifiedDomainName
ProdFullyQualifiedDomainName
DevDatabaseName
ProdDatabaseName
DevUserName
ProdUserName
Thanks
Any sort of trigger can also have parameters attached to it. Check out the following example, assuming you have a custom event trigger and SQL server as a source:
Create a string parameter for the database name field while establishing a SQL server connected service as a dataset.
Create New parameter in dataset, assign the dataset parameter to that same Linked service parameter, which will be used to store the trigger data.
A custom event trigger has the ability to parse and deliver a custom data payload to your pipeline. You define the pipeline parameters and then populate the values on the Parameters page. To parse the data payload and provide values to the pipeline parameters, use the format #triggerBody().event.data. keyName_.
As per Microsoft Official Documents, which could be referred:
Reference trigger metadata in pipelines
System variables in custom event trigger
When you utilize a pipeline activity in a source, it will request you for a dataset parameter. In this case, utilize dynamic content and choose the parameter containing the trigger data.
I would suggest using Azure Key Vault for that.
Create an Azure Key Vault for each environment (dev, prod, etc.)
Create secrets inside both key vaults with the same name but different values.
For example, for the database server name, create the same secret "database-server" in both dev and prod key vaults but with the correct value representing the connection string of the dev and prod server respectively, in the following format:
integrated security=False;encrypt=True;connection timeout=30;data source=<serverName>.database.windows.net;initial catalog=<databaseName>;user id=<userName>;password=<loginPassword>
In your Azure Data Factory, create a Key Vault linked service pointing to your key vault.
In your Azure Data Factory, create a new Azure SQL Database linked service selecting the Key Vault created in step 1 and the secret created in step 2.
Now you can easily switch between dev and prod by simply adjusting your Key Vault linked service to point to the desired environment.
Have fun ;)
Reference:
https://learn.microsoft.com/en-us/azure/data-factory/store-credentials-in-key-vault
Related
I am working on a pipeline and we have DEV,QA and UAT env, so we are trying to use a parameter in the linked service in order to change the conection to the different DB (based on the environment)
we also have different triggers to run the pipeline based on the environment so my question is, is there a way to add a parameter in the trigger, execute the pipeline and send the linked service to connect to a specific environment?
You can have parameters to any type of triggers. Assuming you have a custom event trigger and SQL server as source, checkout the below example:
While creating SQL server linked service as a dataset, create string parameter for database name field
Create New parameter in dataset
Assign dataset parameter to Linked service parameter, which we will then use to store the data from trigger.
Create or use existing trigger, I am using a custom event trigger for example
A custom event trigger can parse and send a custom data payload to your pipeline. You create the pipeline parameters, and then fill in the values on the Parameters page. Use the format #triggerBody().event.data._keyName_ to parse the data payload and pass values to the pipeline parameters.
For a detailed explanation, see the following articles:
Reference trigger metadata in pipelines
System variables in custom event trigger
existing pipeline parameter.
Inside pipeline activity, specific your activity when used in source it would prompt for dataset parameter. Here use dynamic content and select the parameter holding trigger data.
Finally.. when the pipeline is triggered, trigger metadata is passed to pipeline parameter, which will be used in dataset property to switch between database dynamically in a server. Use multiple parameters similar to my example for use in different triggers and pipeline as per your environment.
I have created a simple SSIS project and in this project, I have a package that will delete a particular file in Downloads folder.
I deployed this project to Azure. And when I am trying to execute this package using Azure Data Factory then the pipeline fails with an empty error (I am attaching the screenshot here).
enter image description here
What I have done to fix this error is:
I have added self-hosted IR to Azure-SSIS IR as the proxy to access the data on-premise.
Set the ConnectByProxy as True.
Converted the project to Project Deployment Model.
Please help me out to fix this error and if you need more details then just leave a comment.
Windows Authentication :
To access data stores such as SQL servers/file shares on-premises or Azure Files, check the Windows authentication check box.
If this check box is selected, fill in the Domain, Username, and Password fields with the values for your package execution credentials. The domain is Azure, the username is storage account name>, and the password is storage account key> to access Azure Files, for example.
Using the secrets stored in your Azure Key Vault
As a substitute, you can leverage secrets from your Azure Key Vault as values. Select the AZURE KEY VAULT check box next to them to do so. Create a new key vault connected service or choose or update an existing one. Then choose your value's secret name and version. You can pick or update an existing key vault or create a new one when creating or editing your key vault connected service. If you haven't previously done so, allow Data Factory managed identity access to your key vault. You may also directly input your secret in the format key vault linked service name>/secret name>/secret version>.
Note : If you are using Windows Authentication, there are four methods to
access data stores with Windows authentication from SSIS packages
running on your Azure-SSIS IR: Access data stores and file shares with
Windows authentication from SSIS packages in Azure | Docs
Make Sure it Falls under one of such methods, else it could potentially fail at the Run Time.
I am trying to develop a simple ADF pipeline that copies data from a delimited file to MySQL database, when such a file is uploaded to a Blob Storage Account. I am using parameters to define the name of the Storage Account, the Container that houses the files and file name (inputStorageAccount, inputContainer, inputFile). The name of the Storage Account is a global parameter and the other two are meant to be provided by the trigger. The Linked Service has also been parameterized.
However, I want to define the mappings for this operation. So, I am trying to 'import schemas' by providing the values for these parameters (I have stored a sample file in the Storage Account). But, I keep getting this error when trying to do so,
What am I doing wrong? How can I get this to work?
I would also like to know why I am not being asked to provide a value for the inputContainer parameter when I try to use 'import schema' at the dataset level,
Where you have to add the values Add dynamic content [Alt+P] :
Just as mentioned here in the below Snip, Go to the + Symbol where you will find a window and need to fill in the parameter name, type and value:
Where we can directly select the parameter according to the options :
Here is another detailed scenario which might help: Using Azure DataFactory Parameterized Linked Service | Docs, then you can reset the schema.
I have created azure pipeline using classic editor and executes test.js file using pipeline. I need to retrieve azure COSMOS key which could be used in the js file.
Tried by installing Cosmos DB Key Retriever extension but it doesnt show ADD option in the pipeline.
How can this be resolved? How cosmos key be fetched within js file?
How can this be resolved? How cosmos key be fetched within js file?
We strongly suggest using a config.js file to set your app's configurations, including the PRIMARY KEY of Azure Cosmos DB. Check related official documents here: #1, #2, #3.
It seems that you want to avoid writing the key directly in code, then you can consider:
1.Copy the primary key from this page in Azure Web portal, and then create a variable group in Azure Devops pipelines to store that value. (Change variable type to secret !)
Also you can choose to host that value using Azure key valut and then link secrets from an Azure key vault in current variable group. (If you don't want to host the value in Varibale group directly.)
2.Link the variable group to your current classic pipeline.
3.Then you can use Replace Token task to insert the value of your Primary key into the config.js or xx.js file. You should run your other tasks after this task so that you can use the key in your js file.
Assuming I have the variable TheKeyIfCosmos to store the primary key.
Then specifying this format in config.js file:
key: "#{TheKeyOfCosmos}#",
After running that task, the content in config.js would be real key: "xxxxxxxxxxxxxxxx",.
After above steps you can test/run your app with primary key.
I have a parameterized connection string in Azure Data Factory Linked Services as below:
Integrated
Security=False;Encrypt=True;Connection Timeout=30;Data
Source=xyz;User ID=admin;Password=password;Initial
Catalog=#{linkedService().LSDBName};
The value for database is passed from a pipeline variable at runtime.
I want to save this connection string to Azure Key Vault, but the issue is that after the value is read from the key vault, the linked service parameter "LSDBName" is not dynamically replaced by the actual value and it tries to connect to "#{linkedService().LSDBName}" as the database name.
Is there any way to secure a dynamically parameterized connection string in key vault? Or a workaround to achieve this?
Thanks!
If you want to store the entire connection string in key vault then you have to pass the connection string in "Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword;" format. Create separate connection string for each database and store it in key vault with different secrets and then create parameterized linked service in ADF, giving these secrets as parameter.
My idea is using Set Variable Activity+Azure Function Activity.
First step is using Set Variable Activity to get the LinkedService Connection String.
Second Step is passing the variable as parameter into Azure Function Activity. Then use AKV sdk to store the connection string value in the azure function inside.
Incidentally, I think your connection string has been parameterized already, security issues have been avoided. You don't have to store it into AKV again because mostly we read private information from AKV, rather than write store information into AKV in ADF. Just my own opinion.