Data factory SQL connection String with keyvault - azure

I exported arm template from Datafactory V2, when Importing the template it is asking me to manually enter SQL database connection string. To minimize the human interaction I made the following changes.
{
"name": "[concat(parameters('factoryName'), '/myFactory')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": "[concat('Server=tcp:',parameters('sqlServerName'),'.database.windows.net,1433;Initial Catalog=', parameters('sqlDatabaseName'), ';Persist Security Info=False;User ID=',parameters('sqlServerUserName'),';Password=(password)',';MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30')]",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "sql-password"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureKeyVault1')]"
]
},
So currently when deploying to Datafactory V2 and test connection to this SQL server, I got
Cannot connect to SQL Database: 'tcp:mysqlserver.database.windows.net,1433',
Database: 'mydatabase', User: 'admin'. Check the linked service configuration
is correct, and make sure the SQL Database firewall allows the integration runtime to access.
Login failed for user 'admin'., SqlErrorNumber=18456,
If I manually input all the connections in the portal UI, I can easily connect to the database and test successfully so it is not a firewall issue.
Then I think there could be 2 issue:
1.how the password from keyvault is consumed in the connectionstring. I didn't find much information about it online.
When I open the created Sql Linked service, I notice the Fully qualified domain name is missing,If i manually add it in then the connection works.
The SQL connection UI

Throwing this as an alternative answer/approach.
Store the connection string in it's entirety in Key Vault. If doing this then the reference would look like:
{
"name": "[concat(parameters('factoryName'), '/',parameters('connectionNameAdventureWorks'))]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"annotations": [],
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "[variables('azkDataAnalyticsReferenceName')]",
"type": "LinkedServiceReference"
},
"secretName": "[variables('azkAdventureWorksSecretName')]"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/',variables('azkDataAnalyticsReferenceName'))]"
]
}
And even more secure approach would be to add Data Factory as a Managed Identity and then run a sql script to add the user If doing this then all there is no need for any credentials to be passed at all.
One downside is if the DataFactory is deleted and recreated then the managed identity permissions would need to be reassigned to the sql database.

Related

Failed to edit Key Vault API connection

I am using Azure Key Vault connector in the logic apps and for deploying the logic apps using ARM templates.
In the ARM templates I have added the Microsoft.Web/connections resource to include Key Vault API connection.
The API Connection gets successfully deployed, but when I open it on the portal to Authorize it I get an error :- "Failed to edit Api connection "keyvault"".
The resource template of the key vault looks like below :-
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_keyvault_name')]",
"location": "[parameters('location')]",
"properties": {
"api": {
"id": "[concat('/subscriptions/',subscription().subscriptionId,'/providers/Microsoft.Web/locations/', parameters('location'), '/managedApis/', parameters('connections_keyvault_name'))]"
},
"displayName": "",
"customParameterValues": {}
}
}
The status of the API connection after deployment always shows "Error". However, I am using office365 API connection as well which works fine after deployment i.e. when I authorize it, it allows me to save it.
Can anyone please help me with this issue? This is blocking us to move this logic app to production.
Thanks,
Archana Kolte
You have to add the key vault name to connect to by setting the vaultName property under properties.paramaterValues.
Also, the end of the properties.api.id is the same for all Key Vault API Connections and should be /managedApis/keyvault.
Your ARM template would look something like this
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_keyvault_name')]",
"location": "[parameters('location')]",
"properties": {
"api": {
"id": "[concat('/subscriptions/',subscription().subscriptionId,'/providers/Microsoft.Web/locations/', parameters('location'), '/managedApis/keyvault')]"
},
"displayName": "",
"parameterValues": {
"vaultName": "<name-of-your-key-vault>"
}
}
}

Azure Data Factory - Dynamic Account information - Parameterization of Connection

The documentation demonstrates how to create a parameter for a connected service but not how to actual pass in that parameter from a dataset or activity. Basically the connection string is coming from a lookup foreach loop and I want to connect to a storage table.
The connection looks like this. The test works when passing in a correct parameter:
{
"name": "StatsStorage",
"properties": {
"type": "AzureTableStorage",
"parameters": {
"connectionString": {
"type": "String"
}
},
"annotations": [],
"typeProperties": {
"connectionString": "#{linkedService().connectionString}"
}
}
}
The dataset is the following, I'm struggling to determine how to set the connectionString parameter for the connection. The dataset has two parameters, the connectionstring from the db and the tablename that it needs to connect to:
{
"name": "TestTable",
"properties": {
"linkedServiceName": {
"referenceName": "StatsStorage",
"type": "LinkedServiceReference"
},
"parameters": {
"ConnectionString": {
"type": "string"
},
"TableName": {
"type": "string"
}
},
"annotations": [],
"type": "AzureTable",
"schema": [],
"typeProperties": {
"tableName": {
"value": "#dataset().TableName",
"type": "Expression"
}
}
}
}
How do I set the connection string on the connection?
First, you can't make the whole connection string as an expression. You need provide accountName and accountKey sperately. Refer this post about how to do it. How to provide connection string dynamically for azure table storage/blob storage in Azure data factory Linked service
Then, if you are using ADF UI, it will guide you how to provide value for linked service. For example, if you have two dataset parameters, you could specify it as following.
If you want to see json code, you could click the code icon on the top left corner.
I am using azure blob as an example, but the azure table is almost the same.
Hope it could help.

ARM template Office 365 connection for logic apps

I have a logic app that I am trying to automate through an ARM Template.
The logic app requires a connection to Office 365. Below I have the template for the connection generated from the automation pane of the Azure Portal.
When I run the script it fails - there is an authentication issue between the Azure subscription and the Office 365 subscription.
LinkedAuthorizationFailed
The client has permission to perform action 'Microsoft.Web/locations/managedApis/join/action' on scope ... however the current tenant 'curr-tenant-guid' is not authorized
to access linked subscription 'linked-sub-guid' ...
I wont be able to create this trust to automate the provisioning, but I would like to create the connection as placeholder so that the logic app can be deployed and I can go back to the portal to authorise the connection. Is this possible? Are there any other alternatives?
{
"comments": "Office 365 user for file monitoring",
"type": "Microsoft.Web/connections",
"name": "MyOffice365User",
"apiVersion": "2016-06-01",
"location": "northeurope",
"scale": null,
"properties": {
"displayName": "readuser#example.com",
"customParameterValues": {},
"api": {
"id": "[concat('/subscriptions/a6720ff8-f7cb-4bc8-a542-e7868767686/providers/Microsoft.Web/locations/northeurope/managedApis/', 'MyOffice365User')]"
}
},
"dependsOn": []
}
I found three post related to the same problem:
Deploying a logic app using ARM templates/powershell
Azure Logic Apps - ARM template to deploy filesystem API connection
How to set the connection string for a Service Bus Logic App action in an ARM template?
The problem is the same for all API Connection.
Connection parameters to access the specific service are stored on Azure and when you try to export the ARM Template there is nothing regarding these specific parameters (which make sens as Azure will not expose your secret, password...).
The trick is to query Azure Resource Management API to return the parameters needed for any connection in a Logic App.
Just follow the instructions on this article:
Deploying in the Logic Apps Preview Refresh
Use the below link to install logic app tool which can help to design workflow
https://marketplace.visualstudio.com/items?itemName=VinaySinghMSFT.AzureLogicAppsToolsforVisualStudio-18551
once done you can create office 365 connector and when you open ARM template you can see o365 component in the ARM template.
My template looks like :-
{
"type": "MICROSOFT.WEB/CONNECTIONS",
"apiVersion": "2016-06-01",
"name": "[parameters('office365_1_Connection_Name')]",
"location": "[parameters('location')]",
"properties": {
"api": {
"id": "[concat(subscription().id, '/providers/Microsoft.Web/locations/', parameters('location'), '/managedApis/', 'office365')]"
},
"displayName": "[parameters('office36`enter code here`5_1_Connection_DisplayName')]"
}
}
and inside workflow
"triggers": {
"When_a_new_event_is_created_(V2)": {
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['office365']['connectionId']"
}
},
"method": "get",
"path": "/datasets/calendars/v2/tables/#{encodeURIComponent(encodeURIComponent('AAMkNbPwESLK3F8s5n1Q3BwAhXXXXXXXXXXXXXXXXXXXXXXXXXXX'))}/onnewitems"
},
"recurrence": {
"frequency": "Minute",
"interval": 1
},
"splitOn": "#triggerBody()?['value']"
}
}
================================================================
And the parameters for workflow
"parameters": {
"$connections": {
"value": {
"office365": {
"id": "[concat(subscription().id, '/providers/Microsoft.Web/locations/', parameters('location'), '/managedApis/', 'office365')]",
"connectionId": "[resourceId('Microsoft.Web/connections', parameters('office365_1_Connection_Name'))]",
"connectionName": "[parameters('office365_1_Connection_Name')]"
}
================================================================

What are the properties in an ARM template for a Dynamics 365 CRM logic app connector?

Logic app connectors are closed source and the 'Automation Script' option in the Azure portal strips the authentication portions of the properties node from connectors. This is what the portal hands you when you script out the ARM template for a logic app which talks to CRM.
{
"comments": "Generalized from resource: '/subscriptions/<guid>/resourceGroups/<resource group name>/providers/Microsoft.Web/connections/dynamicsCRMconnector'.",
"type": "Microsoft.Web/connections",
"name": "[parameters('connections_dynamicsCRMconnector_name')]",
"apiVersion": "2016-06-01",
"location": "eastus",
"scale": null,
"properties": {
"displayName": "CRMConnection",
"customParameterValues": {},
"api": {
"id": "/subscriptions/<guid>/providers/Microsoft.Web/locations/eastus/managedApis/dynamicscrmonline"
}
},
"dependsOn": []
}
The other connectors (SFTP, storage account, etc.) have the missing elements node documented here and there (nothing official from MS, but blog posts and sample code) but I can't find the information for the Dynamics connectors. As an example of what I would expect to see, here is how SFTP and storage accounts can be pre-configured with authentication values in ARM:
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[variables('sftp_conn_friendly_name')]",
"location": "[resourceGroup().location]",
"properties": {
"displayName": "SFTP connection",
"parameterValues": {
"hostName": "[variables('sftp_host')]",
"userName": "[variables('sftp_user')]",
"password": "[variables('sftp_pass')]",
"portNumber": "[variables('sftp_port')]",
"giveUpSecurityAndAcceptAnySshHostKey": true,
"disableUploadFilesResumeCapability": false
},
"api": {
"id": "[variables('sftp_conn_managed_id')]"
}
}
},
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[variables('storage_conn_friendly_name')]",
"location": "[resourceGroup().location]",
"properties": {
"displayName": "Blob connection",
"parameterValues": {
"accountName": "[variables('storage_account_name')]",
"accessKey": "[listKeys(variables('storage_account_name'),'2015-05-01-preview').key1]"
},
"api": {
"id": "[variables('storage_conn_managed_id')]"
}
}
}
While not a direct answer to your question, but a more general answer giving you idea how to act in such a situation. If its not documented anywhere your only hope is reversing it (and more often than not it works).
First of all, this connecter is a resource in Azure (like the ones you've written). You can use any of the available ways to get the resource properties (https://resource.azure.com, Get-AzureRmResource, REST API, various SDKs) and see what the values are like there.
Another way of going about this - creating this connector using the portal and capturing traffic with fiddler. That way you will see the exact REST call needed to créate such a connector and would be able to replicate it using ARM Template. You might not know that ARM Templates are basically proxies for REST calls. Each resource you are creating is being converted to a REST call and performed against the appropriate resource provider.

How to set the connection string for a Service Bus Logic App action in an ARM template?

I'm attempting to deploy an Azure Logic App that includes an action to Send a message on a Service Bus using an ARM template.
In addition to deploying the Logic App, the ARM template deploys a Service Bus Namespace, a Queue and two AuthorizationRule (one for sending and one for listening).
I want to dynamically set the connection information for the Send Service Bus Message action to use the Connection string generated for the AuthorizationRule that supports sending.
When I create this in the portal editor (specifying the connection string for sending), I noticed the following is generated in code view...
"Send_message.": {
"conditions": [
{
"dependsOn": "<previous action>"
}
],
"inputs": {
"body": {
"ContentData": "#{encodeBase64(triggerBody())}"
},
"host": {
"api": {
"runtimeUrl": "https://logic-apis-westus.azure-apim.net/apim/servicebus"
},
"connection": {
"name": "#parameters('$connections')['servicebus']['connectionId']"
}
},
"method": "post",
"path": "/#{encodeURIComponent(string('<queuename>'))}/messages"
},
"type": "apiconnection"
}
},
I assume that the connection information is somehow buried in #parameters('$connections')['servicebus']['connectionId']"
I then used resources.azure.com to navigate to the logic app to see if I could get more details as to how #parameters('$connections')['servicebus']['connectionId']" is defined.
I found this:
"parameters": {
"$connections": {
"value": {
"servicebus": {
"connectionId": "/subscriptions/<subguid>/resourceGroups/<rgname>/providers/Microsoft.Web/connections/servicebus",
"connectionName": "servicebus",
"id": "/subscriptions/<subguid>/providers/Microsoft.Web/locations/westus/managedApis/servicebus"
}
}
}
}
But I still don't see where the connection string is set.
Where can I set the connection string for the service bus action in an ARM template using something like the following?
[listkeys(variables('sendAuthRuleResourceId'), variables('sbVersion')).primaryConnectionString]
EDIT: Also, I've referred to was seems to be a promising Azure quick start on github (based on the title), but I can't make any sense of it. It appears to use an older schema 2014-12-01-preview, and the "queueconnector" references an Api Gateway. If there is a newer example out there for this scenario, I'd love to see it.
I've recently worked on an ARM Template for the deployment of logic apps and service bus connection. Here is the sample template for configuring service bus connection string within the type "Microsoft.Web/connections". Hope it helps.
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_servicebus_name')]",
"location": "centralus",
"dependsOn": [
"[resourceId('Microsoft.ServiceBus/namespaces/AuthorizationRules', parameters('ServiceBusNamespace'), 'RootManageSharedAccessKey')]"
],
"properties": {
"displayName": "ServiceBusConnection",
"customParameterValues": {},
"api": {
"id": "[concat(subscription().id, '/providers/Microsoft.Web/locations/centralus/managedApis/servicebus')]"
},
"parameterValues": {
"connectionString": "[listKeys(resourceId('Microsoft.ServiceBus/namespaces/authorizationRules', parameters('ServiceBusNamespace'), 'RootManageSharedAccessKey'), '2017-04-01').primaryConnectionString]"
}
}
}
As you know connections is a resource so it needs to be created first did you refer this https://blogs.msdn.microsoft.com/logicapps/2016/02/23/deploying-in-the-logic-apps-preview-refresh/. Quick start link you are referring is for older schema.

Resources