ADF ARM Template doesn't appear to include the factory itself - azure

I was in the process of configuring DevOps to deploy my Dev ADF to the UAT ADF instance.
I had come across the standard issue of the deploy not deleting out-dated pipelines, and attempted to use "Complete" deployment mode to resolve that.
Whereupon DevOps entirely deleted the UAT ADF instance!
Looking further at the docs, it appears that this is the expected behaviour if the factories are not in the ARM Templates.
And looking at my ARM Template (generated entirely by ADF, and with [AFAIK] entirely standard settings), it confirms that the factory itself is NOT amongst the documented resources to be created.
This seems ... odd.
Am I missing something?
How do I get the factory to be included in the ARM Template?
Or alternatively, how can I use the "Complete" deployment mode without it deleting the target ADF instance?
Note that the reason I don't want to use the "define a separate script to solve this" approach, is that it seems excessively complex when that the "Complete" mode sounds like it should do exactly what I want :) (If it weren't for this one oddity about deleting the factory)

You are correct. I've run into this issue before. To circumnavigate it this I recommend creating a core ARM template that would contain the Data Factory and any necessary linked services solely used by Data Factory. This will ensure the "infrastructure/connections" are deployed when creating a new instance.
If you are following Azure Data Factory CI/CD this would be an additional Azure Resource Group Deployment task before the Pipelines are deployed and reference the ARM template which should be in a separate repository.
Here's a template for Data Factory w/ Log Analytics to get you started. I included Log Analytics as most people don't realize about Log retention until they need it. Plus it's a best practices. Just update the system name as this will create a naming standard of adf-systemName-environment-regionAbrviation. The region abbreviation is dynamic based upon the object and will look up agianst the resource group.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"environment": {
"type": "string",
"metadata": "Name of the environment being deployed to"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
}
},
"variables": {
"systemName": "DataFactoryBaseName",
"regionReference": {
"centralus": "cus",
"eastus": "eus",
"westus": "wus"
},
"dataFactoryName": "[toLower(concat('adf-', variables('systemName'),'-', parameters('environment'),'-',variables('regionDeployment')))]",
"logAnalyticsName": "[toLower(concat('law-', variables('systemName'),'-', parameters('environment'),'-',variables('regionDeployment')))]",
"regionDeployment": "[toLower(variables('regionReference')[parameters('location')])]"
},
"resources": [
{
"name": "[variables('dataFactoryName')]",
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"location": "[parameters('location')]",
"tags": {
"displayName": "Data Factory",
"ProjectName": "[variables('systemName')]",
"Environment":"[parameters('environment')]"
},
"identity": {
"type": "SystemAssigned"
}
},
{
"type": "Microsoft.OperationalInsights/workspaces",
"name": "[variables('logAnalyticsName')]",
"tags": {
"displayName": "Log Analytics",
"ProjectName": "[variables('systemName')]",
"Environment":"[parameters('environment')]"
},
"apiVersion": "2020-03-01-preview",
"location": "[parameters('location')]"
},
{
"type": "microsoft.datafactory/factories/providers/diagnosticsettings",
"name": "[concat(variables('dataFactoryName'),'/Microsoft.Insights/diagnostics')]",
"location": "[parameters('location')]",
"apiVersion": "2017-05-01-preview",
"dependsOn": [
"[resourceID('Microsoft.OperationalInsights/workspaces',variables('logAnalyticsName'))]",
"[resourceID('Microsoft.DataFactory/factories',variables('dataFactoryName'))]"
],
"properties": {
"name": "diagnostics",
"workspaceId": "[resourceID('Microsoft.OperationalInsights/workspaces',variables('logAnalyticsName'))]",
"logAnalyticsDestinationType": "Dedicated",
"logs": [
{
"category": "PipelineRuns",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
},
{
"category": "TriggerRuns",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
},
{
"category": "ActivityRuns",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
}
],
"metrics": [
{
"category": "AllMetrics",
"timeGrain": "PT1M",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
}
]
}
}
]
}

Related

Workflow deployment with enabling Log Analytics in Azure Logic App using Rest API

I am trying to deploy workflows in Azure using workflows Rest API's(https://learn.microsoft.com/en-us/rest/api/logic/workflows ). I want to enable Log Analytics Workspace while deploying this workflow in Logic App. This is straight forward in portal while creating logic app. But I didn't find any document or information to pass log analytics workspace while creating workflow definition(JSON File). Please let me know, if there is a way to pass it
In a comment above you mentioned "If it's possible in ARM Template, then I am fine with it and I will use ARM Template deployment API's to deploy it."
Here's an example, feel free to amend it so that it suits your needs:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logicapp-name": {
"type": "string",
"defaultValue": "testLogicApp",
"metadata": {
"description": "Name of the Logic App"
}
},
"loganalytics-workspace-resourceid": {
"type": "string",
"defaultValue": "/subscriptions/11111111-2222-3333-4444-555555555555/resourcegroups/testResourceGroup/providers/microsoft.operationalinsights/workspaces/testLogAnalyticsWorkspace",
"metadata": {
"description": "Resource ID of the Log Analytics workspace"
}
}
},
"variables": {
"logAnalyticsSettingName": "testDiagnosticSettingName"
},
"resources": [
{
"type": "Microsoft.Logic/workflows/providers/diagnosticSettings",
"name": "[concat(parameters('logicapp-name'),'/Microsoft.Insights/', variables('logAnalyticsSettingName'))]",
"apiVersion": "2017-05-01-preview",
"properties": {
"workspaceId": "[parameters('loganalytics-workspace-resourceid')]",
"logs": [
{
"category": "WorkflowRuntime",
"enabled": true,
"retentionPolicy": {
"days": 0,
"enabled": false
}
}
],
"metrics": [
{
"timeGrain": "PT1M",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
}
]
}
}
]
}

Enable azure diagonstic setting using ARM template for azure data factory,azure sql

When I am enabling the diagonstic setting fromt the azure portal for ADF & Azuresql, in the ARM template I am not able to find anything in ARM with respect to diagonstic setting.Similar way for keyvault and sql I need the ARM template for enabling the diagonstic setting.
I tried from my side for ADF since I new to ARM template I am not able to find the method for enabling the diagonstic setting.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"factoryName": {
"type": "string",
"metadata": {
"description": "The name of the Data Factory"
}
}
},
"resources": [
{
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"name": "[parameters('factoryName')]",
"location": "[resourceGroup().location]",
"identity": {
"type": "SystemAssigned"
},
"properties": {
},
"resources": [
{
"type": "Microsoft.DataFactory/factories/providers/diagnosticSettings",
"apiVersion": "2017-05-01-preview",
"name": "[concat(parameters('factoryName'),'/microsoft.insights/', parameters('settingName'))]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]"
],
"properties": {
"name": "[parameters('DS03')]",
"workspaceId": "[/subscriptions/3xxxxx-xxxxx-x-xxxx--xx/resourceGroups/BDAZxfdfG01]"
}
}
]
}
]
}
The ARM template above is creating the diagnostic settings; however it is not actually configuring the logging of anything. Add the following for all Data Factory metrics after your workspaceID property.
"logAnalyticsDestinationType": "Dedicated",
"logs": [
{
"category": "PipelineRuns",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
},
{
"category": "TriggerRuns",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
},
{
"category": "ActivityRuns",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
}
],
"metrics": [
{
"category": "AllMetrics",
"timeGrain": "PT1M",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
}
]
Besides configuring the diagnostic settings what metrics and diagnostics must be select to send to log analytics. These fields align to those on the diagnostic blade:
The "logAnalyticsDestinationType": "Dedicated" is to ensure the logs go to their own table as opposed to the default AzureDiagnostic table. There is documented limitation in the original table.

Enabling auditing settings on Azure SQL database via ARM Template

I have been working on a template to deploy SQL/XSS injection detection. All is well except for enabling the auditing settings. In the docs I see the following:
{
"name": "default",
"type": "Microsoft.Sql/servers/databases/auditingSettings",
"apiVersion": "2017-03-01-preview",
"properties": {
"state": "string",
"storageEndpoint": "string",
"storageAccountAccessKey": "string",
"retentionDays": "integer",
"auditActionsAndGroups": [
"string"
],
"storageAccountSubscriptionId": "string",
"isStorageSecondaryKeyInUse": boolean
}
}
I believe I've followed this structure. See my full code here or the snippet here:
- apiVersion: 2017-03-01-preview
type: Microsoft.Sql/servers/auditingSettings
name: "[concat(parameters('sqlServerName'), '/auditing-default')]"
dependsOn:
- "[resourceId('Microsoft.Sql/servers', parameters('sqlServerName'))]"
properties:
state: Enabled
storageEndpoint: "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')),
'2018-03-01-preview').PrimaryEndpoints.Blob]"
storageAccountAccessKey: "[listKeys(resourceId('Microsoft.Storage/storageAccounts',
parameters('storageAccountName')), '2018-03-01-preview').keys[0].value]"
retentionDays: 0
storageAccountSubscriptionId: "[subscription().subscriptionId]"
isStorageSecondaryKeyInUse: false'
I am seeing that there is a discrepancy between the servers/databases and just /servers for the type, but I actually borrowed this code from the Azure Quick Starts and the specific file here where the code is the following:
{
"apiVersion": "2017-03-01-preview",
"type": "Microsoft.Sql/servers/auditingSettings",
"name": "[concat(parameters('sqlServerName'), '/', 'default')]",
"properties": {
"state": "Enabled",
"storageEndpoint": "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2018-03-01-preview').PrimaryEndpoints.Blob]",
"storageAccountAccessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2018-03-01-preview').keys[0].value]",
"retentionDays": 0,
"auditActionsAndGroups": null,
"storageAccountSubscriptionId": "[subscription().subscriptionId]",
"isStorageSecondaryKeyInUse": false
}
}
The official docs don't seem to have info on adding the auditingSettings on a server level, but then here the type is directly under server, so I'm a bit lost. I haven't looked into the schema yet, but any help/guidance as to what might be going on here would be much appreciated!
We recently published a template that shows how to deploy an Azure SQL Server with server auditing enabled.
The full example is here: https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.sql/sql-auditing-server-policy-to-blob-storage
As the other answers are returning 404s here's a full list of instructions to get the basics working in ARM for auditing at the SQL Server level. This will therefore audit all databases within the SQL Server.
Firstly, create a parameter for the name of your SQL Server and storage account:
"sqlServerName": {
"type": "string"
},
"auditingStorageAccountName": {
"type": "string"
}
Then in your resources section create a storage account to store your audit records, this example will replicate the audit blobs to the paired region (RA-GRS). It was necessary to add network ACLs explicitly as shown so that Azure can write the audit logs. This example also uses storage account assigned keys but managed identities are also possible:
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2019-06-01",
"name": "[parameters('auditingStorageAccountName')]",
"location": "[resourceGroup().location]",
"sku": {
"name": "Standard_RAGRS",
"tier": "Standard"
},
"kind": "StorageV2",
"properties": {
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [],
"ipRules": [],
"defaultAction": "Allow"
},
"supportsHttpsTrafficOnly": true,
"allowBlobPublicAccess": false,
"encryption": {
"services": {
"blob": {
"keyType": "Account",
"enabled": true
}
},
"keySource": "Microsoft.Storage"
},
"accessTier": "Hot"
}
},
...
Finally add the auditing settings themselves - this example is for a resource added at the root (i.e. directly within "resources": {}), to add it as sub-resource to the SQL Server itself the type needs to be just "auditingSettings". A retention days of zero means audit records will be kept indefinitely. It was necessary to add the subscription ID explicitly otherwise the settings do not appear correctly when viewed in the portal:
{
"type": "Microsoft.Sql/servers/auditingSettings",
"name": "default",
"apiVersion": "2020-11-01-preview",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers/', parameters('sqlServerName'))]",
"[resourceId('Microsoft.Storage/storageAccounts', parameters('auditingStorageAccountName'))]"
],
"properties": {
"retentionDays": 0,
"state": "Enabled",
"storageEndpoint": "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('auditingStorageAccountName'))).primaryEndpoints.blob]",
"storageAccountAccessKey": "[listKeys(parameters('auditingStorageAccountName'), '2019-06-01').keys[0].value]",
"storageAccountSubscriptionId": "[subscription().subscriptionId]"
}
},
...
For those looking for guidance on enabling server level auditing to a Log Analytics workspace, I found this github link

Azure Deployment - Overide parameter with ARM functions

I'm using VSTS to deploy azure resources.
I use task "Azure Resource Group Deployment" to deploy ARM templates.
How can I, for a specific parameter, override the value with a ARM function (concat, listkeys, etc)?
Example: My ARM template has a parameter that is a storage account key and instead of providing the key directly, I want to provide it by passing [listkeys(...)]
You cannot do that, several functions (like listKeys()) are evaluated at runtime only. I don't know what you are trying to achieve, so there are probably ways of doing what you try to achieve.
If you want to hide the keys you can store them in the Key Vault and retrieve at deployment time:
"password": {
"reference": {
"keyVault": {
"id": "[resourceId('kvGroup', 'Microsoft.KeyVault/vaults', 'kvName')]"
},
"secretName": "secret"
}
},
If the storage account isn't created within the same ARM template, I'd use the parameter to supply the name of the storage account and then listkeys() within the ARM template to get at the storage account connection string.
If you're creating the storage account in a previous ARM template deployment in your pipeline you could use output parameters to make the connection string available in the pipeline. Here is an example where xxx represents your company naming prefix:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"environment": {
"type": "string",
"defaultValue": "d",
"metadata": {
"description": "The deployment environment, given by develop (d), testing (t), production (p) or quality assurance (q)"
}
}
},
"variables": {
"busUnit": "vendor_name_here",
//storage account names must be lowercase and are limited to 24 alpha numeric characters
"storage_account_name": "[concat('xxx', parameters('environment'), variables('busUnit'), 'stor')]"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"sku": {
"name": "Standard_LRS", //this is a hard coded SKU
"tier": "Standard" //general purpose versus blob-only
},
"kind": "Storage",
"name": "[variables('storage_account_name')]",
"apiVersion": "2017-06-01",
"location": "[resourceGroup().location]", //add it to the same region/location as the resource group
"properties": {
"encryption": {
"keySource": "Microsoft.Storage",
"services": {
"blob": {
"enabled": true
}
}
},
"networkAcls": {
"bypass": "AzureServices",
"defaultAction": "Allow",
"ipRules": [],
"virtualNetworkRules": []
}
},
"dependsOn": []
}
],
"outputs": {
"storageAccountKey": {
//"description": "This works if the storage account is in the same resource group. It returns the access key for the account",
"type": "securestring",
"value": "[listKeys(variables('storage_account_name'),'2015-05-01-preview').key1]"
},
"storageAccountName": {
//"description": "This is the computed name of the storage account, based on naming conventions in the variables",
"type": "string",
"value": "[variables('storage_account_name')]"
}
}
}

Retrieve Service Bus event hub connection string

I have an existing Service Bus with one queue and event hub deployed using Azure Resource Manager.
Now I am interested to retrieve the primary key and connection string using Azure PowerShell wiithout using the ServiceBus.dll. Is it possible??
As a workaround I have created an ARM template which does not deploy anything but just query the existing resource and retrieve the information I need. The below template retrieves the connection string and primary key of an event hub/queue for a specific service bus namespace
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"serviceBusNamespace": {
"type": "string",
"minLength": 1,
"metadata": {
"description": "The name of the service bus namespace to create."
}
},
"resourceName": {
"type": "string",
"minLength": 1,
"metadata": {
"description": "The name of the resource to be retreived."
}
},
"resourceType": {
"type": "string",
"minLength": 1,
"allowedValues": [
"queues",
"eventhubs"
],
"metadata": {
"description": "The type of the resource"
}
},
"policy": {
"type": "string",
"minLength": 1,
"defaultValue": "ManagePolicy",
"allowedValues": [
"ManagePolicy",
"SendPolicy",
"ListenPolicy"
],
"metadata": {
"description": "The type of the resource"
}
}
},
"variables": {
},
"resources": [ ],
"outputs": {
"connectionString": {
"type": "string",
"value": "[listKeys(resourceId(concat('Microsoft.ServiceBus/namespaces/',parameters('resourceType'),'/authorizationRules'),parameters('serviceBusNamespace'),parameters('resourceName'),parameters('policy')),'2015-08-01').primaryConnectionString]"
},
"primaryKey": {
"type": "string",
"value": "[listKeys(resourceId(concat('Microsoft.ServiceBus/namespaces/',parameters('resourceType'),'/authorizationRules'),parameters('serviceBusNamespace'),parameters('resourceName'),parameters('policy')),'2015-08-01').primaryKey]"
}
}
}
Is it abusing to use ARM template to query for a resource and not actually deploy anything?
EDIT
To capture the output of the ARM template within PowerShell use the below code
$ep = New-AzureRmResourceGroupDeployment -Name "getEventHub" -ResourceGroupName myResourceGroup -Mode Incremental -TemplateFile getEventHub.json -TemplateParameterFile getEventHub.param.json
$RuleConnString = $ep.Outputs.connectionString.value
$RulePrimaryKey = $ep.Outputs.primaryKey.value
Note that the property names connectionString and primaryKey are same as defined in my template file
EDIT 2
If I re-run the ARM template to deploy the event hub second time I get the below error.
I din't find any option other than to use the ARM template to query the details.
I don’t see what’s wrong with what you’re doing. In my view Resource Manager templates in their nature are incremental. So you could author a template to create your existing service bus with the same resources. If the properties are the same then it will leave the existing resources intact and return you the connection string and primary key of the relevant resource.
I have a need to automate the creation of a service bus and queue and separate send/listen shared access policies. You can retrieve the connection string on the service bus itself using PowerShell natively without using the .Net ServiceBus.dll assembly by using Get-AzureSBAuthorizationRule but due to a still current bug this doesn’t work at the queue level.
I tried using the ServiceBus.dll to create the shared access policies but sometimes it would randomly fail but subsequently work if you ran it a second time immediately afterwards. I also tried Resource Manager templates but previously you had to pass in the keys you’d generated yourself. Now I see Microsoft generate those for you but you’re still left trying to get the key in an automated fashion so I like your solution.
One question though, can you capture the Resource Manager template outputs and pass them back to a PowerShell script, do you know?
Cheers
Rob
{ "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": {
"servicebusNamespace": {
"type": "string",
"metadata": {
"description": "The service bus namespace"
}
},
"notificationssmsqueue": {
"type": "string",
"metadata": {
"description": "Notifications SMS queue"
}
} }, "variables": {
"location": "[resourceGroup().location]", }, "resources": [
{
"apiVersion": "2015-08-01",
"name": "[parameters('servicebusNamespace')]",
"type": "Microsoft.ServiceBus/namespaces",
"location": "[variables('location')]",
"properties": {
"messagingSku": 2
},
"resources": [
{
"apiVersion": "2015-08-01",
"name": "[parameters('notificationssmsqueue')]",
"type": "Queues",
"dependsOn": [
"[concat('Microsoft.ServiceBus/namespaces/', parameters('servicebusNamespace'))]"
],
"properties": {
"path": "[parameters('notificationssmsqueue')]"
},
"resources": [
{
"apiVersion": "2015-08-01",
"name": "[concat(parameters('notificationssmsqueue'),'.listen')]",
"type": "AuthorizationRules",
"dependsOn": [
"[parameters('notificationssmsqueue')]"
],
"properties": {
"keyName": "[concat(parameters('notificationssmsqueue'),'.listen')]",
"claimType": "SharedAccessKey",
"claimValue": "None",
"rights": [ "Listen" ],
"revision": -1
}
},
{
"apiVersion": "2015-08-01",
"name": "[concat(parameters('notificationssmsqueue'),'.send')]",
"type": "AuthorizationRules",
"dependsOn": [
"[parameters('notificationssmsqueue')]"
],
"properties": {
"keyName": "[concat(parameters('notificationssmsqueue'),'.send')]",
"claimType": "SharedAccessKey",
"claimValue": "None",
"rights": [ "Send" ],
"revision": -1
}
}
]
}
]
} ], "outputs": {
"connectionString": {
"type": "string",
"value": "[listKeys(resourceId(concat('Microsoft.ServiceBus/namespaces/AuthorizationRules'),parameters('serviceBusNamespace'),'RootManageSharedAccessKey'),'2015-08-01').primaryConnectionString]"
},
"smsSendPrimaryKey": {
"type": "string",
"value": "[listKeys(resourceId(concat('Microsoft.ServiceBus/namespaces/Queues/AuthorizationRules'),parameters('serviceBusNamespace'),parameters('notificationssmsqueue'),concat(parameters('notificationssmsqueue'),'.send')),'2015-08-01').PrimaryKey]"
},
"smsListenPrimaryKey": {
"type": "string",
"value": "[listKeys(resourceId(concat('Microsoft.ServiceBus/namespaces/Queues/AuthorizationRules'),parameters('serviceBusNamespace'),parameters('notificationssmsqueue'),concat(parameters('notificationssmsqueue'),'.listen')),'2015-08-01').PrimaryKey]"
} } }
But I call my templates like this:
New-AzureRMResourceGroupDeployment -ResourceGroupName $ResourceGroupName -TemplateFile "$scripts_folder$SB_create_script" -TemplateParameterObject `
#{ servicebusNamespace = $servicebusNamespace;
notificationssmsqueue = $NotificationSMSqueue }
This is the correct way to get the information you are seeking. The Resource Manager provides a common interface to interact with all the services. It is how the Portal access the services, and each of the language SDKs are just wrappers for similar requests to the one you have created.
I usually use the Python or java SDKs, but I have been told that NodeJS is a very easy way to wrap the Web APIs that ARM calls to construct similar calls like the one you made, if you are looking for a none ARM way to do this.

Resources