I am creating a custom activity pipeline in Azure Data Factory and as a part of that I need to create an AzureBatchLinkedService using the JSON below:
{
"name": "AzureBatchLinkedService",
"properties": {
"hubName": "name_hub",
"type": "AzureBatch",
"typeProperties": {
"accountName": "accountname",
"accessKey": "**********",
"poolName": "?????????",
"batchUri": "https://northeurope.batch.azure.com",
"linkedServiceName": "AzureStorageLinkedService"
}
}
}
When I created this the first time, I didn't have to create a batch pool. It was created for me automatically
My question is how do I get Azure to create this "pool" for me? If I go into the batch account and click "Add+" to add a pool manually, it has all these config options and I don't know what to put for any of those fields.
Related
Trying to deploy ARM Template for a Database Account, SQL Database with two Collections where autoscale throughput setting are set at the database level (shared for collections).
I created this setup in Azure UI and then exported the template.
When importing the template from Powershell using New-AzResourceGroupDeployment it fails with message
Status Message: Entity with the specified id does not exist in the system.
ActivityId: <redacted>, Microsoft.Azure.Documents.Common/2.11.0 (Code:NotFound)
This is ridiculous because I exported the template and did not modify it and then imported. Isn't Azure recognizing it's own format?
I think the problem has to do with this fragment of template:
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/throughputSettings",
"apiVersion": "2020-04-01",
"name": "[concat(parameters('databaseAccounts_an_test_name'), '/', parameters('databaseAccounts_an_test_name'), '-db-2/default')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', parameters('databaseAccounts_an_test_name'), concat(parameters('databaseAccounts_an_test_name'), '-db-2'))]",
"[resourceId('Microsoft.DocumentDB/databaseAccounts', parameters('databaseAccounts_an_test_name'))]"
],
"properties": {
"resource": {
"throughput": 400,
"autoscaleSettings": {
"maxThroughput": 4000
}
}
}
}
Any ideas?
Based on Mark Brown hints this is the exact solution.
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases",
"name": ...
"apiVersion": "2020-04-01"
"dependsOn": ...
"properties": {
"resource": {
"id": ...
},
"options": {
"autoscaleSettings": {
"maxThroughput": 4000
}
}
}
}
Don't use the Microsoft.DocumentDB/databaseAccounts/sqlDatabases/throughputSettings part of yaml from exported template. I'm not sure why Azure exports it and then doesn't allow for import.
If you are creating a new database or container resource you need to pass the throughput in the options for the resource. You can only use the throughput resource directly when updating the throughput.
Here is an example here
I have an ARM template that creates an Azure Key Vault followed by an Azure Kubernetes service. The problem is that the Azure Kubernetes service needs a Service Principle's Client ID and Client Secret passed in the first time I create it. So I run the application.json without the kubernetes_servicePrincipalClientId and kubernetes_servicePrincipalClientSecret parameters in the production.parameters.json file:
application.json
{
"comments": "Kubernetes Service Principal Client ID",
"type": "Microsoft.KeyVault/vaults/secrets",
"name": "[concat(parameters('key_vault_name'), '/KubernetesServicePrincipalClientId')]",
"apiVersion": "2018-02-14",
"properties": {
"contentType": "text/plain",
"value": "[parameters('kubernetes_servicePrincipalClientId')]"
}
},
{
"comments": "Kubernetes Service Principal Client Secret",
"type": "Microsoft.KeyVault/vaults/secrets",
"name": "[concat(parameters('key_vault_name'), '/KubernetesServicePrincipalClientSecret')]",
"apiVersion": "2018-02-14",
"properties": {
"contentType": "text/plain",
"value": "[parameters('kubernetes_servicePrincipalClientSecret')]"
}
}
The second time I run the ARM template, I add the following lines to my production.parameters.json file, so that the Client ID and Client Secret are retrieved from Azure Key Vault where they were stored the first time I ran the ARM template.
production.parameters.json
"kubernetes_servicePrincipalClientId": {
"reference": {
"keyVault": {
"id": "/subscriptions/[Subscription Id]/resourcegroups/[Resource Group Name]/providers/Microsoft.KeyVault/vaults/[Vault Name]"
},
"secretName": "KubernetesServicePrincipalClientId"
}
},
"kubernetes_servicePrincipalClientSecret": {
"reference": {
"keyVault": {
"id": "/subscriptions/[Subscription Id]/resourcegroups/[Resource Group Name]/providers/Microsoft.KeyVault/vaults/[Vault Name]"
},
"secretName": "KubernetesServicePrincipalClientSecret"
}
}
Unfortunately it looks like you can't create a service principal in an ARM template. Is there a better way to configure all this in an automated way, so that regardless of whether or not I'm running the template the first time or second time, I don't have to perform any manual steps?
no, those apis are not exposed to arm, there is no way of managing a service principal with an ARM Template. you can however create a script that will provision the service principal and pass its details to the arm template, or you can use some sort of tool to handle all of this for you (pulumi\terraform\ansible)
The documentation demonstrates how to create a parameter for a connected service but not how to actual pass in that parameter from a dataset or activity. Basically the connection string is coming from a lookup foreach loop and I want to connect to a storage table.
The connection looks like this. The test works when passing in a correct parameter:
{
"name": "StatsStorage",
"properties": {
"type": "AzureTableStorage",
"parameters": {
"connectionString": {
"type": "String"
}
},
"annotations": [],
"typeProperties": {
"connectionString": "#{linkedService().connectionString}"
}
}
}
The dataset is the following, I'm struggling to determine how to set the connectionString parameter for the connection. The dataset has two parameters, the connectionstring from the db and the tablename that it needs to connect to:
{
"name": "TestTable",
"properties": {
"linkedServiceName": {
"referenceName": "StatsStorage",
"type": "LinkedServiceReference"
},
"parameters": {
"ConnectionString": {
"type": "string"
},
"TableName": {
"type": "string"
}
},
"annotations": [],
"type": "AzureTable",
"schema": [],
"typeProperties": {
"tableName": {
"value": "#dataset().TableName",
"type": "Expression"
}
}
}
}
How do I set the connection string on the connection?
First, you can't make the whole connection string as an expression. You need provide accountName and accountKey sperately. Refer this post about how to do it. How to provide connection string dynamically for azure table storage/blob storage in Azure data factory Linked service
Then, if you are using ADF UI, it will guide you how to provide value for linked service. For example, if you have two dataset parameters, you could specify it as following.
If you want to see json code, you could click the code icon on the top left corner.
I am using azure blob as an example, but the azure table is almost the same.
Hope it could help.
I am creating simple pipeline in the data factory that should only run a custom activity. The deployment template for the pipeline looks like this:
{
"type": "pipelines",
"name": "MyCustomActivityPipeline",
"dependsOn": [
"DataFactoryName",
"AzureBatchLinkedService"
],
"apiVersion": "[variables('api-version')]",
"properties": {
"description": "Custom activity sample",
"activities": [
{
"type": "Custom",
"name": "MyCustomActivity",
"linkedServiceName": {
"referenceName": "AzureBatchLinkedService",
"type": "LinkedServiceReference"
},
"typeProperties": {
"command": "cmd /c echo hello world"
}
}
]
}
}
Additionally I have created all the resources needed- the batch account with pools and the storage account. All the resources are in the same resource group and subscription. I try to trigger the pipeline using console command
Invoke-AzureRmDataFactoryV2Pipeline -DataFactory "DataFactory" -PipelineName "PipelineName" -ResourceGroupName "ResourceGroupName"
I am getting this error:
Activity MyCustomActivity failed: Can not access user batch account, please check batch account setiings.
Has anyone ever experienced such an error from ADF execution of a pipeline? The weird part is that all the resources have access to each other and are within the same resource group and subscription.
Please check the settings for the storage linked service used by batch linked service. Make sure the connection string type is SecureString
I'm attempting to deploy an Azure Logic App that includes an action to Send a message on a Service Bus using an ARM template.
In addition to deploying the Logic App, the ARM template deploys a Service Bus Namespace, a Queue and two AuthorizationRule (one for sending and one for listening).
I want to dynamically set the connection information for the Send Service Bus Message action to use the Connection string generated for the AuthorizationRule that supports sending.
When I create this in the portal editor (specifying the connection string for sending), I noticed the following is generated in code view...
"Send_message.": {
"conditions": [
{
"dependsOn": "<previous action>"
}
],
"inputs": {
"body": {
"ContentData": "#{encodeBase64(triggerBody())}"
},
"host": {
"api": {
"runtimeUrl": "https://logic-apis-westus.azure-apim.net/apim/servicebus"
},
"connection": {
"name": "#parameters('$connections')['servicebus']['connectionId']"
}
},
"method": "post",
"path": "/#{encodeURIComponent(string('<queuename>'))}/messages"
},
"type": "apiconnection"
}
},
I assume that the connection information is somehow buried in #parameters('$connections')['servicebus']['connectionId']"
I then used resources.azure.com to navigate to the logic app to see if I could get more details as to how #parameters('$connections')['servicebus']['connectionId']" is defined.
I found this:
"parameters": {
"$connections": {
"value": {
"servicebus": {
"connectionId": "/subscriptions/<subguid>/resourceGroups/<rgname>/providers/Microsoft.Web/connections/servicebus",
"connectionName": "servicebus",
"id": "/subscriptions/<subguid>/providers/Microsoft.Web/locations/westus/managedApis/servicebus"
}
}
}
}
But I still don't see where the connection string is set.
Where can I set the connection string for the service bus action in an ARM template using something like the following?
[listkeys(variables('sendAuthRuleResourceId'), variables('sbVersion')).primaryConnectionString]
EDIT: Also, I've referred to was seems to be a promising Azure quick start on github (based on the title), but I can't make any sense of it. It appears to use an older schema 2014-12-01-preview, and the "queueconnector" references an Api Gateway. If there is a newer example out there for this scenario, I'd love to see it.
I've recently worked on an ARM Template for the deployment of logic apps and service bus connection. Here is the sample template for configuring service bus connection string within the type "Microsoft.Web/connections". Hope it helps.
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[parameters('connections_servicebus_name')]",
"location": "centralus",
"dependsOn": [
"[resourceId('Microsoft.ServiceBus/namespaces/AuthorizationRules', parameters('ServiceBusNamespace'), 'RootManageSharedAccessKey')]"
],
"properties": {
"displayName": "ServiceBusConnection",
"customParameterValues": {},
"api": {
"id": "[concat(subscription().id, '/providers/Microsoft.Web/locations/centralus/managedApis/servicebus')]"
},
"parameterValues": {
"connectionString": "[listKeys(resourceId('Microsoft.ServiceBus/namespaces/authorizationRules', parameters('ServiceBusNamespace'), 'RootManageSharedAccessKey'), '2017-04-01').primaryConnectionString]"
}
}
}
As you know connections is a resource so it needs to be created first did you refer this https://blogs.msdn.microsoft.com/logicapps/2016/02/23/deploying-in-the-logic-apps-preview-refresh/. Quick start link you are referring is for older schema.