Bicep Template ~ Create directories in ADLS - azure

I'm trying to transition from Terraform with the azurerm provider to Bicep and having a tohgh time figuring out how to use Bicep to populate my required ADLS Gen 2 directory structure.
Many of my builds need a new ADLS with lots of containers, then inside each container I need nested directories, sometimes three or four levels deep. In Terraform this is pretty simple using the storage/path resource type: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_data_lake_gen2_path
I cannot see any equivalent using Bicep. What am I missing?
The only possibility that seems remotely feasible to accomplish this is to package a deployment Powershell or CLI script, but that looks like a lot of extra configuration and effort for something this trivial.
I would provide code samples for my bicep, but they would only show the storage and container resources and probably would not be very useful. The container resource works perfectly for creating the root directories - but I cannot see how to get Bicep to create directories/paths within them. Can anyone assist, please?
Update
Does not appear to be possible in Bicep. Happy to be proven wrong.

You will have to work with a deployment script to create directories in ADLS Gen2 file systems. You can include inline Azure CLI scripts in Bicep templates. You can create the directory using the Azure CLI command az storage fs directory create. If we include this command in a Bicep template deploying a ADLS Gen 2 storage account, it would look like this:
param location string = 'westeurope'
param containerName string = 'mycontainer'
param directoryName string = 'mydirectory'
resource storageAccount 'Microsoft.Storage/storageAccounts#2022-09-01' = {
name: uniqueString(resourceGroup().id)
kind: 'StorageV2'
location: location
sku: {
name: 'Standard_LRS'
}
properties: {
isHnsEnabled: true
}
resource container 'blobServices#2022-09-01' = {
name: 'default'
resource container 'containers#2022-09-01' = {
name: containerName
}
}
}
resource createDirectory 'Microsoft.Resources/deploymentScripts#2020-10-01' = {
name: 'createDirectory'
kind: 'AzureCLI'
location: location
properties:{
azCliVersion: '2.42.0'
retentionInterval: 'P1D'
arguments: '\'${storageAccount.name}\' \'${containerName}\' \'${directoryName}\''
scriptContent: 'az storage fs directory create --account-name $1 -f $2 -n $3 --auth-mode key'
environmentVariables: [
{
name: 'AZURE_STORAGE_KEY'
secureValue: storageAccount.listKeys().keys[0].value
}
]
}
}

Related

Error parsing json result from the Azure CLI: launching Azure CLI: exec: "az": executable file not found in %PATH%

I get this error running a "terraform plan". I authenticated terraform to Azure via CLI. I have set up an account subscription. How can I solve this problem?
Error: building AzureRM Client: please ensure you have installed Azure CLI version 2.0.79 or newer. Error parsing JSON result from the Azure CLI: launching Azure CLI: exec: "az": executable file not found in %PATH%.
The error says that az is not found. So for this type of error simple way out is logging-in into Azure like below:
az login
Then the error goes:
And then you can check your subscriptions once like below:
az account list
And following login step i got resolved my problem.
And also check if you are using latest Azure Cli version.
And also try the below command:
az account get-access-token
References taken from:
Terraform apply results in Error populating Client ID from the Azure CLI
azure cli $Path error running in terraform cloud
https://forum.gitlab.com/t/teraform-gitlab-image-no-azure-cli/60534
Terraform cloud needs an Azure access since your plan is running on the cloud.
First, you need to create a service principal for azure
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/SUBSCRIPTION_ID"
See this tutorial: https://developer.hashicorp.com/terraform/tutorials/azure-get-started/azure-build
After service principal is created, you get this in response:
{
"appId": "...", - client_id
"displayName": "...",
"password": "...", - client_secret
"tenant": "..." - tenant_id
}
Then you can provide azure access for terraform using one of these methods:
Add Workspace variables via terraform cloud GUI. They will be treated as environment variables.
ARM_CLIENT_ID="..."
ARM_CLIENT_SECRET="..."
ARM_SUBSCRIPTION_ID="..."
ARM_TENANT_ID="..."
Or include them into your .tf file.
provider "azurerm" {
features {}
subscription_id = '...'
client_id = '...'
client_secret = '...'
tenant_id = '...'
}
Hovever it's not a good idea to sotre sensitive data in config.
That's why you may use method #3:
Declare variables in your .tf file and pass them via command line
provider "azurerm" {
features {}
subscription_id = var.subscription-id
client_id = var.client-id
client_secret = var.secret
tenant_id = var.tenant-id
}
terraform apply -var client-id='...' -var tenant-id='...' -var...
See this answer for details:
https://discuss.hashicorp.com/t/using-the-azure-provider-with-terraform-cloud/18177/2

Create Azure Resource Group using Bicep

I'm trying to create an Azure Resource Group using a .bicep file:
targetScope = 'subscription'
param environment string
param location string = deployment().location
resource resourceGroup 'Microsoft.Resources/resourceGroups#2021-04-01' = {
name: 'snapshot-generator-${environment}-west-eu'
location: location
}
And for deploy I'm using the following command in PowerShell:
New-AzResourceGroupDeployment -TemplateFile resourceGroup.bicep
but the ResourceGroupName is requested and I can't understand why and what I'm doing wrong.
You should use New-AzSubscriptionDeployment instead of New-AzResourceGroupDeployment for a subscription-level deployment. See here for more.

publishing azure function - can you create a template json resource file

I have an Azure function that needs an automated deployment script. We currently have a line in the Powershell script that looks like this:
#create resources defined in JSON -
az deployment group create --resource-group TESTGROUP --template-file resources.json
This is what the resource file has for the function name:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"functionapp_name": {
"defaultValue": "[concat('widgets-',uniqueString(resourceGroup().id))]",
"type": "String"
}
This code works fine but I don't want to use dynamic function names anymore. What I'm trying to accomplish is the following:
if local.settings.json exists, then I want to call the function name by the developer's machine hostname
if there is no local.settings.json file then i know this is for a prod env so I want to use different value.
Is there a simple way to do this?
Try this :
$file = 'path\to\local.settings.json'
$functionAppName = 'nameToGiveYourFunctionAppIfInProd'
#If the file exists, use dev machine hostname.
if (Test-Path -Path $file -PathType Leaf) {
$functionAppName = [System.NET.DNS]::GetHostByName('').HostName
}
#Now run your az deployment script and pass the funcAppName as parameter
#create resources defined in JSON -
az deployment group create --resource-group TESTGROUP --template-file resources.json --parameters functionapp_name=$functionAppName

Arm Template (Bicep): Circular Dependency when merging appsettings (list function)

I'm trying to update the AppSettings of an App Service through a bicep file.
When doing this in my bicep template:
var currentAppSettings = list('Microsoft.Web/sites/appServiceName/config/appsettings', '2018-11-01')
var newAppSettings = {
test: 'testValue'
}
var mergedAppSettings = union(currentAppSettings, newAppSettings)
resource appServiceConfig 'Microsoft.Web/sites/config#2018-11-01' = {
name: 'appServiceName/appSettings'
properties: mergedAppSettings
}
...I get a circular dependency error when deploying the bicep file:
"Deployment template validation failed: 'Circular dependency detected on resource: '/subscriptions/293d7347-b26f-4413-9986-d61717aaff26/resourceGroups/WSAPlayground/providers/Microsoft.Web/sites/playground-fitxp-backend-euw-wa/config/appSettings'. Please see https://aka.ms/arm-template/#resources for usage details.'."
Is there a way to do this without getting the dependency error?
try using modules. Bicep modules are essentially nested deployments. in the top level file (i. e. main) extract the current values and pass them to the submodule (appsettings) as parameter of type object and then execute merge and update.
clue is to deploy the update in a different module than reading current value.
Using modules doesn't seem to help if the deployment both creates the app service / function app and sets the app settings. This causes a circular dependency error.
Instead, the solution I have ended up using is to move the extraction of the current app settings outside of the bicep template, and pass them in as a parameter. Something like this bash script:
existingAppSettings="{}"
functionAppAlreadyDeployed=$(az appservice plan list --query "[?name=='ic-portfolio-service-preprod-app-plan']" | jq length)
if [functionAppAlreadyDeployed -eq 1]
then
existingAppSettings=$(az functionapp config appsettings list --name ic-${{parameters.serviceName}}-${{parameters.environment}} --resource-group ${{parameters.serviceName}}-${{parameters.environment}} | jq -r 'map( { (.name): .value } ) | add')
fi
az deployment group create \
--name "deploymentNameGoesHere" \
--resource-group "resourceGroupNameGoesHere" \
--template-file "$(Pipeline.Workspace)/templatepathgoeshere/main.bicep" \
--parameters "buildNumber=$(Build.BuildNumber)" \
"existingAppSettings=$existingAppSettings" \
--mode Complete
NB - I am using az appservice plan list because appservice doesn't support exists
You can then pass this into the bicep template as an object:
#secure()
param existingAppSettings object
And use as follows:
var appSettings = {
New app settings go here
}
resource functionAppAppsettings 'Microsoft.Web/sites/config#2018-11-01' = {
name: '${functionAppName}/appsettings'
properties: union(existingAppSettings, appSettings)
}

`docker context create aci` `--location` argument seems to have no effect

When using docker compose to deploy a container group to Azure Container Instances, the --location argument specified when creating the docker context for aci, docker context create aci..., appears to have no effect.
I'm following these azure instructions for using docker compose with azure container instances
and this docker documentation showing the additional arguments for the docker context create command
Steps to re-produce
Pre-requisites:
Azure CLI
λ az version
{
"azure-cli": "2.22.1",
"azure-cli-core": "2.22.1",
"azure-cli-telemetry": "1.0.6",
"extensions": {}
}
Docker
λ docker-compose --version
docker-compose version 1.29.0, build 07737305
Docker Compose
λ docker --version
Docker version 20.10.5, build 55c4c88
Steps:
Log into azure via the cli az login
Authenticate docker with azure docker login azure
Create an azure resource group in a specific location az group create -l uksouth -n test-aci-ctx-group
Create a docker context specifying a location different to the resource group location docker context create aci aci-context-eastus --resource-group test-aci-ctx-group --location eastus
Display the information about the new context. docker context inspect aci-context-eastus
Expected result: a context bound to the location specified as --location for the docker context create command, i.e., eastus
Actual result: a context bound to the same location as the azure resource group, i.e., uksouth
λ docker context inspect aci-context-eastus
[
{
"Name": "aci-context-eastus",
"Metadata": {
"Description": "test-aci-ctx-group#uksouth",
"Type": "aci"
},
"Endpoints": {
"aci": {
"Location": "uksouth",
"ResourceGroup": "test-aci-ctx-group",
"SubscriptionID": "xxxxxxxxxxxxxxxxxxx"
},
"docker": {
"SkipTLSVerify": false
}
},
"TLSMaterial": {},
"Storage": {
"MetadataPath": "xxxx",
"TLSPath": "xxxx"
}
}
]
The ultimate manifestation of this issue is that I am unable to create a container group in azure because I'm working in the context of a resource group which I cannot change, but the resource group is in a location that does not support Azure Container Instances.
When I try and issue the docker-compose up command in the aci context that I thought I created in a location that supports azure container instances, I get a location not supported error, listing the location of the resource group and not the location I specified for the docker aci context.
Am I misunderstanding the purpose of the --location parameter?
I know you can mix and match locations between a container instance and a resource group because I am able to create a container instance via azure cli with a location different to that of the resource group that it is linked to.
You misunderstand the steps. The parameter needs to be used like this:
docker context create aci --location "eastus"
So you don't need to create the resource group first. When you do it, then the group is already created with a certain location and you can't change it with the docker-compose command.
You have two choices. One is that create the resource group with the location you want, for example, location "eastus". Or create the resource group with the parameter --location "eastus" without a name.

Resources