I have a deployment template which creates a vnet, a subnet and 4 private endpoints to existing resources.
I create the subnet and the endpoints via a couple of modules.
Via the createSubnet.bicep module I return the id of the created subnet as an output.
Later on in the script I create the endpoints, passing subnetId to it as follows:
// Create Infrastructure Private Endpoints
module createInfrastructureEndpoints 'modules/createPrivateEndpoint.bicep' = [for (item, index) in infrastructureItems: {
name: 'infrastructure-pep-${item.Name}'
params: {
deploymentTags: deploymentTags
endpointName: item.name
groupId: item.groupId
location: location
resourceSuffix: resourceSuffix
privateLinkServiceId: item.privateLinkServiceId
subnetId: defaultSubnet.outputs.subnetId
}
}]
However, it seems two of the endpoint creations fail with a ReferencedResourceNotProvisioned error associated with the defaultSubnet. I note they are the first two chronologically, however the last modified date is after that of the subnet deployment.
Sorry I can't yet post images
[](https://i.stack.imgur.com/fhECR.png)
I've checked the generated ARM template and to me it indicates it's waiting for the subnet-default module to finish before it can continue
{
"copy": {
"name": "createInfrastructureEndpoints",
"count": "[length(variables('infrastructureItems'))]"
},
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"name": "[format('infrastructure-pep-{0}', variables('infrastructureItems')[copyIndex()].Name)]",
...
"dependsOn": [
"[resourceId('Microsoft.Resources/deployments', 'subnet-default')]"
]
}
How can I get the deployment to wait for the actual completion of the resources created in the module?
I would have expected the creation of the infrastructure private endpoints to wait until the subnet was created.
I imagine there are some issues when creating the subnet resource outside of the vnet resource. I would advice creating the subnet inside the vnet resource.
Here are few related issues regarding VNET/Subnet creation.
Create VNET without destroying all subnets
resource virtualNetworks with subnets wants to delete the existing subnet
Related
A good example is a subnet declaration within a vnet resource. I can add a subnet to a vnet via this ARM syntax
//vnet declaration with subnet declaration in subnets arrary
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2020-11-01",
"name": "vnetName",
...
"properties": {
"subnets": [
"name": "subnetName",
"properties": {
...
}
],
}
}
I can also add it outside of the vnet declaration like this:
//vnet declaration w/out subnet
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2020-11-01",
"name": "vnetName",
...
"properties": {
...
}
}
//separate subnet declaration as specific type outside of vnet declaration, but dependsOn vnet
{
"type": "Microsoft.Network/virtualNetworks/subnets",
"apiVersion": "2020-11-01",
"name": "subnetName",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks', 'vnetName')]"
],
"properties": {
...
}
}
A quick google on this landed me on this Microsoft documentation page which shows you how to do it by nesting a resources array within the parent resource, not utilizing the subnets array.
I'd like to know what's required b/c if the same subnet needs to be declared both within the vnet declaration as well as outside the vnet declaration as its own type, that adds a lot of verbosity to the ARM template.
Thanks!
TL;DR Both are not required the external declaration is there to help with avoiding circular dependencies and give control on deploy order.
In your example adding the subnet definitions within the VNet declaration will always be okay as you wouldn't create a circular dependency.
However let's say we have a Key Vault and a web app that references secrets in that vault.
Firstly we create the Key Vault with the needed secrets. However with this we also need the access policy setup to allow the app service to reference the secrets.
If we were to try and do this internally then it would be trying to reference an app service that doesn't exist.
To fix this we use an external declaration for the access policy which is dependent on both the Key Vault and App service. This allows both to be created and then access to be granted; when all the Ids have been generated.
This article from Microsoft explains all of this in more detail.
#blockingHD is correct - and there's more nuance to this as well.
In the case of a vnet - a subnet property is always "evaulated" and if you don't provide it, it will be treated as if you supplied an empty array. That has the effect of removing your subnets if they exist. When not empty they are replaced with the contents of the property.
In the case of KeyVault, the accessPolicies property is required, you can't omit it from the template. If you supply it (empty or not) it replaces the accessPolicies on the vault -- the same in both cases since the template is declarative.
What this all means is that you need to be aware of out-of-band changes, like adding an access policy for a managed identity, and whether your [initial] declarations will overwrite them when deployed again.
For both subnets and accessPolicies the child resources are good for "additive" scenarios but you need to be aware of when the parent resource (vault or vnet) will be deployed without those additions as if they are not included, will be lost.
That help?
As per this announcement, the support for NSG flow logs creation using ARM template is now released and hence was trying the quick-start ARM template which can do the same, however it appears that there is a pre-requisite that needs to be there before the template is deployed. i.e. Network watcher resource in the region (the region of the target NSG) needs to be enabled.
This could be highly restrictive for replicating deployments across multiple regions where you are not sure if the network watcher is enabled or not. This might end up in resulting deployment failure all-together.
Is there any way to auto-enable the network watcher resource when the NSG flow log gets deployed be it with Powershell or with ARM template?
Azure Network Watcher enabled by default for subscriptions that contain virtual networks. There is no impact to your resources or associated charge for automatically enabling Network Watcher. This will simplify and improve your network troubleshooting experience.
Recommend you enable Network Watcher in each region. Here’s what the ARM template piece to enable Network Watcher:
{
"type": "Microsoft.Network/networkWatchers",
"name": "[concat('net-watcher-',parameters('environmentCode'),'-',variables('regionFullCode'))]",
"apiVersion": "2018-07-01",
"location": "[parameters('vnetLocation')]",
"tags": {
"environmentCode": "[parameters('environmentCode')]",
"serviceCode": "net",
"deploymentARMTemplateVersion": "[parameters('deploymentARMTemplateVersion')]",
"deploymentARMTemplateParameterFileVersion": "[parameters('deploymentARMTemplateParameterFileVersion')]",
"deploymentDateTime": "[parameters('deploymentDateTime')]"
},
"properties": {},
"dependsOn": []
}
I am attempting to deploy a new Azure Virtual Network Gateway to an existing VNET that includes several subnets. I am configuring this in a test environment first with a dummy subnet. I am using ARM to create a .json template and parameters file, which I am deploying via Jenkins. Currently the template attempts to redeploy the whole VNET when it deploys the Virtual Network Gateway. I do not want it to do this. I want it to deploy the Virtual Network Gateway to the existing VNET. Please see below for how I am coding the VNET in the template.
{
"apiVersion": "2019-04-01",
"type": "Microsoft.Network/virtualNetworks",
"name": "[parameters('virtualNetworkName')]",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[parameters('azureVNetAddressPrefix')]"
]
},
"subnets": [
{
"name": "GatewaySubnet",
"properties": {
"addressPrefix": "[parameters('gatewaySubnetPrefix')]"
}
}
]
}
}
I am getting the following error in Jenkins when deploying this template:
"code": "InUseSubnetCannotBeDeleted",
"message": "Subnet testing-subnet is in use by /subscriptions/****/resourceGroups/networks-dev-rg/providers/Microsoft.Network/networkInterfaces/dev-jmp-d31653/ipConfigurations/ipconfig1 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet."
I've looked at the Microsoft knowledgebase but I've struggled to find an explanation of how I can do this, or whether it's even possible. Ideally, I'd like to avoid listing all of the subnets in the vnet, as this is a template I want to apply to different vnets with different subnets.
Can anyone provide answers or advice? Thanks.
Unfortunately, this does not seem to be supported very well in ARM. This is because a VNET is a resource and a subnet is a property of that resource. When an ARM template is deployed, any resources not mentioned are ignored (in iterative mode, at least).
However, properties of existing resources that are mentioned MUST BE SPECIFIED. This is because Azure tries to implement the resource as specified in the template. If a property is different, it will alter it. If a property is absent, it will REMOVE it.
Potential solutions:
Have multiple templates for each of your vnets. When you make a change, you update the whole vnet. This requires you to track several templates and is not ideal for infrastructure as code, but is a simple solution.
Use a powershell solution instead:
https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-tutorial-create-gateway-powershell. I haven't tried this myself as I've been told to use ARM by my superiors, but it has been suggested on several forums as an alternative.
You could also attempt to use a copyloop as per this guidance, but this has limited utility and I haven't yet verified if you can use a name array rather than a number array:
https://pkm-technology.com/azure-vnet-json/
Update your subnets as part of a separate template. This requires you to also update your master vnet template as well, otherwise your new subnets will be removed if you ever redeploy the master vnet template. Also, you can only add subnets in this way. It doesn't help if you want to do something else, such as deploy a VPN gateway.
The following ARM template will add a subnet to a virtual network with existing subnets and will not disturb the existing subnets.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"virtualNetworkName": {
"type": "string",
"defaultValue": "VNet1"
},
"gatewaySubnetPrefix": {
"type": "string",
"defaultValue": "10.0.2.0/24"
}
},
"variables": {},
"resources": [
{
"apiVersion": "2019-04-01",
"type": "Microsoft.Network/virtualNetworks/subnets",
"name": "[concat(parameters('virtualNetworkName'), '/GatewaySubnet')]",
"location": "[resourceGroup().location]",
"properties": {
"addressPrefix": "[parameters('gatewaySubnetPrefix')]"
}
}
]
}
I'm trying to learn Azure Resource Templates and am trying to understand the workflow behind when to use them and when to use the REST API.
My sense is that creating a Virtual Network and Subnets in Azure is a fairly uncommon occurance, once you get that set up as you want you don't modify it too frequently, you deploy things into that structure.
So with regard to an ARM Template let's say I have a template with resources for VNET and Subnet. To take the example from https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-template-walkthrough#virtual-network-and-subnet I might have:
{
"apiVersion": "2015-06-15",
"type": "Microsoft.Network/virtualNetworks",
"name": "[parameters('vnetName')]",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"10.0.0.0/16"
]
},
"subnets": [
{
"name": "[variables('subnetName')]",
"properties": {
"addressPrefix": "10.0.0.0/24"
}
}
]
}
}
which I deploy to a Resource Group. Let's say I then add a Load Balancer and redeploy the template. In this case the user then gets asked to supply the value for the vnetName parameter again and of course cannot supply the same value so we would end up with another VNET which is not what we want.
So is the workflow that you define your ARM Template (VNET, LBs, Subnets, NICs etc) in one go and then deploy? Then when you want to deploy VMs, Scale Sets etc you use the REST API to deploy then to the Resource Group / VNET Subnet? Or is there a way to incrementally build up an ARM Template and deploy it numerous times in a way that if a VNET already exists (for example) the user is not prompted to supply details for another one?
I've read around and seen incremental mode (default unless complete is specified) but not sure if this is relevant and if it is how to use it.
Many thanks for any help!
Update
OK so I can now use azure group deployment create -f azuredeploy.json -g ARM-Template-Tests -m Incremental and have modified the VNET resource in my template from
{
"apiVersion": "2016-09-01",
"type": "Microsoft.Network/virtualNetworks",
"name": "[variables('virtualNetworkName')]",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('addressPrefix')]"
]
},
"subnets": [
{
"name": "[variables('subnetName')]",
"properties": {
"addressPrefix": "[variables('subnetPrefix')]"
}
}
]
}
},
to
{
"apiVersion": "2015-05-01-preview",
"type": "Microsoft.Network/virtualNetworks",
"name": "[parameters('virtualNetworkName')]",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[parameters('addressPrefix')]"
]
},
"subnets": [
{
"name": "[parameters('subnet1Name')]",
"properties": {
"addressPrefix": "[parameters('subnet1Prefix')]"
}
},
{
"name": "[parameters('gatewaySubnet')]",
"properties": {
"addressPrefix": "[parameters('gatewaySubnetPrefix')]"
}
}
]
}
},
but the subnets don't change. Should they using azure group deployment create -f azuredeploy.json -g ARM-Template-Tests -m Incremental
I am going to piggy back on this Azure documentation. Referencing appropriate section below:
Incremental and complete deployments
When deploying your resources,
you specify that the deployment is either an incremental update or a
complete update. By default, Resource Manager handles deployments as
incremental updates to the resource group.
With incremental deployment, Resource Manager
leaves unchanged resources that exist in the resource group but are not specified in the template
adds resources that are specified in the template but do not exist in the resource group
does not reprovision resources that exist in the resource group in the same condition defined in the template
reprovisions existing resources that have updated settings in the template
With complete deployment, Resource Manager:
deletes resources that exist in the resource group but are not specified in the template
adds resources that are specified in the template but do not exist in the resource group
does not reprovision resources that exist in the resource group in the same condition defined in the template
reprovisions existing resources that have updated settings in the template
To choose Incremental update or Complete update it depends on if you have resources that are in use. If devops requirement is to always have resources in sync with what is defined in the json template then Complete Update mode should be used. The biggest benefit of using templates and source code for deploying resources is to prevent configuration drift and it is beneficial to use Complete Update mode.
As for specifying the parameters if you specify in parameters file then you don't have to specify them again.
A new template can be deployed in incremental mode which would add new resources to the existing resource group. Define only the new resources in the template, existing resources would not be altered.
From powershell use the following cmdlet
New-AzureRmResourceGroupDeployment -ResourceGroupName "YourResourceGroupName" -TemplateFile "path\to\template.json" -Mode Incremental -Force
My rule of thumb is for things that I want to tear down, or for things I want to replicate across Subscriptions, I use ARM templates.
For example we want things in test environments, I just ARM it up, build on the scripts as developers request things ("Hey I need a cache", "Oh by the way I need to start using a Service Bus"), using incremental mode we can just push it out to Dev, then as we migrate up to different environments you just deploy to a different Subscription in Azure, and it should have everything ready to go.
Also, we've started provisioning our own Cloud Load Test agents in a VMSS, a simple ARM template that's called by a build to scale up to x number of machines, then when done, we just trash the Resource Group. It's repeatable and reliable, sure you can script it up, but as TFS has a task to deploy these things (also with schedules)...
One of the beautiful things I've come across is Key Vault, when you ARM it up and poke all the values from your service busses, storage accounts/whatevers, you can simply get the connection strings/keys/whatevers and put them straight into the Key Vault, so you never need to worry about it, and if you want to regenerate anything (say a dev wants to change the name of a cache or whatever, or accidentally posted the key to GitHub), just redeploy (often I'll just trash the whole Resource Group) and it updates the vault for you.
I have successfully deployed a VMSS with an AD server in the same VLAN. However, when I try to deploy a second VMSS it create a new virtual network, placing it away from my existing AD.
How do I deploy the second VMSS into the same virtual network as the first and AD? If I edit a template to have the same virtual network as the existing one, then deploy it, will the deployment be into the existing virtual network or will it attempt to create a new one with the same name and fail or duplicate it?
Alternatively, is there any way to migrate the NIC in a VMSS to another network? I can't find a cmdlet to do it and in the web interface it only shows the virtual network created at deployment in the drop-down and not the original virtual network.
If you deploy a template that has an identical vnet configuration a second time it will essentially no-op. If you change the config slightly on an existing work it will attempt to update that config.
Essentially what you want to do here is author the template such that it references the existing network via the NICs and not define the network itself. Your declaration is that the network "exists" so you do not need to define it only reference it. This is a snippet of the JSON:
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "vmss-nic",
"properties": {
"primary": true,
"ipConfigurations": [
{
"name": "vmss-ipconfig",
"properties": {
"subnet": {
"id": "[concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
}
The "id" property is the resourceId of the subnet for the vnet you want to place the VMSS in. If the vnet is in the same resource group as the VMSS you can shorten that expression to simply:
Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName')
But the fully qualified id will work in all scenarios.
re: moving the VMSS, you can always try by deploying a template configured as above. Same VMSS name in the same RG but with a different network profile.
HTH