VMs lose outbound connectivity after subsequent template deploy - azure

I'm using an ARM template to deploy a virtual network, VPN gateway, and a number of virtual machines (I've tried a standalone VM, and a VMSS). I'm also deploying a PowerShell DSC module to each VM which copies over some code and installs it as a service.
There is a recurring issue where on subsequent deployments, probably half of the time, the deployment script fails because the deployment of the DSC extension fails due to lack of network connectivity and being unable to resolve the hostname of the storage account where the code is being offered.
When I connect to the vnet VPN and remote into the VM in question, there is always no outbound network activity. If I compare the /ipconfig all settings with the other VMs, the settings are identical (except with slightly different local IP). However one VM will be unable to ping any public IPs or resolve any hosts. Just trying to enter an nslookup session fails to connect to the DNS server itself immediately, even though other VMs are using that same DNS server just fine.
Usually just restarting the VM in question fixes the issue.
My vnet setup is pretty basic and I haven't specified my own DNS, so I'm just using "Azure DNS".
Currently my VMs in the template are configured as dependent on the virtual network. I'm not sure if I'm also supposed to make them dependent on the gateway as well.
Here's the config I'm using on the VMs:
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "[concat(variables('scalesetName'), '-nic')]",
"properties": {
"primary": "true",
"ipConfigurations": [
{
"name": "[concat(variables('scalesetName'), '-ipconfig')]",
"properties": {
"subnet": {
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
}
}
}
]
}
}
]
},
And the configuration of the virtual network:
{
"type": "Microsoft.Network/virtualNetworks",
"name": "[variables('virtualNetworkName')]",
"location": "[resourceGroup().location]",
"apiVersion": "[variables('networkApiVersion')]",
"tags": {
"displayName": "VirtualNetwork"
},
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('addressPrefix')]"
]
},
"subnets": [
{
"name": "[variables('subnetName')]",
"properties": {
"addressPrefix": "[variables('subnetPrefix')]"
}
},
{
"name": "GatewaySubnet",
"properties": {
"addressPrefix": "[variables('gatewaySubnetPrefix')]"
}
}
]
}
},
{
"type": "Microsoft.Network/virtualNetworkGateways",
"name": "[variables('gatewayName')]",
"apiVersion": "[variables('networkApiVersion')]",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "VpnGateway"
},
"dependsOn": [
"[concat('Microsoft.Network/publicIPAddresses/', variables('gatewayPublicIPName'))]",
"[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
],
"properties": {
"gatewayType": "Vpn",
"vpnType": "RouteBased",
"enableBgp": "false",
"sku": {
"name": "[variables('gatewaySku')]",
"tier": "[variables('gatewaySku')]"
},
"ipConfigurations": [
{
"name": "vnetGatewayConfig",
"properties": {
"privateIPAllocationMethod": "Dynamic",
"subnet": {
"id": "[concat(resourceId('Microsoft.Network/virtualNetworks', variables('virtualNetworkName')),'/subnets/', 'GatewaySubnet')]"
},
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses', variables('gatewayPublicIPName'))]"
}
}
}
],
"vpnClientConfiguration": {
"vpnClientAddressPool": {
"addressPrefixes": [
"[variables('vpnClientAddressPoolPrefix')]"
]
},
"vpnClientRootCertificates": [
{
"name": "RootCertificate",
"properties": {
"PublicCertData": "<snip>"
}
}
]
}
}
},
Any ideas why deploying is randomly breaking outbound VM traffic until I restart the VM?

Please note that azure blocks 'Ping' to/from the azure network.
Please answer the following:
a. Do you have any UDR rules on your subnet that force route your traffic to VPN gateway?
b. What is the storage account configuration for the VM. Is it using premium storage or standard storage.

Related

ARM Template: Firewall rule is not being created for database server

I'm deploying an Azure database server with a ARM template which includes a firewall rule to allow all IP addresses. The deployment finishes without errors, however, the firewall rule doesn't seem to be created when I check the Portal.
"resources": {
"name": "mydbserver",
"type": "Microsoft.Sql/servers",
"apiVersion": "2014-04-01",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "mydbserver"
},
"properties": {
"administratorLogin": "[parameters('dbserverUsername')]",
"administratorLoginPassword": "[parameters('dbserverPassword')]"
},
"resources": [
{
"type": "firewallRules",
"apiVersion": "2014-04-01",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers', 'mydbserver')]"
],
"location": "[resourceGroup().location]",
"name": "AllowAllWindowsAzureIps",
"properties": {
"startIpAddress": "0.0.0.0",
"endIpAddress": "0.0.0.0"
}
}
]
},
Azure portal
Adding this rule (AllowAllWindowsAzureIps) manually in the portal resulted in an error stating "This rule name is already in use for 'Allow Azure services and resources to access this server' setting.".
So, it is clear that the rule name is reserved for the below outlined setting. Everything was working as intended after all.

Deploying Azure Firewall IP Group changes fails with conflict

I am attempting to deploy an Azure Firewall with a Policy, a Rule and a set of IPGroups. When I deploy the ARM templates to start everything works.. Later If I want to change something in one of the IPGroups, and I try to deploy that IPGroup change, the Azure Deployment fails with a Status: Conflict with message:
{
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'."
}
}
I've attempted to both manage the IPGroups distinctly in their own ARM Template, and place them in with the Azure Policy Rule Collection ARM Template with a DependsOn to see if deploying them all together would help, but either way we just get "Conflict".. I Guess I am wondering what is the appropriate way to update an IPGroup that is a part of a Firewall Network rule? If I can't simply update the IPGroup?
Here is an example of my full ARM Template for my Policy with the IPGroups..
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"firewallPolicyName": {
"defaultValue": "[concat('onelucki-fw-parent-policy', uniqueString(resourceGroup().id))]",
"type": "String"
},
"DevSubnets": {
"defaultValue": "DevSubnets",
"type": "String"
},
"AzureSubnets": {
"defaultValue": "AzureSubnets",
"type": "String"
}
},
"variables": {
"fwPolicyName": "[parameters('firewallPolicyName')]"
},
"resources": [
{
"type": "Microsoft.Network/ipGroups",
"apiVersion": "2020-05-01",
"name": "AzureSubnets",
"location": "centralus",
"tags": { "Zone": "MixedZones" },
"properties": {
"ipAddresses": [
"10.99.1.1"
]
}
},
{
"type": "Microsoft.Network/ipGroups",
"apiVersion": "2020-05-01",
"name": "DevSubnets",
"location": "centralus",
"tags": { "Zone": "Dev" },
"properties": {
"ipAddresses": [
"10.99.2.2"
]
}
},
{
"type": "Microsoft.Network/firewallPolicies",
"apiVersion": "2020-11-01",
"name": "[parameters('firewallPolicyName')]",
"location": "centralus",
"properties": {
"sku": {
"tier": "Standard"
},
"threatIntelMode": "Alert"
}
},
{
"type": "Microsoft.Network/firewallPolicies/ruleCollectionGroups",
"apiVersion": "2020-11-01",
"name": "[concat(parameters('firewallPolicyName'), '/DefaultNetworkRuleCollectionGroup')]",
"location": "westus",
"dependsOn": [
"[resourceId('Microsoft.Network/ipGroups', parameters('AzureSubnets'))]",
"[resourceId('Microsoft.Network/ipGroups', parameters('DevSubnets'))]",
"[resourceId('Microsoft.Network/firewallPolicies', parameters('firewallPolicyName'))]"
],
"properties": {
"priority": 200,
"ruleCollections": [
{
"ruleCollectionType": "FirewallPolicyFilterRuleCollection",
"action": {
"type": "Allow"
},
"rules": [
{
"ruleType": "NetworkRule",
"name": "DemoRule",
"ipProtocols": [
"TCP"
],
"sourceAddresses": [],
"sourceIpGroups": [
"/subscriptions/<subscriptionIDHere>/resourceGroups/onelucki-fw/providers/Microsoft.Network/ipGroups/DevSubnets"
],
"destinationAddresses": [],
"destinationIpGroups": [
"/subscriptions/<subscriptionIDHere>/resourceGroups/onelucki-fw/providers/Microsoft.Network/ipGroups/AzureSubnets"
],
"destinationFqdns": [],
"destinationPorts": [
"135",
"445"
]
}
],
"name": "DemoDeployRuleCollection",
"priority": 1300
}
]
}
}
]
}
IP groups need to be deployed one at a time. Also the firewall policy needs a depends on the IP groups being used despite it not having them listed.
The deploy of the IP groups seems to do some validation/update on the firewall policy during deploy.
Deploy nested resources in Azure using DependsOn

Get public IP address of API Management from ARM template

I got this API Management ARM template
{
"apiVersion": "2017-03-01",
"name": "[variables('am-apimanagement-service-name')]",
"type": "Microsoft.ApiManagement/service",
"location": "North Europe",
"sku": {
"name": "[parameters('am-sku')]",
"capacity": "[parameters('am-skuCount')]"
},
"properties": {
"publisherEmail": "[parameters('am-publisher-email-p')]",
"publisherName": "[parameters('am-publisher-name-p')]"
},
"resources": [
{
"type": "apis",
"apiVersion": "2017-03-01",
"name": "test",
"dependsOn": [
"[concat('Microsoft.ApiManagement/service/',variables('am-apimanagement-service-name'))]"
],
"properties": {
"displayName": "test",
"description": "",
"serviceUrl": "[concat('https://test-',parameters('environment'),'.azurewebsites.net')]",
"path": "test",
"protocols": [
"https"
],
"isCurrent": true
},
"resources": [
{
"apiVersion": "2017-03-01",
"type": "operations",
"name": "GetTEst",
"dependsOn": [
"[concat('Microsoft.ApiManagement/service/', variables('am-apimanagement-service-name'), '/apis/test')]"
],
"properties": {
"displayName": "GET",
"method": "GET",
"urlTemplate": "/api/sites",
"description": "Get"
}
}
]
}
]
}
And this Web API ARM template
{
"apiVersion": "2016-03-01",
"name": "[variables('swa-name')]",
"type": "Microsoft.Web/sites",
"properties": {
"name": "[variables('swa-name')]",
"serverFarmId": "[variables('swa-hosting-plan-name')]",
"hostingEnvironment": "[parameters('swa-hosting-environment')]",
"siteConfig": {
"appSettings": [
{
"name": "Test:ConnectionString",
"value": "[concat('Server=tcp:', reference(resourceId('Microsoft.Sql/servers/', variables('db-serverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', variables('db-databaseName'), ';Persist Security Info=False;User ID=', parameters('db-administratorLogin'), ';Password=', parameters('db-administratorLoginPassword'), ';MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;')]"
}
],
"ipSecurityRestrictions": [
{
"ipAddress": "[variables('test-ip-address')]",
"subnetMask": "255.255.255.255"
}
]
}
},
"location": "[parameters('swa-location')]",
"tags": {
},
"kind": "api",
"dependsOn": [
"[concat('Microsoft.Web/serverfarms/', variables('swa-hosting-plan-name'))]"
]
}
How do I use the reference function to get IP address of the API Management resource from the Web API ? I want to put the API Management IP address in the ipSecurityRestrictions. I can not figure it out, and it is frustrating me quite a bit.
I have tried this from the web api resource resource section of the ARM template:
"ipSecurityRestrictions": [
{
"ipAddress": "[variables('test-ip-address')]",
"subnetMask": "255.255.255.255"
},
{
"ipAddress": "[reference(variables('am-apimanagement-service-name')).publicIPAddresses[0]]",
"subnetMask": "255.255.255.255"
}
]
But it doesnt work. Maybe I am not able to get the IP address in this step of the process? I am reading about outputs and linked templates. Maybe this is the solution?
The crazy thing here is that I need two things:
apiVersion to the reference
I need to reference the staticIp property
The thing is that a free tier API Management resource is actually not assigned a static IP, but a dynamic ip. Still the static ip address is assigned to the staticIp property of the referenced object:
"ipAddress": "[reference(variables('am-apimanagement-service-name'),'2017-03-01').staticIps[0]]"
This will reference the IP correctly.
I think you should add the api version to your reference, as the API management is not deployed in the same template as where you're refering to it.
https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-functions-resource#reference
Something like this
{
"ipAddress": "[reference(variables('am-apimanagement-service-name'), '2017-03-01').publicIPAddresses[0]]",
"subnetMask": "255.255.255.255"
}

Deploy Azure Resource Manager Template to existing VNET via CLI

I have a VNET set up in Azure with a number of subnets each with their own NSG defining inbound and outbound rules.
Into these subnets I would like to deploy VM scale sets with autoscale rules (based on https://raw.githubusercontent.com/gbowerman/azure-myriad/master/vmss-ubuntu-scale/azuredeploy.json for example) with certain extensions (perhaps pulling some repos from github/docker).
In my template how do I define that the scale set / VM should be assigned to an existing subnet/NSG etc?
Well, that's fairly straight forward, you just need to specify the ID of the resource you are referencing.
Let's say you want to use existing subnet:
"parameters": {
...
"existingVirtualNetworkName": {
"type": "string",
"metadata": {
"description": "Name of the existing VNET"
}
},
"existingVirtualNetworkResourceGroup": {
"type": "string",
"metadata": {
"description": "Name of the existing VNET resource group"
}
},
"subnetName": {
"type": "string",
"metadata": {
"description": "Name of the subnet in the virtual network you want to use"
}
},
...
},
"variables": {
...
"vnetID": "[resourceId(parameters('existingVirtualNetworkResourceGroup'), 'Microsoft.Network/virtualNetworks', parameters('existingVirtualNetworkName'))]",
"subnetRef": "[concat(variables('vnetID'),'/subnets/', parameters('subnetName'))]",
...
}
"resources": [
...
{
"apiVersion": "[variables('api-version')]",
"type": "Microsoft.Network/networkInterfaces",
"name": "[variables('nicName')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]"
],
"tags": {
"displayName": "NetworkInterface"
},
"properties": {
"ipConfigurations": [{
"name": "ipconfig1",
"properties": {
"privateIPAllocationMethod": "Dynamic",
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
},
"subnet": {
"id": "[variables('subnetRef')]"
}
}
}]
}
},
You would use the same approach for the Network Security Group.
Take a look here for more: https://github.com/Azure/azure-quickstart-templates/blob/master/201-vm-specialized-vhd-existing-vnet/azuredeploy.json

How to create a Virtual Machine in Microsoft Azure without PUBLIC VIRTUAL IP (VIP) address?

Is there any way to create a virtual machine without Public IP address in Windows Azure?
Even if you are creating a VM in virtual network we cannot control the Public IP assigned by Azure. Is there any way to disable or delete the public IP assigned by Azure?
This is incorrect as of 11/29/2016 You can now choose in the provisioning stage through Azure portal to set the public IP Address as NONE. This will allow only communication within the virtual network that the VM is living in.
No. A VM will always have Public IP Address. But you can not assign any Endpoints (leave the VM without any Endpoints defined) - this will effectively block all and any Internet traffic for your VM before it event reaches the Internal Data Centre network.
There will always be a Public IP address assigned to any Cloud Service with something deployed. It is up to you to decide whether to allow Internet traffic to your VM or not. And this decision is made via defining Endpoints.
EDIT
Using Azure Resource Manager you can create a VM without Public IP. You can control this setting on the Network Adapter. Within the IP Configurations for your Network Adapter in Azure Resource Manager you mus specify NONE for the Public IP Address setting.
With the Resource Groups approach you can assign the public ip as None.
It'll still have a network interface, which can be used to access it within the virtual network.
I stumbled across this question via a post on Twitter and saw there wasn't a recent, complete, answer.
Nowadays it's possible to create a virtual machine in Azure without a Public IP address. Check out the sample below. It'll create a VM, a NIC, and a VNet where the VM will be created in.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"my-super-secure-password": {
"type": "securestring"
}
},
"variables": {
},
"resources": [
{
"apiVersion": "2018-10-01",
"dependsOn": [
"my-project-prefix-nic"
],
"location": "westeurope",
"name": "my-project-prefix",
"properties": {
"diagnosticsProfile": {
"bootDiagnostics": {
"enabled": false
}
},
"hardwareProfile": {
"vmSize": "Standard_D8ds_v4"
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces','my-project-prefix-nic')]"
}
]
},
"osProfile": {
"adminPassword": "[parameters('my-super-secure-password')]",
"adminUsername": "jan",
"computerName": "my-project-prefix"
},
"storageProfile": {
"dataDisks": [
{
"createOption": "Empty",
"diskSizeGB": 512,
"lun": 0,
"managedDisk": {
"storageAccountType": "StandardSSD_LRS"
},
"name": "my-project-prefix-datadisk-0"
}
],
"imageReference": {
"publisher": "MicrosoftWindowsDesktop",
"offer": "Windows-10",
"sku": "20h1-pro-g2",
"version": "latest"
},
"osDisk": {
"createOption": "FromImage",
"diskSizeGB": 256,
"managedDisk": {
"storageAccountType": "StandardSSD_LRS"
},
"name": "my-project-prefix-osdisk"
}
}
},
"type": "Microsoft.Compute/virtualMachines"
},
{
"apiVersion": "2018-11-01",
"dependsOn": [
"my-project-prefix-vnet"
],
"location": "westeurope",
"name": "my-project-prefix-nic",
"properties": {
"ipConfigurations": [
{
"name": "ipconfig1",
"properties": {
"privateIPAllocationMethod": "Dynamic",
"subnet": {
"id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', 'my-project-prefix-vnet', 'my-project-prefix-subnet')]"
}
}
}
]
},
"type": "Microsoft.Network/networkInterfaces"
},
{
"apiVersion": "2018-11-01",
"location": "westeurope",
"name": "my-project-prefix-vnet",
"properties": {
"addressSpace": {
"addressPrefixes": [
"10.0.0.0/16"
]
},
"subnets": [
{
"name": "my-project-prefix-subnet",
"properties": {
"addressPrefix": "10.0.0.0/24",
"delegations": []
}
}
]
},
"type": "Microsoft.Network/virtualNetworks"
}
]
}
As you can see, I'm omitting the publicIPAddress property on creation of the VM. Seeing it's not a mandatory property, it's still a valid resource.

Resources