Azure VMSS custom script extension error - unable to download script files - azure

Question

I'm running the custom template deployment which includes a Linux VM and a Linux-based VMSS.
Both operations (VM creation and VMSS creation) involves the CustomScriptExtension.
The scripts used for post-configuration are the same for VM and VMSS.
They have the same reference inside the template.
However, the deployment for VM is completed succesfully but for VMSS is ended with an error.
When checking resource group in Azure portal, VMSS is created successfully and shows activity (CPU, memory etc).
In "Extensions" blade I can see my predefined extension:
However the state is "Failed". When clicking on "Failed" to see the details the following error is shown:
Message: VM has reported a failure when processing extension 'filesextension'. Error message: "Enable failed: processing file downloads failed: failed to download file[1]: failed to download file: unexpected status code: actual=404 expected=200" More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot
Azure portal deployment error:
{
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "VMExtensionProvisioningError",
"message": "VM has reported a failure when processing extension 'filesextension'. Error message: \"Enable failed: processing file downloads failed: failed to download file[1]: failed to download file: unexpected status code: actual=404 expected=200\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "
}
]
}
}
The same error is when deploying via CLI:
Deployment failed. Correlation ID: f077af77-405b-49fe-9f95-bf42a722c7ec. {
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "VMExtensionProvisioningError",
"message": "VM has reported a failure when processing extension 'filesextension'. Error message: \"Enable failed: processing file downloads failed: failed to download file[0]: failed to download file: unexpected status code: actual=404 expected=200\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "
}
ARM template itself:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"defaultValue": "[resourceGroup().location]",
"type": "String",
"metadata": {
"description": "Location for all resources"
}
},
"dnsNameForJumpBox": {
"type": "String",
"metadata": {
"description": "Unique DNS Name for the Public IP used to access the Docker Virtual Machine (master node)."
}
},
"vmImageReference": {
"defaultValue": {
"publisher": "Canonical",
"offer": "UbuntuServer",
"sku": "16.04-LTS",
"version": "latest"
},
"type": "Object",
"metadata": {
"description": "The image to use for VMs created. This can be marketplace or custom image",
"link": "https://learn.microsoft.com/en-us/nodejs/api/azure-arm-compute/imagereference?view=azure-node-2.2.0"
}
},
"vmNodeSku": {
"defaultValue": "Standard_F8s_v2",
"type": "String",
"metadata": {
"description": "Size of VMs in the VM Scale Set."
}
},
"vmMasterSku": {
"defaultValue": "Standard_F16s_v2",
"type": "String",
"metadata": {
"description": "Size of the master node."
}
},
"vmMasterDiskType": {
"defaultValue": "Premium_LRS",
"allowedValues": [
"Premium_LRS",
"Standard_LRS"
],
"type": "String",
"metadata": {
"description": "Choose between a standard disk for and SSD disk for the master node's NFS fileshare"
}
},
"vmMasterDiskSize": {
"defaultValue": 256,
"allowedValues": [
32,
64,
128,
256,
512,
1000,
2000,
4000,
10000
],
"type": "Int",
"metadata": {
"description": "The SSD Size to be used for the NFS file share. For pricing details see https://azure.microsoft.com/en-us/pricing/details/managed-disks/"
}
},
"vmAdditionalInstallScriptUrl": {
"defaultValue": "",
"type": "String",
"metadata": {
"description": "An additional installs script (bash run as root) to be run after nodes/master are configured. Can be used to mount additional storage or do additional setup"
}
},
"vmAdditionalInstallScriptArgument": {
"defaultValue": "",
"type": "String",
"metadata": {
"description": "An argument to be passed to the additional install script"
}
},
"nextflowInstallUrl": {
"defaultValue": "https://get.nextflow.io",
"type": "String",
"metadata": {
"description": "The install URL for nextflow, this can be used to pin nextflow versions"
}
},
"instanceCount": {
"defaultValue": 2,
"maxValue": 100,
"type": "Int",
"metadata": {
"description": "Number of cluster VM instances (100 or less)."
}
},
"adminUsername": {
"type": "String",
"metadata": {
"description": "Admin username on all VMs."
}
},
"vnetName": {
"defaultValue": "nfvnet",
"type": "String",
"metadata": {
"description": "Name of the virtual network to deploy the scale set into."
}
},
"subnetName": {
"defaultValue": "nfsubnet",
"type": "String",
"metadata": {
"description": "Name of the subnet to deploy the scale set into."
}
},
"shareName": {
"defaultValue": "sharedstorage",
"type": "String",
"metadata": {
"description": "Azure file share name."
}
},
"mountpointPath": {
"defaultValue": "/datadisks/disk1",
"type": "String",
"metadata": {
"description": "Path on VM to mount file shares. '/datadisks/disk1/' is a Premium Managed disk with high iops, this will suit most uses."
}
},
"nodeMaxCpus": {
"defaultValue": 2,
"type": "Int",
"metadata": {
"description": "Sets the cluster.maxCpus setting on all cluster nodes"
}
},
"_artifactsLocation": {
"defaultValue": "https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master",
"type": "String",
"metadata": {
"description": "*Advanced* This is best left as default unless you are an advanced user. The base URI where artifacts required by this template are located."
}
},
"_artifactsLocationSasToken": {
"defaultValue": "",
"type": "SecureString",
"metadata": {
"description": "*Advanced* This should be left as default unless you are an advanced user. The sasToken required to access _artifactsLocation. When the template is deployed using the accompanying scripts, a sasToken will be automatically generated."
}
},
"_artifactsSharedFolder": {
"defaultValue": "shared_scripts/ubuntu",
"type": "String",
"metadata": {
"description": "*Advanced* This should be left as default unless you are an advanced user. The folder in the artifacts location were shared scripts are stored."
}
},
"_artifactsNextflowFolder": {
"defaultValue": "nextflow-genomics-cluster-ubuntu/scripts",
"type": "String",
"metadata": {
"description": "*Advanced* This should be left as default unless you are an advanced user. The folder in the artifacts location were nextflow scripts are stored."
}
},
"authenticationType": {
"defaultValue": "sshPublicKey",
"allowedValues": [
"sshPublicKey",
"password"
],
"type": "String",
"metadata": {
"description": "Type of authentication to use on the Virtual Machine. SSH key is recommended."
}
},
"adminPasswordOrKey": {
"type": "SecureString",
"metadata": {
"description": "SSH Key or password for the Virtual Machine. SSH key is recommended."
}
}
},
"variables": {
"nextflowInitScript": "[uri(parameters('_artifactsLocation'), concat(parameters('_artifactsNextflowFolder'), '/init.sh', parameters('_artifactsLocationSasToken')))]",
"diskInitScript": "[uri(parameters('_artifactsLocation'), concat(parameters('_artifactsSharedFolder'), '/vm-disk-utils-0.1.sh', parameters('_artifactsLocationSasToken')))]",
"jumpboxNICName": "jumpboxNIC",
"addressPrefix": "10.0.0.0/16",
"subnetPrefix": "10.0.0.0/24",
"vmssName": "[concat('cluster', uniqueString(parameters('dnsNameForJumpBox')))]",
"storageAccountType": "Standard_LRS",
"storageAccountName": "[concat('nfstorage', uniqueString(resourceGroup().id))]",
"publicIPAddressName": "jumpboxPublicIP",
"publicIPAddressType": "Dynamic",
"jumpboxVMName": "jumpboxVM",
"subnetRef": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('subnetName'))]",
"linuxConfiguration": {
"disablePasswordAuthentication": true,
"ssh": {
"publicKeys": [
{
"path": "[concat('/home/', parameters('adminUsername'), '/.ssh/authorized_keys')]",
"keyData": "[parameters('adminPasswordOrKey')]"
}
]
}
},
"networkSecurityGroupName": "default-NSG"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2016-01-01",
"name": "[variables('storageAccountName')]",
"location": "[parameters('location')]",
"sku": {
"name": "[variables('storageAccountType')]"
},
"kind": "Storage"
},
{
"type": "Microsoft.Network/publicIPAddresses",
"apiVersion": "2017-06-01",
"name": "[variables('publicIPAddressName')]",
"location": "[parameters('location')]",
"properties": {
"publicIPAllocationMethod": "[variables('publicIPAddressType')]",
"dnsSettings": {
"domainNameLabel": "[parameters('dnsNameForJumpBox')]"
}
}
},
{
"type": "Microsoft.Network/networkSecurityGroups",
"apiVersion": "2019-08-01",
"name": "[variables('networkSecurityGroupName')]",
"location": "[parameters('location')]",
"properties": {
"securityRules": [
{
"name": "default-allow-22",
"properties": {
"priority": 1000,
"access": "Allow",
"direction": "Inbound",
"destinationPortRange": "22",
"protocol": "Tcp",
"sourceAddressPrefix": "*",
"sourcePortRange": "*",
"destinationAddressPrefix": "*"
}
}
]
}
},
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2017-06-01",
"name": "[parameters('vnetName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]"
],
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('addressPrefix')]"
]
},
"subnets": [
{
"name": "[parameters('subnetName')]",
"properties": {
"addressPrefix": "[variables('subnetPrefix')]",
"networkSecurityGroup": {
"id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]"
}
}
}
]
}
},
{
"type": "Microsoft.Network/networkInterfaces",
"apiVersion": "2017-06-01",
"name": "[variables('jumpboxNICName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]",
"[concat('Microsoft.Network/virtualNetworks/', parameters('vnetName'))]"
],
"properties": {
"ipConfigurations": [
{
"name": "ipconfig1",
"properties": {
"privateIPAllocationMethod": "Dynamic",
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
},
"subnet": {
"id": "[variables('subnetRef')]"
}
}
}
]
}
},
{
"type": "Microsoft.Compute/virtualMachines",
"apiVersion": "2017-03-30",
"name": "[variables('jumpboxVMName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName'))]",
"[concat('Microsoft.Network/networkInterfaces/', variables('jumpboxNICName'))]"
],
"properties": {
"hardwareProfile": {
"vmSize": "[parameters('vmMasterSKU')]"
},
"osProfile": {
"computerName": "[variables('jumpboxVMName')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPasswordOrKey')]",
"linuxConfiguration": "[if(equals(parameters('authenticationType'), 'password'), json('null'), variables('linuxConfiguration'))]"
},
"storageProfile": {
"imageReference": "[parameters('vmImageReference')]",
"osDisk": {
"createOption": "FromImage"
},
"dataDisks": [
{
"lun": 0,
"name": "jumpboxdatadisk",
"diskSizeGB": "[parameters('vmMasterDiskSize')]",
"caching": "None",
"createOption": "Empty",
"managedDisk": {
"storageAccountType": "[parameters('vmMasterDiskType')]"
}
}
]
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces',variables('jumpboxNICName'))]"
}
]
}
}
},
{
"type": "Microsoft.Compute/virtualMachines/extensions",
"apiVersion": "2017-03-30",
"name": "[concat(variables('jumpboxVMName'),'/nfinit')]",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', variables('jumpboxVMName'))]"
],
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.0",
"autoUpgradeMinorVersion": true,
"forceUpdateTag": "rerunnow",
"settings": {
"fileUris": [
"[variables('nextflowInitScript')]",
"[variables('diskInitScript')]"
]
},
"protectedSettings": {
"commandToExecute": "[concat('bash init.sh ', variables('storageAccountName'), ' ', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2016-01-01').keys[0].value, ' ', parameters('shareName'), ' ', parameters('mountpointPath'), ' false ', parameters('adminUsername'), ' 0 ', parameters('nextflowInstallUrl'), ' ', parameters('vmAdditionalInstallScriptUrl'), ' ', parameters('vmAdditionalInstallScriptArgument'))]"
}
}
},
{
"type": "Microsoft.Compute/virtualMachineScaleSets",
"apiVersion": "2017-03-30",
"name": "[variables('vmssName')]",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Network/virtualNetworks/', parameters('vnetName'))]",
"[concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName'))]"
],
"sku": {
"name": "[parameters('vmNodeSKU')]",
"capacity": "[parameters('instanceCount')]"
},
"properties": {
"overprovision": true,
"upgradePolicy": {
"mode": "Manual"
},
"virtualMachineProfile": {
"storageProfile": {
"osDisk": {
"createOption": "FromImage",
"caching": "ReadWrite"
},
"imageReference": "[parameters('vmImageReference')]"
},
"osProfile": {
"computerNamePrefix": "[variables('vmssName')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPasswordOrKey')]",
"linuxConfiguration": "[if(equals(parameters('authenticationType'), 'password'), json('null'), variables('linuxConfiguration'))]"
},
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "nic",
"properties": {
"primary": true,
"ipConfigurations": [
{
"name": "ipconfig",
"properties": {
"subnet": {
"id": "[resourceId(resourceGroup().name, 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('subnetName'))]"
}
}
}
]
}
}
]
},
"extensionProfile": {
"extensions": [
{
"name": "filesextension",
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.0",
"autoUpgradeMinorVersion": true,
"forceUpdateTag": "rerunnow",
"settings": {
"fileUris": [
"[variables('nextflowInitScript')]",
"[variables('diskInitScript')]"
]
},
"protectedSettings": {
"commandToExecute": "[concat('bash init.sh ', variables('storageAccountName'), ' ', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2016-01-01').keys[0].value, ' ', parameters('shareName'), ' ', parameters('mountpointPath'), ' true ', parameters('adminUsername'), ' ', parameters('nodeMaxCpus'), ' ', parameters('nextflowInstallUrl'), ' ', parameters('vmAdditionalInstallScriptUrl'), ' ', parameters('vmAdditionalInstallScriptArgument'))]"
}
}
}
]
}
}
}
}
],
"outputs": {
"JumpboxConnectionString": {
"type": "String",
"value": "[concat('ssh ', parameters('adminUsername'), '#', reference(variables('publicIPAddressName')).dnsSettings.fqdn)]"
},
"ExampleNextflowCommand": {
"type": "String",
"value": "[concat('nextflow run hello -process.executor ignite -cluster.join path:', parameters('mountpointPath'), '/cifs/cluster', ' -with-timeline runtimeline.html -with-trace -cluster.maxCpus 0')]"
},
"ExampleNextflowCommandWithDocker": {
"type": "String",
"value": "[concat('nextflow run nextflow-io/rnatoy -with-docker -process.executor ignite -cluster.join path:', parameters('mountpointPath'), '/cifs/cluster', ' -with-timeline runtimeline.html -with-trace -cluster.maxCpus 0')]"
}
}
}
I have only one suggestion that the extensionProfile part of the template isn't working as expected however was unable to find any proofs.
"extensionProfile": {
"extensions": [
{
"name": "filesextension",
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.0",
"autoUpgradeMinorVersion": true,
"forceUpdateTag": "rerunnow",
"settings": {
"fileUris": [
"[variables('nextflowInitScript')]",
"[variables('diskInitScript')]"
]
},
"protectedSettings": {
"commandToExecute": "[concat('bash init.sh ', variables('storageAccountName'), ' ', listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2016-01-01').keys[0].value, ' ', parameters('shareName'), ' ', parameters('mountpointPath'), ' true ', parameters('adminUsername'), ' ', parameters('nodeMaxCpus'), ' ', parameters('nextflowInstallUrl'), ' ', parameters('vmAdditionalInstallScriptUrl'), ' ', parameters('vmAdditionalInstallScriptArgument'))]"
}
}
}
]
}
}

Answer 1

Has the url with sas token worked at all? Check it by attempting to download the script in a browser after you have created the sas token.
If it works and then doesn't work later - this could be down to the sas token expiring after a set period. You can attempt to log the full url out after deploying of your vmss. Then try again and download the script from a browser.