When running a bicep resource of type Microsoft.Resources/deploymentScripts
that runs a script that needs access to a keyvault which only allows selected networks how can we make the following script work?
resource exampleScript 'Microsoft.Resources/deploymentScripts#2020-10-01' = {
name: 'KeyVaultSecretFromProduct'
location: resourceGroup().location
kind: 'AzurePowerShell'
identity: {
type: 'UserAssigned'
userAssignedIdentities: {
'/subscriptions/${subscription().subscriptionId}/resourcegroups/${managedIdentity.scope}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${managedIdentity.name}': {}
}
}
properties: {
arguments: '-ResourceGroupName \\"${keyVaultSecretFromProduct.scope}\\" -SubscriptionKey \\"${subscriptionKey}\\" -KeyVault \\"${keyVaultSecretFromProduct.keyVault}\\"'
azPowerShellVersion: '3.0'
scriptContent: loadTextContent('../../membership-optimization/create-secret-for-product-key.ps1')
retentionInterval: 'P1D'
}
}
After running it fails with the error:
New-AzResourceGroupDeployment: 15:37:50 - The deployment 'test_keyvault' failed with error(s). Showing 1 out of 1 error(s).
Status Message: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details. (Code: DeploymentFailed)
- {
"status": "failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'failed'.",
"details": [
{
"code": "DeploymentScriptError",
"message": "The provided script failed with the following error:\r\nMicrosoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'\n at Microsoft.Azure.Commands.KeyVault.Models.KeyVaultDataServiceClient.SetSecret(String vaultName, String secretName, SecureString secretValue, PSKeyVaultSecretAttributes secretAttributes)\n at Microsoft.Azure.Commands.KeyVault.SetAzureKeyVaultSecret.ExecuteCmdlet()\n at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)\n at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)\n at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)\n at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()\r\nat <ScriptBlock>, /mnt/azscripts/azscriptinput/userscript.ps1: line 46\r\nat <ScriptBlock>, <No file>: line 1\r\nat <ScriptBlock>, /mnt/azscripts/azscriptinput/DeploymentScript.ps1: line 264. Please refer to https://aka.ms/DeploymentScriptsTroubleshoot for more deployment script information."
}
]
}
} (Code:Conflict)
CorrelationId: xxxxxxxxxxxxxxxxxxxx
A Vnet with some subnets used for app services were configured so that those app services can have access to key vault secrets.
Is there a way to solve this problem? Any workaround? Maybe a command that we can run that allows us to connect to the vnet?
A work-around could be to change the VNET settings on the vault, then run the script, and then re-set the VNET settings to its original state. It kinda sucks but it is the only thing I got working for me when handling this situation (though in my case It was a Powershell script).
Or, you could run the script on a VM, that is in a authorised subnet.
Related
I used https://learn.microsoft.com/en-us/azure/aks/certificate-rotation this link to rotate certificates in AKS. Certificate got updated but my cluster is in failed state. Because of this my application is down.
I am getting below mentioned error when I am running this command az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
ERROR: "error": { "code": "ErrorCodeRotateClusterCertificates", "message": "VMASAgentPoolReconciler retry failed: Category: ClientError; SubCode: OutboundConnFailVMExtensionError; Dependency: Microsoft.Compute/virtualMachines/extensions; OrginalError: Code=\"VMExtensionProvisioningError\" Message=\"VM has reported a failure when processing extension 'cse-agent-0'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=50\\n[stdout]\\n\\n[stderr]\\ncurl: option --proxy-insecure: is unknown\\ncurl: try 'curl --help' or 'curl --manual' for more information\\nCommand exited with non-zero status 2\\n0.00user 0.00system 0:00.00elapsed 100%!!(MISSING)C(string=VMAS agent pools reconciling)PU (0avgtext+0avgdata 7044maxresident)k\\n0inputs+8outputs (0major+372minor)pagefaults 0swaps\\n\\\"\\r\\n\\r\\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot \"; AKSTeam: NodeProvisioning, Retriable: false" } }
Kubernetes version: 1.14.8
Please help to resolved this issue.
What version of Ubuntu are you running on your nodes? From that error, guessing Ubuntu 16.04 or older.
I'm not sure if it will work, but instead of trying to rotate certificates, can you try upgrading the nodes?
You might also want to consider just creating a new cluster, and using VMSS instead of VMAS.
I'm attempting to deploy to my Virtual machine scale set using the custom script extension as below.
az vmss extension set --debug --name 'CustomScriptExtension' `
--resource-group 'my-rg' `
--publisher 'Microsoft.Compute' `
--version '1.9.5' `
--vmss-name 'myvmss' `
--settings '{\"commandToExecute\": \"powershell.exe ./download-package.ps1\", \"fileUris\": [\"https://[REDACTED].blob.core.windows.net/upload/download-package.ps1\"]}' `
--protected-settings '{\"managedIdentity\": {\"objectId\": \"[REDACTED]\"}}'
When running I get the following error:
cli.azure.cli.core.azclierror : Deployment failed. Correlation ID: 73f4d16b-afe0-4373-8773-1d7dd7d26940. VM has reported a failure when processing extension 'CustomScriptExtension'. Error message: "Failed to download all specified files. Exiting. Error Message: Exception of type 'Microsoft.WindowsAzure.GuestAgent.Plugins.CustomScriptHandler.Downloader.MsiNotFoundException' was thrown."
More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot
Deployment failed. Correlation ID: 73f4d16b-afe0-4373-8773-1d7dd7d26940. VM has reported a failure when processing extension 'CustomScriptExtension'. Error message: "Failed to download all specified files. Exiting. Error Message: Exception of type 'Microsoft.WindowsAzure.GuestAgent.Plugins.CustomScriptHandler.Downloader.MsiNotFoundException' was thrown."
The file to be downloaded requires authentication so I have given the scale set a system assigned identity and granted it the Storage Blob Data Reader role on the storage account hosting the powershell file.
The custom extension logs on the VM suggest that it was unable to get the identity of the vm:
[7108+00000001] [11/20/2020 09:12:28.79] [INFO] Handler successfully enabled
[7108+00000001] [11/20/2020 09:12:28.80] [INFO] Loading configuration for sequence number 1
[7108+00000001] [11/20/2020 09:12:28.84] [INFO] HandlerSettings = ProtectedSettingsCertThumbprint: [REDACTED], ProtectedSettings: {[REDACTED]}, PublicSettings: {FileUris: [https://[REDACTED].blob.core.windows.net/upload/download-package.ps1], CommandToExecute: powershell.exe ./download-package.ps1}
[7108+00000001] [11/20/2020 09:12:29.26] [INFO] Downloading files specified in configuration...
[7108+00000001] [11/20/2020 09:12:30.90] [INFO] Attempting to get MSI from IMDS
[7108+00000001] [11/20/2020 09:12:31.04] [WARN] WebClient: non retryable error occurred System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadString(Uri address)
at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.<>c__DisplayClass3_0.<DownloadStringWithRetries>b__0()
at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)
[7108+00000001] [11/20/2020 09:12:31.14] [ERROR] Unknown exception occurred while attempting to get MSI token System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadString(Uri address)
at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.<>c__DisplayClass3_0.<DownloadStringWithRetries>b__0()
at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)
at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.DownloadStringWithRetries(Uri address)
at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.MsiProvider.GetMsiHelper(NameValueCollection queries)
[7108+00000001] [11/20/2020 09:12:31.14] [INFO] Msi was not obtained
I can retrieve the identity token from the metadata endpoint via Invoke-WebRequest -Method Get -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/' so that appears to be set up correctly.
Any advice on what the problem could be or how to further diagnose this issue would be greatly appreciated.
Here are the few fixes you can try
The object ID of the managed identity might be incorrect.
Please also move commandToExecute and FileUris into protected settings with managed identities.
If want to use system assigned managed identity, you don't need to pass a clientId or objectID, more info here https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows#property-managedidentity
edit: please explicitly pass an empty json object as settings when you add commandToExecute and fileUris to protected settings. Extensions would fail otherwise due to duplicated settings.
I want to create 2 instance (an instance Zone A, and an instance Zone B), Private IP, SLB (Public IP), the VServer is both instance (for SLB), Frontend and Backend Port are HTTP. Regarding my terraform scripts are here: https://github.com/gablooge/alibaba-terraform-challenge
But I have a problem when applying my scripts. It turned out like this when creating the instance and SLB on applying my terraform scripts. Here are the error messages:
Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_instance.go:423: Resource alicloud_instance RunInstances Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
SDK.ServerError
ErrorCode: Forbidden.RiskControl
Recommend:
RequestId: 09B3E699-BC3E-457B-B266-54AC66325BE7
Message: This operation is forbidden by Aliyun RiskControl system.
on main.tf line 1, in resource "alicloud_instance" "instance_a":
1: resource "alicloud_instance" "instance_a" {
Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_instance.go:423: Resource alicloud_instance RunInstances Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
SDK.ServerError
ErrorCode: Forbidden.RiskControl
Recommend:
RequestId: 5520BCA6-6A03-4E3A-A87E-1794AC38AB11
Message: This operation is forbidden by Aliyun RiskControl system.
on main.tf line 15, in resource "alicloud_instance" "instance_b":
15: resource "alicloud_instance" "instance_b" {
Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_slb.go:244: Resource alicloud_slb CreateLoadBalancer Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
SDK.ServerError
ErrorCode: RISK.RISK_CONTROL_REJECTION
Recommend:
RequestId: D677A5EB-F3A7-48A2-98D1-91988BAF6B53
Message: The Account is rejected by risk control system.
on main.tf line 30, in resource "alicloud_slb" "default":
30: resource "alicloud_slb" "default" {
Alibaba Cloud Terraform Apply - ErrorCode: RISK.RISK_CONTROL_REJECTION & ErrorCode: Forbidden.RiskControl
Anybody know how to resolve this problem?
This looks like access related issue. You might want to check with raise a ticket with Alibaba Cloud support team to confirm all the required access is enabled.
I am deploying a Service Fabric Cluster on Nano Server using the secure-cluster-5-node template (https://github.com/Azure/azure-quickstart-templates/tree/master/service-fabric-secure-cluster-5-node-1-nodetype)
I get the following error:
Operation xxx
Tracking xxx
StatusConflict
Provisioning StateFailed
Timestamp6/22/2017 13:05:14
Duration6 minutes 11 seconds
TypeMicrosoft.Compute/virtualMachineScaleSets
Resource Id/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachineScaleSets/nt1vm
StatusMessage{
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "VMExtensionHandlerNonTransientError",
"message": "Handler 'Microsoft.Azure.ServiceFabric.ServiceFabricNode' has reported failure for VM Extension 'ServiceFabricNodeVmExt_vmNodeType0Name' with terminal error code '1007' and error message: 'Install failed for plugin (name: Microsoft.Azure.ServiceFabric.ServiceFabricNode, version 1.0.0.35). Exception:\nSystem.ComponentModel.Win32Exception: The subsystem needed to support the image type is not present\r\n at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)\r\n at Microsoft.Azure.Agent.StateMachine.HandlerStateMachine.InvokeCommand(String command, PluginArtifacts pluginArtifact, String pluginVersion, String pluginFolder, String pluginLogFolder, Int32 processWaitTimeout, PluginEventType startType, PluginEventType endType)\r\n at Microsoft.Azure.Agent.StateMachine.HandlerStateMachine.InstallHandler(PluginArtifacts artifact)'"
}
]
}
}
The settings are using in the ARM template for the os is:
"vmImagePublisher": {
"value": "MicrosoftWindowsServer"
},
"vmImageOffer": {
"value": "WindowsServer"
},
"vmImageSku": {
"value": "2016-Nano-Server"
},
"vmImageVersion": {
"value": "latest"
},
Any idea on how to troubleshoot this?
For now, Nano Server is not supported by Service Fabric Cluster. Please refer to this link.
You are able to create clusters on VMs running these operating
systems:
Windows Server 2012 R2
Windows Server 2016
Linux Ubuntu 16.04(in public preview)
You also could check it on Azure Portal. 2016-Nano-Server could not be selected.
I have used the following command to achieve it :
azure group deployment create <my-resource-group> <my-deployment-name> --template-uri https://raw.githubusercontent.com/azure/azure-quickstart-templates/master/active-directory-new-domain/azuredeploy.json
This link has a template which does it :
https://raw.githubusercontent.com/azure/azure-quickstart-templates/master/active-directory-new-domain/azuredeploy.json
I ran it several times but it shows this error :
info: Resource 'adVM/CreateADForest' of type 'Microsoft.Compute/virtualMachines/extensions' provisioning status is Running
info: Resource 'adVM' of type 'Microsoft.Compute/virtualMachines' provisioning status is Succeeded
info: Resource 'adNic' of type 'Microsoft.Network/networkInterfaces' provisioning status is Succeeded
info: Resource 'adLoadBalancer' of type 'Microsoft.Network/loadBalancers' provisioning status is Succeeded
info: Resource 'csyh7daynrjiaadsa' of type 'Microsoft.Storage/storageAccounts' provisioning status is Succeeded
info: Resource 'adPublicIP' of type 'Microsoft.Network/publicIPAddresses' provisioning status is Succeeded
info: Resource 'adAvailabiltySet' of type 'Microsoft.Compute/availabilitySets' provisioning status is Succeeded
info: Resource 'VNet' of type 'Microsoft.Resources/deployments' provisioning status is Succeeded
info: Resource 'adVNET' of type 'Microsoft.Network/virtualNetworks' provisioning status is Succeeded
error: getaddrinfo ENOTFOUND management.azure.com management.azure.com:443
ApplicationInsights:Sender [ { [Error: getaddrinfo ENOENT dc.services.visualstudio.com:80]
code: 'ENOENT',
errno: 'ENOENT',
syscall: 'getaddrinfo',
hostname: 'dc.services.visualstudio.com',
host: 'dc.services.visualstudio.com',
port: 80 } ]
error: Error information has been recorded to C:\Users\admin\.azure\azure.err
error: group deployment create command failed
The extensions part is failing. Could anyone tell what I have to do now???
I have solved it....I was not in the arm mode.
azure config mode arm
And then type the commands posted above. This worked.