Azure VM Details :
OS : Windows Server 2019 Datacenter Core
Size: Standard D4s v3 (4 vcpus, 16 GiB memory)
Location: Australia East
VM generation: V1
Agent status: Ready
Agent version: 2.7.41491.1010
Azure disk encryption: Not Enabled
Extensions already installed :
DependencyAgentWindows
IaaSAntimalware
MDE.Windows
MicrosoftMonitoringAgent
Have an existing recovery services vault with 10s of other VMs getting backed up.
Trying to enable the backup from Azure Portal for this VM ( From the VM Blade > Operations > Backup ) but it's failing with the following error code:
I have tried it multiple times.
Provisioning state: Failed
Duration: 1 minute 3 seconds
Status: Conflict
{
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
"details": [
{
"code": "BMSUserErrorContainerObjectNotFound",
"message": "Item not found"
}
]
}
All the information on troubleshooting backup relates issues # https://learn.microsoft.com/en-us/azure/backup/backup-azure-vms-troubleshoot talk about After the "Enable Backup" step.
I have also tried to enable the backup using azure cli:
az backup protection enable-for-vm --vm "/subscriptions/xxx/resourceGroups/yyy/providers/Microsoft.Compute/virtualMachines/vm_name" -v vaultname -g vault_resourcegroup -p backuppolicy_name
It throws the following error:
The specified Azure Virtual Machine Not Found. Possible causes are
1. VM does not exist
2. The VM name or the Service name needs to be case sensitive
3. VM is already Protected with same or other Vault.
Please Unprotect VM first and then try to protect it again.
Please contact Microsoft for further assistance.
None of the Point 1,2 or 3 are true.
VM exists, the name is used as shown in the portal, no other VM protection service is in use.
Note: I have faced this issue a few days back on another subscription, but luckily no one was yet using that VM, so I destroyed and re-deployed the VM, and the error went away.
I can't do the same for this VM as it's already in use.
Any help/guidance will be appreciated.
Seems like a portal error or the VM is not able to communicate with Azure Platform. I would suggest you try the "Reapply" feature to update the platform status.
[Snippet of Reapply in Azure Porta][1]
Else, you can try initiating a backup from the "Recovery Services vaults" blade and add the VM to it.
The solution was to contact Microsoft support. Their engineer after some analysis ( aka to and fro, screenshots exchange over email..etc) replied with:
I check from the backend and notice that the VM status is not in synchronize state. I’ve requested the VM engineer xxxxx resync the VM from the backend. Please try to reenable the VM backup again in the Azure portal recovery service Vault page. If you encounter the same issue, please try to configure the VM backup in the Azure Virtual Machine Panel page and let me know the results. Thanks!
After this when I attempted to enable the backup it worked.
So for anyone who faces this problem, it looks like the only option is to get in touch with MS Support.
Related
I have an Azure Windows 10 VM (Standard D2as v4 (2 vcpus, 8 GiB memory)). I have tried using the automated tasks section to start this VM up and shut it down automatically. However, the auto start task repeatedly fails. I do not understand why. Here is the error that I have found:
Unauthorized!
{
"error": {
"code": "InvalidAuthenticationTokenTenant",
"message": "The access token is from the wrong issuer 'https://sts.windows.net/xxxxxxxxxxxxxxxxxxxxx/'. It must match the tenant 'https://sts.windows.net/xxxxxxxxxxxxxxxxxxxxxxx/' associated with this subscription. Please use the authority (URL) 'https://login.windows.net/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' to get the token. Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). If you just transferred your subscription and see this error message, please try back later."
}
}
I do not understand what is going on at all am afraid. If someone could please explain and help resolve!
Thanks!
I have tried in my environment and got below results:
Initially I have created Virtual machine in my environment and added two tasks like Azure start VM and Azure stop VM
Task1 - start VM
Home -> Virtual machine -> Automation -> task
I have Authenticated connection both Azure VM (azurevm) and Office 365 Outlook (office365)
Configuration:
After created task1 it will appears on the Tasks list.
After the selected task runs, you get an email that looks like this example:
In the tasks list-> task1->In that task's Runs column, select View.
Task2 - Stop VM
Similar process which I took to reproduce:
Mail:
View:
{ "error": { "code": "InvalidAuthenticationTokenTenant", "message": "The access token is from the wrong issuer
'https://sts.windows.net/xxxxxxxxxxxxxxxxxxxxx/'. It must match the tenant 'https://sts.windows.net/xxxxxxxxxxxxxxxxxxxxxxx/' associated with this subscription. Please use the authority (URL)'https://login.windows.net/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' to get the token. Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). If you just transferred your subscription and see this error message, please try back later."} }
The above error occurs due to not have enough permission.
You must have certain permissions to enable VMs for the Start&Stop VMs.
Kindly check and make sure you have role of Contributor to your logic app.
In permission they has some of the pre-existing Automation account and new Automation account for log analytics workspace kindly check the below reference link.
Logic app:
logic app -> task1->Run history
Reference:
https://learn.microsoft.com/en-us/azure/automation/automation-solution-vm-management
Azure Logic App: Azure VM Connector - InvalidAuthenticationTokenTenant - Microsoft Q&A
I'd like to add an offsite Windows VM to Azure Arc for health monitoring. The VM is hosted by Vultr and runs Windows Server 2016 Standard Build 14393.
However, installing AzureConnectedMachineAgent.msi on the target VM fails with error code 1603. Installation log also contains this error:
Start-Service : Service 'Guest Configuration Extension service
WixQuietExec64: (ExtensionService)' cannot be started due to the following error: Cannot start
WixQuietExec64: service ExtensionService on computer '.'.
WixQuietExec64: At C:\Program Files\AzureConnectedMachineAgent\ExtensionService\GC\Modules\Exte
WixQuietExec64: nsionService\ServiceHelper.psm1:367 char:5
Any suggestions on how to fix this?
You may Check if the user with which you are logged into the VM have
sufficient permissions to start a system service
If you find the following in the
%ProgramData%\AzureConnectedMachineAgent\Log\himds.log or in installation logs :
time="2021-02-11T08:39:38-08:00" level=error msg="Cannot open event source: Azure Hybrid Instance Metadata Service."
You can verify the permissions by collecting the following registry
key from an impacted server.
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Eventlog\Application\CustomS
Mitigation can be to grant the permission to write to the
SECURITY_SERVICE_RID S-1-5-6 which would grant the required
permissions to the himds service account.
https://learn.microsoft.com/en-us/windows/win32/secauthz/well-known-sids.
If the registry key does NOT exist on the impacted VM, then this
resolution will NOT apply as there will be a separate root cause such
as AV interference.
If the root cause is not found here ,then a procmon trace needs to be
taken to analyze the root cause for the msi not being able to start a
service.
( In case a procmon trace has to be analyzed , please open an MS
Support ticket)
To get support for Windows Agent and extensions in Azure, the Windows
Agent on the Windows VM must be later than or equal to version
2.7.41491.911. However the cause for the failure of agent installation is different in this case.
You may also want to check %programdata%\ext_mgr_logs\gc_ext_telemetry.txt log which must have had an entry something like this :
<GCLOG>........ Not starting Extension Service since machine is an Azure VM</GCLOG>
Cause:
This can happen while attempting to install the agent on an Azure VM.This is an unsupported production scenario.One Should not be installing this agent on an Azure VM as it conflicts with the Azure Guest Agent and interferes with Azure VM management.
If one wishes to use an Azure VM simply for testing purposes then
they can follow the below document for guidance
https://learn.microsoft.com/en-us/azure/azure-arc/servers/plan-evaluate-on-azure-virtual-machine
I have a new vm, Operating system Windows (Windows Server 2016 Datacenter).
When I try to enable backup and select new Recovery Service Vault, I get deployment error:
Deployment to resource group test failed.
Additional details from the underlying API that might be helpful: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
Resource
vault242/Azure/iaasvmcontainer;iaasvmcontainerv2;test;web01/vm;iaasvmcontainerv2;test;web01
Type
Microsoft.RecoveryServices/vaults/backupFabrics/protectionContainers/protectedItems
Status
Conflict
Status message
{
"status": "Failed",
"error": {
"code": "BMSUserErrorContainerObjectNotFound",
"message": "Item not found"
}
}
Can't find any information for code BMSUserErrorContainerObjectNotFound and why a protected item not created automatically
My apologies for the delay in the response.
Were you able to resolve the issue?
If not, let's review it.
As I understood, you are enabling the Azure VM Back Up by following the next steps:
There could be multiple reasons why you are getting this failure.
Did you perform these steps manually using the Azure Portal? Template deployment? Scripting? I suspect most likely you are doing the template deployment or any kind of scripting and this one is the syntax issue.
Second thought, it was the transmitted issue due to the load of request on the Azure end. In this case, you need to retry the operation.
Additional question to ask, do you get the failure on one specific machine or all machines? Specific region?
Do you get the same failure when you use the existing vault?
If you still can provide information above, it's going to be helpful to narrow down the root cause.
I ran into this error as well today and I think it is is a Azure portal bug when enabling the Backup from the VM blade.
Instead, you can initiate a Backup from the "Recovery Services vaults" blade and add the VM to it.
I've successfully deployed Azure Stack TP2 and created a few VMs on it using the provided "WindowsServer-2012-R2-Datacenter" image, I was even able to connect to them.
However, after a few weeks I was having problems with VPN connection so I've restarted the server. The VPN connectivity started to work again but I've noticed that all of my Virtual Machines are stopped and they switch to "Failed" state when I've tried to start them. Also I'm unable to successfully create another VM and this is the error message I get:
{
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "InternalExecutionError",
"message": "Failed to change the diagnostics profile for VM 'testVM1'"
}
]
}
}
I've read on the Microsoft Azure Stack troubleshooting page that VMs not starting after boot may be related to Failover Cluster Manager not starting, however when opening Failover Cluster Manager as the instruction suggests I don't see any Clusters there so I'm not sure if it's even configured to use it.
Has anyone got this issue before?
Microsoft Azure Stack troubleshooting page was right indeed, however it didn't mention that:
you need to type the name of the default cluster yourself
the problem is also related to creating new Virtual Machines
After the reboot of Azure Stack TP2 server every Virtual Machine that stays in "Saved" state in Failover Cluster Manager needs to be started in order to be able to create new Virtual Machines or use available ones. To do this:
Open Start Menu and type "Failover Cluster Manager".
Choose "Connect to Cluster...".
Enter cluster name: "s-cluster.azurestack.local" and press "OK".
Open "Roles".
Right-click on every role that has its Status set to "Saved" and click "Start" from the context menu.
Moreover, if a Virtual Machine was going to be created on a rebooted server when there were any Virtual Machines in "Saved" state and failed, then it's not possible to delete this failed VM from Azure Stack Portal until the VM is started from Failover Cluster Manager.
Can't seem to figure out how to change the availability set of an existing Azure VM in the Resource Manager stack. There's no interface for it. Set-AzureAvailabilitySet does not exist in the Azure Powershell tools when in ResourceManager mode. It does exist in service stack mode. But that doesn't help me.
AFAIK, this feature may be addressed by the end of this year. It's a big challenge for the MS team to allow such operation. Changing the availability Set requires a review of the VM mobility architecture on Azure. Fore example, adding a VM in an Availability Set already containing a VM means putting it to different default domain. Becasue VM mobilty is a matter on Azure (No Live Migration), it's not an easy operation.
I have written a Powershell script which let you change the AS of an ARM VM by recreating it.
Give it a try and enjoy:
How to use it ?
1- Download the script and save it to local location
2- Run it and provide the requested parameters
or
2- ./Set-ArmVmAvailabilitySet.ps1 –VmName ‘The VM Name’ –ResourceGroup
‘Resource Group’ –AvailabilitySetName ‘As Name’ –SubscriptionName
‘The Subscription name’
To remove a VM from an AvailabilitySet:
./Set-ArmVmAvailabilitySet.ps1 –VmName ‘The VM Name’ –ResourceGroup
‘Resource Group’ –AvailabilitySetName 0 –SubscriptionName ‘The
Subscription name’
Download Link
Version 1.01 :
https://gallery.technet.microsoft.com/Set-Azure-Resource-Manager-f7509ec4
Source
That feature isn't implemented yet in the ARM stack, that's why you're not seeing the cmdlet...