I use the Microsoft Azure and I subscribed the HDInsight and its location is in Japan. A couple of days ago, I mistakenly removed all the VMs that I have been used and I determined to recover the VMs using the vhdl files. However, I changed my mind to create new VMs instead of restoring them. I successfully completed the installation of the new VMs and I installed the Hadoop and Spark and used them very well. However, a few days later, when I started my VMs, it was strangely slow to turn on and eventually it throw error indicating below message.
Provisioning failed. Allocation failed. Please try reducing the VM
size or number of VMs, retry later, or try deploying to a different
Availability Set or different Azure location.. AllocationFailed.
I tried to follow the documents in Azure and changed the VMs size and delete the VMs that I made but deleting or changing the VMs failed throwing below message.
Provisioning failed. Delete/Deallocate operation on VM 'hadoop-master' failed because the remaining VMs in the Availability Set 'spark-avs' cannot be allocated together. Changes in Availability Set allocation need to be executed atomically. Please deallocate or delete some or all of these VMs before retrying the current operation.
Please note that this VM is not allocated and won't accrue any charges.
Details: {
"resourceType": "Microsoft.WindowsAzure.ComputeResourceProvider.Core.Strings, CRP.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"ResourceCode": "ComputeAllocationFailure",
"ResourceParameters": []
}. CannotAllocateRemainingVMsInAvailabilitySet
Note that all the VMs that I re-installed is in the same availability set and I selected the south-korea for the new region (previous one is japan).
I tried to start many times but failed.
How to resolve this issue?
Note: When you create a VM, restart stopped (de-allocated) VMs, resize a VM, or when you add new instances, Microsoft Azure allocates compute resources to your subscription. You may occasionally receive errors when performing these operations even before you reach the Azure subscription limits.
This article “Troubleshoot allocation failures when you create, restart, or resize Linux VMs in Azure” explains the causes of some of the common allocation failures and suggests possible remediation.
If you assign Extra small, Small, and medium, you will receive validation failed error message.
Note: Head node recommended size is D3 v2, D4 v2, and D12 v2.
For more details, refer “Default node configuration and virtual machine sizes for clusters”.
Before deploying an HDInsight cluster, plan for the desired cluster capacity by determining the needed performance and scale. This planning helps optimize both usability and costs.
Some cluster capacity decisions cannot be changed after deployment. If the performance parameters change, a cluster can be dismantled and re-created without losing stored data.
For more details, refer “Capacity planning for HDInsight clusters”.
Related
I am new at working with Microsoft Azure and I am trying to open a Notebook from the Azure Machine learning studio.
Every time I try to create a new compute it says Creation failed so I cannot work. My region is francecentral and I have tried different Virtual Machine size
Your reason might be explained here:
As demand continues to grow, if we are faced with any capacity
constraints in any region during this time, we have established clear
criteria for the priority of new cloud capacity. Top priority will be
going to first responders, health and emergency management services,
critical government infrastructure organizational use, and ensuring
remote workers stay up and running with the core functionality of
Teams.
If you qualify for this category, you should reach out to Azure Support or your Microsoft representative. If not, you need to keep retrying (might work better at night) or try a different region.
I've "Visual Studio Enterprise" azure subscription. I initially got few credits. I think it was $200 for free. Earlier it used to show static remaining credit i.e. if I built two VMs and it cost $50 it would show $150. But since few days may be just past day remaining credit is continuously reducing.
The activities I have done so far. Earlier I was unable to connect to VM as outgoing connection was not allowed from my organisation. Since I have got permission I'm able to connect. After that I installed Active Directory Services on one of the VM.
I don't think till this step my credit amount got reduced except for the VMs I built.
I was trying to start and stop the VM using powershell but I wasn't able to do so. So I installed powershell azure modules. using Connect-AzAccount to connect and then used Start-AzVM and Stop-AzVm to start and stop the VMs. Just to clarify I keep the VMs in shut down mode so I don't keep getting billed.
Still the amount is being deducted. I ran Disconnect-AzAccount on suspicion that it might be causing the issue but still it hasn't fixed the issue.
Can any one help here?
There are mainly two ways to stop VM:
Stop the Azure VM locally: When you connect to the Azure VM with remote desktop, go to the Power options within Windows and select Shutdown, this will essentially “turn off” the VM and stop it from running. However, even though the VM won’t be running you WILL still be paying for the Virtual machine hardware allocation. Doing this will cause the Azure Portal to report the status of the VM to be “Stopped”.
Another way is to click Stop in the virtual machine page on the portal(or use Azure PowerShell or Azure CLI). Instead of just shutting down the Operating System, Azure will also deallocate the hardware (CPU and Memory) allocation. Doing this will cause the Azure Portal to report the status of the VM to be “Stopped (Deallocated)”.
Overall, both methods will reduce cost but not none of cost. You do still pay for the Azure Storage account usage, which is used for storig the VM’s .vhd disk image file. As a result, you will still incur some cost for the storage, but at least you will save on the VM resources.
Ref: Properly Shutdown Azure VM to Save Money
You need to identify what resource are been billed when VM is stopped or Stopped De-allocated.
Check the Cost analysis under Cost Management to identify the resource. Note that when VM is stopped you still pay for storage and other resources depending in your setup.
We are working on an application that processes excel files and spits off output. Availability is not a big requirement.
Can we turn the VM sets off during night and turn them on again in the morning? Will this kind of setup work with service fabric? If so, is there a way to schedule it?
Thank you all for replying. I've got a chance to talk to a Microsoft Azure rep and documented the conversation in here for community sake.
Response for initial question
A Service Fabric cluster must maintain a minimum number of Primary node types in order for the system services to maintain a quorum and ensure health of the cluster. You can see more about the reliability level and instance count at https://azure.microsoft.com/en-gb/documentation/articles/service-fabric-cluster-capacity/. As such, stopping all of the VMs will cause the Service Fabric cluster to go into quorum loss. Frequently it is possible to bring the nodes back up and Service Fabric will automatically recover from this quorum loss, however this is not guaranteed and the cluster may never be able to recover.
However, if you do not need to save state in your cluster then it may be easier to just delete and recreate the entire cluster (the entire Azure resource group) every day. Creating a new cluster from scratch by deploying a new resource group generally takes less than a half hour, and this can be automated by using Powershell to deploy an ARM template. https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cluster-creation-via-arm/ shows how to setup the ARM template and deploy using Powershell. You can additionally use a fixed domain name or static IP address so that clients don’t have to be reconfigured to connect to the cluster. If you have need to maintain other resources such as the storage account then you could also configure the ARM template to only delete the VM Scale Set and the SF Cluster resource while keeping the network, load balancer, storage accounts, etc.
Q)Is there a better way to stop/start the VMs rather than directly from the scale set?
If you want to stop the VMs in order to save cost, then starting/stopping the VMs directly from the scale set is the only option.
Q) Can we do a primary set with cheapest VMs we can find and add a secondary set with powerful VMs that we can turn on and off?
Yes, it is definitely possible to create two node types – a Primary that is small/cheap, and a ‘Worker’ that is a larger size – and set placement constraints on your application to only deploy to those larger size VMs. However, if your Service Fabric service is storing state then you will still run into a similar problem that once you lose quorum (below 3 replicas/nodes) of your worker VM then there is no guarantee that your SF service itself will come back with all of the state maintained. In this case your cluster itself would still be fine since the Primary nodes are running, but your service’s state may be in an unknown replication state.
I think you have a few options:
Instead of storing state within Service Fabric’s reliable collections, instead store your state externally into something like Azure Storage or SQL Azure. You can optionally use something like Redis cache or Service Fabric’s reliable collections in order to maintain a faster read-cache, just make sure all writes are persisted to an external store. This way you can freely delete and recreate your cluster at any time you want.
Use the Service Fabric backup/restore in order to maintain your state, and delete the entire resource group or cluster overnight and then recreate it and restore state in the morning. The backup/restore duration will depend entirely on how much data you are storing and where you export the backup.
Utilize something such as Azure Batch. Service Fabric is not really designed to be a temporary high capacity compute platform that can be started and stopped regularly, so if this is your goal you may want to look at an HPC platform such as Azure Batch which offers native capabilities to quickly burst up compute capacity.
No. You would have to delete the cluster and recreate the cluster and deploy the application in the morning.
Turning off the cluster is, as Todd said, not an option. However you can scale down the number of VM's in the cluster.
During the day you would run the number of VM's required. At night you can scale down to the minimum of 5. Check this page on how to scale VM sets: https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cluster-scale-up-down/
For development purposes, you can create a Dev/Test Lab Service Fabric cluster which you can start and stop at will.
I have also been able to start and stop SF clusters on Azure by starting and stopping the VM scale sets associated with these clusters. But upon restart all your applications (and with them their state) are gone and must be redeployed.
Is it possible to downgrade an Azure VM A8 (high compute) to a lower version like an A3? I keep getting the following error message when I try. I don't have an availability set setup. Thanks!
"Unable to upgrade the deployment. The requested VM size 'Large' may not be available in the resources supporting the existing deployment. Please try again later, try with a different VM size or smaller number of role instances, or create a deployment under an empty hosted service with a new affinity group or no affinity group binding. The long running operation tracking ID was: b2024fe9e93f6764bec3aa008756f0b7."
I recently discovered (via MS support tickets) that there are different "clusters" within Azure data centers, with different VM size compatibilities. In my case I had some cloud services in older clusters which didn't allow the newer "D-Series" VM sizes I wanted. The only solution was to create brand new cloud service instances from scratch and use Azure traffic manager to achieve a transition from the old servers to new ones.
I am trying to change the Virtual machine size from - 8 Cores, 14 GB to A 6 - 4 core, 28 GB for one of of my MSSQL Azure IaaS server.
I am getting the following error:
Unable to upgrade the deployment. The requested VM size may not be
available in the resources supporting the existing deployment. Please
try again later, try with a smaller VM size or smaller number of role
instances, or create a deployment under an empty hosted service with a
new affinity group or no affinity group binding. The long running
operation tracking ID was: 1d8145d1977877978d1d8dffdd045d83.
I understand that there is a limitation on how much one can get from one subscription. However, this is the live server and I have another 4 servers running under same subscription. Is there any way I can move this Virtual machine from one subscription to an another?
Otherwise, what is the right approach on increasing the size of this server?
Please advise the earliest.
I got the answer from Support as :
When customer initially deployed service it got deployed under the
cluster which does not support high memory VM's. Since customer is
having deployment under the hosted service it cannot be
pinned/migrated to a cluster which supports A6 or higher VM size. This
is a by design behaviour as of now. Unfortunately, the only way
customer can deploy a A6 VM is to delete and recreate deployment with
A6 size under the given hosted service. When customer tries to create
it, then he/she will be allocated a cluster which supports A6 or
higher VM size.