Virtual Machine scale set in Azure doesn't scale. The CPU usage on the VM is 99%, but the utilziation shown on Azure for only one is around 10%-20%.
I have tried adjusting the scale lower for the average cpu to 40%, but it doesn't change anything.
When the virtual machine starts, there is a load of around %80-90 instantly from my code. The scale set should scale to a second machine after 5 minutes and keep repeating until the load decreases and all runs are complete. What happens the machine starts and if you RDP to the machine, task manager shows the process using 80-90% cpu constantly. If you view Azure to see the scale set as a whole it only list 10-20% cpu usage. The virtual machine is a server 2016 core edition built-in to Azure. It is almost like it isn't reporting back to azure the CPU usage, so it never scales beyond 1 maybe 2 every once in a while.
Related
I have 2 azure vm's created, 1 windows server 2012 r2, and 1 Ubuntu 14.
It takes both those vm's approximately 5 minutes to startup.
Is there a way I can speed up the process?
I don't need the vm's running continuously, I prefer to start/stop, as per need.
There are no steps you can take to speed up the VM start. Resources must be allocated and the VM provisioned.
What I can recommend is setting up a script to auto start / stop your VM's based on a schedule. For example, if you are using them in a class room environment you can set it up so that they start early (6am?) and shut down each day by 5pm.
You can find some more information about this here.
I have a simple Umbraco 7.2 website (no plugins/custom code/etc) on a Shared Azure Website. Azure has suspended the website twice in the last week because of over quota memory usage. I scaled it up to 6 instances for now. Looking at the dashboard right now, it shows me it's using 1.5gigs. I went on the Kudu interface (.scm.azurewebsites.net) and in the Process Explorer it shows that the the process is only taking ~150mb of both private memory and working set. It also says virtual memory is taking ~750mb.
Why is Azure saying it's taking up so much memory? Does increasing instance count actually mean more memory for my app or does it just mean more instances are running the same app... so it's basically 200mb*6 instances = 1200megs?
Thanks!
When you scale up the number of instances for an Azure Website it changes the number of VMs your website is running on - so it means more instances running the same app.
Similarly, when you log into the Kudu interface this is going to connect to the process running on one particular instance - so it won't show you total memory being used by all instances at once, just what's used by the one particular instance you're connected to.
I am using a azure websites solution with 20 websites. Hosted on 4 cores, 8 GB RAM standard instance. I would like to know how I could do scaling in Azure websites and when to do it ?
Also I am reading some values from the new azure portal.
Can someone guide me on the values that I see here ?
Thank you
Averages
The Avg % is telling you, on average, how much of that resource is being used. So, if you have 8GB of ram, and you are typically using 66% of it, then you are averaging 5.28 Gb of ram used. Same goes for the CPU average listed below.
For the totals, I have no idea.
You're not using much of the CPU available to you here, but you are definitely taking advantage of the RAM. I'm not sure of what kind of web application you are running though, so it's dificult to determine what could be causing this.
Scaling
In terms of scaling, I always suggest starting with a small machine, then gradually scaling up.
Based on your usage, I'd drop to a machine that has fewer CPU cores, but more available RAM. From within your dashboard, you can see how to scale by clicking no your web app, then scrolling down. Click on the scale tab and it should appear as it does below:
You can now adjust what you want to scale by. The default setting is CPU Percentage, but that isn't particularly useful in this case. Instead, select Schedule and performance rules and a new panel wioll appear. On the right hand side, select Metric name and look for Memory Percentage.
In your particular case, this is helpful as we saw that your RAM is consistently being used.
Look at Action and you will want to Increase count by and change the number of VMs to 1. What this does is when your RAM reaches a certain usage %, Azure will auto-scale and generate a new VM for you. After a cool down period of 5 minutes (the default, listed at the bottom), your machine will revert to 1 machine.
Conclusion
With these settings, each time your website uses <= (Select your percentage) of RAM, Azure will increase the size of your machines.
In your case, I suggest using fewer cores, but more RAM.
Make sure you save your settings, with the Save button above.
Scott Hanselman as a great blog post on how to make sense of all of this.
Microsoft Azure's Web Sites reports it's CPU consumption as CPU time and it's measured in milliseconds. I understand it's the time taken to run code but it doesn't help make scaling decision. We are on the "Standard" plan which gives us the entire VM (no sharing).
On the other hand, Azure Cloud Services reports its CPU consumption as "CPU %" which clearly communicates total capacity as well as current consumption to easily scale (eg: over 80% => scale up. Under 15% => scale down)
So the question is, given a somewhat slippery measure like CPU time without a measure of capacity, how do you determine what's the residual capacity to make scale up/scale down decisions for azure websites?
I just took a look at my Azure websites setup, and it appears to me that while the monitoring metrics only give us CPU time, the automatic scaling is setup by targeting CPU percentage.
I am limited by a piece of software that utilizes a single core per instance of the program run. It will run off an SQL server work queue and deposit results to the server. So the more instances I have running the faster the overall project is done. I have played with Azure VMs a bit and can speed up the process in two ways.
1) I can run the app on a single core VM, clone that VM and run it on as many as I feel necessary to speed up the job sufficiently.
OR
2) I can run the app 8 times on an 8-core VM, ...again clone that VM and run it on as many as I feel necessary to speed up the job sufficiently.
I have noticed in testing that the speed-up is roughly the same for adding 8 single core VMs and 1 8-core VM. Assuming this is true, would it better better price-wise to have single core machines?
The pricing is a bit of a mystery, whether it is real cpu usage time, or what. It is a bit easier using the 1 8-core approach as spinning up machines and taking them down takes time, but I guess that could be automated.
It does seem from some pricing pages that the multiple single core VM approach would cost less?
Side question: so could I do like some power shell scripts to just keep adding VMs of a certain image and running the app, and then start shutting them down once I get close to finishing? After generating the VMs would there be some way to kick off the app without having to remote in to each one and running it?
I would argue that all else being equal, and this code truly being CPU-bound and not benefitting from any memory sharing that running multiple processes on the same machine would provide, you should opt for the single core machines rather than multi-core machines.
Reasons:
Isolate fault domains
Scaling out rather than up is better to do when possible because it naturally isolates faults. If one of your small nodes crashes, that only affects one process. If a large node crashes, multiple processes go down.
Load balancing
Windows Azure, like any multi-tenant system, is a shared resource. This means you will likely be competing for CPU cycles with other workloads. Having small VMs gives you a better chance of having them distributed across physical servers in the datacenter that have the best resource situation at the time the machines are provisioned (you would want to make sure to stop and deallocate the VMs before starting them again to allow the Azure fabric placement algorithms to select the best hosts). If you used large VMs, it would be less likely to find a suitable host with optimal contention to accommodate many virtual cores.
Virtual processor scheduling
It's not widely understood how scheduling a virtual CPU is different than scheduling a physical one, but it is something worth reading up on. The main thing to remember is that hypervisors like VMware ESXi and Hyper-V (which runs Azure) schedule multiple virtual cores together rather than separately. So if you have an 8-core VM, the physical host must have 8 physical cores free simultaneously before it can allow the virtual CPU to run. The more virtual cores, the more unlikely the host will have sufficient physical cores at any given time (even if 7 physical cores are free, the VM cannot run). This can result in a paradoxical effect of causing the VM to perform worse as more virtual CPU cores are added to it. http://www.perfdynamics.com/Classes/Materials/BradyVirtual.pdf
In short, a single vCPU machine is more likely to get a share of the physical processor than an 8 vCPU machine, all else equal.
And I agree that the pricing is basically the same, except for a little more storage cost to store many small VMs versus fewer large ones. But storage in Azure is far less expensive than the compute, so likely doesn't tip any economic scale.
Hope that helps.
Billing
According to Windows Azure Virtual Machines Pricing Details, Virtual Machines are charged by the minute (of wall clock time). Prices are listed as hourly rates (60 minutes) and are billed based on total number of minutes when the VMs run for a partial hour.
In July 2013, 1 Small VM (1 virtual core) costs $0.09/hr; 8 Small VMs (8 virtual cores) cost $0.72/hr; 1 Extra Large VM (8 virtual cores) cost $0.72/hr (same as 8 Small VMs).
VM Sizes and Performance
The VMs sizes differ not only in number of cores and RAM, but also on network I/O performance, ranging from 100 Mbps for Small to 800 Mbps for Extra Large.
Extra Small VMs are rather limited in CPU and I/O power and are inadequate for workloads such as you described.
For single-threaded, I/O bound applications such as described in the question, an Extra Large VM could have an edge because of faster response times for each request.
It's also advisable to benchmark workloads running 2, 4 or more processes per core. For instance, 2 or 4 processes in a Small VM and 16, 32 or more processes in an Extra Large VM, to find the adequate balance between CPU and I/O loads (provided you don't use more RAM than is available).
Auto-scaling
Auto-scaling Virtual Machines is built-into Windows Azure directly. It can be based either on CPU load or Windows Azure Queues length.
Another alternative is to use specialized tools or services to monitor load across the servers and run PowerShell scripts to add or remove virtual machines as needed.
Auto-run
You can use the Windows Scheduler to automatically run tasks when Windows starts.
The pricing is "Uptime of the machine in hours * rate of the VM size/hour * number of instances"
e.g. You have a 8 Core VM (Extra Large) running for a month (30 Days)
(30 * 24) * 0.72$ * 1= 518.4$
for 8 single cores it will be
(30 * 24) * 0.09 * 8 = 518.4$
So I doubt if there will be any price difference. One advantage of using smaller machines and "scaling out" is when you have more granular control over scalability. An Extra-large machine will eat more idle dollars than 2-3 small machines.
Yes you can definitely script this. Assuming they are IaaS machines you could add the script to windows startup, if on PaaS you could use the "Startup Task".
Reference