I have a virtual machine running Windows on Microsoft Windows Azure. I am noticing that one of the hard drives shows as completely full. Do these drives automatically expand as data is added, or do I need to increase storage, and if so--how?
Thanks for any tips.
In Azure, both the OS Disk and any attached Data Disks are fixed format VHDs. They are not resized automatically and, in fact, there is no supported process to modify the size. Since these disks are allocated using sparse storage - i.e., you are only billed for space actually used - the general recommendation is to use 1TB disks. If the disk is empty you will not be billed.
Windows Server 2012 provides TRIM support which clears the space occupied in Azure Storage by deleted files. Without this support there could still be a charge for files which have been deleted from the filesystem but which still occupy pages in the page blob backing the VHD in Azure Storage.
Martin Balliauw has written up instructions for modifying the size of a VHD in Azure. He has also created a utility that helps with this task. This comes with a do it at your own risk warning.
Related
I created a Azure Virtual Machine and it in turn created a temporary storage drive for me (D:). Now, If I have C:, D:, E:,F: drive in my VM how can I differentiate which is Azure temporary Storage which is not.
I have tried to Use DeviceType=3 but it lists all the Logical Drives.
I'm assuming you're talking about Virtual Machines (you didn't specify). For VMs:
Your OS disk is backed by durable blob storage. All the time. For Windows, this is approx. 127GB. For Linux, this is approx. 32GB.
Your temp drive is always in-chassis, and at risk. The temp disk size is advertised in the VM sizing specs.
Beyond that: You'd be taking the specific action of mounting additional disks, which are all durable, blob-backed. So you'll know exactly which drives there are.
After some research, it looks there isn't any programmatic way to identify the Azure temporary drive. May be the hack is to look for "DATALOSS_WARNING_README.txt" file in all the drives.
I am using two Microsoft Azure Virtual Machines (marked as classic), both running on Linux. One is used for test purposes and internal demos, the other is production and running few of clients' instances.
What I would like to do is change the size of Virtual Machine. I understand this is quite common process and can easily be done from the Azure Management Portal and that this is not affecting data. However, when I have changed the size of our testing machine, exactly this has happened and we have lost all data.
Azure Support answer received was:
"We recommend you delete the VM by keeping the attached disks and create a new VM with the required size." Not sure why this would be better?
Any data stored on the ephemeral (internal-to-chassis) scratch disk is at risk, as it's a non-durable disk (and will in all likelihood be destroyed/recreated upon resizing a VM).
The only way to have durable data is to use Azure Storage (blobs, vhd as attached disk, Azure File storage) or external database. Azure Storage is durable (minimum 3 copies), and is not stored with your VM.
One more thing: The VM's OS Disk is a VHD in Azure Storage (so the OS disk is durable, just like attached vhd's).
You have more than one way to do that and keep in mind what David said, data on OS disks, attached disks and blobs is the only durable one.
To prevent losing data and since you're using Classic VMs, you can do the following:
1- Go to your VM on portal and capture an image out of it.
2- Go to your new image and create a new VM out of it, while specifying the new specs that you need.
3- When done, connect to your new VM while keeping the old one without termination.
4- Check if all your data is there, if yes, then you can remove the old one. (In case you need the old IP, you can still assign it to the new one).
Cheers.
I have created a VM in hyperv using dynamically expanding VHD option.Now I have exported that VHD to azure cloud and created a VM in azure.My VM is not booting properly. Is it recommended to use dynamically expanding VHD to create VM in azure?
OS and Data disks (ephemeral storage is local to the hardware) are thin provisioned in Azure. That is they are dynamically expanding. This is a decision made by Azure (so they don't have to store hundreds of thousands of mostly empty 120gb C: drives, which also makes it cheaper for you. ) and there is no option to change that.
Disks that you upload need to be a fixed size. (so you will pay for the full size)
According to Azure Documentation - Upload a Windows VM image to Microsoft Azure for Resource Manager deployments
Microsoft Azure can only accept images for Generation 1 virtual machines saved in the VHD file format. The VHD size must be fixed and a whole number of MB. The maximum size allowed for the VHD is 1023GB.
If you are having problems with an uploaded disk, it is likely that the problem is that you have created a dynamically expanding one and it simply isn't booting.
Usually, you want to create the VM with a fixed size disk as the dynamically expanding disks have a performance cost.
More information here:
https://technet.microsoft.com/en-us/magazine/ff458359.aspx
When you create an Azure VM, it creates a D: drive named "Temporary Storage". It's unclear to me just how temporary this is though. Is it reset between reboots, start/stop, or just if the image is recreated? Or arbitrarily? Also, what are some use cases for this temporary storage?
Resets, reboots, fail overs etc. can wipe it. It is only to be used for swap files etc.
Performance Best Practices for SQL Server in Azure Virtual Machines
Temporary Disk
The temporary storage drive, labeled as the D: drive is not persisted and is not saved in the Windows Azure Blob storage. It is used primarily for the page file and its performance is not guaranteed to be predictable. Management tasks such as a change to the virtual machine size, resets the D: drive. In addition, Windows Azure erases the data on the temporary storage drive when a virtual machine fails over. The D: drive is not recommended for storing any user or system database files, including tempdb.
EDIT: Interestingly enough the above has now been updated to the below which says that is OK to store your tempdb on if you are using the D-Series Virtual Machines.
The temporary storage drive, labeled as the D: drive, is not persisted to Azure blob storage. Do not store your data or log files on the D: drive.
Only store tempdb and/or Buffer Pool Extensions on the D drive when using the D-Series Virtual Machines (VMs). Unlike the other VM series, the D drive in the D-Series VMs is SSD-based. This can improve the performance of workloads that heavily use temporary objects or that have working sets which don't fit in memory. For more information, see Using SSDs in Azure VMs to store SQL Server TempDB and Buffer Pool Extensions.
Some Azure Virtual Machine Basics Please
The Temporary storage drive is a local drive on the physical box that should only serve as scratch space. For example, we put the paging file onto this drive to save on transaction costs for each Windows page-in but nothing else. I would recommend you don’t put anything that you need persisted on this drive as it will be deleted upon any hardware failure the system sees whereas the OS disk and any data disks attached will remain persisted in storage even on hardware failures.
This disk is a local disk, non-replicated. If it crashes, anything on it is gone. If your VM is moved to another server, the content is gone. If you scale a VM size down, it's likely this data will be gone, as the allocation size decreases.
As for use cases: Think about database servers that need a scratch disk. Or maybe your workflow involves grabbing content from a blob, processing it, then shipping it off somewhere (e.g. photo/video/audio processing). You could download to temporary storage, do all of your transform work on that disk, then pick up the results, saving the final file somewhere durable. Temp storage use is really open-ended - it's really up to you how you use it; just realize that it's not durable storage (unlike your VHD, which is in durable storage).
When you resize the VM, When you shutdown or restart your VM, when your VM is moved to a different host server(due to service healing, shutdown and restart), when the host is updated, when the host experiences a hardware failure etc.
Source: Azure Pricing page
I am confused about the Azure VM setup. I am trying to setup a SQL Server and the guidelines suggest that if your DBs are larger than 10GB, that you should setup a seperate Data Disk in Azure Storage. But all the documentation explicitly says not to use the D: Temporary Storage as it is volatile across reboots.
I completely understand this. The issue I have is that when I create a new VM, (I just created a SQL 2012 Web on 2008 R2 SP1 from the gallery), I get a single C: drive of about 128GB. When I then attach an empty data disk through the portal, it appears as D: and is called Temporary Storage.
My understanding is that this drive is not temporary storage (volatile) as I have created it through the portal as a data disk.
Is this a hangover from a past Azure configuration? I gather the VMs used to come with a 30GB OS drive but now come with a 128GB OS drive. Is this something to do with it?
I'm pretty confused!
The way it works, the D drive is the 70GB temp (volatile) drive (at least with Windows Server 2012):
Here, I just attached an empty disk and refreshed the windows Server disk manager. I then go to format it:
Once formatted, my new 20GB disk is assigned to F (and I still have a 70GB temp drive). This drive, backed by blob storage, is durable.
When you are using Azure VMs - the OS drive & the Data drives are backed by Azure Blob Storage (the VHDs are Page Blobs). The OS disk size limit during most of the CTP was 10GB, but was raised around the time the feature shipped to the larger 128GB. The deciding factor for Data Drive/No Data Drive/Lots of Data Drives (Max = 16) for SQL is more a function of your IOPS requirements than either the size of the DB corpus or the relative drive size.
For SQL workloads in a VM, I would strongly recommend reviewing:
http://go.microsoft.com/fwlink/?LinkId=306266
This is a performance paper based on the latest Azure bits, developed by the SQL team (updated June 2013).
Pat
In the interest of providing an answer to this question.
I think it was just an anomoly. #DavidMakogon helped me go through what was expected and it seems that my first VM simply didn't initialize the Temporary Drive on first boot, so this caused lots of confusion.
It's all working as expected now.