Azure VM D4 is advertised as having 400GB of SSD backed storage. Provisioning the VM I end up with 127GB for the OS and 400GB for temp storage.
Is this normal? I need the full 400GB on the OS drive and dont see an obvious way to reconfigure storage.
That is correct. However, because the local SSD is not guaranteed to be persistent, you will not want this to be responsible for your OS drive.
In the D-series announcements, http://azure.microsoft.com/blog/2014/09/22/new-d-series-virtual-machine-sizes/
"Local Storage SSD Drive
On these new sizes, the temporary drive (D:\ on Windows, /mnt or /mnt/resource on Linux) are local SSDs. This high-speed local disk is best used for workloads that replicate across multiple instances, like MongoDB, or can leverage this high I/O disk for a local and temporary cache, like SQL Server 2014’s Buffer Pool Extensions. Note, these drives are not guaranteed to be persistent. Thus, while physical hardware failure is rare, when it occurs, the data on this disk may be lost, unlike your OS disk and any attached durable disks that are persisted in Azure Storage." (emphasis mine)
Found this post that explains how this is normal for the OS drive.
https://azure.microsoft.com/blog/2015/03/25/azure-vm-os-drive-limit-octupled/
So for marketplace images the guidance is to provision new data drives.
Related
I installed the Azure Local Cluster on my Windows 10 Home edition.
The installation created directory C:/SfDevCluster/
When I run the Properties command on C:/SfDevCluster/, Size is 8.06 TB, Size on disk is 10 GB. However, the physical size of my disk is only 1 TB.
I went through all the tabs in the Properties dialog box looking for
some hints about virtual size but no clues. I went through all
system/administrative tools and no clue about virtual disks.
What feature of Windows 10 enables this virtual size?
Whatever it is, there has to be a way to map it into some
external storage but how? What is the name for this feature so I can research it.
storage spaces thin provisioning probably does, possibly in combination with dynamic vhd(x) underneath.
lotsa resources lying around:
https://social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-questions-faq.aspx#What_happens_if_I_run_out_of_physical_capacity_on_a_thinly_provisioned_storage_space
https://serverfault.com/questions/591125/how-can-i-safely-use-storage-thin-provisioning (more general)
Hi the official Spark documentation state:
While Spark can perform a lot of its computation in memory, it still
uses local disks to store data that doesn’t fit in RAM, as well as to
preserve intermediate output between stages. We recommend having 4-8
disks per node, configured without RAID (just as separate mount
points). In Linux, mount the disks with the noatime option to reduce
unnecessary writes. In Spark, configure the spark.local.dir variable
to be a comma-separated list of the local disks. If you are running
HDFS, it’s fine to use the same disks as HDFS.
I wonder what is the purpose of 4-8 per node
Is it for parallel write ? I am not sure to understand the reason why as it is not explained.
I have no clue for this: "If you are running HDFS, it’s fine to use
the same disks as HDFS".
Any idea what is meant here...
Purpose of usage 4-8 RAID disks to mirror partitions adding redundancy to prevent data lost in case of fault on hardware level. In case of HDFS the redundancy that RAID provides is not needed, since HDFS handles it by replication between nodes.
Reference
I am quite a novice on Azure and I am bit stuck trying to understand virtual machine size and features.
I have just deployed "Hortonworks Sandbox with HDP 2.4" virtual machine template on a DS3_v2 machine, which seems to have following features: 4Cores, 14GB Ram, 8 data disks and 28Gb ssds that it is pretty decent to run a proof of concept, however i have some doubts. I am not sure about the total disk size available on this machine if its 200GB or 100GB, does this size include the os vhd? I understand i cant attach till 8 data disk from my storage account sum up either 100GB or 200GB.
DS3_v2 machine also includes Azure premium storage that i think it referes to 28GB ssds, I guess i could have two ssd data disks of 14 GB each?
I really appreciate any insight about these doubts.
Thank you very much.
Your OS disk is a different one, the 28 GB SSD is a local disk and is a temporary disk(think of it as D:\ where your OS is in C:) and the data present on this is not guaranteed during hardware failures. The data on 8 disks you can attach are persisted and you can choose (GRS or RA-GRS Geo Redundant storage) and each of these disks can be upto 1 TB (around 1023 GB) which means you can attach a total of around 8 TB storage data disks to DS3_V2 instance.
I have installed mysql in a A4 Linux VM in Azure and I'm running out of disk space. I saw that the primary disk space is 29GB only and the temporary disk space is 600GB, but I don't want to store mysql's data to the temporary disk because of the possibility of losing them.
I want to attach a new disk to my VM and I have two questions:
1) What type of storage is the best for my case?
2) If I attach a disk, will mysql data be distributed to this disk automatically when the primary disk space runs out or should I make further adjustments (i.e. change mysql's data_dir or something)?
You can attach a new data disk to your VM. The steps are described here in the Azure documentation: http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-tutorial/#attachdisk
No, the newly attached disk won't expand the existing one.
AWS Amazon offers 160GB space for small instance. On booting Suse linux the total root partition space I got is 10GB. On df -h I only see/dev/sda1 with 10GB space. Where is rest 150GB? How can I claim this space? I dont want to use EBS as it cost extra and 160GB space suffice my need. Please help.
The extra 150GB is given as an ephemeral storage, e.g. data on this storage won't survive reboots in contrast to the data on your root storage. During launching, you can select where your ephemeral disks should be made available as a device in your machine (this is the -boption when using the command line or in the "Instance Storage" tab when launching via the S3 console. You can then simply mount it in your running instance.