I am quite a novice on Azure and I am bit stuck trying to understand virtual machine size and features.
I have just deployed "Hortonworks Sandbox with HDP 2.4" virtual machine template on a DS3_v2 machine, which seems to have following features: 4Cores, 14GB Ram, 8 data disks and 28Gb ssds that it is pretty decent to run a proof of concept, however i have some doubts. I am not sure about the total disk size available on this machine if its 200GB or 100GB, does this size include the os vhd? I understand i cant attach till 8 data disk from my storage account sum up either 100GB or 200GB.
DS3_v2 machine also includes Azure premium storage that i think it referes to 28GB ssds, I guess i could have two ssd data disks of 14 GB each?
I really appreciate any insight about these doubts.
Thank you very much.
Your OS disk is a different one, the 28 GB SSD is a local disk and is a temporary disk(think of it as D:\ where your OS is in C:) and the data present on this is not guaranteed during hardware failures. The data on 8 disks you can attach are persisted and you can choose (GRS or RA-GRS Geo Redundant storage) and each of these disks can be upto 1 TB (around 1023 GB) which means you can attach a total of around 8 TB storage data disks to DS3_V2 instance.
Related
I have this issue from lot longer, when purchasing Azure managed disks I have a requirement of allocating 512GB premium disk. I'm wondering if I allocate two P15 (256GB) premium disk will also give the same as P20 capacity with small pricing different, IOPS and throughput. I need to answer the following questions :
Which approach is best to match 512GB is it allocating single
(P20)512GB or two 256GB (P15) disk ?
If I allocate two 256GB (P15) disks will that double the IOPS and Throughput ?
From the managed disk price, two 256GiB (P15) is a larger price than one P20 512GiB premium disk. Also, the two P15 disks have 1100*2 less than P20 disk 2300 IOPS but have double throughput.
Take into account considerations listed below when making the choice.
Scale Limits (IOPS and Throughput) The IOPS and Throughput limits of
each Premium disk size is different and independent from the VM scale
limits. Make sure that the total IOPS and Throughput from the disks
are within scale limits of the chosen VM size.
For example, if an application requirement is a maximum of 250 MB/sec
Throughput and you are using a DS4 VM with a single P30 disk. The DS4
VM can give up to 256 MB/sec Throughput. However, a single P30 disk
has Throughput limit of 200 MB/sec. Consequently, the application will
be constrained at 200 MB/sec due to the disk limit. To overcome this
limit, provision more than one data disks to the VM or resize your
disks to P40 or P50.
So If you have a high scale VM, your application requires a larger throughput and it supports write or operate data to both disks at the same time to optimize the two disks' throughput. You could select two P15 disks, otherwise, generally, it prefers to use single P20 than two P15 disks.
For more information, you can see Azure premium storage: design for high performance.
I installed the Azure Local Cluster on my Windows 10 Home edition.
The installation created directory C:/SfDevCluster/
When I run the Properties command on C:/SfDevCluster/, Size is 8.06 TB, Size on disk is 10 GB. However, the physical size of my disk is only 1 TB.
I went through all the tabs in the Properties dialog box looking for
some hints about virtual size but no clues. I went through all
system/administrative tools and no clue about virtual disks.
What feature of Windows 10 enables this virtual size?
Whatever it is, there has to be a way to map it into some
external storage but how? What is the name for this feature so I can research it.
storage spaces thin provisioning probably does, possibly in combination with dynamic vhd(x) underneath.
lotsa resources lying around:
https://social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-questions-faq.aspx#What_happens_if_I_run_out_of_physical_capacity_on_a_thinly_provisioned_storage_space
https://serverfault.com/questions/591125/how-can-i-safely-use-storage-thin-provisioning (more general)
Azure VM D4 is advertised as having 400GB of SSD backed storage. Provisioning the VM I end up with 127GB for the OS and 400GB for temp storage.
Is this normal? I need the full 400GB on the OS drive and dont see an obvious way to reconfigure storage.
That is correct. However, because the local SSD is not guaranteed to be persistent, you will not want this to be responsible for your OS drive.
In the D-series announcements, http://azure.microsoft.com/blog/2014/09/22/new-d-series-virtual-machine-sizes/
"Local Storage SSD Drive
On these new sizes, the temporary drive (D:\ on Windows, /mnt or /mnt/resource on Linux) are local SSDs. This high-speed local disk is best used for workloads that replicate across multiple instances, like MongoDB, or can leverage this high I/O disk for a local and temporary cache, like SQL Server 2014’s Buffer Pool Extensions. Note, these drives are not guaranteed to be persistent. Thus, while physical hardware failure is rare, when it occurs, the data on this disk may be lost, unlike your OS disk and any attached durable disks that are persisted in Azure Storage." (emphasis mine)
Found this post that explains how this is normal for the OS drive.
https://azure.microsoft.com/blog/2015/03/25/azure-vm-os-drive-limit-octupled/
So for marketplace images the guidance is to provision new data drives.
AWS Amazon offers 160GB space for small instance. On booting Suse linux the total root partition space I got is 10GB. On df -h I only see/dev/sda1 with 10GB space. Where is rest 150GB? How can I claim this space? I dont want to use EBS as it cost extra and 160GB space suffice my need. Please help.
The extra 150GB is given as an ephemeral storage, e.g. data on this storage won't survive reboots in contrast to the data on your root storage. During launching, you can select where your ephemeral disks should be made available as a device in your machine (this is the -boption when using the command line or in the "Instance Storage" tab when launching via the S3 console. You can then simply mount it in your running instance.
What is MongoDB's storage size limit on 64bit platforms? Can MongoDB store 500-900 Gb of data within one instance (node)? What was the largest amount of data you've stored in MongoDB, and what was your experience?
The "production deployments" page on MongoDB's site may be of interest to you. Lots of presentations listed with infrastructure information. For example:
http://blog.wordnik.com/12-months-with-mongodb says they're storing 3 TB per node.
The MongoDB's storage limit on different operating systems are tabulated below as per the MongoDB 3.0 MMAPv1 storage engine limits.
The MMAPv1 storage engine limits each database to no more than 16000 data files. This means that a single MMAPv1 database has a maximum size of 32TB. Setting the storage.mmapv1.smallFiles option reduces this limit to 8TB.
Using the MMAPv1 storage engine, a single mongod instance cannot manage a data set that exceeds maximum virtual memory address space provided by the underlying operating system.
Virtual Memory Limitations
Operating System Journaled Not Journaled
Linux 64 terabytes 128 terabytes
Windows Server 2012 R2
and Windows 8.1 64 terabytes 128 terabytes
Windows (otherwise) 4 terabytes 8 terabytes
Reference: MongoDB Database Limit.
Note:The WiredTiger storage engine is not subject to this limitation.
Another way to have more than 2GB on a single node is to run multiple mongod processes. So sharding is one option or doing some manual partitioning across processes.
Hope This helps.
You won't run anywhere near hitting the cap with 1TB on 64 bit systems, however Mongo does store the indexes in memory so a smooth experience depends on your index size and how much memory you have. But if you have a beefy enough system it won't be a problem.