I created a .vhd image and downloading through the portal using Edge.
The .vhd file in the portal shows it's 127gb (which is the size of the hard drive on the VM).
The actual usage is 33gb - as shown below. The problem is, other than being very slow to download, it's currently showing 40gb downloaded and is still continuing.
Can anyone tell me why this is or suggest a good, quick, alternative to downloading?!
Thanks - Rob
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 131045924 32279488 93405860 26% /
none 4 0 4 0% /sys/fs/cgroup
udev 3563804 8 3563796 1% /dev
tmpfs 713764 380 713384 1% /run
none 5120 0 5120 0% /run/lock
none 3568800 0 3568800 0% /run/shm
none 102400 0 102400 0% /run/user
The size of the file is 127GB as you mentioned. So when you directly download via browser, it will always download the entire file i.e. 127GB and not the data that is currently being occupied as the browser doesn't know how much full your VHD is.
What you need to do is use the tools which support Sparse Download. In this method, first the tool identifies the occupied bytes (or rather occupied pages as it is a Page Blob) and downloads only those bytes.
On Windows, you can use AzCopy tool. It supports sparse download. On Mac/Linux platform, try downloading the VHD via CLI tools. Though I have not used it but my guess is that it also supports sparse download.
You may find this blog post from Azure Storage Team on how to efficiently upload/download Page Blobs: https://blogs.msdn.microsoft.com/windowsazurestorage/2010/04/10/using-windows-azure-page-blobs-and-how-to-efficiently-upload-and-download-page-blobs/.
Related
We are using GItHub Enterprise as our source code management system and its installed as an appliance in Vmware Esxi.
For monitoring purpose we are using 2 tools for getting the performance metrics.
1) Nagios over SNMP ( GHE provides the resource metric by default over SNMP)
2) vROPS for montoring.
But its noticed that GHE will create a much cached memory which will result in unwanted alerts. There is 4.6 G of cached memory, how to configure threshold for GHE (used-cached) memory instead of just used memory.
Current memory usage on the system is:
free -h
total used free shared buffers cached Mem:
15G 14G 909M 1.5M 375M 4.6G
-/+ buffers/cache: 9.8G 5.9G
Swap: 0B 0B 0B
But in monitoring side is there any way to exclude these linux consumed memory and only the GHe consumed memory alone?
I have Varnish 5.1.1 on Centos 6.5 and want to Use a fresh SSD for file storage, (my RAM 64GB get full quickly as I have a lot of objects to cache)
As said in Varnish Doc I have mounted a tmpfs partition for the working directory :
"The shmlog usually is located in /var/lib/varnish and you can feel free to remove any data found in this directory if needed. Varnish suggests make sure this location is mounted on tmpfs by using /etc/fstab to make sure this gets mounted on server reboot / start."
I have a dedicated 256 GB SSD drive for cache storage.
Do I have to mount it as tmpf with noatime like working dir ?
I did not find any suggestion on how to configure SSD for Varnish needs.
No, you do not have to mount anything special if you're just going to use your SSD. tempfs is specific for a RAM drive only, and if you're not going to take advantage of the superior speed of RAM over SSD, then leaving /var/lib/varnish as is on the default install is good enough.
/var/lib/varnish is used for logging, and Varnish logs a lot of data. Since it uses a circular buffer, size isn't an issue, however the I/O will wear your disks down.
TL;DR: always mount the work directory as tmpfs.
Created a Standard DS1 virtual machine in Azure. It is supposed to have a 7GB SSD disk which is mounted on /mnt.
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 29G 1.5G 27G 6% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 1.7G 8.0K 1.7G 1% /dev
tmpfs 345M 392K 344M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.7G 0 1.7G 0% /run/shm
none 100M 0 100M 0% /run/user
none 64K 0 64K 0% /etc/network/interfaces.dynamic.d
/dev/sdb1 6.8G 16M 6.4G 1% /mnt
However, there is a warning placed into the /mnt location:
ls /mnt
cdrom DATALOSS_WARNING_README.txt lost+found
cat /mnt/DATALOSS_WARNING_README.txt
WARNING: THIS IS A TEMPORARY DISK.
Any data stored on this drive is SUBJECT TO LOSS and THERE IS NO WAY TO
RECOVER IT.
Please do not use this disk for storing any personal or application data.
For additional details to please refer to the MSDN documentation at:
http://msdn.microsoft.com/en-us/library/windowsazure/jj672979.aspx
The link in the description leads to a resource that has been removed.
I've tried rebooting the virtual machine and data on /mnt seems to be persisted between reboots. I need to store persistent data on the SSD, is the warning accurate and can I expect data to be deleted from the /mnt partition? If so, what are my choices to have a persistant SSD disk on Azure virtual machine?
You're looking at the local scratch disk. Yes it's SSD, but it's still a scratch disk. If that SSD crashes, your data is gone in an instant. If you resize your VM, you will likely lose what's on that disk. It's not a permanent-storage disk. If you want permanent storage, you must attach a disk, either from premium storage (SSD) or regular storage). Attached disks, just like the OS disk vhd, are stored in persistent blob storage.
That is a temporary disk:
Most VMs contain a temporary disk, which is not a managed disk. The temporary disk provides short-term storage for applications and processes, and is intended to only store data such as page or swap files. Data on the temporary disk may be lost during a maintenance event or when you redeploy a VM. During a successful standard reboot of the VM, data on the temporary disk will persist. For more information about VMs without temporary disks, see Azure VM sizes with no local temporary disk.
https://learn.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview#temporary-disk
I have just created a linux (Ubuntu 14.4) virtual machine in Azure (SE Asia)
Issue: I only have 29GB not 127GB
It is a Basic Tier, A0 (smallest size)
The advertised disk drive size is 127GB (+20GB tmp)
http://msdn.microsoft.com/en-us/library/azure/dn197896.aspx
I find (after running out of disk space) that I only have around 29GB.
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 29G 24G 4.1G 86% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 323M 12K 323M 1% /dev
tmpfs 68M 388K 67M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 336M 0 336M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sdb1 20G 4.3G 15G 23% /mnt
Running cfdisk shows there is no other free space on the drive.
I can't find any documentation to suggest why only 29GB.
Is this a bug/issue/problem with my VM?
Or is it something to do with linux/ubuntu 14.4/basic tier A0 ?
The VM's operating system drive is backed by a blob in your Azure storage account. The blob is a VHD file. When you created the VM, the appropriate VHD was copied from the gallery into your storage account.
The gallery-provided VHD file has a logical capacity of 30GB by design. The documentation states that the maximum allowed size is 127GB, but that is incidental - the gallery images are 30GB.
The solution is two steps, resize the VHD itself (and corresponding blob), then use Linux tooling to resize the partition. This may help:
Resizing a Windows Azure virtual disk
It's now possible to resize via the Azure UI.
That being said, after resizing, the Ubuntu VM will not see the new size by default, you will see that cfdisk sees the unallocated space, and to do it I would use fdisk like it's explained in this answer.
You need to execute follow Azure PowerShell command:
Update-AzureDisk –DiskName "<Disk name>" -Label "ResiZedOS" -ResizedSizeInGB <Size in GB>
Example:
Update-AzureDisk –DiskName "dimitar-linux-dimitar-linux-os" -Label "ResiZedOS" -ResizedSizeInGB 524
Note: Maximum size is 1023 GB.
Note 2: Virtual machine should be power off.
Documentation:
How to install and configure Azure PowerShell
Update-AzureVM
Update-AzureDisk
Question: How to get disk name ?
Answer: You can use azure cli or azure portal.
azure-cli command:
azure vm disk list <virtual machine name> # In my case: azure vm disk list dimitar-centos
As mentioned in Eron Wright's answer, the disk's size is determined from the Gallery image.
To resize the disk via the Azure Portal web UI:
Shut down (deallocate) the VM.
Open the VM blade and click Disks
Select the OS disk. For me, the size of the disk was blank.
Enter a new Size and click Save.
I tested this with an Standard_D2_V2 sized Ubuntu VM. Once I resized the disk and started the VM, I could access the full size of the disk.
The answer by #danailov work brilliantly for me (Azure Linux VM resized correctly)
Just to add this bit that might help somebody; here is how I found out my disk name via Azure Powershell;
Get-AzureDisk | Get-AzureDisk | Format-list DiskName, AttachedTo, DiskSizeInGB, OS > c:\files\disks.txt
The above command will output a text file at the stated location in your local pc with following details for each disks/VMs in your Azure account - DiskName, AttachedTo, DiskSizeInGB and OS the VM is running.
The Azure VM table...
https://www.windowsazure.com/en-us/pricing/details/#header-2
...says that, for example, a Medium Instance comes with 490GB of local storage. So I was expecting the usual 30GB Azure BLOB OS disk, and then a 490GB /mnt/resource.
But no:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
28G 1.7G 25G 7% /
tmpfs 1.7G 0 1.7G 0% /dev/shm
/dev/sda1 485M 68M 392M 15% /boot
/dev/sdb1 133G 188M 126G 1% /mnt/resource
That's on the CentOS image, but it's the same for other images.
Am i missing something? I don't see the space in a volume group or anything, and there are no sd* devices that aren't mounted.
Neil Mackenzie's answer in the comments appears to be correct
All Windows Azure IaaS VMs come with two disks - an OS disk backed by a VHD
persisted in Windows Azure Blob storage and a temp disk physically
attached to the hosting server. This temp disk is truly ephemeral and
any contents on it will be lost if the VM is deallocated (shutdown on
the portal or API) or moved for server healing. For Linux VMs this
temporary disk /dev/sdb.
I found this link to corroborate:
http://blogs.msdn.com/b/wats/archive/2013/12/07/understanding-the-temporary-drive-on-windows-azure-virtual-machines.aspx