We have very slow connection and very small hard disk, How can I create 1 TB VHD for cloud drive on azure?
Do you need to upload an existing VHD or you just need a 1 TB Azure drive for your application in the cloud? If it is former, Rinat is probably right. Look at this blog post for how to write a console app: blogs.msdn.com/b/windowsazurestorage/archive/2010/04/11/using-windows-azure-page-blobs-and-how-to-efficiently-upload-and-download-page-blobs.aspx.
However if you just need 1 TB Azure drive for your application, you can just create one using your code running in the cloud. You can write code similar to what I have written below:
string pageBlobName = "testpageblob";// Guid.NewGuid().ToString();
string blobUri = string.Format("{0}/{1}", blobContainer.Uri.AbsoluteUri,
pageBlobName);
CloudDrive cloudDrive = new CloudDrive(new Uri(blobUri), csa.Credentials);
for (int i = 0; i < 30; i++)
{
try
{
cloudDrive.Create(20);
break;
}
catch (CloudDriveException ex)
{
if (!ex.Message.Equals("ERROR_UNSUPPORTED_OS") || i == 29)
throw;
Thread.Sleep(10000);
}
}
string driveLetter = cloudDrive.Mount(25, DriveMountOptions.Force);
For efficient uploads you can write a simple console app that will upload your VHD to Azure Blob Storage as page blobs, transferring only non-zero bytes.
CSUpload does exactly that: uploads nonzero pages only.
You can create locally a (small) 1TB dynamic VHD and then upload with CSUpload (expanding to fixed size on-the-fly, without sending zero bytes). See also this question about having an 1TB disk in Azure.
you can also create a vm-disk and attach an vm to it and prepare it from there
then afterwards remove the disk from the vm(this will not remove it from storage) and mount it with cloudDrive
(you can delete the vm afterwards)
1TB is a extremely large size for vhd, however a brand new vhd only contain very little data(512 bytes for fixed hard disk image) at the end of file. In other words, the most data of your vhd is zero and there is no need to upload the entire 1TB data to azure storage.
For vhd file format, please refer to http://en.wikipedia.org/wiki/VHD_(file_format)
If you are familiar with vhd and azure storage client, you can write your own application to do these things.
Create an empty 1TB page blob on azure storage with PutBlob operation
http://msdn.microsoft.com/en-us/library/windowsazure/dd179451.aspx
Write the VHD footer to end of the Page Blob with PutPage operation
http://msdn.microsoft.com/en-us/library/windowsazure/ee691975.aspx
If not, you can use diskmgmt.msc(Disk Management) to create a vhd at first, and then use Set-AzureStorageBlobContent in windows azure powershell to efficiently upload your vhd.
Set-AzureStorageBlobContent will skip the useless data when uploading page blob.
Set-AzureStorageBlobContent -Container upload -File .\a.vhd -BlobType Page
For reference,
http://msdn.microsoft.com/en-us/library/dn408487.aspx
Related
The challenge is to have a way to copy osdisk and datadisk from vm located in first location, to another one and of course spawn there a new virtual machine.
So far I found disks-upload-vhd-to-managed-disk-cli article and was able to copy disk between different location by using azcopy utility and creating sas uri links.
As I use terraform everywhere I don't like to use external tools for such job.
I try already abuse azurerm_managed_disk to be able make a copy of my disk to another location but it seams that it's not possible, and those disk need to be in the same place.
So maybe some of you have idea how to make such copy of the disks (or entire vm) in different location just by terraform way and of course I don't mean here to use local-exec to wrap azcopy in it :)
Best Regards.
To copy the managed disk to another region, except the AzCopy command, you can only copy the disk with a generated SAS URL into a storage page blob in another region. And then create a managed disk from the VHD file in that storage blob.
The steps here:
export the disk that you want to copy and then it will generate a SAS URL;
create a storage page blob in the storage container which in the target region from the SAS URL;
create a managed disk from the page blob in the same region.
Question background:
This may be a simple question but I cant find an answer to it. I've just started using Azure storage (for storing images) and want to know if one 'blob' holds a maximum of one file?
This is my container called fmfcpics:
Within the container I have a block blob named myBlob and within this I have one image:
Through the following code, if I upload another image file to the myBlob block blob then it overwrites the image already in there:
CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob");
using (var fileStream = System.IO.File.OpenRead(#"C:\Users\Me\Pictures\Image1.jpg"))
{
blockBlob.UploadFromStream(fileStream);
}
Is this overwriting correct? Or should I be able to store multiple files at the myBlob?
Each blob is a completely separate entity, direct-addressable via uri:
http(s)://storageaccountname.blob.core.windows.net/containername/blobname
If you want to manage multiple entities (such as image jpg's in your case), you would upload each one to a separate blob name (and you're free to store as many as you want within a single container, and you may have as many containers as you want).
Note: These are block blobs. There are also page blobs that have random-access capability, and this is the basis for vhd storage (and in that case, the vhd would have a formatted file system within it, with multiple files).
In blob Azure Documentation you understand how blob service works and the concepts about this storage service.
In some minutes you can use the service easily
I have uploaded successfully my 1TB vhd (not containing Windows files) to Azure storage.
Now I want to attach it as a second drive to my virtual machine but in the attach list I can find only the "attach an empty disk" option!
I used Add-AzureVhd to upload the vhd file:
Creating new page blob of size 999653638656...
I linked the storage resource in Cloud Service but the vhd is still not available to mount.
The container of the storage where I uploaded my vhd is the same with the one where C: drive of my VM is saved.
The container access is set to private.
Will it help if I change it to Public Blob or Public Container?
What else to try?
Thanks
Take a look at the PowerShell command Add-AzureDataDisk. This should be what you're looking for, as you can specify media location of the uploaded vhd.
Alternatively, in the portal, go to Virtual Machines and navigate to the Disks tab, where you can create a new disk:
At this point, you can navigate to your uploaded vhd:
After this is done, the new disk should become available for you to add to a Virtual Machine.
It should show the options to attach (Empty Disk and Existing Disk) as show in this link from Azure documentation.
Assuming the above not possible for what ever may be the reason, the alternative is
As the you already claim you are able to see the Attach Empty Disk; you can attach a 1 TB disk and download and put the blob contents there.
You wont be charged for the out-bandwidth as it is all internal
Make sure you used CSUpload and not just pushed the VHD to blob storage. See: http://msdn.microsoft.com/en-us/library/windowsazure/gg466228.aspx
When I attempt to delete a blob from my storage account container, I get an error message, "There is currently a lease on the blob and no lease ID was specified in the request."
I have 4 virtual machine instances. I also have 8 virtual machine disks, 4 of which are in use (one by each of the virtual machine instances). Strangely, I have 10 blobs listed in my single storage account's lone container, called vhds. Here is a screenshot of the 10 blobs, highlighting the two that I cannot delete.
Can anyone give me guidance on how to delete these blobs? I have no use for them and I'd like to cut down on my storage costs for my subscription.
You need to delete the disks from the Virtual Machines section of the portal.
Navigate to Virtual Machines -> Disks
Delete the disks
Check this MSDN blog post for the complete instructions:
http://blogs.msdn.com/b/windows_azure_technical_support_wats_team/archive/2013/02/05/iaas-unable-to-delete-vhd-there-is-currently-a-lease-on-the-blob.aspx
Alternatively, you can just kill the lease on the Blobs with PowerShell:
(Get-AzureRmStorageAccount -Name "STORAGE_ACCOUNT_NAME" | Get-AzureStorageBlob -name "CONTAINER_NAME").ICloudBlob.BreakLease()
Just realize when you do this, the VM's that use this storage will not be able to turn on. (And you should turn them off if they aren't already before you do this.
However, if you might use the VM's again in the future this technique allows you to:
Stop the VM in question.
Download a copy of the VHD.
Release the lease on the VHD
Delete the VHD in the storage account.
Insert arbitrary time period where you don't need the VM
Upload the VHD to the same storage account with the same container and same file name.
Start the VM back up and have it work :-).
There is an alternate (easier) way to break a lease if you use (or download) Microsoft Azure Storage Explorer (a really cool tool to manage Azure Storage).
You can browse to the Storage Account and find the relevant file (vhd) and then select the Break Lease option.
The same CAUTIONS above apply and the Explorer tool makes these clear.
You should have images associated to your VMs. Even if you have deleted your VMs, the images have to be explicitly deleted.
Once the images are deleted, you should see VHD getting cleared as well
I would like to copy a very large storage container from one Azure storage account into another (which also happens to be in another subscription).
I would like an opinion on the following options:
Write a tool that would connect to both storage accounts and copy blobs one at a time using CloudBlob's DownloadToStream() and UploadFromStream(). This seems to be the worst option because it will incur costs when transferring the data and also be quite slow because data will have to come down to the machine running the tool and then get re-uploaded back to Azure.
Write a worker role to do the same - this should theoretically be faster and not incur any cost. However, this is more work.
Upload the tool to a running instance bypassing the worker role deployment and pray the tool finishes before the instance gets recycled/reset.
Use an existing tool - have not found anything interesting.
Any suggestions on the approach?
Update: I just found out that this functionality has finally been introduced (REST APIs only for now) for all storage accounts created on July 7th, 2012 or later:
http://msdn.microsoft.com/en-us/library/windowsazure/dd894037.aspx
You can also use AzCopy that is part of the Azure SDK.
Just click the download button for Windows Azure SDK and choose WindowsAzureStorageTools.msi from the list to download AzCopy.
After installing, you'll find AzCopy.exe here: %PROGRAMFILES(X86)%\Microsoft SDKs\Windows Azure\AzCopy
You can get more information on using AzCopy in this blog post: AzCopy – Using Cross Account Copy Blob
As well, you could remote desktop into an instance and use this utility for the transfer.
Update:
You can also copy blob data between storage accounts using Microsoft Azure Storage Explorer as well. Reference link
Since there's no direct way to migrate data from one storage account to another, you'd need to do something like what you were thinking. If this is within the same data center, option #2 is the best bet, and will be the fastest (especially if you use an XL instance, giving you more network bandwidth).
As far as complexity, it's no more difficult to create this code in a worker role than it would be with a local application. Just run this code from your worker role's Run() method.
To make things more robust, you could list the blobs in your containers, then place specific file-move request messages into an Azure queue (and optimize by putting more than one object name per message). Then use a worker role thread to read from the queue and process objects. Even if your role is recycled, at worst you'd reprocess one message. For performance increase, you could then scale to multiple worker role instances. Once the transfer is complete, you simply tear down the deployment.
UPDATE - On June 12, 2012, the Windows Azure Storage API was updated, and now allows cross-account blob copy. See this blog post for all the details.
here is some code that leverages the .NET SDK for Azure available at http://www.windowsazure.com/en-us/develop/net
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.WindowsAzure.StorageClient;
using System.IO;
using System.Net;
namespace benjguinAzureStorageTool
{
class Program
{
private static Context context = new Context();
static void Main(string[] args)
{
try
{
string usage = string.Format("Possible Usages:\n"
+ "benjguinAzureStorageTool CopyContainer account1SourceContainer account2SourceContainer account1Name account1Key account2Name account2Key\n"
);
if (args.Length < 1)
throw new ApplicationException(usage);
int p = 1;
switch (args[0])
{
case "CopyContainer":
if (args.Length != 7) throw new ApplicationException(usage);
context.Storage1Container = args[p++];
context.Storage2Container = args[p++];
context.Storage1Name = args[p++];
context.Storage1Key = args[p++];
context.Storage2Name = args[p++];
context.Storage2Key = args[p++];
CopyContainer();
break;
default:
throw new ApplicationException(usage);
}
Console.BackgroundColor = ConsoleColor.Black;
Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine("OK");
Console.ResetColor();
}
catch (Exception ex)
{
Console.WriteLine();
Console.BackgroundColor = ConsoleColor.Black;
Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine("Exception: {0}", ex.Message);
Console.ResetColor();
Console.WriteLine("Details: {0}", ex);
}
}
private static void CopyContainer()
{
CloudBlobContainer container1Reference = context.CloudBlobClient1.GetContainerReference(context.Storage1Container);
CloudBlobContainer container2Reference = context.CloudBlobClient2.GetContainerReference(context.Storage2Container);
if (container2Reference.CreateIfNotExist())
{
Console.WriteLine("Created destination container {0}. Permissions will also be copied.", context.Storage2Container);
container2Reference.SetPermissions(container1Reference.GetPermissions());
}
else
{
Console.WriteLine("destination container {0} already exists. Permissions won't be changed.", context.Storage2Container);
}
foreach (var b in container1Reference.ListBlobs(
new BlobRequestOptions(context.DefaultBlobRequestOptions)
{ UseFlatBlobListing = true, BlobListingDetails = BlobListingDetails.All }))
{
var sourceBlobReference = context.CloudBlobClient1.GetBlobReference(b.Uri.AbsoluteUri);
var targetBlobReference = container2Reference.GetBlobReference(sourceBlobReference.Name);
Console.WriteLine("Copying {0}\n to\n{1}",
sourceBlobReference.Uri.AbsoluteUri,
targetBlobReference.Uri.AbsoluteUri);
using (Stream targetStream = targetBlobReference.OpenWrite(context.DefaultBlobRequestOptions))
{
sourceBlobReference.DownloadToStream(targetStream, context.DefaultBlobRequestOptions);
}
}
}
}
}
Its very simple with AzCopy. Download latest version from https://azure.microsoft.com/en-us/documentation/articles/storage-use-azcopy/
and in azcopy type:
Copy a blob within a storage account:
AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer1 /Dest:https://myaccount.blob.core.windows.net/mycontainer2 /SourceKey:key /DestKey:key /Pattern:abc.txt
Copy a blob across storage accounts:
AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1 /Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt
Copy a blob from the secondary region
If your storage account has read-access geo-redundant storage enabled, then you can copy data from the secondary region.
Copy a blob to the primary account from the secondary:
AzCopy /Source:https://myaccount1-secondary.blob.core.windows.net/mynewcontainer1 /Dest:https://myaccount2.blob.core.windows.net/mynewcontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt
I'm a Microsoft Technical Evangelist and I have developed a sample and free tool (no support/no guarantee) to help in these scenarios.
The binaries and source-code are available here: https://blobtransferutility.codeplex.com/
The Blob Transfer Utility is a GUI tool to upload and download thousands of small/large files to/from Windows Azure Blob Storage.
Features:
Create batches to upload/download
Set the Content-Type
Transfer files in parallel
Split large files in smaller parts that are transferred in parallel
The 1st and 3rd feature is the answer to your problem.
You can learn from the sample code how I did it, or you can simply run the tool and do what you need to do.
Write your tool as a simple .NET Command Line or Win Forms application.
Create and deploy a dummy we/worker role with RDP enabled
Login to the machine via RDP
Copy your tool over the RDP connection
Run the tool on the remote machine
Delete the deployed role.
Like you I am not aware of any of the off the shelf tools supporting a copy between function.
You may like to consider just installing Cloud Storage Studio into the role though and dumping to disk then re-uploading. http://cerebrata.com/Products/CloudStorageStudiov2/Details.aspx?t1=0&t2=7
Use could 'Azure Storage Explorer' (free) or some other such tool. These tools provide a way to download and upload content. You will need to manually create containers and tables - and of course this will incur a transfer cost - but if you are short on time and your contents are of reasonable size then this is a viable option.
I recommend use azcopy, you can copy the all the storage account, a container, a directory or a single blob. Here al example of cloning all the storage account:
azcopy copy 'https://{SOURCE_ACCOUNT}.blob.core.windows.net{SOURCE_SAS_TOKEN}' 'https://{DESTINATION_ACCOUNT}.blob.core.windows.net{DESTINATION_SAS_TOKEN}' --recursive
You can get SAS token from Azure Portal. Navigate to storage account overviews (source and destination), then in the sidenav click on "Shared access sigantura" and generate your own.
More examples here
I had to do somethign similar to move 600 GB of content from a local file system to Azure Storage. After a couple iterations of code I finally ended up with taking the 'Azure Storage Explorer' and extended it with ability to select folders instead of just files and then have it recursively drill into the multiple selected folders, loaded a list of Source / Destination copy item statements into an Azure Queue. Then in the upload section in 'Azure Storage Explorer', in the Queue section to pull from the queue and execute the copy operation.
Then I launched like 10 instances of the 'Azure Storage Explorer' tool and had each pulling from the queue and executing the copy operation. I was able to move the 600 GB of items in just over 2 days. Added in smarts to utilize the modified time stamps on files and have it skip over files that have already been both copied from the queue and not add to the queue if it is in sync. Now I can run "updates" or syncs within an hour or two across the whole library of content.
Try CloudBerry Explorer. It copies blob within and between subscriptions.
For copying between subscriptions, edit the storage account container's access from Private to Public Blob.
The copying process took few hours to complete. If you choose to reboot your machine, the process will continue. Check status by refreshing the target storage account container in Azure management UI by checking the timestamp, the value gets updated until the copy process completes.