I've got some files sitting in an Azure File storage.
I'm trying to programatically archive them to Azure Blobs and I'm not sure how to do this efficiently.
I keep seeing code samples about copying from one blob container to another blob container .. but not from a File to a Blob.
Is it possible to do without downloading the entire File content locally and then uploading this content? Maybe use Uri's or something?
More Info:
The File and Blob containers are in the same Storage account.
The storage account is RA-GRS
Here was some sample code I was thinking of doing .. but it just doesn't feel right :( (pseudo code also .. with validation and checks omitted).
var file = await ShareRootDirectory.GetFileReference(fileName);
using (var stream = new MemoryStream())
{
await file.DownloadToStreamAsync(stream);
// Custom method that basically does:
// 1. GetBlockBlobReference
// 2. UploadFromStreamAsync
await cloudBlob.AddItemAsync("some-container", stream);
}
How to copy an Azure File to an Azure Blob?
We also can use CloudBlockBlob.StartCopy(CloudFile). CloudFile type is also can be accepted by the CloudBlockBlob.StartCopy function.
How to copy CloudFile to blob please refer to document. The following demo code is snippet from the document.
// Parse the connection string for the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
Microsoft.Azure.CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create a CloudFileClient object for credentialed access to File storage.
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
// Create a new file share, if it does not already exist.
CloudFileShare share = fileClient.GetShareReference("sample-share");
share.CreateIfNotExists();
// Create a new file in the root directory.
CloudFile sourceFile = share.GetRootDirectoryReference().GetFileReference("sample-file.txt");
sourceFile.UploadText("A sample file in the root directory.");
// Get a reference to the blob to which the file will be copied.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("sample-container");
container.CreateIfNotExists();
CloudBlockBlob destBlob = container.GetBlockBlobReference("sample-blob.txt");
// Create a SAS for the file that's valid for 24 hours.
// Note that when you are copying a file to a blob, or a blob to a file, you must use a SAS
// to authenticate access to the source object, even if you are copying within the same
// storage account.
string fileSas = sourceFile.GetSharedAccessSignature(new SharedAccessFilePolicy()
{
// Only read permissions are required for the source file.
Permissions = SharedAccessFilePermissions.Read,
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24)
});
// Construct the URI to the source file, including the SAS token.
Uri fileSasUri = new Uri(sourceFile.StorageUri.PrimaryUri.ToString() + fileSas);
// Copy the file to the blob.
destBlob.StartCopy(fileSasUri);
Note:
If you are copying a blob to a file, or a file to a blob, you must use a shared access signature (SAS) to authenticate the source object, even if you are copying within the same storage account.
Use the Transfer Manager:
https://msdn.microsoft.com/en-us/library/azure/microsoft.windowsazure.storage.datamovement.transfermanager_methods.aspx
There are methods to copy from CloudFile to CloudBlob.
Add the "Microsoft.Azure.Storage.DataMovement" nuget package
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.File;
using Microsoft.WindowsAzure.Storage.DataMovement;
private string _storageConnectionString = "your_connection_string_here";
public async Task CopyFileToBlob(string blobContainer, string blobPath, string fileShare, string fileName)
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(_connectionString);
CloudFileShare cloudFileShare = storageAccount.CreateCloudFileClient().GetShareReference(fileShare);
CloudFile source = cloudFileShare.GetRootDirectoryReference().GetFileReference(fileName);
CloudBlobContainer blobContainer = storageAccount.CreateCloudBlobClient().GetContainerReference(blobContainer);
CloudBlockBlob target = blobContainer.GetBlockBlobReference(blobPath);
await TransferManager.CopyAsync(source, target, true);
}
Related
How do I copy a block (or page) blob in azure storage to an Azure file share?
My example code works fine if I download the block blob to a local file, but there does not appear to be a method to download to an Azure file share. I've looked at the Azure data movements library but there's no example of how to do this.
void Main()
{
string myfile =#"Image267.png";
CredentialEntity backupCredentials = Utils.GetBackupsCredentials();
CloudStorageAccount backupAccount = new CloudStorageAccount(new StorageCredentials(backupCredentials.Name, backupCredentials.Key), true);
CloudBlobClient backupClient = backupAccount.CreateCloudBlobClient();
CloudBlobContainer backupContainer = backupClient.GetContainerReference(#"archive");
CloudBlockBlob blob = backupContainer.GetBlockBlobReference(myfile);
CredentialEntity fileCredentails = Utils.GetFileCredentials();
CloudStorageAccount fileAccount = new CloudStorageAccount(new StorageCredentials(fileCredentails.Name,fileCredentails.Key), true);
CloudFileClient fileClient = fileAccount.CreateCloudFileClient();
CloudFileShare share = fileClient.GetShareReference(#"xfer");
if (share.Exists())
{
CloudFileDirectory rootDir = share.GetRootDirectoryReference();
CloudFileDirectory sampleDir = rootDir.GetDirectoryReference("hello");
if (sampleDir.Exists())
{
CloudFile file = sampleDir.GetFileReference(myfile);
// blob.DownloadToFile(file.ToString());
}
}
}
The part that does not work is the commented out line blob.DownloadToFile
Any ideas on how I can do this?
There is an example in the official docs here: it's used to copy files from file share to blob storage, but you can make a little change to copy from blob storage to file share. I also write a sample code which is used copy from blob storage to file share, you can take a look at it as below.
You can use SAS token(for the source blob or source file) to copy files to blob / or copy blob to files, in the same storage account or different storage account.
A sample code as below(copy blob to file in the same storage account, and you can make a little change if they are in different storage account):
static void Main(string[] args)
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("connection string");
var blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = blobClient.GetContainerReference("t0201");
CloudBlockBlob sourceCloudBlockBlob = cloudBlobContainer.GetBlockBlobReference("test.txt");
//Note that if the file share is in a different storage account, you should use CloudStorageAccount storageAccount2 = CloudStorageAccount.Parse("the other storage connection string"), then use storageAccount2 for the file share.
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
CloudFileShare share = fileClient.GetShareReference("testfolder");
CloudFile destFile = share.GetRootDirectoryReference().GetFileReference("test.txt");
//Create a SAS for the source blob
string blobSas = sourceCloudBlockBlob.GetSharedAccessSignature(new SharedAccessBlobPolicy()
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24)
});
Uri blobSasUri = new Uri(sourceCloudBlockBlob.StorageUri.PrimaryUri.ToString()+blobSas);
destFile.StartCopy(blobSasUri);
Console.WriteLine("done now");
Console.ReadLine();
}
it works well at my side, hope it helps.
There are a few samples in here but in dot net. A non code related alternative would be to use Azcopy it would help transfer data from blob to File Shares and vice versa.
External Suggestions:
The following tool called Blobxfer seems to support the transfer but in python.
This is how I try to upload an image to Azure blog storage, then upload an empty file located there.
I try to upload this image here:
CloudStorageAccount storageAccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(Accountname, KeyValue), true);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.GetContainerReference(ContainerValue);
CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);
using (var f = System.IO.File.Open(model.FileToUpload.FileName, FileMode.Create))
{
await blockBlob.UploadFromStreamAsync(f);
}
But if i try to open this image say this.
From danish to english: We can not open this file
As Marco said, FileMode.Create specifies that the operating system should create a new file. If the file already exists, it will be overwritten.
For more details, you could refer to this article.
According to the pictures you provided, your blob size is 0 B. So, you would always couldn't find the file and open it.
FileMode specifies how the operating system should open a file.
So I suggest that you could delete it and use OpenRead to open an existing file for reading. You could refer to the code as below:
using (var f = System.IO.File.OpenRead(model.FileToUpload.FileName))
{
await blockBlob.UploadFromStreamAsync(f);
}
This is how I try to upload an image to Azure blog storage, then upload an empty file located there.
I try to upload this image here:
CloudStorageAccount storageAccount = new CloudStorageAccount(new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(Accountname, KeyValue), true);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.GetContainerReference(ContainerValue);
CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);
using (var f = System.IO.File.Open(model.FileToUpload.FileName, FileMode.Create))
{
await blockBlob.UploadFromStreamAsync(f);
}
But if i try to open this image say this.
From danish to english: We can not open this file
As Marco said, FileMode.Create specifies that the operating system should create a new file. If the file already exists, it will be overwritten.
For more details, you could refer to this article.
According to the pictures you provided, your blob size is 0 B. So, you would always couldn't find the file and open it.
FileMode specifies how the operating system should open a file.
So I suggest that you could delete it and use OpenRead to open an existing file for reading. You could refer to the code as below:
using (var f = System.IO.File.OpenRead(model.FileToUpload.FileName))
{
await blockBlob.UploadFromStreamAsync(f);
}
I have rather foolishly uploaded a vhd to Azure file storage thinking I can create a virtual machine from it only to find out it really needs to be in blob storage.
I know I can just upload it again - but it is very large and my upload speed is very slow.
My question is - can I move a file from file storage to blob storage without downloading/uploading again? I.e. is there anything in the Azure portal UI to do it, or even a PowerShell command?
You can try AzCopy:
AzCopy.exe /Source:{*URL to source container*} /Dest:{*URL to dest container*} /SourceKey:{*key1*} /DestKey:{*key2*} /S
When copying from File Storage to Blob Storage, the default blob type is block blob, user can specify option /BlobType:page to change the destination blob type.
AzCopy by default copies data between two storage endpoints asynchronously. Therefore, the copy operation will run in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob will be copied, and AzCopy will periodically check the copy status until the copying is completed or failed. The /SyncCopy option ensures that the copy operation will get consistent speed.
In c#:
public static CloudFile GetFileReference(CloudFileDirectory parent, string path)
{
var filename = Path.GetFileName(path);
var fullPath = Path.GetDirectoryName(path);
if (fullPath == string.Empty)
{
return parent.GetFileReference(filename);
}
var dirReference = GetDirectoryReference(parent, fullPath);
return dirReference.GetFileReference(filename);
}
public static CloudFileDirectory GetDirectoryReference(CloudFileDirectory parent, string path)
{
if (path.Contains(#"\"))
{
var paths = path.Split('\\');
return GetDirectoryReference(parent.GetDirectoryReference(paths.First()), string.Join(#"\", paths.Skip(1)));
}
else
{
return parent.GetDirectoryReference(path);
}
}
The code to copy:
// Source File Storage
string azureStorageAccountName = "shareName";
string azureStorageAccountKey = "XXXXX";
string name = "midrive";
CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials(azureStorageAccountName, azureStorageAccountKey), true);
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
CloudFileShare fileShare = fileClient.GetShareReference(name);
CloudFileDirectory directorio = fileShare.GetRootDirectoryReference();
CloudFile cloudFile = GetFileReference(directorio, "SourceFolder\\fileName.pdf");
// Destination Blob
string destAzureStorageAccountName = "xx";
string destAzureStorageAccountKey = "xxxx";
CloudStorageAccount destStorageAccount = new CloudStorageAccount(new StorageCredentials(destAzureStorageAccountName, destAzureStorageAccountKey), true);
CloudBlobClient destClient = destStorageAccount.CreateCloudBlobClient();
CloudBlobContainer destContainer = destClient.GetContainerReference("containerName");
CloudBlockBlob destBlob = destContainer.GetBlockBlobReference("fileName.pdf");
// copy
await TransferManager.CopyAsync(cloudFile, destBlob, true);
Another option is to use Azure CLI...
az storage copy -s /path/to/file.txt -d https://[account].blob.core.windows.net/[container]/[path/to/blob]
More info here: az storage copy
Thanks to Gaurav Mantri for pointing me in the direction of AzCopy.
This does allow me to copy between file and blob storage using the command:
AzCopy.exe /Source:*URL to source container* /Dest:*URL to dest container* /SourceKey:*key1* /DestKey:*key2* /S
However as Gaurav also rightly points out in the comment the resulting blob will be of type Block Blob and this is no good for me. I need one of type Page Blob in order to create a VM out of it using https://github.com/Azure/azure-quickstart-templates/tree/master/201-vm-specialized-vhd
There is no way to change the blob type as far as I can see once it is up there in the cloud, so it looks like my only option is to wait for a lengthy upload again.
I have created container on azure storage and with code below made it work to upload my files(blobs) in container. Now it says upload was successful however I can't figure out how to reach those files to download it. There is no documentation on the internet about it.
Any help would be appreciated to solve it.
// create Azure Storage
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=bireddy;AccountKey=8OMbjBeJR+SIYaSt0YtBUzivLKPX5ZbsGJeEY9vsX0BPbX3uy9KxOckK7LuLeH3ZbOh+NoEaiEIV/NWvZbFOrA==");
// create a blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// create a container
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
// make it public
container.SetPermissions(
new BlobContainerPermissions {
PublicAccess = BlobContainerPublicAccessType.Container
});
// create a block blob
CloudBlockBlob blockBlob = container.GetBlockBlobReference(FileUpload1.FileName);
// upload to Azure Storage
// this has to be changed sometime later
blockBlob.Properties.ContentType = FileUpload1.PostedFile.ContentType;
blockBlob.UploadFromStream(FileUpload1.FileContent);
Grateful to your discussion.
There are a number of options to download your blobs:
Using PowerShell: See the Azure Storage Powershell guide here:
http://azure.microsoft.com/en-us/documentation/articles/storage-powershell-guide-full/
Using Command Line: See the documentation for the AzCopy tool here:
http://aka.ms/azcopy
Using your C# code: There is a guide here, that includes a section on how to download blobs:
http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/
Using GUI-based explorers: There are a number of third-party explorers, some of them are listed here:
http://blogs.msdn.com/b/windowsazurestorage/archive/2014/03/11/windows-azure-storage-explorers-2014.aspx
// Parse the connection string and return a reference to the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
//Create the Blob service client
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
// Retrieve reference to a blob named
CloudBlockBlob blockBlob2 = container.GetBlockBlobReference(document);
//using (var np = File.Open(#"C:\mydocuments\"+ document, FileMode.Create))
// blockBlob2.DownloadToStream(np, null, options, null);
byte[] fileBytes;
using (var fileStream = new MemoryStream())
{
blockBlob2.DownloadToStream(fileStream, null, options, null);
fileBytes = fileStream.ToArray();
}
return fileBytes;