How can I read a file to stream from one blob and upload to another blob? My requirement is to copy a file from one blob to another blob with different file name? In C#
The easiest way to achieve it is using "Azure Storage Data Movement Library" (you can get it thru nuget package).
This is a simple-sample to make the transfer:
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.DataMovement;
using System;
namespace BlobClient
{
class Program
{
static void Main(string[] args)
{
const string storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=juanktest;AccountKey=loHQwke4lSEu1p2W3gg==";
const string container1 = "juankcontainer";
const string sourceBlobName = "test.txt";
const string destBlobName = "newTest.txt";
//Setup Account, blobclient and blobs
CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = account.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference(container1);
blobContainer.CreateIfNotExists();
CloudBlockBlob sourceBlob = blobContainer.GetBlockBlobReference(sourceBlobName);
CloudBlockBlob destinationBlob = blobContainer.GetBlockBlobReference(destBlobName);
//Setup data transfer
TransferContext context = new TransferContext();
Progress<TransferProgress> progress = new Progress<TransferProgress>(
(transferProgress) => {
Console.WriteLine("Bytes uploaded: {0}", transferProgress.BytesTransferred);
});
context.ProgressHandler = progress;
// Start the transfer
try
{
TransferManager.CopyAsync(sourceBlob, destinationBlob,
false /* isServiceCopy */,
null /* options */, context);
}
catch (Exception e)
{
Console.WriteLine("The transfer is cancelled: {0}", e.Message);
}
Console.WriteLine("CloudBlob {0} is copied to {1} ====successfully====",
sourceBlob.Uri.ToString(),
destinationBlob.Uri.ToString());
Console.ReadLine();
}
}
}
Note that "Azure Storage Data Movement Library" is very robust so you can track the transfer progress, cancel the operation or even suspend it to resume it later ;)
One of the easiest ways to copy files is with the AzCopy utility.
I would like to recommend another method other than the above to get this done.
That is azure functions. (serverless compute service).
As a prerequisite, you should have some knowledge in azure functions, creating them, deploying them. 1. What is azure function 2. create an Azure function app
And the following code snippet is the simplest and basic way to perform this action. (In here, when a user uploading a new file to the "demo" blob, the function will be triggered and read that uploaded file from the demo blob and copy to the "output" blob.)
namespace Company.Function{
public static class NamalFirstBlobTrigger
{
[FunctionName("NamalFirstBlobTrigger")]
public static void Run([BlobTrigger("demo/{name}", Connection = "AzureWebJobsStorage")]Stream myBlob,
[Blob("output/testing.cs",FileAccess.Write, Connection = "AzureWebJobsStorage")]Stream outputBlob,
string name,
ILogger log)
{
myBlob.CopyTo(outputBlob);
}
}}
Related
I am not able to download azure blob file to my local disk. Please find below the java code I am using to download the file. When I run and do a HTTP trigger test, I do not see the file downloaded into my Local path. Also, I have given the authentication as public access.And , I am able to read the contents of the txt file using the blob.downloadText(). But I am not able to download it to a file.
My requirement is to download a pdf in the Blob Storage to my Local Disk.
#FunctionName("BlobDownload-Java")
public void run1(
#HttpTrigger(name = "req", methods = {HttpMethod.GET, HttpMethod.POST}, authLevel = AuthorizationLevel.FUNCTION) HttpRequestMessage<Optional<String>> request,
final ExecutionContext context) {
context.getLogger().info("Java HTTP trigger processed a request.");
try {
// Retrieve storage account from connection-string.
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
// Get a reference to a container.
// The container name must be lower case
CloudBlobContainer container = blobClient.getContainerReference("doccontainer");
CloudBlockBlob blob1 = container.getBlockBlobReference("AssembledDoc.pdf");
context.getLogger().info("File Name Check 1 ----->" + blob1.getName());
CloudBlockBlob blob2 = container.getBlockBlobReference("Test.txt");
String s= blob2.downloadText();
context.getLogger().info("Text Document content ----->" + s );
File file = new File("C:\\Users\\wb541348\\Project\\Temp.txt");
blob2.downloadToFile(file.getAbsolutePath());
If you just want to download PDF file to local, here is my test code , you could have a try.
#Test
public void downloadBlob(){
try {
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
//Create the service client object for credentialed access to the Blob service.
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.getContainerReference("blobcontainer");
CloudBlob blob1 =container.getBlockBlobReference("thinking-in-java.pdf");
blob1.download(new FileOutputStream("C:\\Users\\georgec\\Documents\\" + blob1.getName()));
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (InvalidKeyException e) {
e.printStackTrace();
} catch (StorageException e) {
e.printStackTrace();
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
If you still have questions, please let me know.
#Prasanna, Your can't download the blob into your localpath because the pattern of your path is wrong, it should be "c:/Users/wb541348/Project/Temp.txt", there is another point to pay attention, that is the filename is needed, which means that path shouldn't be only folder, hope it benefits.
you can also click this sample of azure blob for java sdk for reference.
I've created a storage account on Azure for uploading files to. I am using KenoUI Async upload widget which allows multiple file uploads. What I'd like to do is have the system create a container based on the passing in Id parameter if it doesn't already exist.
I'm struggling with the actual upload part however, I'm not sure how to pass the enumerable files to the storage account. Here is my code so far:
public ActionResult Async_Save(IEnumerable<HttpPostedFileBase> files, string id)
{
//Connect to Azure
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("my_AzureStorageConnectionString"));
//Create Blob Client
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
//Retrieve a reference to a container
CloudBlobContainer blobContainer = blobClient.GetContainerReference("vehicle_" + id);
try
{
// Create the container if it doesn't already exist
blobContainer.CreateIfNotExists();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.InnerException);
}
foreach (var file in files)
{
//This doesn't seem right to me and it's where I'm struggling
var fileName = Path.GetFileName(file.FileName);
var physicalPath = Path.Combine(blobContainer, fileName);
file.SaveAs(physicalPath);
}
// Return an empty string to signify success
return Content("");
}
What I've attempted to do is create a method that connects to my Azure storage account and:
Check for the existence of a container with the same ID as the
the parameter that's passed in, if it doesn't exist, create it.
Upload the files to the existing or newly created directory and give them a prefix of the ID i.e. "_".
As Gaurav Mantri said above, we will need to use Azure Storage SDK.
For better performance, we can do as below:
Parallel.ForEach(files, file =>
{
CloudBlockBlob blob = blobContainer.GetBlockBlobReference(file.FileName);
blob.UploadFromStream(file.InputStream);
file.InputStream.Close();
});
Your code will be better as below:
public ActionResult Async_Save(List<HttpPostedFileBase> files, string id)
{
// Connect to Azure
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("my_AzureStorageConnectionString"));
try
{
//Create Blob Client
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
//Retrieve a reference to a container
CloudBlobContainer blobContainer = blobClient.GetContainerReference("vehicle_" + id);
// Create the container if it doesn't already exist
blobContainer.CreateIfNotExists();
Parallel.ForEach(files, file =>
{
CloudBlockBlob blob = blobContainer.GetBlockBlobReference(file.FileName);
blob.UploadFromStream(file.InputStream);
file.InputStream.Close();
});
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.InnerException);
}
// Return an empty string to signify success
return new ContentResult();
}
More information about how to use Azure Storage SDK, we can refer to:
Get Quick Start with Storage SDK
To upload a blob in Azure Storage, you will need to use Azure Storage SDK.
Please replace the following code:
foreach (var file in files)
{
//This doesn't seem right to me and it's where I'm struggling
var fileName = Path.GetFileName(file.FileName);
var physicalPath = Path.Combine(blobContainer, fileName);
file.SaveAs(physicalPath);
}
With something like:
foreach (var file in files)
{
var blob = blobContainer.GetBlockBlobReference(file.FileName);
using (var s = file.InputStream)
{
blob.UploadFromStream(s);
}
}
I have the ability to download files from my Azure Blob storage from a web application.
Once I get my Azure Blob storage list back, it's not downloaded.
Error entering download blob.
I would appreciate it if you could look at my code and let me know what the problem is.
public string DownloadBlob()
{
CloudBlobContainer container = GetCloudBlobContainer();
CloudBlockBlob blob = container.GetBlockBlobReference("myBlob");
using (var fileStream = System.IO.File.OpenWrite(#"C:\BlobTest"))
{
blob.DownloadToStream(fileStream);
}
return "success!";
}
I have tested your code and it works in my MVC web application as below.
public ActionResult Download()
{
ContentResult contentResult = new ContentResult();
contentResult.Content = DownloadBlob();
return contentResult;
}
public static string DownloadBlob()
{
try
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=leeliublob;AccountKey=O7xB6ebGq8e86XQSy2vkvSi/x/e9l9FhLqayXcbz1R+E0mIcJ5Wjkly1DsQPYY5dF2JrAVHtBozbJo29ZrrGJA==;EndpointSuffix=core.windows.net");
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = cloudBlobClient.GetContainerReference("mycontainer");
CloudBlockBlob blob = container.GetBlockBlobReference("4.PNG");
using (var fileStream = System.IO.File.OpenWrite(#"C:\Test\BlobTest.PNG"))
{
blob.DownloadToStream(fileStream);
}
return "success!";
}
catch(Exception ex)
{
return ex.StackTrace;
}
}
I think this is the problem about the permission of your MVC application.
Your MVC application may use a virtual account of IIS as below, it may have no access to your C disk.
To solve this problem, here is two ways:
Please try to download to other disk like "E" or "D".
Please add the user group "Authenticated Users" to that file or folder like below.
Update:
I have reproduced your problem as below:
We need to create a folder under root directory of C disk as below, because your web application doesn't have the permission to the root directory of C disk:
I want to upload file in azure blob storage and retrieve that file in my own application.
My Code:
protected void btnUploadtoCloud_Click(object sender, EventArgs e)
{
UploadFile();
}
public string UploadFile()
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("vest");
blobContainer.CreateIfNotExist();
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
containerPermissions.PublicAccess = BlobContainerPublicAccessType.Container;
blobContainer.SetPermissions(containerPermissions);
CloudBlob blobText = blobContainer.GetBlobReference(fupPicture.FileName);
blobText.Properties.ContentType = fupPicture.PostedFile.ContentType;
string blobURL = blobText.Uri.ToString();
blobText.UploadFromStream(fupPicture.FileContent);
imgTest.ImageUrl = blobURL;
return blobURL;
}
But my problem is first line of UploadFile.I include image because I want to show my ref dll too.
.
Why error in RoleEnvironment. May i missing some thing to include.
Note : I have window azure account and I deploy other sites easily.
EDIT
After Gaurav suggestion I try :
string cnn = "<add name=\"testconnectionstring\" DefaultEndpointsProtocol=\"http;AccountName=mystorage;AccountKey=A9Jlajksdh69szut9zOfsxWMD/qLaP3Gy6ID9mIarNmCDasdhlyNlAR9wV0NGjqJUZeM4x6x5bTumLv+1nC51clHDl5GTYYPA==" + " />";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(cnn);
but unable to work I got error invalid combination.
The reason you're getting this error is because RoleEnvironment is only available when your code is running in context of a Cloud Service (Web Role) either in local compute emulator or in cloud.
Since it is a web application and I am guessing that you will be deploying it in an Azure Website, you can simply use AppSettingsReader to read from Web.Config file. Something like:
var rdr = new AppSettingsReader();
string connectionString = (string) rdr.GetValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", typeof(String));
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
If your application is connecting to a cloud storage account, your connection string would look something like:
DefaultEndpointsProtocol=https;AccountName=your-account-name;AccountKey=your-account-key
For connecting to local storage emulator, your connection string would look like:
UseDevelopmentStorage=true
I am coding a MVC5 internet application and would like some help to upload a file from my own filesystem to an Azure Blob.
Here is my Azure upload code function:
public void UploadFileToBlobStorage(string containerName, string blockBlogName, string fileName)
{
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
// Create the container if it doesn't already exist.
container.CreateIfNotExists();
container.SetPermissions(
new BlobContainerPermissions
{
PublicAccess =
BlobContainerPublicAccessType.Blob
});
// Retrieve reference to a blob named "myblob".
CloudBlockBlob blockBlob = container.GetBlockBlobReference(blockBlogName);
// Create or overwrite the "myblob" blob with contents from a local file.
using (var fileStream = System.IO.File.OpenRead(fileName))
{
blockBlob.UploadFromStream(fileStream);
}
}
Here is my function to upload a test file:
public void UploadTestFile(string localFileName)
{
string containerName = "TestContainer";
string blockBlogName = "Test.txt";
AzureService azureService = new AzureService();
azureService.UploadFileToBlobStorage(containerName, blockBlogName, localFileName);
}
I am not sure how to call the UploadTestFile() function from a MVC View where the user can browse to a file to upload.
Do I need to use Ajax, or can I simply upload a file by calling the method from a MVC view? Can I please have some help with this?
Thanks in advance
One way to call your UploadTestFile() function from an MVC View is by using the Html.BeginForm() method. I am including an example below:
#using (Html.BeginForm("UploadTestFile", "INSERT_YOUR_CONTROLLER_NAME_HERE", FormMethod.Post, new { enctype = "multipart/form-data" })) {
<span>
<input type="file" name="myFile" multiple /> <br>
<input type="submit" value="Upload" />
</span>
}
Also, a couple of suggestions on your code:
UploadFileToBlobStorage(): The code checks for container existence and setting permissions on every request. I would recommend separating the container.CreateIfNotExists() and container.SetPermissions(…) logic into a separate initialization function that needs to be executed only once on first deployment.
UploadFileToBlobStorage(): It looks like the code will try to upload the localFileName from the VM file system and not the multi-part form data. One approach would be to use the HttpFileCollectionBase class and the Controller.Request property. Example below:
public void UploadFileToBlobStorage(
string containerName,
string blockBlogName,
HttpFileCollectionBase files)
{
// .....
// Use this:
blockBlob.UploadFromStream(files[0].InputStream);
/* uploading the first file:
you can enumerate thru the files collection
if you are uploading multiple files */
/* Instead of this:
Create or overwrite the "myblob" blob with contents
from a local file. */
using (var fileStream = System.IO.File.OpenRead(fileName))
{
blockBlob.UploadFromStream(fileStream);
}
}
[HttpPost]
public void UploadTestFile()
{
string containerName = "TestContainer";
string blockBlogName = "Test.txt";
AzureService azureService = new AzureService();
// Notice the Request.Files instead of localFileName
azureService.UploadFileToBlobStorage(
containerName, blockBlogName, Request.Files);
}
Please let me know if that works on your end.