How can I dynamically create Ziparchive in Azure - azure

I have multiple files in my Azure storage account say its Master container, I have created a dynamic container which will have the required files copied from Master Container, and those coped files needed to be zipped inside that created container. See below code, where in I have created the zip archive. Also the zip archive is getting created, but when I download manually and see the files, it seems corrupted (for ex. the actual size of the individual files are more than 1Mb but the files which I get after download seems 22Kb), and File formats would be .ipt, .iam (Autodesk Inventor Files)
CloudBlobContainer destContainer = blobClient.GetContainerReference(AzureContainer);
bool isCreated = destContainer.CreateIfNotExists();
var zipblob = destContainer.GetBlockBlobReference("inputAssembly.zip");
using (var stream = await zipblob.OpenWriteAsync())
using (var zip = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var fileName in inputfile)
{
using (var fileStream = new MemoryStream())
{
if (destContainer.GetBlockBlobReference(fileName).Exists())
{
destContainer.GetBlockBlobReference(fileName).DownloadToStream(fileStream);
}
var newZip = new ZipArchive(fileStream, ZipArchiveMode.Create);
var entry = newZip.CreateEntry(fileName, CompressionLevel.NoCompression);
using (var innerFile = entry.Open())
{
fileStream.CopyTo(innerFile);
}
fileStream.Close();
}
}
}

CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(fixedpartContainer);
CloudBlobContainer destContainer1 = blobClient.GetContainerReference(AzureContainer);
bool isCreated = destContainer1.CreateIfNotExists();
var zipblob = destContainer1.GetBlockBlobReference("inputAssembly.zip");
using (var stream = await zipblob.OpenWriteAsync())
{
using (var zipArchive = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var blobName in blobFileNames)
{
var blob = destContainer.GetBlockBlobClient(blobName);
var zipEntry = zipArchive.CreateEntry(blobName);
using var zipStream = zipEntry.Open();
using var fileStream = new MemoryStream();
await blob.DownloadToAsync(fileStream);
await zipStream.WriteAsync(fileStream.ToArray());
AssemblyCreated = true;
}
}
}

Related

Migrate from Azure to SharePoint using Migration API - "Must specify IV in Manifest or blob metadata'"

I am trying to use the Migration API with C# to migrate from the default free Azure Cloud (given to all sharepoint accounts) to SharePoint.
I can successfully generate a cloud container, and upload files, but the migration fails. Here is my code:
private void MigrateToAzureThenSharePoint()
{
//Provision The default Azure Containers and Migation Queue
var migrationContainers = destinationCtx.Site.ProvisionMigrationContainers();
var migrationQueue = destinationCtx.Site.ProvisionMigrationQueue();
destinationCtx.ExecuteQuery();
var containerInfoList = migrationContainers.Value;
var dataContainerUri = containerInfoList.DataContainerUri;
var metadataContainerUri = containerInfoList.MetadataContainerUri;
var queueInfo = migrationQueue.Value;
//Instansiate Cloud Queue
var cloudQueue = new CloudQueue(new Uri(queueInfo.JobQueueUri));
//Instansiate CloudBlobContainer
CloudBlobContainer dataContainer = new CloudBlobContainer(new Uri(dataContainerUri));
CloudBlobContainer manifestContainer = new CloudBlobContainer(new Uri(metadataContainerUri));
//Test files to upload
var testfiles = new[]
{
new SourceFile
{
Filename = "test.txt",
LastModified = DateTime.Now,
Contents = Encoding.UTF8.GetBytes("Hi, this is a test text-file"),
Title = "Title of file 1"
},
new SourceFile
{
Filename = "test2.txt",
LastModified = DateTime.Now.AddDays(-1),
Contents = Encoding.UTF8.GetBytes("Tesfile2"),
Title = "Second title"
}
};
//Upload Test Files to Cloud
foreach (var testfile in testfiles)
{
var blobReference = dataContainer.GetBlockBlobReference(testfile.Filename);
blobReference.UploadFromByteArray(testfile.Contents, 0, testfile.Contents.Length, null);
blobReference.CreateSnapshot();
}
var manifestPackage = new ManifestPackage(destOrganisationList.Id,
destinationCtx.Web.Id,
destinationCtx.Web.Title,
destOrganisationList.Title,
#"\",
destOrganisationList.RootFolder.UniqueId,
destOrganisationList.RootFolder.ParentFolder.UniqueId
);
//Create XML Files for Azure Migration (An easier way here would be great? I manualy write them)
var filesInManifestPackage = manifestPackage.GetManifestPackageFiles(testfiles);
//Uploaded manually created XML files
foreach (var migrationPackageFile in filesInManifestPackage)
{
var blobReference = manifestContainer.GetBlockBlobReference(migrationPackageFile.Filename);
blobReference.UploadFromByteArray(migrationPackageFile.Contents, 0, migrationPackageFile.Contents.Length, null);
blobReference.CreateSnapshot();
}
//Start Migration
var result = destinationCtx.Site.CreateMigrationJobEncrypted(destinationCtx.Web.Id,
containerInfoList.DataContainerUri,
containerInfoList.MetadataContainerUri,
queueInfo.JobQueueUri,
new EncryptionOption()
{
AES256CBCKey = containerInfoList.EncryptionKey
});
destinationCtx.ExecuteQuery();
//Check for messags
while (true) {
var msg = cloudQueue.GetMessage();
if (msg == null)
{
Task.Delay(TimeSpan.FromSeconds(1));
continue;
}
var message = JsonConvert.DeserializeObject<EncryptedMessage>(msg.AsString);
cloudQueue.DeleteMessage(msg);
//Decode encrypted message
var decoded = JsonConvert.DeserializeObject<UpdateMessage>(Decrypt(message.Content, containerInfoList.EncryptionKey, message.IV));
//Error
}
}
There are a few errors, but they all are similar to this:
"Unable to download SystemData.xml with exception 'Must specify IV in
Manifest or blob metadata'"
I believe I may have to encrypt the files when I upload from this method:
blobReference.UploadFromByteArray
There is an option:
var blobRequestOptions = new BlobRequestOptions();
blobRequestOptions.EncryptionPolicy =
But I don't know how to instantiate the encryption policy (and I may be going down a rabbit hole)
Any help would be greatly appreciated, as documentation online is shocking minimal.

Zipping files and uploading to Azure Blobstorage using filestream

I am creating a website with ASP.NET Core, and I need to take a bunch of files, zip them together and then upload them to Azure Blobstorage.
It succeeds, I get a downloadlink but when I download the file, I just get a "File" that I cannot open with anything.
Here's the code that handles the uploading:
var moduleVersionReleaseFiles = _appDbContext
.ModuleVersionReleaseFile
.Where(x => x.ModuleVersionReleaseId == moduleVersionReleaseId)
.ToList();
string AzureBlobStorageConfig = _configuration["AzureBlobStorage"];
var connectionString = _configuration["StorageConnectionString"];
string containerName = moduleVersionReleaseId + "-" + Guid.NewGuid().ToString();
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName.ToLowerInvariant(), Azure.Storage.Blobs.Models.PublicAccessType.BlobContainer);
BlobClient blobClient = containerClient.GetBlobClient(moduleVersionReleaseId.ToString());
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var moduleVersionReleaseFile in moduleVersionReleaseFiles)
{
var entry = archive.CreateEntry(moduleVersionReleaseFile.File);
using (var entryStream = entry.Open())
using (var fileStream = File.OpenRead($"{moduleVersionReleaseFile.Path}\\{moduleVersionReleaseFile.File}"))
{
await fileStream.CopyToAsync(entryStream);
}
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);
await stream.FlushAsync();
}
return blobClient.Uri.ToString();
I am using Azurite for testing, so I can't log onto the blobstorage and check there.
Why am I not getting a zip file when downloading the file as expected?
The reason you're not able to open the zip file is because it is not getting saved properly. Can you please try with the following code:
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var moduleVersionReleaseFile in moduleVersionReleaseFiles)
{
var entry = archive.CreateEntry(moduleVersionReleaseFile.File);
using (var entryStream = entry.Open())
var fileContents = File.ReadAllBytes($"{moduleVersionReleaseFile.Path}\\{moduleVersionReleaseFile.File}"));
using (BinaryWriter zipFileBinary = new BinaryWriter(entryStream))
{
zipFileBinary.Write(fileContents);
}
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);
}

How to unzip file in Azure File Share using Azure Function?

I have an Azure Storage Account with Azure File share. I want to extract zip archive file to another dir in file share using Azure functions. I wrote this code in C#:
CloudFileDirectory rootDirectory = cloudFileShare.GetRootDirectoryReference();
CloudFileDirectory output = rootDirectory.GetDirectoryReference("output");
CloudFile cloudFile = input.GetFileReference("archive1.zip");
using (var stream = await cloudFile.OpenReadAsync())
{
var file1 = new ZipArchive(stream);
foreach (var zipEntry in file1.Entries)
{
var file2 = output.GetFileReference(zipEntry.Name);
var fileStream = zipEntry.Open();
await file2.UploadFromStreamAsync(fileStream); //error is in this line
}
}
But I got the error:
System.Private.CoreLib: Exception while executing function: HttpTriggerExtract. Microsoft.WindowsAzure.Storage:
Operation is not valid due to the current state of the object.
How to fix this?
Edit: I fix the error using MemoryStream in addition, this code works:
foreach (var zipEntry in file1.Entries) {
var fsz = output.GetFileReference(zipEntry.Name);
using (var ms = new MemoryStream())
{
using (var fileStream = zipEntry.Open())
{
await fileStream.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
await fsz.UploadFromStreamAsync(ms);
}
}
Regarding the issue, please refer to the following code (I use package WindowsAzure.Storage 9.3.1 to do that)
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudFileClient cloudFileClient = storageAccount.CreateCloudFileClient();
CloudFileShare cloudFileShare = cloudFileClient.GetShareReference("share2");
CloudFileDirectory rootDirectory = cloudFileShare.GetRootDirectoryReference();
CloudFileDirectory input = rootDirectory.GetDirectoryReference("input");
CloudFileDirectory output = rootDirectory.GetDirectoryReference("output");
CloudFile cloudFile = input.GetFileReference("sample.zip");
using (var stream = await cloudFile.OpenReadAsync())
using (var zipArchive = new ZipArchive(stream)) {
foreach (var entry in zipArchive.Entries)
{
if (entry.Length > 0) {
CloudFile extractedFile = output.GetFileReference(entry.Name);
using (var entryStream = entry.Open())
{
byte[] buffer = new byte[16 * 1024];
using (var ms = await extractedFile.OpenWriteAsync(entry.Length))
{
int read;
while ((read = await entryStream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
}
}
}
}
}
The above answer helped me for my problem.
With the new Azure Library (12.7.0) you have to code this way:
string srcDir = #"sourcePath";
string destDir = #"sourcePath\testStorageUnzip";
string srcFileName = "AzureStorageZip.zip";
string azureConnectionString = ConfigurationManager.AppSettings["beecloudfileshare_AzureStorageConnectionString"];
StorageSharedKeyCredential credential = BeeFileManager.GetAzureStorageKeyCredential(azureConnectionString);
Uri srcUri = new Uri("https:" + Path.Combine(srcDir, srcFileName).Replace("\\", "/"), UriKind.Absolute);
Uri destDirUri = new Uri("https:" + Path.Combine(destDir).Replace("\\", "/"), UriKind.Absolute);
// Get a reference to the file we created previously
ShareFileClient sourceFile = new ShareFileClient(srcUri, credential);
ShareDirectoryClient shareDirectoryClient = new ShareDirectoryClient(destDirUri,credential);
shareDirectoryClient.CreateIfNotExistsAsync().GetAwaiter().GetResult();
using (var stream = sourceFile.OpenRead())
using (var zipArchive = new ZipArchive(stream))
{
foreach (var entry in zipArchive.Entries)
{
if (entry.Length > 0)
{
//CloudFile extractedFile = output.GetFileReference(entry.Name);
Uri destUri = new Uri("https:" + Path.Combine(destDir, entry.Name).Replace("\\", "/"), UriKind.Absolute);
ShareFileClient extractedFile = new ShareFileClient(destUri, credential);
using (var entryStream = entry.Open())
{
using (MemoryStream ms = new MemoryStream())
{
entryStream.CopyTo(ms);
//
//Sorry I have this part in another method
//
Uri fileUri = new Uri("https:" + Path.GetDirectoryName(filePath).Replace("\\", "/"), UriKind.Absolute);
// Get a reference to the file we created previously
ShareDirectoryClient directory = new ShareDirectoryClient(fileUri, credential);
ShareFileClient file = directory.GetFileClient(Path.GetFileName(filePath));
ms.Seek(0, SeekOrigin.Begin);
file.Create(ms.Length);
file.Upload(ms);
//
//
//
}
}
}
}
}
}

Download Zip file from azure Storage c#

I am looping through the file names from my database and the same file i have in azure storage. I am zipping those n number of files and download from azure storage. I saves the zipped file to my local storage. When i extract and want to see a file, it say damaged/corrupt.
public ActionResult Download(string productid, string YearActiveid)
{
HomePageModel homepagemodel = new HomePageModel();
homepagemodel.ProdHeaderDetail = GetProductHeaderDetail(productid, YearActiveid);
homepagemodel.PriorYearsActive = GetPriorYearActive(productid, YearActiveid);
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=<name>;AccountKey=<key>;EndpointSuffix=core.windows.net");
CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("product");
var blobFileNames = new string[] { "file1.png", "file2.png", "file3.png", "file4.png" };
var outputMemStream = new MemoryStream();
var zipOutputStream = new ZipOutputStream(outputMemStream);
foreach (var ProdHeaderDetail in homepagemodel.ProdHeaderDetail)
{
zipOutputStream.SetLevel(5);
var blob = cloudBlobContainer.GetBlockBlobReference(ProdHeaderDetail.FileName);
var entry = new ZipEntry(ProdHeaderDetail.FileName);
zipOutputStream.PutNextEntry(entry);
blob.DownloadToStreamAsync(zipOutputStream);
}
zipOutputStream.Finish();
//zipOutputStream.Close();
//zipOutputStream.CloseEntry();
zipOutputStream.IsStreamOwner = false;
outputMemStream.Position = 0;
return File(outputMemStream, "application/zip", "filename.zip");
}
I resolved the issue by adding async and wait
public async Task Download(string productid, string YearActiveid)
await blob.DownloadToStreamAsync(zipOutputStream);

zip all files in azure blob container

I am trying to use azure function to zip all files inside a blob container using System.IO.Compression.
I could list all files inside the container using the below CloudBlob code
CloudStorageAccount storageAccount = CloudStorageAccount.Parse (storageConn);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("<container>");
BlobContinuationToken blobToken = null;
var blobs = await container.ListBlobsSegmentedAsync(blobToken);
var fileList = new List<string>();
var blobpath1 = #"https://<pathtocontainer>/test.zip";
foreach (var blbitem in blobs.Results)
{
if (blbitem is CloudBlockBlob)
{
var blobFileName = blbitem.Uri.Segments.Last().Replace("%20", " ");
var blobFilePath = blbitem.Uri.AbsolutePath.Replace(blbitem.Container.Uri.AbsolutePath + "/", "").Replace("%20", " ");
var blobPath = blobFilePath.Replace("/" + blobFileName, "");
log.LogInformation("blob path : " + blbitem.Uri.ToString());
fileList.Add(blbitem.Uri.ToString());
string rootpath = #"D:\home\site\wwwroot\ZipandSendFile\temp\";
string path = rootpath + blobPath;
log.LogInformation("saving in " + path);
//Add to zip
/*
CloudBlobContainer container = cloudBlobClient.GetContainerReference("<container>");
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
using (FileStream fs = new FileStream(rootpath, FileMode.Create))
{
blob. DownloadToStream(fs);
}
*/
}
}
</code>
After getting each file details inside the blob , I am trying to add them into zip archive
using the below System.IO.Compression package
My attempt to download file
<code>
public static void AddFilesToZip(string zipPath, string[] files,ILogger log)
{
if (files == null || files.Length == 0)
{
return;
}
log.LogInformation("Executing add files to zip");
log.LogInformation(zipPath);
using (var zipArchive = ZipFile.Open(zipPath, ZipArchiveMode.Update))
{
log.LogInformation("in Zip archive");
foreach (var file in files)
{
var fileInfo = new FileInfo(file);
log.LogInformation(fileInfo.FullName);
zipArchive.CreateEntryFromFile(fileInfo.FullName, fileInfo.Name);
}
}
}
</code>
But I am getting access denied. Any pointers on this ?
Resolved issue by logging into kudo cmdshell and cd into directory and change file attribute
with attrib +A .

Resources