Migrate from Azure to SharePoint using Migration API - "Must specify IV in Manifest or blob metadata'" - azure

I am trying to use the Migration API with C# to migrate from the default free Azure Cloud (given to all sharepoint accounts) to SharePoint.
I can successfully generate a cloud container, and upload files, but the migration fails. Here is my code:
private void MigrateToAzureThenSharePoint()
{
//Provision The default Azure Containers and Migation Queue
var migrationContainers = destinationCtx.Site.ProvisionMigrationContainers();
var migrationQueue = destinationCtx.Site.ProvisionMigrationQueue();
destinationCtx.ExecuteQuery();
var containerInfoList = migrationContainers.Value;
var dataContainerUri = containerInfoList.DataContainerUri;
var metadataContainerUri = containerInfoList.MetadataContainerUri;
var queueInfo = migrationQueue.Value;
//Instansiate Cloud Queue
var cloudQueue = new CloudQueue(new Uri(queueInfo.JobQueueUri));
//Instansiate CloudBlobContainer
CloudBlobContainer dataContainer = new CloudBlobContainer(new Uri(dataContainerUri));
CloudBlobContainer manifestContainer = new CloudBlobContainer(new Uri(metadataContainerUri));
//Test files to upload
var testfiles = new[]
{
new SourceFile
{
Filename = "test.txt",
LastModified = DateTime.Now,
Contents = Encoding.UTF8.GetBytes("Hi, this is a test text-file"),
Title = "Title of file 1"
},
new SourceFile
{
Filename = "test2.txt",
LastModified = DateTime.Now.AddDays(-1),
Contents = Encoding.UTF8.GetBytes("Tesfile2"),
Title = "Second title"
}
};
//Upload Test Files to Cloud
foreach (var testfile in testfiles)
{
var blobReference = dataContainer.GetBlockBlobReference(testfile.Filename);
blobReference.UploadFromByteArray(testfile.Contents, 0, testfile.Contents.Length, null);
blobReference.CreateSnapshot();
}
var manifestPackage = new ManifestPackage(destOrganisationList.Id,
destinationCtx.Web.Id,
destinationCtx.Web.Title,
destOrganisationList.Title,
#"\",
destOrganisationList.RootFolder.UniqueId,
destOrganisationList.RootFolder.ParentFolder.UniqueId
);
//Create XML Files for Azure Migration (An easier way here would be great? I manualy write them)
var filesInManifestPackage = manifestPackage.GetManifestPackageFiles(testfiles);
//Uploaded manually created XML files
foreach (var migrationPackageFile in filesInManifestPackage)
{
var blobReference = manifestContainer.GetBlockBlobReference(migrationPackageFile.Filename);
blobReference.UploadFromByteArray(migrationPackageFile.Contents, 0, migrationPackageFile.Contents.Length, null);
blobReference.CreateSnapshot();
}
//Start Migration
var result = destinationCtx.Site.CreateMigrationJobEncrypted(destinationCtx.Web.Id,
containerInfoList.DataContainerUri,
containerInfoList.MetadataContainerUri,
queueInfo.JobQueueUri,
new EncryptionOption()
{
AES256CBCKey = containerInfoList.EncryptionKey
});
destinationCtx.ExecuteQuery();
//Check for messags
while (true) {
var msg = cloudQueue.GetMessage();
if (msg == null)
{
Task.Delay(TimeSpan.FromSeconds(1));
continue;
}
var message = JsonConvert.DeserializeObject<EncryptedMessage>(msg.AsString);
cloudQueue.DeleteMessage(msg);
//Decode encrypted message
var decoded = JsonConvert.DeserializeObject<UpdateMessage>(Decrypt(message.Content, containerInfoList.EncryptionKey, message.IV));
//Error
}
}
There are a few errors, but they all are similar to this:
"Unable to download SystemData.xml with exception 'Must specify IV in
Manifest or blob metadata'"
I believe I may have to encrypt the files when I upload from this method:
blobReference.UploadFromByteArray
There is an option:
var blobRequestOptions = new BlobRequestOptions();
blobRequestOptions.EncryptionPolicy =
But I don't know how to instantiate the encryption policy (and I may be going down a rabbit hole)
Any help would be greatly appreciated, as documentation online is shocking minimal.

Related

Zipping files and uploading to Azure Blobstorage using filestream

I am creating a website with ASP.NET Core, and I need to take a bunch of files, zip them together and then upload them to Azure Blobstorage.
It succeeds, I get a downloadlink but when I download the file, I just get a "File" that I cannot open with anything.
Here's the code that handles the uploading:
var moduleVersionReleaseFiles = _appDbContext
.ModuleVersionReleaseFile
.Where(x => x.ModuleVersionReleaseId == moduleVersionReleaseId)
.ToList();
string AzureBlobStorageConfig = _configuration["AzureBlobStorage"];
var connectionString = _configuration["StorageConnectionString"];
string containerName = moduleVersionReleaseId + "-" + Guid.NewGuid().ToString();
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName.ToLowerInvariant(), Azure.Storage.Blobs.Models.PublicAccessType.BlobContainer);
BlobClient blobClient = containerClient.GetBlobClient(moduleVersionReleaseId.ToString());
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var moduleVersionReleaseFile in moduleVersionReleaseFiles)
{
var entry = archive.CreateEntry(moduleVersionReleaseFile.File);
using (var entryStream = entry.Open())
using (var fileStream = File.OpenRead($"{moduleVersionReleaseFile.Path}\\{moduleVersionReleaseFile.File}"))
{
await fileStream.CopyToAsync(entryStream);
}
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);
await stream.FlushAsync();
}
return blobClient.Uri.ToString();
I am using Azurite for testing, so I can't log onto the blobstorage and check there.
Why am I not getting a zip file when downloading the file as expected?
The reason you're not able to open the zip file is because it is not getting saved properly. Can you please try with the following code:
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var moduleVersionReleaseFile in moduleVersionReleaseFiles)
{
var entry = archive.CreateEntry(moduleVersionReleaseFile.File);
using (var entryStream = entry.Open())
var fileContents = File.ReadAllBytes($"{moduleVersionReleaseFile.Path}\\{moduleVersionReleaseFile.File}"));
using (BinaryWriter zipFileBinary = new BinaryWriter(entryStream))
{
zipFileBinary.Write(fileContents);
}
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);
}

How can I dynamically create Ziparchive in Azure

I have multiple files in my Azure storage account say its Master container, I have created a dynamic container which will have the required files copied from Master Container, and those coped files needed to be zipped inside that created container. See below code, where in I have created the zip archive. Also the zip archive is getting created, but when I download manually and see the files, it seems corrupted (for ex. the actual size of the individual files are more than 1Mb but the files which I get after download seems 22Kb), and File formats would be .ipt, .iam (Autodesk Inventor Files)
CloudBlobContainer destContainer = blobClient.GetContainerReference(AzureContainer);
bool isCreated = destContainer.CreateIfNotExists();
var zipblob = destContainer.GetBlockBlobReference("inputAssembly.zip");
using (var stream = await zipblob.OpenWriteAsync())
using (var zip = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var fileName in inputfile)
{
using (var fileStream = new MemoryStream())
{
if (destContainer.GetBlockBlobReference(fileName).Exists())
{
destContainer.GetBlockBlobReference(fileName).DownloadToStream(fileStream);
}
var newZip = new ZipArchive(fileStream, ZipArchiveMode.Create);
var entry = newZip.CreateEntry(fileName, CompressionLevel.NoCompression);
using (var innerFile = entry.Open())
{
fileStream.CopyTo(innerFile);
}
fileStream.Close();
}
}
}
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(fixedpartContainer);
CloudBlobContainer destContainer1 = blobClient.GetContainerReference(AzureContainer);
bool isCreated = destContainer1.CreateIfNotExists();
var zipblob = destContainer1.GetBlockBlobReference("inputAssembly.zip");
using (var stream = await zipblob.OpenWriteAsync())
{
using (var zipArchive = new ZipArchive(stream, ZipArchiveMode.Create))
{
foreach (var blobName in blobFileNames)
{
var blob = destContainer.GetBlockBlobClient(blobName);
var zipEntry = zipArchive.CreateEntry(blobName);
using var zipStream = zipEntry.Open();
using var fileStream = new MemoryStream();
await blob.DownloadToAsync(fileStream);
await zipStream.WriteAsync(fileStream.ToArray());
AssemblyCreated = true;
}
}
}

How to stream Video that are stored in Azure storage Blob using Xamarin

I have uploaded video files to Azur Blob(Containers),I want to access them in the mobile app via Streaming.I have files with extention .mp4 . I have done code to download from blob and Store in local drive then play using default player , But I want to give user a option to stream instead of download.
I have used this method
var credentials = new StorageCredentials("myaccountname", "mysecretkey");
var account = new CloudStorageAccount(credentials, true);
var container = account.CreateCloudBlobClient().GetContainerReference("yourcontainername");
var blob = container.GetBlockBlobReference("yourmp4filename");
var sas = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy()
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1),//Set this date/time according to your requirements
});
var urlToBePlayed = string.Format("{0}{1}", blob.Uri, sas);//This is the URI which should be embedded in your video player.
Problem:-
If I browse the Url(Blob Url) , It downloads the file instead of directly playing it .But in App , Nothing appears. Blank screen.
I am using
<WebView Source="{Binding VideoUrl}" HeightRequest="200" WidthRequest="200"/>
In Vm:
VideoUrl=url;
First Change content Type : like #Zhaoxing Lu - Microsoft said
public async Task ChangeContentTypeAsync()
{
try
{
UserDialogs.Instance.ShowLoading();
BlobContinuationToken blobContinuationToken = null;
var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference("videos");
var results = await container.ListBlobsSegmentedAsync(null, blobContinuationToken);
blobContinuationToken = results.ContinuationToken;
BlobResultSegment blobs = await blobClient
.GetContainerReference("videos")
.ListBlobsSegmentedAsync(blobContinuationToken)
;
foreach (CloudBlockBlob blob in blobs.Results)
{
if (Path.GetExtension(blob.Uri.AbsoluteUri) == ".mp4")
{
blob.Properties.ContentType = "video/mp4";
}
//// repeat and resume
await blob.SetPropertiesAsync();
}
UserDialogs.Instance.HideLoading();
}
catch (Exception ex)
{
var m = ex.Message;
}
}
Then Use this method :
private async Task StreamVideo(string filename)
{
try
{
UserDialogs.Instance.ShowLoading();
await azureBlob.ChangeContentTypeAsync();
var secretkey = "xxxx";
var credentials = new StorageCredentials("chatstorageblob", secretkey);
var account = new CloudStorageAccount(credentials, true);
var container = account.CreateCloudBlobClient().GetContainerReference("videos");
var blob = container.GetBlockBlobReference(filename);
var sas = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy()
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1),//Set this date/time according to your requirements
});
var urlToBePlayed = string.Format("{0}{1}", blob.Uri, sas);//This is the URI which should be embedded in your video player.
await Navigation.PushAsync(new VideoPlayerPage(urlToBePlayed));
UserDialogs.Instance.HideLoading();
}
catch (Exception ex)
{
var m = ex.Message;
}
}

Email Attachments missing on Azure, works locally

I have implemented something similar to the following question and I can get it working locally on my server, but when I deploy to Azure it doesn't work. I don't get any errors: just an email without the attachment.
Sending attachments using Azure blobs
Are there restrictions on what types of files can be sent with SendGrid (file is only 56k)?
Does the Azure App service have to be at a particular level or can it be done on Basic?
The blob URL definitley exists and I am setting the stream to zero as suggested in that previous question.
MailMessage mm = new MailMessage("receiver address", "someone");
mm.From = new MailAddress("myAddress", "My Name");
mm.Subject = content.Subject;
mm.Body = content.Body;
mm.IsBodyHtml = true;
mm.BodyEncoding = UTF8Encoding.UTF8;
mm.DeliveryNotificationOptions = DeliveryNotificationOptions.OnFailure;
var attachmentAsStream = _storageAccessService.GetAssetAsStreamForEmail("myContainer", "fileThatExists.pdf");
var attachment = new Attachment(attachmentAsStream, "File.pdf", MediaTypeNames.Application.Pdf);
mm.Attachments.Add(attachment);
public MemoryStream GetAssetAsStreamForEmail(string containerName, string fileName)
{
// Create the blob client.
CloudBlobClient blobClient = StorageAccountReference().CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
var memoryStream = new MemoryStream();
try
{
using (var stream = new MemoryStream())
{
blob.DownloadToStream(memoryStream);
memoryStream.Seek(0, SeekOrigin.Begin);
}
}
catch (Exception ex)
{
Elmah.ErrorSignal.FromCurrentContext().Raise(new Exception("Failed to download Email Atatchment: "+ ex.Message));
}
memoryStream.Position = 0;
return memoryStream;
}
using (SmtpClient client = new SmtpClient())
{
client.Port = 587;
client.Host = "smtp.sendgrid.net";
client.EnableSsl = true;
client.Timeout = 10000;
client.DeliveryMethod = SmtpDeliveryMethod.Network;
client.UseDefaultCredentials = false;
client.Credentials = new System.Net.NetworkCredential("hidden", "hidden");
await client.SendMailAsync(message);
}
Update
Update after Yasir's suggestion below. Downloading the blob from Azure as a Stream only seems to work locally. But if I change to download as a ByteArray then it works everywhere, nonetheless...
public MemoryStream GetAssetAsStreamForEmail(string containerName, string fileName)
{
// Create the blob client.
CloudBlobClient blobClient = StorageAccountReference().CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
try
{
blob.FetchAttributes();
var fileStream = new byte[blob.Properties.Length];
for (int i = 0; i < blob.Properties.Length; i++)
{
fileStream[i] = 0x20;
}
blob.DownloadToByteArray(fileStream, 0) ;
MemoryStream bufferStream = new MemoryStream(fileStream);
}
catch (Exception ex)
{
Elmah.ErrorSignal.FromCurrentContext().Raise(new Exception("Failed to download Email Atatchment: " + ex.Message));
}
return null;
}
The following code works for me for sending emails using Sendgrid from Azure functions. I attach a CSV file from Blob Storage. It should work for you as well. All you will have to do is ensure that you read the pdf file as a byte[].
public interface IEmailAttachment
{
string Name { get; }
byte[] FileData { get; }
}
public static void Send(MailMessage mailMessage, IEnumerable<IEmailAttachment> attachments)
{
try
{
// Get the configuration data
string server = ConfigReader.EmailServer;
int port = ConfigReader.EmailPort;
string username = ConfigReader.SendGridUserName;
string password = ConfigReader.SendGridPassword;
smtpClient.EnableSsl = false;
smtpClient.Credentials = new NetworkCredential(username, password);
// Create the SMTP Client
SmtpClient smtpClient = new SmtpClient(server, port);
// Prepare the MailMessage
mailMessage.From = new MailAddress(ConfigReader.FromEmail);
var toEmails = ConfigReader.ToEmail.Split(',');
foreach (var toEmail in toEmails)
{
mailMessage.To.Add(toEmail);
}
var ccEmails = ConfigReader.EmailCc.Split(',');
foreach (var ccEmail in ccEmails)
{
mailMessage.CC.Add(ccEmail);
}
// Add attachments
List<MemoryStream> files = new List<MemoryStream>();
if (attachments != null)
{
foreach (IEmailAttachment file in attachments)
{
MemoryStream bufferStream = new MemoryStream(file.FileData);
files.Add(bufferStream);
Attachment attachment = new Attachment(bufferStream, file.Name);
mailMessage.Attachments.Add(attachment);
}
}
mailMessage.IsBodyHtml = true;
// Send the email
smtpClient.Send(mailMessage);
foreach (MemoryStream stream in files)
{
stream.Dispose();
}
}
catch (Exception)
{
throw;
}
}

How to use StartCopyFromBlob between different accounts?

I am using this code to copy blobs from one account to another... but it throws an exception.
var srcAccount = CloudStorageAccount.Parse("connection string 1");
var dstAccount = CloudStorageAccount.Parse("connection string 2");
var srcBlobClient = srcAccount.CreateCloudBlobClient();
var dstBlobClient = dstAccount.CreateCloudBlobClient();
foreach (var srcCloudBlobContainer in srcBlobClient.ListContainers())
{
var dstCloudBlobContainer = dstBlobClient
.GetContainerReference(srcCloudBlobContainer.Name);
dstCloudBlobContainer.CreateIfNotExists();
foreach (var srcBlob in srcCloudBlobContainer.ListBlobs())
{
if (srcBlob.GetType() == typeof(CloudBlockBlob))
{
var srcBlockBlock = (CloudBlockBlob)srcBlob;
var dstBlockBlock = dstCloudBlobContainer
.GetBlockBlobReference(srcBlockBlock.Name);
// throws exception StorageException:
// The remote server returned an error: (404) Not Found.
dstBlockBlock.StartCopyFromBlob(srcBlockBlock.Uri);
}
}
}
Microsoft states that cross account copy is supported, but I cannot get it to work.
What am I doing wrong?
Can you check the source blob container's ACL? If it's Private you may either need to change the ACL to Public / Blob or create a SAS URL. You can use the following code if you wish to keep your blob container's ACL as Private and make use of SAS URL:
var srcAccount = CloudStorageAccount.Parse("connection string 1");
var dstAccount = CloudStorageAccount.Parse("connection string 2");
var srcBlobClient = srcAccount.CreateCloudBlobClient();
var dstBlobClient = dstAccount.CreateCloudBlobClient();
foreach (var srcCloudBlobContainer in srcBlobClient.ListContainers())
{
var dstCloudBlobContainer = dstBlobClient
.GetContainerReference(srcCloudBlobContainer.Name);
dstCloudBlobContainer.CreateIfNotExists();
//Assuming the source blob container ACL is "Private", let's create a Shared Access Signature with
//Start Time = Current Time (UTC) - 15 minutes to account for Clock Skew
//Expiry Time = Current Time (UTC) + 7 Days - 7 days is the maximum time allowed for copy operation to finish.
//Permission = Read so that copy service can read the blob from source
var sas = srcCloudBlobContainer.GetSharedAccessSignature(new SharedAccessBlobPolicy()
{
SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-15),
SharedAccessExpiryTime = DateTime.UtcNow.AddDays(7),
Permissions = SharedAccessBlobPermissions.Read,
});
foreach (var srcBlob in srcCloudBlobContainer.ListBlobs())
{
if (srcBlob.GetType() == typeof(CloudBlockBlob))
{
var srcBlockBlock = (CloudBlockBlob)srcBlob;
var dstBlockBlock = dstCloudBlobContainer
.GetBlockBlobReference(srcBlockBlock.Name);
//Create a SAS URI for the blob
var srcBlockBlobSasUri = string.Format("{0}{1}", srcBlockBlock.Uri, sas);
// throws exception StorageException:
// The remote server returned an error: (404) Not Found.
dstBlockBlock.StartCopyFromBlob(new Uri(srcBlockBlobSasUri));
}
}
}

Resources