how to pass Body for ADLS Gen-2 as a stream - azure

I'm Using ADLS Gen2 Path-Update API to update file from ADLS which is already created.
As a body i can easily pass string which is working fine but the same with Stream is not working.
i'm reading local file data and trying to store it into stream and pass as a body but getting an error of Http request header is invalid

I have a quick test at my side, the following code which read local file as stream, then upload the stream to adls gen2. It works fine. Please try it at your side, and let me know if you have more issues.
static void Main(string[] args)
{
var auth = new AzureServiceTokenProvider();
const string url = "https://storage.azure.com/";
string token = auth.GetAccessTokenAsync(url).Result;
string requestUri = "https://xxx.dfs.core.windows.net/t11/b.txt?action=append&position=0";
var method = new HttpMethod("PATCH");
// read local file as stream
var mystream = File.OpenRead(#"D:\temp\1\test1.txt");
Console.WriteLine($"the stream length is: {mystream.Length}");
Console.WriteLine($"the position of the stream is: {mystream.Position}");
var stream_length = mystream.Length;
var request = new HttpRequestMessage(method, requestUri)
{
//Content = new StringContent(upload_string)
Content = new StreamContent(mystream)
};
// Add some defined headers
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/plain"));
var i = request.Content.AsString().Length;
Console.WriteLine(request.Content.AsString());
var httpClient = new HttpClient();
var result = httpClient.SendAsync(request).Result;
Console.WriteLine("append result status code: "+ (int)result.StatusCode);
//for flush
string requestUri_2 = "https://xxx.dfs.core.windows.net/t11/b.txt?action=flush&position="+stream_length;
var request_2 = new HttpRequestMessage(method,requestUri_2);
using (HttpClient httpClient_2 = new HttpClient())
{
httpClient_2.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
HttpResponseMessage response = httpClient_2.SendAsync(request_2).Result;
Console.WriteLine("flush result status code: " + (int)response.StatusCode);
}

Related

Migrate from Azure to SharePoint using Migration API - "Must specify IV in Manifest or blob metadata'"

I am trying to use the Migration API with C# to migrate from the default free Azure Cloud (given to all sharepoint accounts) to SharePoint.
I can successfully generate a cloud container, and upload files, but the migration fails. Here is my code:
private void MigrateToAzureThenSharePoint()
{
//Provision The default Azure Containers and Migation Queue
var migrationContainers = destinationCtx.Site.ProvisionMigrationContainers();
var migrationQueue = destinationCtx.Site.ProvisionMigrationQueue();
destinationCtx.ExecuteQuery();
var containerInfoList = migrationContainers.Value;
var dataContainerUri = containerInfoList.DataContainerUri;
var metadataContainerUri = containerInfoList.MetadataContainerUri;
var queueInfo = migrationQueue.Value;
//Instansiate Cloud Queue
var cloudQueue = new CloudQueue(new Uri(queueInfo.JobQueueUri));
//Instansiate CloudBlobContainer
CloudBlobContainer dataContainer = new CloudBlobContainer(new Uri(dataContainerUri));
CloudBlobContainer manifestContainer = new CloudBlobContainer(new Uri(metadataContainerUri));
//Test files to upload
var testfiles = new[]
{
new SourceFile
{
Filename = "test.txt",
LastModified = DateTime.Now,
Contents = Encoding.UTF8.GetBytes("Hi, this is a test text-file"),
Title = "Title of file 1"
},
new SourceFile
{
Filename = "test2.txt",
LastModified = DateTime.Now.AddDays(-1),
Contents = Encoding.UTF8.GetBytes("Tesfile2"),
Title = "Second title"
}
};
//Upload Test Files to Cloud
foreach (var testfile in testfiles)
{
var blobReference = dataContainer.GetBlockBlobReference(testfile.Filename);
blobReference.UploadFromByteArray(testfile.Contents, 0, testfile.Contents.Length, null);
blobReference.CreateSnapshot();
}
var manifestPackage = new ManifestPackage(destOrganisationList.Id,
destinationCtx.Web.Id,
destinationCtx.Web.Title,
destOrganisationList.Title,
#"\",
destOrganisationList.RootFolder.UniqueId,
destOrganisationList.RootFolder.ParentFolder.UniqueId
);
//Create XML Files for Azure Migration (An easier way here would be great? I manualy write them)
var filesInManifestPackage = manifestPackage.GetManifestPackageFiles(testfiles);
//Uploaded manually created XML files
foreach (var migrationPackageFile in filesInManifestPackage)
{
var blobReference = manifestContainer.GetBlockBlobReference(migrationPackageFile.Filename);
blobReference.UploadFromByteArray(migrationPackageFile.Contents, 0, migrationPackageFile.Contents.Length, null);
blobReference.CreateSnapshot();
}
//Start Migration
var result = destinationCtx.Site.CreateMigrationJobEncrypted(destinationCtx.Web.Id,
containerInfoList.DataContainerUri,
containerInfoList.MetadataContainerUri,
queueInfo.JobQueueUri,
new EncryptionOption()
{
AES256CBCKey = containerInfoList.EncryptionKey
});
destinationCtx.ExecuteQuery();
//Check for messags
while (true) {
var msg = cloudQueue.GetMessage();
if (msg == null)
{
Task.Delay(TimeSpan.FromSeconds(1));
continue;
}
var message = JsonConvert.DeserializeObject<EncryptedMessage>(msg.AsString);
cloudQueue.DeleteMessage(msg);
//Decode encrypted message
var decoded = JsonConvert.DeserializeObject<UpdateMessage>(Decrypt(message.Content, containerInfoList.EncryptionKey, message.IV));
//Error
}
}
There are a few errors, but they all are similar to this:
"Unable to download SystemData.xml with exception 'Must specify IV in
Manifest or blob metadata'"
I believe I may have to encrypt the files when I upload from this method:
blobReference.UploadFromByteArray
There is an option:
var blobRequestOptions = new BlobRequestOptions();
blobRequestOptions.EncryptionPolicy =
But I don't know how to instantiate the encryption policy (and I may be going down a rabbit hole)
Any help would be greatly appreciated, as documentation online is shocking minimal.

Issue with Updating file in ADLS Gen2 using Rest-API

I'm using ADLS Gen2 using rest api Path-Update i'm trying to update data into already created created blank file into ADLS.
but whenever i'm trying to use the API i'm getting below response from API.
{StatusCode: 202, ReasonPhrase: 'Accepted'}
but still the file will be empty.
string requestUri = "https://XXXXXXX.dfs.core.windows.net/XXXXX/abc.txt?action=append&position=0";// &retainUncommittedData=false&close=true";
dynamic method = new HttpMethod("PATCH");
dynamic request = new HttpRequestMessage(method, requestUri)
{
Content = new StringContent("\"requestBody\":\"test\"")
};`enter code here`
// Add some defined headers
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", authenticationToken);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));
// Add some other headers or custom headers
request.Headers.TryAddWithoutValidation("Content-Length", "0");
dynamic httpClient = new HttpClient();
dynamic result = httpClient.SendAsync(request).Result;
i expect the data should be updated in file but now i'm getting 202 Accepted as a response code but file is not updated with data
also i tried append with flush operation below is the code i'm getting 405 error
string requestUri = "https://XXXXXX.dfs.core.windows.net/XXXXX/abc.txt?action=append&position=0";// &retainUncommittedData=false&close=true";
var method = new HttpMethod("PATCH");
var request = new HttpRequestMessage(method, requestUri)
{
Content = new StringContent("\"requestBody\":\"test\"")
};
// Add some defined headers
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", authenticationToken);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/plain"));
// Add some other headers or custom headers
// request.Headers.TryAddWithoutValidation("Content-Length", "0");
var httpClient = new HttpClient();
var result = httpClient.SendAsync(request).Result;
string requestUri1 = "https://XXXXX.dfs.core.windows.net/XXXXXX/abc.txt?action=flush&position=0";//&retainUncommittedData=false&close=true";
using (HttpClient httpClient1 = new HttpClient())
{
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", authenticationToken);
HttpResponseMessage response = (httpClient.PutAsync(requestUri1, null)).Result;
}
Update:
If the files in adls gen2 is empty, you can use the method below:
static void Main(string[] args)
{
var auth = new AzureServiceTokenProvider();
const string url = "https://storage.azure.com/";
string token = auth.GetAccessTokenAsync(url).Result;
string requestUri = "https://xxx.dfs.core.windows.net/t11/c.txt?action=append&position=0";
var method = new HttpMethod("PATCH");
string upload_string = "have a nice day!";
var request = new HttpRequestMessage(method, requestUri)
{
Content = new StringContent(upload_string)
};
// Add some defined headers
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/plain"));
var i = request.Content.AsString().Length;
Console.WriteLine(request.Content.AsString());
var httpClient = new HttpClient();
var result = httpClient.SendAsync(request).Result;
Console.WriteLine("append result status code: "+ (int)result.StatusCode);
//for flush
string requestUri_2 = "https://xxx.dfs.core.windows.net/t11/c.txt?action=flush&position="+upload_string.Length;
var request_2 = new HttpRequestMessage(method,requestUri_2);
using (HttpClient httpClient_2 = new HttpClient())
{
httpClient_2.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
HttpResponseMessage response = httpClient_2.SendAsync(request_2).Result;
Console.WriteLine("flush result status code: " + (int)response.StatusCode);
}
Console.ReadLine();
}
Test result as below, and I also check in azure, the data is flushed into the file.
Once you have received the 202 Accepted you can then call the action=flush and pass the position that you want the data to be flushed to. like below:
https://$STORAGE_ACCOUNT_NAME.dfs.core.windows.net/mydata/data/file1?action=flush&position=10

Not able to get excel file from web api dot net core 2.1 using epplus excel package

I'm trying to generate an excel file / stream in my web api and return it in a HttpResponseMessage to serve it to the client in Angular 5 as a download.
The generation succeeds and an xlsx file is generated and saved on the server, but when I return it in the Content of my httpResponseMessage, my browser shows just some json instead of the whole excel file.
{"version":{"major":1,"minor":1,"build":-1,"revision":-1,"majorRevision":-1,"minorRevision":-1},"content":{"headers":[{"key":"Content-Disposition","value":["attachment; filename=636742856488421817.xlsx"]},{"key":"Content-Type","value":["application/ms-excel"]},{"key":"Content-Length","value":["22780"]}]},"statusCode":200,"reasonPhrase":"OK","headers":[],"requestMessage":null,"isSuccessStatusCode":true}
This is how I create the excel file and returns it:
var dataBytes = File.ReadAllBytes(fileName);
var dataStream = new MemoryStream(dataBytes);
HttpResponseMessage httpResponseMessage = new HttpResponseMessage(HttpStatusCode.OK);
httpResponseMessage.Content = new StreamContent(dataStream);
httpResponseMessage.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
httpResponseMessage.Content.Headers.ContentDisposition.FileName = fileName;
httpResponseMessage.Content.Headers.ContentType = new MediaTypeHeaderValue("application/ms-excel");
httpResponseMessage.Content.Headers.ContentLength = dataStream.Length;
I solved this:
Here is what I did
Helper class which creates the excel package and converts it to a Stream
var fileName = DateTime.Now.Ticks + ".xlsx";
FileInfo file = new FileInfo(fileName);
FileInfo templateFile = new FileInfo(#"Templates/ReportTemplate.xlsx");
ExcelPackage package = new ExcelPackage(file, templateFile);
ExcelWorksheet worksheet = package.Workbook.Worksheets.FirstOrDefault();
... filling rows and cells goed here ...
var dataBytes = package.GetAsByteArray();
Stream dataStream = new MemoryStream(dataBytes);
dataStream.Seek(0, SeekOrigin.Begin);
return dataStream;
In the controller I return the file to the Angular client like this:
var stream = _helperClass.GenerateReport(exportDate, exportTitle);
return File(stream, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", $"Report-{DateTime.Now.ToShortDateString()}.xlsx");
In the Angular component I do this after I receive the response:
var blob = new Blob([res], { type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" });
var blobURL = window.URL.createObjectURL(blob);
var anchor = document.createElement("a");
anchor.download = `Report-${new Date().toISOString()}.xlsx`;
anchor.href = blobURL;
anchor.click();
Try this following code:
For conroller:
[Route("DownLoadExcel")]
public IActionResult DownLoadExcel()
{
var pack = new ExcelPackage();
ExcelWorksheet worksheet = pack.Workbook.Worksheets.Add("sample");
//First add the headers
worksheet.Cells[1, 1].Value = "ID";
worksheet.Cells[1, 2].Value = "Name";
//Add values
worksheet.Cells["A2"].Value = 1000;
worksheet.Cells["B2"].Value = "Jon";
return File(pack.GetAsByteArray(), "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", "Sample.xlsx");
}
For client side:
window.open("../.../DownLoadExcel");

Microsoft Face Detect API code example Bad Request

I have been trying to solve this bad request error. I am able to make the request call and Azure reports total calls correctly and also reports total errors.
I can not get this code example to work; however if I send this via their online console all is fine:
static async void MakeRequest()
{
string key1 = "YourKey"; // azure the one should work
string data = "https://pbs.twimg.com/profile_images/476054279438868480/vvv5YG0Q.jpeg";
var client = new HttpClient();
var queryString = HttpUtility.ParseQueryString(string.Empty);
// Request parameters
queryString["returnFaceId"] = "true";
// Request headers
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", key1);
Console.Beep();
var uri = "https://westus.api.cognitive.microsoft.com/face/v1.0/detect?" + queryString;
//string statusURL = HttpContext.Current.Request.Url.Host;
//console.WriteLine("Your Status URL address is :" + statusURL);
HttpResponseMessage response;
// Request body
// byte[] byteData = Encoding.UTF8.GetBytes("{url: https://pbs.twimg.com/profile_images/476054279438868480/vvv5YG0Q.jpeg}");
byte[] byteData = Encoding.UTF8.
GetBytes("{"+ "url"+":"+"https://pbs.twimg.com/profile_images/476054279438868480/vvv5YG0Q.jpeg" + "}");
using (var content = new ByteArrayContent(byteData))
{
content.Headers.ContentType =
new MediaTypeHeaderValue("application/json");
response = await client.PostAsync(uri, content);
}
HttpRequestMessage request =
new HttpRequestMessage(HttpMethod.Post, uri);
request.Content = new StringContent("{body}",
Encoding.UTF8,
"application/json");
//CONTENT-TYPE header
await client.SendAsync(request)
.ContinueWith(responseTask =>
{
Console.WriteLine("Response: {0}", responseTask.Result);
Console.WriteLine("-----------------------------------");
Console.ForegroundColor = ConsoleColor.Blue;
Console.WriteLine("End of Post return from MS");
Console.WriteLine("Hit ENTER to exit...");
Console.ReadKey();
});
}// end of Make request
Your JSON is malformed. Your fields and non-scalar fields must be quoted. You also have some unnecessary code. Here's code that works:
static async void MakeRequest()
{
string key1 = "YourKey"; // azure the one should work
string imageUri = "https://pbs.twimg.com/profile_images/476054279438868480/vvv5YG0Q.jpeg";
var client = new HttpClient();
var queryString = HttpUtility.ParseQueryString(string.Empty);
// Request parameters
queryString["returnFaceId"] = "true";
// Request headers
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", key1);
var uri = "https://westus.api.cognitive.microsoft.com/face/v1.0/detect?" + queryString;
string body = "{\"url\":\"" + imageUri + "\"}";
using (var content = new StringContent(body, Encoding.UTF8, "application/json"))
{
await client.PostAsync(uri, content)
.ContinueWith(async responseTask =>
{
var responseBody = await responseTask.Result.Content.ReadAsStringAsync();
Console.WriteLine("Response: {0}", responseBody);
Console.WriteLine("-----------------------------------");
Console.ForegroundColor = ConsoleColor.Blue;
Console.WriteLine("End of Post return from MS");
Console.WriteLine("Hit ENTER to exit...");
Console.ReadKey();
});
}
}// end of Make request
If you're using Visual Studio, I would recommend the NuGet package as this will handle much of the mundane details for you, including C# types for responses.

Azure File Share REST API for Xamarin

In Xamarin client app, I want to access Azure Files using SAS token with Portable Class Library. It seems I can not do it using latest WindowAzure.Storage nuget package as it may only works with Blob, Table,... and it requires lots of dependencies.
Is there anyway to accomplish this?
I ended up with using Azure File Storage REST API.
Basically we request SAS token generated from Azure Share first then using that SAS token in URL to send http request to Azure Files Storage:
https://[yourshare].file.core.windows.net/[yourdirectory]/[yourfile]?[your_sas_token]
I have created a class to help client doing some basic operations as below (it is portable class so can use anywhere in client side):
public class AzureFileREST
{
private AzureSASToken _azureShareToken;
public AzureFileREST(AzureSASToken azureShareToken)
{
_azureShareToken = azureShareToken;
}
public async Task CreateIfNotExist(string directoryName)
{
var existed = await CheckDirectoryExists(directoryName);
if (!existed)
{
await CreateDirectory(directoryName);
}
}
public async Task<bool> CheckDirectoryExists(string directoryName)
{
using (var client = new HttpClient())
{
//Get directory (https://msdn.microsoft.com/en-us/library/azure/dn194272.aspx)
var azureCreateDirUrl = _azureShareToken.Url + directoryName + _azureShareToken.SASToken + "&restype=directory";
var response = await client.GetAsync(azureCreateDirUrl).ConfigureAwait(false);
return (response.StatusCode != System.Net.HttpStatusCode.NotFound);
}
}
public async Task CreateDirectory(string directoryName)
{
using (var client = new HttpClient())
{
//Create directory (https://msdn.microsoft.com/en-us/library/azure/dn166993.aspx)
var azureCreateDirUrl = _azureShareToken.Url + directoryName + _azureShareToken.SASToken + "&restype=directory";
var response = await client.PutAsync(azureCreateDirUrl, null).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
}
}
public async Task UploadFile(string fileName, byte[] fileBytes)
{
using (var client = new HttpClient())
{
//Create empty file first (https://msdn.microsoft.com/en-us/library/azure/dn194271.aspx)
var azureCreateFileUrl = _azureShareToken.Url + fileName + _azureShareToken.SASToken;
client.DefaultRequestHeaders.Add("x-ms-type", "file");
client.DefaultRequestHeaders.Add("x-ms-content-length", fileBytes.Length.ToString());
var response = await client.PutAsync(azureCreateFileUrl, null).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
//Then upload file (https://msdn.microsoft.com/en-us/library/azure/dn194276.aspx)
var azureUploadFileUrl = azureCreateFileUrl + "&comp=range";
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Add("x-ms-write", "update");
client.DefaultRequestHeaders.Add("x-ms-range", String.Format("bytes=0-{0}", (fileBytes.Length - 1).ToString()));
var byteArrayContent = new ByteArrayContent(fileBytes);
response = await client.PutAsync(azureUploadFileUrl, byteArrayContent).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
}
}
}
In server side, use the following function to generate SAS token from Share:
public AzureSASToken GetSASFromShare(string shareName)
{
var share = _fileclient.GetShareReference(shareName);
share.CreateIfNotExists();
string policyName = "UPARSharePolicy";
// Create a new shared access policy and define its constraints.
var sharedPolicy = new SharedAccessFilePolicy()
{
SharedAccessExpiryTime = DateTime.UtcNow.AddDays(15),
Permissions = SharedAccessFilePermissions.Read | SharedAccessFilePermissions.Write
};
// Get existing permissions for the share.
var permissions = share.GetPermissions();
// Add the shared access policy to the share's policies.
// Note that each policy must have a unique name.
// Maximum 5 policies for each share!
if (!permissions.SharedAccessPolicies.Keys.Contains(policyName))
{
if (permissions.SharedAccessPolicies.Count > 4)
{
var lastAddedPolicyName = permissions.SharedAccessPolicies.Keys.Last();
permissions.SharedAccessPolicies.Remove(lastAddedPolicyName);
}
permissions.SharedAccessPolicies.Add(policyName, sharedPolicy);
share.SetPermissions(permissions);
}
var sasToken = share.GetSharedAccessSignature(sharedPolicy);
//fileSasUri = new Uri(share.StorageUri.PrimaryUri.ToString() + sasToken);
return new AzureSASToken ()
{
Name = shareName,
Url = share.StorageUri.PrimaryUri.ToString() + "/",
SASToken = sasToken
};
}
Finally using class like this:
var azureFileRest = new AzureFileREST(sasToken);
await azureFileRest.CreateIfNotExist(directoryName);
await azureFileRest.UploadFile(directoryName + "/" + fileName, bytes);

Resources