Issue in creating a file and writing content into it in azure function - azure

I am writing an azure function for creating and uploading text on azure storage but I got error 500 Internal server error. Below is my code of the azure function.
#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.File;
using System;
public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string strFileName = "321rahila.csv";//req.Query["name"];
string Content ="Hello File";
//string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
// dynamic data = JsonConvert.DeserializeObject(requestBody);
// strFileName = strFileName ?? data?.strFileName;
string StorageAccountName = "xyz";
string StorageKey = "i0PNZ6Ykse7oSSfUzFeA36rQfAv9UZnJ5wybQWh5Jol0NRM4sal4s8B3ipkjvfzcsP8/gnI6A==";`enter code here`
string strShareName = "lables";
//string StorageScheme = "SharedKey";
// string FileEndPoint = string.Format("https://{0}.file.core.windows.net/", StorageAccountName);
CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials(StorageAccountName, StorageKey), true);
var fileClient = storageAccount.CreateCloudFileClient();
var share = fileClient.GetShareReference(strShareName);
// if (share.Exists())
{
var rootDir = share.GetRootDirectoryReference();
CloudFile file = rootDir.GetFileReference(strFileName);
var fileToCreate = rootDir.GetFileReference(strFileName);
**fileToCreate.UploadText(Content);**
}
return strFileName != null
? (ActionResult)new OkObjectResult($"Hello, {strFileName}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}
I get the error when I uncomment the line fileToCreate.UploadText(Content); and without it I am unable to create a file and upload text into it. The same is working fine on visual studio.

The problem is the Azure Function in the portal could not find the Microsoft.WindowsAzure.Storage package. The right way is to create project.json file and reference NuGet package explicitly. The below is my project.json file.
{
"frameworks": {
"net46":{
"dependencies": {
"Microsoft.WindowsAzure.Storage": "9.3.3"
}
}
}
}
Then in the run.csx use the assembly.
#r "Microsoft.WindowsAzure.Storage"
And the below is my work code.
#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.File;
using Microsoft.WindowsAzure.Storage.Auth;
public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
string strFileName = "321rahila.csv";//req.Query["name"];
string Content ="Hello File";
string StorageAccountName = "my account";
string StorageKey = "my key";
string strShareName = "windows";
//string StorageScheme = "SharedKey";
// string FileEndPoint = string.Format("https://{0}.file.core.windows.net/", StorageAccountName);
CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials(StorageAccountName, StorageKey), true);
var fileClient = storageAccount.CreateCloudFileClient();
var share = fileClient.GetShareReference(strShareName);
// if (share.Exists())
{
var rootDir = share.GetRootDirectoryReference();
CloudFile file = rootDir.GetFileReference(strFileName);
var fileToCreate = rootDir.GetFileReference(strFileName);
await fileToCreate.UploadTextAsync(Content);
}
return name != null
? (ActionResult)new OkObjectResult($"Hello, {name}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}
Hope this could help you.

Related

Performance of Azure SDK v12 vs Storage Data Movement Library?

I know that the Storage Data Movement Library is supposed to be faster when uploading and downloading files to and from blob storage, but I am not seeing the performance benefits of it when compared to Azure SDK v12. I got an average of 37.463 seconds with Azure SDK v12 and 41.863 seconds using Storage Data Movement Library (SDML).
Here is the code using SDML:
namespace FunctionApp
{
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Blob;
using Microsoft.Azure.Storage.DataMovement;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using System;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Web.Http;
public static class Function1
{
[FunctionName("A")]
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "testRoute")] HttpRequestMessage req,
ILogger log)
{
Stopwatch timer = new Stopwatch();
timer.Start();
try
{
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
TransferManager.Configurations.ParallelOperations = 64;
string fileToDownload = "<URI to zip file in blob storage containing two 300MB files";
string connectionString = "<connection string to storage account>";
string containerName = "<container to upload files to>";
using MemoryStream test = new MemoryStream();
CloudBlockBlob sourceBlob = new CloudBlockBlob(new Uri(fileToDownload));
await TransferManager.DownloadAsync(sourceBlob, test);
CloudStorageAccount account = CloudStorageAccount.Parse(connectionString);
CloudBlobClient blobClient = account.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
using ZipArchive zipArchive = new ZipArchive(test);
foreach (ZipArchiveEntry file in zipArchive.Entries)
{
if (!string.IsNullOrEmpty(file.Name))
{
CloudBlockBlob destBlob = container.GetBlockBlobReference(file.FullName);
using Stream stream = file.Open();
await TransferManager.UploadAsync(stream, destBlob);
}
}
}
catch (Exception exception)
{
return new InternalServerErrorResult();
}
timer.Stop();
return new OkObjectResult(timer.ElapsedMilliseconds);
}
}
}
Here is the code using Azure SDK v12:
namespace FunctionApp
{
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Specialized;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using System;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Web.Http;
public static class Function1
{
[FunctionName("A")]
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "testRoute")] HttpRequestMessage req,
ILogger log)
{
Stopwatch timer = new Stopwatch();
timer.Start();
try
{
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
string fileToDownload = "<URI to zip file in blob storage containing two 300MB files";
string connectionString = "<connection string to storage account>";
string containerName = "<container to upload files to>";
using MemoryStream test = new MemoryStream();
BlockBlobClient client = new BlockBlobClient(new Uri(fileToDownload));
await client.DownloadToAsync(test);
BlobContainerClient containerClient = new BlobContainerClient(connectionString, containerName);
using ZipArchive zipArchive = new ZipArchive(test);
foreach (ZipArchiveEntry file in zipArchive.Entries)
{
if (!string.IsNullOrEmpty(file.Name))
{
BlockBlobClient blockBlobClient = containerClient.GetBlockBlobClient(file.FullName);
using Stream stream = file.Open();
await blockBlobClient.UploadAsync(stream);
}
}
}
catch (Exception exception)
{
return new InternalServerErrorResult();
}
timer.Stop();
return new OkObjectResult(timer.ElapsedMilliseconds) ;
}
}
}
For Data Movement library, you may set ParallelOperations and BlockSize, like below:
TransferManager.Configurations.ParallelOperations = 20;
TransferManager.Configurations.BlockSize = 20971520*2; //20M
I did the test at my side, SDML is more faster.

How to parse Excel spreadsheets in Azure Logic Apps

I need to parse and extract Column information from an Excel spreadsheet using Azure Logic Apps
I have already setup the ability for my Logic App to retrieve the latest unread Emails from my Outlook. Also, my Logic App does a FOR EACH to read all attachments (from unread emails) and make sure they are Excel files (based on filename extension).
I have a basic Excel file that contains 3 columns "Product, Description, Price" I need to parse each row (only Product and Price) column.
I will add the ability to store that parsed into into my SQL table hosted on Azure.
Here you go, you could use HTTP Request as well.
using ExcelDataReader;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Nancy.Json;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Data;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
namespace ConvertExcelToJSon
{
public static class Function1
{
[FunctionName("ConvertToJson")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
string blobName = data?.blobName;
string[] splitBlob = blobName.Split('/');
Stream blob = await GetBlobStreamAsync(splitBlob[1], splitBlob[2] + "/" + splitBlob[3]);
DataSet ds = CreateDataSet(blob);
List<Simple> simpleList = new List<Simple>();
foreach (DataTable table in ds.Tables)
{
return (ActionResult)new OkObjectResult(DataTableToJSON(table));
}
return blobName != null
? (ActionResult)new OkObjectResult($"Hello, {blobName}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}
public static string DataTableToJSON(DataTable table)
{
List<Dictionary<string, object>> list = new List<Dictionary<string, object>>();
foreach (DataRow row in table.Rows)
{
Dictionary<string, object> dict = new Dictionary<string, object>();
foreach (DataColumn col in table.Columns)
{
dict[col.ColumnName] = (Convert.ToString(row[col]));
}
list.Add(dict);
}
JavaScriptSerializer serializer = new JavaScriptSerializer();
return serializer.Serialize(list);
}
public static string AppSetting(this string Key)
{
string ret = string.Empty;
if (Environment.GetEnvironmentVariable(Key) != null)
{
ret = Environment.GetEnvironmentVariable(Key);
}
return ret;
}
public static async Task<Stream> GetBlobStreamAsync(string containerName, string blobName)
{
Stream myBlob = new MemoryStream();
if (CloudStorageAccount.TryParse("AzureWebJobsStorage".AppSetting(), out CloudStorageAccount storageAccount))
{
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = cloudBlobClient.GetContainerReference(containerName);
CloudBlob myBloab = container.GetBlobReference(blobName);
await myBloab.DownloadToStreamAsync(myBlob);
myBlob.Seek(0, SeekOrigin.Begin);
}
return myBlob;
}
public static DataSet CreateDataSet(Stream stream)
{
DataSet ds;
System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);
IExcelDataReader reader = null;
reader = ExcelReaderFactory.CreateOpenXmlReader(stream);
ds = reader.AsDataSet(new ExcelDataSetConfiguration()
{
ConfigureDataTable = (tableReader) => new ExcelDataTableConfiguration()
{
UseHeaderRow = true,
}
});
return ds;
}
}
}
I suggest you call an Azure Function from your logic App and use the Function to convert the Excel into a JSON Object. (I am currently doing this very succesfully) I use ExcelDataReader parse to parse a Blob that the Logic App creates. Send the blob location in the request and respond back with the JSON.

Getting next message in an Azure Service Bus Subscription with an Azure Function, with an HTTP Trigger

I want to create an Azure Function that will fulfill the following requirements:
Trigger upon an HTTP request
Looks at an Azure Service Bus Subscription and gets the next message based on a set of filters specified in the HTTP Request.
If you are using C# you can do something like this:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
namespace HttpTriggerSBRead
{
public static class ReadSBOnHttpTrigger
{
const string ServiceBusConnectionString = "{service bus connection string}";
const string TopicName = "{name of your topic}";
const string SubscriptionName = "{name of your subscription}";
[FunctionName("ReadSBOnHttpTrigger")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
string filter = req.Query["filter"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
filter = filter ?? data?.filter;
SubscriptionClient sqlFilterOnlySubscriptionClient = new SubscriptionClient(ServiceBusConnectionString,
TopicName, SubscriptionName);
await sqlFilterOnlySubscriptionClient.AddRuleAsync(new RuleDescription
{
Filter = new SqlFilter(filter),
Name = filter
});
await ReceiveMessagesAsync(SubscriptionName, log);
await sqlFilterOnlySubscriptionClient.RemoveRuleAsync(filter);
return filter != null
? (ActionResult)new OkObjectResult($"{filter}")
: new BadRequestObjectResult("Please pass a filter on the query string or in the request body");
}
static async Task ReceiveMessagesAsync(string subscriptionName, ILogger log)
{
string subscriptionPath = EntityNameHelper.FormatSubscriptionPath(TopicName, subscriptionName);
IMessageReceiver subscriptionReceiver = new MessageReceiver(ServiceBusConnectionString, subscriptionPath, ReceiveMode.ReceiveAndDelete);
log.LogInformation($"{DateTime.Now} :: Receiving Messages From Subscription: {subscriptionName}");
var receivedMessage = await subscriptionReceiver.ReceiveAsync(TimeSpan.FromSeconds(30));
if (receivedMessage != null)
{
log.LogInformation($"Lable Property = {receivedMessage.Label}");
}
log.LogInformation($"{DateTime.Now} :: Messages From Subscription: {subscriptionName}");
}
}
}
To use it you need to pass in the filter parameter to your function. For example something like this:
http://localhost:7071/api/ReadSBOnHttpTrigger?filter=sys.Label=%27test%27 or http://localhost:7071/api/ReadSBOnHttpTrigger?filter=sys.To=%27test%27
Just for the reference, I used this code as source with some small modifications:
https://github.com/Azure/azure-service-bus/tree/master/samples/DotNet/GettingStarted/Microsoft.Azure.ServiceBus/TopicSubscriptionWithRuleOperationsSample

How to upload a file to a storage location through URL using Azure function app

I want to upload the upload a file to a storage location through URL using Azure function app from Azure blob storage. I'm able to pull the file from Azure blob. But not able to upload the file through url.
Below I have attached the code which i have written. Could anyone help me on this?
#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"
#r "System.IO"
using System;
using System.IO;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Auth;
using System.Xml;
using System.Collections.Generic;
using Newtonsoft.Json;
using System.Net;
public static void Run(string input, TraceWriter log)
{
log.Info($"C# manual trigger function processed\n");
const string StorageAccountName = "";
const string StorageAccountKey = "";
var storageAccount = new CloudStorageAccount(new StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobClient = storageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference("hannahtest");
var Destcontainer = blobClient.GetContainerReference("hannahtestoutput");
var blobs = container.ListBlobs();
log.Info($"Creating Client and Connecting");
foreach (IListBlobItem item in container.ListBlobs(null, false))
{
if (item is CloudBlockBlob blockBlob)
{
using (StreamReader reader = new StreamReader(blockBlob.OpenRead())
{
//old content string will read the blockblob (xml)till end
string oldContent1 = reader.ReadToEnd();
log.Info(oldContent1);
var content = new FormUrlEncodedContent(oldContent1);
var response = await client.PostAsync("http://www.example.com/recepticle.aspx", content);
var responseString = await response.Content.ReadAsStringAsync();
log.Info($"Success");
}
}
}
}
Have a look at Blob Output Binding - that's how the blobs are intended to be uploaded from Azure Functions, without messing with Azure Storage SDK.
Azure function to upload multiple image file to blob storage.
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
public static class ImageUploadFunction
{
[FunctionName("ImageUploadFunction")]
public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post")]HttpRequestMessage req, ILogger log)
{
var provider = new MultipartMemoryStreamProvider();
await req.Content.ReadAsMultipartAsync(provider);
var files = provider.Contents;
List<string> uploadsurls = new List<string>();
foreach (var file in files) {
var fileInfo = file.Headers.ContentDisposition;
Guid guid = Guid.NewGuid();
string oldFileName = fileInfo.FileName;
string newFileName = guid.ToString();
var fileExtension = oldFileName.Split('.').Last().Replace("\"", "").Trim();
var fileData = await file.ReadAsByteArrayAsync();
try {
//Upload file to azure blob storage method
var upload = await UploadFileToStorage(fileData, newFileName + "." + fileExtension);
uploadsurls.Add(upload);
}
catch (Exception ex) {
log.LogError(ex.Message);
return new BadRequestObjectResult("Somthing went wrong.");
}
}
return uploadsurls != null
? (ActionResult)new OkObjectResult(uploadsurls)
: new BadRequestObjectResult("Somthing went wrong.");
}
private static async Task<string> UploadFileToStorage(byte[] fileStream, string fileName)
{
// Create storagecredentials object by reading the values from the configuration (appsettings.json)
StorageCredentials storageCredentials = new StorageCredentials("<AccountName>", "<KeyValue>");
// Create cloudstorage account by passing the storagecredentials
CloudStorageAccount storageAccount = new CloudStorageAccount(storageCredentials, true);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Get reference to the blob container by passing the name by reading the value from the configuration (appsettings.json)
CloudBlobContainer container = blobClient.GetContainerReference("digital-material-library-images");
// Get the reference to the block blob from the container
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName);
// Upload the file
await blockBlob.UploadFromByteArrayAsync(fileStream,0, fileStream.Length);
blockBlob.Properties.ContentType = "image/jpg";
await blockBlob.SetPropertiesAsync();
return blockBlob.Uri.ToString();
//return await Task.FromResult(true);
}
}

A namespace cannot directly contain members in the Azure Function App

TARGET: Do the Azure Function tutorial on and copied code, but got several errors when executing locally on VS2017. I appreciate you help.
https://www.cyotek.com/blog/upload-data-to-blob-storage-with-azure-functions
ERROR 1 - related to Run:
CS0116 A namespace cannot directly contain members such as fields or methods UploadToBlobFunctionApp C:\AzureFunctions\UploadToBlobFunctionApp\UploadToBlobFunctionApp\UploadToBlobFunction.cs 15 Active
ERROR 2 - related to Task CreateBlob:
CS0116 A namespace cannot directly contain members such as fields or methods UploadToBlobFunctionApp
C:\AzureFunctions\UploadToBlobFunctionApp\UploadToBlobFunctionApp\UploadToBlobFunction.cs 45 Active
ERROR 3 - related to await CreateBlob:
CS0103 The name 'CreateBlob' does not exist in the current context UploadToBlobFunctionApp C:\AzureFunctions\UploadToBlobFunctionApp\UploadToBlobFunctionApp\UploadToBlobFunction.cs 36 Active
CODE Function.cs:
using System;
using System.Configuration;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
HttpStatusCode result;
string contentType;
result = HttpStatusCode.BadRequest;
contentType = req.Content.Headers?.ContentType?.MediaType;
if (contentType == "application/json")
{
string body;
body = await req.Content.ReadAsStringAsync();
if (!string.IsNullOrEmpty(body))
{
string name;
name = Guid.NewGuid().ToString("n");
await CreateBlob(name + ".json", body, log);
result = HttpStatusCode.OK;
}
}
return req.CreateResponse(result, string.Empty);
}
private async static Task CreateBlob(string name, string data,
TraceWriter log)
{
string accessKey;
string accountName;
string connectionString;
CloudStorageAccount storageAccount;
CloudBlobClient client;
CloudBlobContainer container;
CloudBlockBlob blob;
accessKey = "qwertyw4VhRajxlZn9C4hTMB8oSwE4klNUsvTy9VeTCIQ11111vFVVGExDwJ+JUboFv2B79j+W6foqLWE92w==";
accountName = "mystorage";
connectionString = "DefaultEndpointsProtocol=https;AccountName=" + accountName + ";AccountKey=" + accessKey + ";EndpointSuffix=core.windows.net";
storageAccount = CloudStorageAccount.Parse(connectionString);
client = storageAccount.CreateCloudBlobClient();
container = client.GetContainerReference("functionupload");
await container.CreateIfNotExistsAsync();
blob = container.GetBlockBlobReference(name);
blob.Properties.ContentType = "application/json";
using (Stream stream = new MemoryStream(Encoding.UTF8.GetBytes(data)))
{
await blob.UploadFromStreamAsync(stream);
}
}
The example that you are referencing is using scripted functions (csx file). They are mostly used while editing code directly in Azure portal.
I think you are trying to create a precompiled application with csproj and cs files. In this case, your code should be a valid C#, i.e. all methods should be inside classes.
Have a look at this example.
You can also use attributes to mark your functions and triggers instead of authoring function.json manually, see examples here.

Resources