Azure Function: How to add row to cloud table on blob creating? - azure

I'm developing an Azure Function which should add new line to an Azure table when new a new blob is added. The application has many containers in Blob Storage, and my Azure Function should process all blobs from all containers.
I tried to implement event getting with EventGrid, but it gives an error.
My Azure function:
#r "D:\home\site\wwwroot\BlobCreatedFunction\Microsoft.Azure.EventGrid.dll"
#r"D:\home\site\wwwroot\BlobCreatedFunction\Microsoft.WindowsAzure.Storage.dll"
using Microsoft.Azure.EventGrid.Models;
using Microsoft.WindowsAzure.Storage.Table;
using System;
public class TemporaryBlobEntity : TableEntity
{
public TemporaryBlobEntity(string partitionKey, string rowKey)
{
this.PartitionKey = partitionKey;
this.RowKey = rowKey;
}
public string BlobUrl { get; set; }
public DateTime BlobUploaded { get; set; }
}
public static TemporaryBlobEntity Run(EventGridEvent eventGridEvent, ILogger log)
{
if (eventGridEvent.Data is StorageBlobCreatedEventData eventData)
{
log.LogInformation(eventData.Url);
log.LogInformation(eventGridEvent.Data.ToString());
var temporaryBlob = new TemporaryBlobEntity("blobs", eventData.Url)
{
BlobUrl = eventData.Url,
BlobUploaded = DateTime.UtcNow
};
return temporaryBlob;
}
return null;
}
Here is my integration JSON:
{
"bindings": [
{
"type": "eventGridTrigger",
"name": "eventGridEvent",
"direction": "in"
},
{
"type": "table",
"name": "$return",
"tableName": "temporaryBlobs",
"connection": "AzureWebJobsStorage",
"direction": "out"
}
]
}
In my Azure Function settings, I added the value for AzureWebJobsStorage.
When I press Run in the test section, logs show:
2019-07-08T13:52:16.756 [Information] Executed 'Functions.BlobCreatedFunction' (Succeeded, Id=6012daf1-9b98-4892-9560-932d05857c3e)
Looks good, but there is no new item in cloud table. Why?
Then I tried to connect my function with EventGrid topic. I filled new subscription form, selected "Web Hook" as endpoint type, and set subscriber endpoint at: https://<azure-function-service>.azurewebsites.net/runtime/webhooks/EventGrid?functionName=<my-function-name>. Then I got the following error message:
Deployment has failed with the following error: {"code":"Url validation","message":"The attempt to validate the provided endpoint https://####.azurewebsites.net/runtime/webhooks/EventGrid failed. For more details, visit https://aka.ms/esvalidation."}
As far as I can understand, the application needs some kind of request validation. Do I really need to implement validation in each of my azure functions? Or shoudl I use another endpoint type?

When you enter a webhook into Event Grid it sends out a request to verify that you actually have permissions on that endpoint. The easiest way to connect a Function to Event Grid is to create the subscription from the Functions app instead of the Event Grid blade.
Opening up the Function in the portal you should find a link at the top to "Add Event Grid subscription". Even if the Functions app was created locally and published to Azure so the code isn't viewable the link will be available.
This will open up the screen for creating an Event Grid subscription. The difference is that instead of the Event Grid topic info being prefilled, the web hook info is prepopulated for you. Fill in the info about the Event Grid topic to finish creating the subscription.
If you decide you want to implement the validation response for whatever reason, it is possible to do this by checking the type of the message.
// Validate whether EventType is of "Microsoft.EventGrid.SubscriptionValidationEvent"
if (eventGridEvent.EventType == "Microsoft.EventGrid.SubscriptionValidationEvent")
{
var eventData = (SubscriptionValidationEventData)eventGridEvent.Data;
// Do any additional validation (as required) such as validating that the Azure resource ID of the topic matches
// the expected topic and then return back the below response
var responseData = new SubscriptionValidationResponse()
{
ValidationResponse = eventData.ValidationCode
};
if (responseData.ValidationResponse != null)
{
return Ok(responseData);
}
}
else
{
//Your code here
}
There is also an option to validate the link manually by getting the validation link out of the validation message and navigating to it in your browser. This method is primarily for 3rd party services where you can't add the validation code.

The following are changes in your EventGridTrigger function:
#r "Microsoft.WindowsAzure.Storage"
#r "Microsoft.Azure.EventGrid"
#r "Newtonsoft.Json"
using System;
using Newtonsoft.Json.Linq;
using Microsoft.Azure.EventGrid.Models;
using Microsoft.WindowsAzure.Storage.Table;
public static TemporaryBlobEntity Run(EventGridEvent eventGridEvent, ILogger log)
{
log.LogInformation(eventGridEvent.Data.ToString());
var eventData = (eventGridEvent.Data as JObject)?.ToObject<StorageBlobCreatedEventData>();
if(eventData?.Api == "PutBlob")
{
log.LogInformation(eventData.Url);
return new TemporaryBlobEntity("blobs", eventData.Sequencer)
{
BlobUrl = eventData.Url,
BlobUploaded = DateTime.UtcNow
};
}
return null;
}
public class TemporaryBlobEntity : TableEntity
{
public TemporaryBlobEntity(string partitionKey, string rowKey)
{
this.PartitionKey = partitionKey;
this.RowKey = rowKey;
}
public string BlobUrl { get; set; }
public DateTime BlobUploaded { get; set; }
}
Notes:
You don't need to validate an EventGridTrigger function for AEG subscription webhook endpoint. This validation is built-in the preprocessing of the EventGridTrigger function.
The eventGridEvent.Data property is a JObject and must be converted (deserialized) to the StorageBlobCreatedEventData object, see here.
For RowKey (and PartitionKey) see the restriction characters in here, so I changed it to the Sequencer value in this example.
The AEG subscription webhook endpoint for the EventGridTrigger function has the following format:
https://{azure-function-service}.azurewebsites.net/runtime/webhooks/EventGrid?functionName={my-function-name}&code={systemkey}

Related

Adding Custom Dimension to Request Telemetry - Azure functions

I am creating a new Function app using v2.x and I am integrating Application Insights for request logging that is automatically being done as Azure Function is now integrated with App Insights (as mentioned in the documentation link). What I would need to do is log few custom fields in the custom dimensions in Application Insights Request Telemetry. Is it possible without using Custom Request logging (using TrackRequest method)
About adding custom properties, you could refer to this tutorial:Add properties: ITelemetryInitializer. The below is my test a HTTP trigger function.
public static class Function1
{
private static string key = "Your InstrumentationKey";
private static TelemetryClient telemetry = new TelemetryClient() { InstrumentationKey = key };
[FunctionName("Function1")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
if (!telemetry.Context.Properties.ContainsKey("Function_appName"))
{
telemetry.Context.Properties.Add("Function_appName", "testfunc");
}
else
{
telemetry.Context.Properties["Function_appName"] = "testfunc";
}
telemetry.TrackEvent("eventtest");
telemetry.TrackTrace("tracetest");
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
return name != null
? (ActionResult)new OkObjectResult($"Hello, {name}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}
}
After running this function, go to the Application Insights Search could check the data Or go to Logs(Analytics).
Update:
You should use ITelemetry Initializer(which can add custom dimension to a specified telemetry like only for request) in function app, please follow the steps below:
1.In Visual studio, create a function app(In my test, I create a blob triggerd function), and install the following nuget packages:
Microsoft.ApplicationInsights, version 2.10.0
Microsoft.NET.Sdk.Functions, version 1.0.29
2.Then in the Function1.cs, write code like below:
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using System.IO;
[assembly: WebJobsStartup(typeof(FunctionApp21.MyStartup))]
namespace FunctionApp21
{
public static class Function1
{
[FunctionName("Function1")]
public static void Run([BlobTrigger("samples-workitems/{name}", Connection = "AzureWebJobsStorage")]Stream myBlob, string name, ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
}
}
internal class MyTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
//use telemetry is RequestTelemetry to make sure only add to request
if (telemetry != null && telemetry is RequestTelemetry && !telemetry.Context.GlobalProperties.ContainsKey("my_custom_dimen22"))
{
telemetry.Context.GlobalProperties.Add("my_custom_dimen22", "Hello, this is custom dimension for request!!!");
}
}
}
public class MyStartup : IWebJobsStartup
{
public void Configure(IWebJobsBuilder builder)
{
builder.Services.AddSingleton<ITelemetryInitializer, MyTelemetryInitializer>();
}
}
}
3.Publish it to azure, then nav to azure portal -> the published function app -> Monitor -> Add an application insights.
4.Run the function from azure. And wait for a few minutes -> nav to the application insights portal, check the telemetry data, and you can see the custom dimension is only added to request telemetry:
The other solutions don't quite answer the question, how to add custom properties to the request telemetry. There is a very simple solution, add the following within your function's code:
Activity.Current?.AddTag("my_prop", "my_value");
You'll need:
using System.Diagnostics;
This then can be dynamic per function invocation / request, rather a fixed global property.

read SB message with azure functions service bus trigger

I just created a simple azure function using service bus trigger. I am using the default example provided. I can read the messageid in the code below
public static void Run(string mySbMsg, TraceWriter log)
{
log.Info($"C# ServiceBus topic trigger function processed message:
{mySbMsg}");
}
I`m struggling to find codes showing how to read the json message that was posted.
Thanks for you help
You can use the BrokeredMessage parameter to get the message body in Azure Function Service Bus Trigger.
This will return the message with the BrokeredMessage.GetBody() method.
Get more information here.
In Azure portal, add "project.json" in View Files. This the library which contains BrokeredMessage object.
The project.json should look like
{
"frameworks":
{
"net46":
{
"dependencies":
{
"WindowsAzure.ServiceBus": "4.1.2"
}
}
}
}
When you save, you can see the package gets restored.
Inside the Run method, add BrokeredMessage as parameter. The method should look like
public static void Run(BrokeredMessage message, TraceWriter log)
{
string messageBody = message.Properties["Message"].ToString();
string messageId = message.Properties["Id"].ToString();
log.Info($"message - " + messageBody + " Id " + messageId);
}
Don't forget to add Using Microsoft.ServiceBus.Messaging in "Run.csx" and change the name property to message in "Function.json"

Azure Event Grid - Event Delivery Security at Azure Function

I have been working on a POC related to Azure Event Grid integration with Azure Function. I am stuck on the Event delivery Security as mentioned here.
I am using Event Grid Trigger which is sent by built-in Event grid Subscription in Azure Blob Storage. I have added an access token as a query parameter in WebHook endpoint as mentioned in the above URL.
But I cannot access that parameter in the Function code. Can someone share a sample for doing this?
FYI - Below is the function definition in my code.
[FunctionName("EventGridFunc")]
public static void Run([EventGridTrigger]EventGridEvent eventGridEvent,
TraceWriter log)
{
log.Info("Received a trigger.");
log.Info(eventGridEvent.Data.ToString());
}
The full subscriberUrl for EventGridTrigger function has the following format:
https://{FunctionAppName}.azurewebsites.net/admin/extensions/EventGridExtensionConfig?functionName={EventGridTriggerFunctionName}&code={masterKey}
As you can see, the EventGridTrigger is basically a special HttpTrigger (push) function with a "hidden pre-processing" of the event message for its validation.
Update:
I didn't see a way how to obtain a query string in the EventGridTrigger. However, there are few workarounds for your solution such as:
using an Application settings
using an Azure Key Vault for storing a secret
using a HttpTrigger instead of EventGridTrigger
The following code snippet shows an example of the HttpTrigger function for EventGrid (version 2018-05-01-preview) subscriber:
#r "Newtonsoft.Json"
using System.Net;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Linq;
using System.Threading.Tasks;
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, IDictionary<string, string> query, TraceWriter log)
{
log.Info("C# HTTP trigger function processed an EventGrid request.");
log.Info($"\nHeaders:\n\t{string.Join("\n\t", req.Headers.Where(i => i.Key.StartsWith("aeg-")).Select(i => $"{i.Key}={i.Value.First()}"))}");
log.Info($"\nQuery:\n\t{string.Join("\n\t", query.Select(i => $"{i.Key}={i.Value}"))}");
string eventGridValidationHeader = req.Headers.FirstOrDefault( x => string.Compare(x.Key,"Aeg-Event-Type", true) == 0).Value?.FirstOrDefault().Trim();
// media type = application/json or application/cloudevents+json
string jsontext = null;
var jtoken = JToken.Parse(await req.Content.ReadAsStringAsync());
log.Info($"\n{jtoken.ToString(Newtonsoft.Json.Formatting.Indented)}");
if (jtoken is JArray)
jsontext = jtoken.SingleOrDefault<JToken>().ToString();
else if (jtoken is JObject)
jsontext = jtoken.ToString();
var eventGridEvent = JsonConvert.DeserializeAnonymousType(jsontext, new { EventType = "", Data = new JObject()});
if (string.IsNullOrEmpty(eventGridValidationHeader) || string.IsNullOrEmpty(eventGridEvent?.EventType) || eventGridEvent?.Data == null)
{
return req.CreateErrorResponse(HttpStatusCode.BadRequest, "No EventGrid message.");
}
if (eventGridValidationHeader == "SubscriptionValidation" && eventGridEvent.EventType == "Microsoft.EventGrid.SubscriptionValidationEvent")
{
log.Verbose(#"Event Grid Validation event received.");
return req.CreateResponse(HttpStatusCode.OK, JsonConvert.DeserializeObject(JsonConvert.SerializeObject(new { validationResponse = ((dynamic)eventGridEvent.Data).validationCode })));
}
#region Event Processing
// for testing a retry delivery policy
//return req.CreateResponse(HttpStatusCode.BadRequest, "Testing");
#endregion
return req.CreateResponse(HttpStatusCode.NoContent);
}

Azure function to route message to queue namebased on message data

Im looking to write an Azure function that takes in a message from IoTHub and places it on a Service bus queue based on data in the message. The thing is, the queue name will changed based on the parameters of the message and the documentation I’ve looked at, such as this all set the queue name as part of the output settings.
Has anyone done anything similar or have any code samples to write to a queue based upon a value in the incoming message? I'm trying to set a function up but it wont even let me set the output type to service bus and not put the queue name in, which I can't, as it isn't fixed!
So if the input was:
{
“queue”: “MyQ12345”,
“data”: “some data here”
}
The function should write the data field (or the entire incoming message) to the service bus queue “MyQ12345” which already exists on the same subscription/resource group etc.
Thanks
EDIT1: Here's what I've got thus far:
So I've tried what #Mikhail has suggested, here is what I'm doing:
run.csx
using System;
public static string Run(MyPoco myEventHubMessage, TraceWriter log, out string queue)
{
var queueName = myEventHubMessage.QueueName;
queue = queueName;
log.Info($"<IoT Hub => ServiceBus> C# Event Hub trigger function processed a message: {queueName}");
return QueueName;
}
public class MyPoco
{
public string QueueName { get; set; }
public double Other { get; set; }
public double Props { get; set; }
public int Here { get; set; }
}
function.json:
{
"type": "serviceBus",
"name": "$return",
"direction": "out",
"queueName": "{queue}",
"connection": "ServiceBusConnectionString",
"accessRights": "manage"
}
but this gives me the following error:
Microsoft.Azure.WebJobs.Host: Cannot bind parameter 'queue' to type
String&. Make sure the parameter Type is supported by the binding.
You can define a class to make your message strongly typed:
public class MyItem
{
public string Queue { get; set; }
public string Data { get; set; }
}
then accept it as function input parameter:
public string Run(MyItem item)
{
return item.Data;
}
and define the output binding like this:
{
"type": "serviceBus",
"name": "$return",
"queueName": "{queue}",
"connection": "...",
"accessRights_": "Manage",
"direction": "out"
}
Another way to do is use the custom endpoints and message routing for Azure IoT Hub and directly have the messages sent to the SB queue of your choice from IoT Hub without passing through a Function.
For now IoT Hub routing only works on message properties, not yet on the messages body (which is in the plan of record) but you can put the queue name in a message property and simply define a query to route the messages to the right queue based on this property.
There are some limitations such as the number of custom endpoints you can set for IoT Hub as well as predefine what the queues should be, but depending on your scenario that could work for you.

Azure Functions and DocumentDB triggers

Is it possible to specify the DocumentDB is to fire triggers when writing to DocumentDB?
I have an Azure function that pulls JSON messages off a Service Bus Queue and puts them into DocumentDB like so:
using System;
using System.Threading.Tasks;
public static string Run(string myQueueItem, TraceWriter log)
{
log.Info($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
return myQueueItem;
}
This inserts new documents into the database as they are added to the service bus queue, however I need DocumentDB to process these as they are added and add attachments. This cannot be done in the present setup and I would like to tell DocumentDB to fire a trigger.
I have tried something like this:
using System;
using System.Threading.Tasks;
public static string Run(string myQueueItem, TraceWriter log)
{
log.Info($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
return "x-ms-documentdb-post-trigger-include: addDocument\n" + myQueueItem;
}
It doesn't work and gives me errors like this:
Exception while executing function:
Functions.ServiceBusQueueTriggerCSharp1. Microsoft.Azure.WebJobs.Host:
Error while handling parameter _return after function returned:.
Newtonsoft.Json: Unexpected character encountered while parsing value:
x. Path '', line 0, position 0.
I like this setup because I can saturate the queue with requests to add records and they just buffer until the database can deal with it, which deals with spikes in demand, but it allows data offload from the client machine as fast as the network can carry it and then the queue/database combination gets caught up when demand drops again.
You could refer to the following code sample to create document with the trigger enabled in Azure Functions.
using System;
using System.Threading.Tasks;
using Microsoft.Azure.Documents;
using Microsoft.Azure.Documents.Client;
public static void Run(string myQueueItem, TraceWriter log)
{
string EndpointUri = "https://{documentdb account name}.documents.azure.com:443/";
string PrimaryKey = "{PrimaryKey}";
DocumentClient client = new DocumentClient(new Uri(EndpointUri), PrimaryKey);
client.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri("{databaseid}", "{collenctionid}"), new MyChunk { MyProperty = "hello" },
new RequestOptions
{
PreTriggerInclude = new List<string> { "YourTriggerName" },
}).Wait();
log.Info($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
}
public class MyChunk
{
public string MyProperty { get; set; }
}
Note: for using Microsoft.Azure.DocumentDB NuGet package in a C# function, please upload a project.json file to the function's folder in the function app's file system.
project.json
{
"frameworks": {
"net46":{
"dependencies": {
"Microsoft.Azure.DocumentDB": "1.13.1"
}
}
}
}
Besides, please make sure you have created triggers in your DocumentDB, for details about creating triggers, please refer to this article.

Resources