Azure function to route message to queue namebased on message data - azure

Im looking to write an Azure function that takes in a message from IoTHub and places it on a Service bus queue based on data in the message. The thing is, the queue name will changed based on the parameters of the message and the documentation I’ve looked at, such as this all set the queue name as part of the output settings.
Has anyone done anything similar or have any code samples to write to a queue based upon a value in the incoming message? I'm trying to set a function up but it wont even let me set the output type to service bus and not put the queue name in, which I can't, as it isn't fixed!
So if the input was:
{
“queue”: “MyQ12345”,
“data”: “some data here”
}
The function should write the data field (or the entire incoming message) to the service bus queue “MyQ12345” which already exists on the same subscription/resource group etc.
Thanks
EDIT1: Here's what I've got thus far:
So I've tried what #Mikhail has suggested, here is what I'm doing:
run.csx
using System;
public static string Run(MyPoco myEventHubMessage, TraceWriter log, out string queue)
{
var queueName = myEventHubMessage.QueueName;
queue = queueName;
log.Info($"<IoT Hub => ServiceBus> C# Event Hub trigger function processed a message: {queueName}");
return QueueName;
}
public class MyPoco
{
public string QueueName { get; set; }
public double Other { get; set; }
public double Props { get; set; }
public int Here { get; set; }
}
function.json:
{
"type": "serviceBus",
"name": "$return",
"direction": "out",
"queueName": "{queue}",
"connection": "ServiceBusConnectionString",
"accessRights": "manage"
}
but this gives me the following error:
Microsoft.Azure.WebJobs.Host: Cannot bind parameter 'queue' to type
String&. Make sure the parameter Type is supported by the binding.

You can define a class to make your message strongly typed:
public class MyItem
{
public string Queue { get; set; }
public string Data { get; set; }
}
then accept it as function input parameter:
public string Run(MyItem item)
{
return item.Data;
}
and define the output binding like this:
{
"type": "serviceBus",
"name": "$return",
"queueName": "{queue}",
"connection": "...",
"accessRights_": "Manage",
"direction": "out"
}

Another way to do is use the custom endpoints and message routing for Azure IoT Hub and directly have the messages sent to the SB queue of your choice from IoT Hub without passing through a Function.
For now IoT Hub routing only works on message properties, not yet on the messages body (which is in the plan of record) but you can put the queue name in a message property and simply define a query to route the messages to the right queue based on this property.
There are some limitations such as the number of custom endpoints you can set for IoT Hub as well as predefine what the queues should be, but depending on your scenario that could work for you.

Related

Azure Event Grid Cloud Events Schema V1.0 with Azure Event Grid Domains and Custom Topics Documentation Not Clear

I have elected to leverage Azure Event Grid in an enterprise multi-tenant model application. I also want to use Cloud Events instead of the proprietary AEG format. I am using AEG domains for each tenant and then I want a custom topic and subject for my messages. The v0.1 of cloud events had a "#" delimited property for cloud events topic and subjects. It looks like V1.0 does not anymore? It really is not clear in the Azure docs.
Secondarily, with Azure Event Grid Domains it seems you can only create a Domain Topic via Powershell (https://learn.microsoft.com/en-us/cli/azure/eventgrid/domain/topic?view=azure-cli-latest) and not in the portal. I can't find a clear way to create a topic for an event domain any other way.
My topic is currently set to : /providers/Microsoft.EventGrid/domains/{tenantname}/topics/refresh.
Do domain topics just appear once they're published for the first time?
Any insight on the format of the cloud events schema and managing topics would be great!
So I now see the UI for this in Azure Portal. You simply, add an event subscription on the domain and one of the options is to filer by topic, where you add your topics.
https://learn.microsoft.com/bs-latn-ba/azure/event-grid/how-to-event-domains?tabs=azurecli#create-topics-and-subscriptions
It is clear here that "There's no separate step to create a topic in a domain.".
Secondarily, I was able to set source = topic for Cloud Events v1.0 and separate out subject as well. Here is my CloudEvent generic class :
public class CloudEvent<T> where T : class
{
[JsonProperty("id")]
public string EventId
{
protected set { }
get => Guid.NewGuid().ToString();
}
[JsonProperty("specversion")]
public string CloudEventVersion
{
protected set { }
get => "1.0";
}
[JsonProperty("type")]
public string EventType { get; set; }
[JsonProperty("eventTypeVersion")]
public string EventTypeVersion
{
protected set { }
get => "1.0";
}
[JsonProperty("source")]
public string Source { get; set; }
[JsonProperty("subject")]
public string Subject { get; set; }
[JsonProperty("time")]
public string Time
{
protected set { }
get => DateTime.UtcNow.ToString(CultureInfo.InvariantCulture);
}
[JsonProperty("data")]
public T Data { get; set; }
}
My topic (which is set to source property for cloud events) is :
/resourceGroups/{rgname}/providers/Microsoft.EventGrid/domains/{domainname}/topics/{topic}
I think this would also set subject properly according to this schema.
https://learn.microsoft.com/en-us/azure/event-grid/cloudevents-schema

Azure Function: How to add row to cloud table on blob creating?

I'm developing an Azure Function which should add new line to an Azure table when new a new blob is added. The application has many containers in Blob Storage, and my Azure Function should process all blobs from all containers.
I tried to implement event getting with EventGrid, but it gives an error.
My Azure function:
#r "D:\home\site\wwwroot\BlobCreatedFunction\Microsoft.Azure.EventGrid.dll"
#r"D:\home\site\wwwroot\BlobCreatedFunction\Microsoft.WindowsAzure.Storage.dll"
using Microsoft.Azure.EventGrid.Models;
using Microsoft.WindowsAzure.Storage.Table;
using System;
public class TemporaryBlobEntity : TableEntity
{
public TemporaryBlobEntity(string partitionKey, string rowKey)
{
this.PartitionKey = partitionKey;
this.RowKey = rowKey;
}
public string BlobUrl { get; set; }
public DateTime BlobUploaded { get; set; }
}
public static TemporaryBlobEntity Run(EventGridEvent eventGridEvent, ILogger log)
{
if (eventGridEvent.Data is StorageBlobCreatedEventData eventData)
{
log.LogInformation(eventData.Url);
log.LogInformation(eventGridEvent.Data.ToString());
var temporaryBlob = new TemporaryBlobEntity("blobs", eventData.Url)
{
BlobUrl = eventData.Url,
BlobUploaded = DateTime.UtcNow
};
return temporaryBlob;
}
return null;
}
Here is my integration JSON:
{
"bindings": [
{
"type": "eventGridTrigger",
"name": "eventGridEvent",
"direction": "in"
},
{
"type": "table",
"name": "$return",
"tableName": "temporaryBlobs",
"connection": "AzureWebJobsStorage",
"direction": "out"
}
]
}
In my Azure Function settings, I added the value for AzureWebJobsStorage.
When I press Run in the test section, logs show:
2019-07-08T13:52:16.756 [Information] Executed 'Functions.BlobCreatedFunction' (Succeeded, Id=6012daf1-9b98-4892-9560-932d05857c3e)
Looks good, but there is no new item in cloud table. Why?
Then I tried to connect my function with EventGrid topic. I filled new subscription form, selected "Web Hook" as endpoint type, and set subscriber endpoint at: https://<azure-function-service>.azurewebsites.net/runtime/webhooks/EventGrid?functionName=<my-function-name>. Then I got the following error message:
Deployment has failed with the following error: {"code":"Url validation","message":"The attempt to validate the provided endpoint https://####.azurewebsites.net/runtime/webhooks/EventGrid failed. For more details, visit https://aka.ms/esvalidation."}
As far as I can understand, the application needs some kind of request validation. Do I really need to implement validation in each of my azure functions? Or shoudl I use another endpoint type?
When you enter a webhook into Event Grid it sends out a request to verify that you actually have permissions on that endpoint. The easiest way to connect a Function to Event Grid is to create the subscription from the Functions app instead of the Event Grid blade.
Opening up the Function in the portal you should find a link at the top to "Add Event Grid subscription". Even if the Functions app was created locally and published to Azure so the code isn't viewable the link will be available.
This will open up the screen for creating an Event Grid subscription. The difference is that instead of the Event Grid topic info being prefilled, the web hook info is prepopulated for you. Fill in the info about the Event Grid topic to finish creating the subscription.
If you decide you want to implement the validation response for whatever reason, it is possible to do this by checking the type of the message.
// Validate whether EventType is of "Microsoft.EventGrid.SubscriptionValidationEvent"
if (eventGridEvent.EventType == "Microsoft.EventGrid.SubscriptionValidationEvent")
{
var eventData = (SubscriptionValidationEventData)eventGridEvent.Data;
// Do any additional validation (as required) such as validating that the Azure resource ID of the topic matches
// the expected topic and then return back the below response
var responseData = new SubscriptionValidationResponse()
{
ValidationResponse = eventData.ValidationCode
};
if (responseData.ValidationResponse != null)
{
return Ok(responseData);
}
}
else
{
//Your code here
}
There is also an option to validate the link manually by getting the validation link out of the validation message and navigating to it in your browser. This method is primarily for 3rd party services where you can't add the validation code.
The following are changes in your EventGridTrigger function:
#r "Microsoft.WindowsAzure.Storage"
#r "Microsoft.Azure.EventGrid"
#r "Newtonsoft.Json"
using System;
using Newtonsoft.Json.Linq;
using Microsoft.Azure.EventGrid.Models;
using Microsoft.WindowsAzure.Storage.Table;
public static TemporaryBlobEntity Run(EventGridEvent eventGridEvent, ILogger log)
{
log.LogInformation(eventGridEvent.Data.ToString());
var eventData = (eventGridEvent.Data as JObject)?.ToObject<StorageBlobCreatedEventData>();
if(eventData?.Api == "PutBlob")
{
log.LogInformation(eventData.Url);
return new TemporaryBlobEntity("blobs", eventData.Sequencer)
{
BlobUrl = eventData.Url,
BlobUploaded = DateTime.UtcNow
};
}
return null;
}
public class TemporaryBlobEntity : TableEntity
{
public TemporaryBlobEntity(string partitionKey, string rowKey)
{
this.PartitionKey = partitionKey;
this.RowKey = rowKey;
}
public string BlobUrl { get; set; }
public DateTime BlobUploaded { get; set; }
}
Notes:
You don't need to validate an EventGridTrigger function for AEG subscription webhook endpoint. This validation is built-in the preprocessing of the EventGridTrigger function.
The eventGridEvent.Data property is a JObject and must be converted (deserialized) to the StorageBlobCreatedEventData object, see here.
For RowKey (and PartitionKey) see the restriction characters in here, so I changed it to the Sequencer value in this example.
The AEG subscription webhook endpoint for the EventGridTrigger function has the following format:
https://{azure-function-service}.azurewebsites.net/runtime/webhooks/EventGrid?functionName={my-function-name}&code={systemkey}

Azure EventHub and Durable Functions

Literally trying out to make do of something I am not good at.
I have read upon the durable function overview here - https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview.
There is a topic on using Bindings to use it on an Event Hub Trigger, but there is not really a working example, I followed what was there and came up with this binding in my function.json,
{
"bindings": [
{
"type": "eventHubTrigger",
"name": "myEventHubMessage",
"direction": "in",
"path": "testinhub",
"connection": "Endpoint=sb://dev-testingeventhubinns.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=lLassdff5Y/esH8/CaXDOWH0jF2JtZBQhQeFoCtfqYs=",
"consumerGroup": "$Default"
},
{
"type": "eventHub",
"name": "outputEventHubMessage",
"connection": "Endpoint=sb://dev-testingeventhuboutns.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=4yuasdff7Lzu+mQJFVlnlozUItqFY1L3WW/kJnpTjq8=",
"path": "testouthub",
"direction": "out"
}
],
"disabled": false,
"entryPoint": "EventTriggerFunction.EventHubTriggerClass.Run"
}
My code in entirety is as follows,
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.EventHubs;
using System;
namespace EventTriggerFunction
{
public static class EventHubTriggerClass
{
[FunctionName("EventHubTrigger")]
public static async Task<List<string>> Run([OrchestrationTrigger] DurableOrchestrationContext context)
{
await context.CallActivityAsync<string>("EventHubTrigger_Send", "Hello World");
return null;
}
[FunctionName("EventHubTrigger_Send")]
public static void SendMessages([EventHubTrigger("testinhub", Connection = "ConnectionValue")] EventData[] eventHubMessages, ILogger log)
{
var exceptions = new List<Exception>();
foreach (EventData message in eventHubMessages)
{
try
{
log.LogInformation($"C# Event Hub trigger function processed a message: {Encoding.UTF8.GetString(message.Body)}");
}
catch (Exception e)
{
// We need to keep processing the rest of the batch - capture this exception and continue.
// Also, consider capturing details of the message that failed processing so it can be processed again later.
exceptions.Add(e);
}
}
}
}
}
If I send a message using the function, I can not see it on my testouthub event hub. Not really sure how this Durable function and EventHub Trigger works hand in hand.
I think you are mixing it up a bit. When using the attributes like FunctionName and EventHubTrigger you do not need to supply the function.json. It's either function.json or with attributes.
If you are trying to receive messages from 1 eventhub and passing it on to the next, then something like this below will also do the trick.
There's no need to use DurableFunctions for this and the Azure Function runtime will scale by itself if there are many messages, see this
[FunctionName("EventHubTriggerCSharp")]
[return: EventHub("outputEventHubMessage", Connection = "EventHubConnectionAppSetting")]
public static void Run([EventHubTrigger("samples-workitems", Connection = "EventHubConnectionAppSetting")] string myEventHubMessage, ILogger log)
{
log.LogInformation($"C# Event Hub trigger function processed a message: {myEventHubMessage}");
return myEventHubMessage;
}
An additional tip: I would not paste your full connectionstring into StackOverflow. Maybe it's wise to immediately create new AccessKeys for your eventhubs.

read SB message with azure functions service bus trigger

I just created a simple azure function using service bus trigger. I am using the default example provided. I can read the messageid in the code below
public static void Run(string mySbMsg, TraceWriter log)
{
log.Info($"C# ServiceBus topic trigger function processed message:
{mySbMsg}");
}
I`m struggling to find codes showing how to read the json message that was posted.
Thanks for you help
You can use the BrokeredMessage parameter to get the message body in Azure Function Service Bus Trigger.
This will return the message with the BrokeredMessage.GetBody() method.
Get more information here.
In Azure portal, add "project.json" in View Files. This the library which contains BrokeredMessage object.
The project.json should look like
{
"frameworks":
{
"net46":
{
"dependencies":
{
"WindowsAzure.ServiceBus": "4.1.2"
}
}
}
}
When you save, you can see the package gets restored.
Inside the Run method, add BrokeredMessage as parameter. The method should look like
public static void Run(BrokeredMessage message, TraceWriter log)
{
string messageBody = message.Properties["Message"].ToString();
string messageId = message.Properties["Id"].ToString();
log.Info($"message - " + messageBody + " Id " + messageId);
}
Don't forget to add Using Microsoft.ServiceBus.Messaging in "Run.csx" and change the name property to message in "Function.json"

Why Azure function is not writing to Service Bus Topic

My function is like
[FunctionName("MyFunctionName")]
[return: ServiceBus("mytopic", Connection = "ServiceBusConnectionString")]
public static async Task<string> MyFunctionAsync([QueueTrigger("my-input-queue")] string msgIn, TraceWriter log)
{
My local.settings.json has
{
"IsEncrypted": false,
"Values": {
"ServiceBusConnectionString": "[my connection string]"
}
}
where [my connection string] is copy-pasted from a Primary Connecting String under one of the Shared access policies with a Send claim.
This just silently fails: Messages get stuck in my-input-queue and no errors are written to log streaming. However I'm 100% sure the attribute is the issue because I've deployed 100 different combinations of this to try and make it work :).
Any ideas?
Based on my test,it should work with servicebus attribute. The following is my test code.
[return: ServiceBus("topicName",Connection = "ServiceBusConnectionString", EntityType = EntityType.Topic)]
public static async Task<string>Run([QueueTrigger("queueName")]string myQueueItem, TraceWriter log)
{
...
return myQueueItem; // write to the Topic.
}
local.settings.json
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "xxxxxx",
"AzureWebJobsDashboard": "xxxxxxxx",
"ServiceBusConnectionString": "xxxxxx"
}
}
You could get more information about Azure Service Bus output binding from this tutorial. You also could do that with follwoing way
[FunctionName("ServiceBusOutput")]
public static void Run([[QueueTrigger("queueName")]string myQueueItem,
TraceWriter log,
[ServiceBus("topicName",Connection = "ServiceBusConnectionString", EntityType = EntityType.Topic)]out string queueMessage)
{
log.Info("Azure Function Demo - Azure Service Bus Queue Topic");
queueMessage = myQueueItem;
}
You are missing the required settings for your QueueTrigger, so your function isn't triggering on new items in the queue. You should have values for AzureWebJobsStorage and AzureWebJobsDashboard, and your QueueTrigger should have a value for the Connection field.
For more information about how to wire up QueueTriggers and test locally, see this answer.

Resources