Azure - Debugging Iot Hub - Function trigger - azure

I'm trying to get the payload of Azure IoT Hub telemetry to a Function. I tried using this documentation, but I must be missing something. While I see data coming through, my function is not executed. I tried to put a Service Bus in between, so I created a Message Route in my IoT Hub and used that according to the same documentation, but for Service Bus instead of IoT Hub. I see the messages from a simulated device in the IoT Hub and the Service Bus, but somehow, the function is not executed. I also have no idea how to debug this problem, why the function is not executed. Any help with debugging tips or documentation tips would be much appreciated.
I added the Service Bus parameters in host.json:
...
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
},
"batchOptions": {
"maxMessageCount": 1000,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
...
And set the right trigger binding in functions.json:
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"queueName": "[MyQueueName]",
"connection": "Endpoint=sb://[MyServiceBusName].servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[MyServiceBusSAS]"
}
]
}

So what the instructions lack to tell you, what I did wrong, is that you should give your connection an arbitrary name, say queue_conn_str. Then on the Azure Portal, you go to your function app, and set there the application setting with the actual connection string (Endpoint...), with the same name (queue_conn_str).
With that you can actually connect your IoT hub directly to your function, no need for an Event Hub or Service Bus in between.

Related

Azure Function's log streams from Azure Event Hub can't be consumed by Filebeat

Our team wants to import the log of Azure Function by Azure Event Hub and Filebeat into Elastic Search. We followed this references to set up an Event Hub for Azure function. But we faced an issue with the wrong format of the log stream.
Firstly, Let me show what's the correct format we expect. Take Azure PostgresSQL's log from Event Hub for example:
[
{
"records": [
{
"time": "2023-01-04T03:45:31.1040000Z",
"properties": {
"timestamp": "2023-01-04 03:45:31.104 UTC",
"processId": 8909,
"errorLevel": "LOG",
"sqlerrcode": "00000",
"message": "2023-01-04 03:45:31 UTC-63b4f65b.22cd-LOG: connection received: host=<host> port=<port>"
},
"resourceId": "/SUBSCRIPTIONS/<my subscription id>/RESOURCEGROUPS/<my resource group>/PROVIDERS/MICROSOFT.DBFORPOSTGRESQL/FLEXIBLESERVERS/<postgres server>",
"category": "PostgreSQLLogs",
"operationName": "LogEvent"
}
]
}
]
Notice that the properties is a flattened json so that it can be consumed by Filebeat. We want this kind of properties. But the Azure Function's log looks like the following, which is a string rather than flattened json:
[
{
"records": [
{
"level": "Informational",
"resourceId": "/SUBSCRIPTIONS/<my subscription id>/RESOURCEGROUPS/<my resource group>/PROVIDERS/MICROSOFT.WEB/SITES/<my azure function>"
"operationName": "Microsoft.Web/sites/functions/log",
"category": "FunctionAppLogs",
"time": "01/04/2023 01:55:00",
"properties": "{'appName':'<my azure function>','roleInstance':'<id>','message':'Host Status: {\\n \\'id\\': \\'<function app id>\\',\\n \\'state\\': \\'Running\\',\\n \\'version\\': \\'4.13.0.0\\',\\n \\'versionDetails\\': \\'4.13.0+da9a765ed67be48c79440526f78fa1b5c6efdeea\\',\\n \\'platformVersion\\': \\'99.0.10.764\\',\\n \\'instanceId\\': \\'<instance id>\\',\\n \\'computerName\\': \\'<computer name>\\',\\n \\'processUptime\\': 69254486,\\n \\'functionAppContentEditingState\\': \\'Unknown\\'\\n}','category':'Host.Controllers.Host','hostVersion':'4.13.0.0','hostInstanceId':'<host id>','level':'Information','levelId':2,'processId':1}",
"EventStampType": "Stamp",
"EventPrimaryStampName": "waws-prod-ty1-081",
"EventStampName": "waws-prod-ty1-081",
"Host": "<host name>",
"EventIpAddress": "<ip address>"
},
The string value of properties can't be processed by decode_json_fields of Filebeat either because the format is not json (the format is 'key': value rather than "key": value). Is there any way to correct the format of properties before it is consumed by Filebeat? By the way, our Azure Function is deployed using a container.

I have 2 service bus, i have to set one of the service bus to maxConcurrentSessions to 1. how can i do that

I have 2 service bus SerBus1,SerBus2. I have to configure service bus (SerBus1) to limit the message to 1 at a time. how can i set in the host to consider only SerBus1 but not SerBus2? is there a possibility to name the id/name of the service bus.
Here is the code I have in the host. this applies to both the service bus. but i want only for SerBus1
{
"version": "2.0",
"extensions": {
"serviceBus": {
"prefetchCount": 1,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 1,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": true,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 1
}
}
}
}
host.json is applied on a Function App level. Every service bus triggered function will respect those settings. If you need to have different settings per function/triggering queue, you should split the Function App and configure each with the required concurrency.

Event Grid System Topic with Service Bus Geo-disaster recovery

I have an Event Grid System Topic that sends blob created events to a Service Bus Topic. The messages are processed by a dotnet application sitting in AKS.
Now the requirement came for high availability so we decided to turn on Service Bus Geo-disaster recovery.
We set the Primary and Secondary namespaces and want to use the Service Bus through the Geo-DR Alias.
Unfortunately I cannot find a way to connect the Event Grid Topic to this Service Bus Geo-DR Alias, it only allows me to select a single service bus namespace.
We are using ARM templates, and I tried different ways, but it is not allowing me to target the alias as a destination resource.
"resources": [{
"type": "Microsoft.EventGrid/systemTopics/eventSubscriptions",
"apiVersion": "2020-10-15-preview",
"name": "[concat(parameters('systemTopicName'), '/', parameters('eventSubscriptionName'))]",
"properties": {
"deliveryWithResourceIdentity": {
"identity": {
"type": "SystemAssigned"
},
"destination": {
"properties": {
"resourceId": "[concat(resourceId(resourceGroup().name, 'Microsoft.ServiceBus/namespaces', parameters('serviceBusNameSpace')), '/topics/blobcreated-event')]"
},
"endpointType": "ServiceBusTopic"
}
},
"filter": {
"subjectBeginsWith": "/blobServices/default/containers/stage",
"includedEventTypes": [
"Microsoft.Storage.BlobCreated"
],
"enableAdvancedFilteringOnArrays": true,
"advancedFilters": [{
"operatorType": "StringIn",
"key": "data.api",
"values": [
"PutBlob",
"PutBlockList",
"FlushWithClose"
]
}]
},
"labels": [],
"eventDeliverySchema": "EventGridSchema",
"retryPolicy": {
"maxDeliveryAttempts": 30,
"eventTimeToLiveInMinutes": 1440
}
}
}]
I have tried to add the alias to the path:
"resourceId": "[concat(resourceId(resourceGroup().name, 'Microsoft.ServiceBus/namespaces', parameters('serviceBusNameSpace')), '/disasterRecoveryConfigs/{aliassname}/topics/blobcreated-event')]"
But that gives me 'Invalid ARM Id.' error.
Other idea was to use Webhook, but EventGrid requires handshake to prove ownership to the Webhook endpoint which I am not sure if Service Bus is capable of.
https://learn.microsoft.com/en-us/azure/event-grid/webhook-event-delivery
Has anybody got this working or found a workaround?
We had a discussion with Microsoft and got confirmation that SB Aliases are not supported with Event Grid
(they are looking into options to enable this).
The options therefore would be either in a DR scenario changing the subscription or having multiple subscribers in different regions.
I am planning to write a script which switches the subscription and executing the failover after.

Unable to write message to Service Bus Queue from Azure Function

I'm trying to follow the documentation to write a message to an Azure Service Bus queue from an Azure Function (https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-output?tabs=csharp).
I started off with the "File->New Project" for an HTTP Trigger and added the binding:
[FunctionName("Message")]
[return: ServiceBus("namequeue")]
public static async Task<string> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
string name = data?.name ?? "DefaultName";
return name;
}
My host.json and local.settings.json file contains:
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
},
"batchOptions": {
"maxMessageCount": 1000,
"operationTimeout": "00:01:00",
"autoComplete": "true"
}
}
},
"Values": {
"AzureWebJobsServiceBus": "Endpoint=<redacted>"
}
When running locally I get a timeout exception (which might be a corporate firewall).
When deployed to Azure, I can POST to the function, get a 204 reply, but no messages are added to the queue.
I think I've missed a key step as my function.json in the Azure Portal has:
{
"generatedBy": "Microsoft.NET.Sdk.Functions-3.0.13",
"configurationSource": "attributes",
"bindings": [
{
"type": "httpTrigger",
"methods": [
"post"
],
"authLevel": "anonymous",
"name": "req"
}
],
"disabled": false,
"scriptFile": "../bin/AppMapServiceBusCreate.dll",
"entryPoint": "AppMapServiceBus.CreateMessageFunction.Run"
}
And when I click on Integration within the Portal there are no output and adding one gives me a warning of "In order to see the entire list of available function templates, you must set up extension bundles for your app.".
I thought Extension Bundles were non .NET code and the fact I've added the following via NuGet did the same thing?
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="4.3.0" />
If that's accurate, how/what do I add to the function.json?
For Azure Functions using an output trigger to write to Service Bus, you'll need to add the connection string as an application setting in the function's configuration. By default, the expected setting name is AzureWebJobsServiceBus:

Azure IoT hub cloud to device message using Stream Analytics

I am sending data through the path as below.
Android -> IoT Hub -> Stream Analytics -> SQL
I am calling machine learning function at the query of Stream Analytics. Now, I want to return the result of machine learning to the Android device. For receiving cloud-to-device message at Android device, I have done and tested with Device Explorer. I am looking at official tutorials, 1, 2 but still no clue on how to send cloud to device message using the Stream Analytics. He said using the service bus and function app but did not give the details. I am new to Azure. Hoping someone will give me some guidance or any link so I can understand more on how to implement it. Thanks in advance.
You can use an Azure Function (Preview) to output an ASA job for sending a cloud-to-device message via the Azure IoT Hub service-facing endpoint.
The following is an example of this function.
run.csx:
#r "Newtonsoft.Json"
using System.Configuration;
using System.Text;
using System.Net;
using Microsoft.Azure.Devices;
using Newtonsoft.Json;
// create proxy
static Microsoft.Azure.Devices.ServiceClient client = ServiceClient.CreateFromConnectionString(ConfigurationManager.AppSettings["myIoTHub"]);
public static async Task<HttpResponseMessage> Run(string input, HttpRequestMessage req, TraceWriter log)
{
log.Info($"ASA Job: {input}");
var data = JsonConvert.DeserializeAnonymousType(input, new[] { new { xyz = "", IoTHub = new { ConnectionDeviceId = ""}}});
if(!string.IsNullOrEmpty(data[0]?.IoTHub?.ConnectionDeviceId))
{
string deviceId = data[0].IoTHub.ConnectionDeviceId;
log.Info($"Device: {deviceId}");
// cloud-to-device message
var msg = JsonConvert.SerializeObject(new { temp = 20.5 });
var c2dmsg = new Microsoft.Azure.Devices.Message(Encoding.ASCII.GetBytes(msg));
// send AMQP message
await client.SendAsync(deviceId, c2dmsg);
}
return req.CreateResponse(HttpStatusCode.NoContent);
}
function.json:
{
"bindings": [
{
"authLevel": "function",
"name": "input",
"type": "httpTrigger",
"direction": "in"
},
{
"name": "$return",
"type": "http",
"direction": "out"
}
],
"disabled": false
}
project.json:
{
"frameworks": {
"net46":{
"dependencies": {
"Microsoft.Azure.Devices": "1.3.2"
}
}
}
}
Appendix A
For the test purpose, make the following steps:
Add the setting myIoTHub in your Azure Function App for your
connection string to the Azure IoT Hub.
Create HttpTrigger function, let call it as HttpASA_1
Update run.csx, function.json and project.json files with the above contents
Use the following sample for Test Request Body:
[
{
"time": "2017-11-26T12:52:23.4292501Z",
"counter": 57,
"windSpeed": 8.7358,
"temperature": 16.63,
"humidity": 79.42,
"EventProcessedUtcTime": "2017-11-26T12:52:21.3568252Z",
"PartitionId": 2,
"EventEnqueuedUtcTime": "2017-11-26T12:52:22.435Z",
"IoTHub": {
"MessageId": null,
"CorrelationId": null,
"ConnectionDeviceId": "Device1",
"ConnectionDeviceGenerationId": "636189812948967054",
"EnqueuedTime": "2017-11-26T12:52:21.562Z",
"StreamId": null
}
}
]
Change the value Device1 for your actually deviceId.
Now, the AF is ready to test it. Press the button Run to run the sample and look at the Logs progress.
You should see a C2D Message sent by AF on your device, or you can download a small tester Azure IoT Hub Tester to simulate your MQTT Devices. The following screen snippet shows this tester:
Now, in this step we can go to the ASA job for invoking this HttpASA_1 function. Note, that the ASA job invoking only the HttpTrigger function. The following screen snippet shows adding an output for our Azure Function:
You should see this function in the combobox when you selected your subscription in the Import option combobox. Once you done, press the Save button and watch the notification message on the screen. The ASA will send a validation message to your AF and its response status should be 20x code.
Finally, you can go to the Query to generate an output for your AF. The following screen shows a simple output for all telemetry data to the AF:
Note, the inpsim is my iothub input.
The ASA outputs payload for HttpTrigger Function in the following format, see my example:
[
{
"time": "2017-11-26T12:52:23.4292501Z",
"counter": 57,
"windSpeed": 8.7358,
"temperature": 16.63,
"humidity": 79.42,
"EventProcessedUtcTime": "2017-11-26T12:52:21.3568252Z",
"PartitionId": 2,
"EventEnqueuedUtcTime": "2017-11-26T12:52:22.435Z",
"IoTHub": {
"MessageId": null,
"CorrelationId": null,
"ConnectionDeviceId": "Device1",
"ConnectionDeviceGenerationId": "636189812948967054",
"EnqueuedTime": "2017-11-26T12:52:21.562Z",
"StreamId": null
}
}
]
Note, that my telemetry data (*) are counter, temperature, humidity and timestamp time, so the other properties are created implicitly by ASA job. Based on the query, you can create any business properties for C2D message.

Resources