Azure IoT hub cloud to device message using Stream Analytics - azure

I am sending data through the path as below.
Android -> IoT Hub -> Stream Analytics -> SQL
I am calling machine learning function at the query of Stream Analytics. Now, I want to return the result of machine learning to the Android device. For receiving cloud-to-device message at Android device, I have done and tested with Device Explorer. I am looking at official tutorials, 1, 2 but still no clue on how to send cloud to device message using the Stream Analytics. He said using the service bus and function app but did not give the details. I am new to Azure. Hoping someone will give me some guidance or any link so I can understand more on how to implement it. Thanks in advance.

You can use an Azure Function (Preview) to output an ASA job for sending a cloud-to-device message via the Azure IoT Hub service-facing endpoint.
The following is an example of this function.
run.csx:
#r "Newtonsoft.Json"
using System.Configuration;
using System.Text;
using System.Net;
using Microsoft.Azure.Devices;
using Newtonsoft.Json;
// create proxy
static Microsoft.Azure.Devices.ServiceClient client = ServiceClient.CreateFromConnectionString(ConfigurationManager.AppSettings["myIoTHub"]);
public static async Task<HttpResponseMessage> Run(string input, HttpRequestMessage req, TraceWriter log)
{
log.Info($"ASA Job: {input}");
var data = JsonConvert.DeserializeAnonymousType(input, new[] { new { xyz = "", IoTHub = new { ConnectionDeviceId = ""}}});
if(!string.IsNullOrEmpty(data[0]?.IoTHub?.ConnectionDeviceId))
{
string deviceId = data[0].IoTHub.ConnectionDeviceId;
log.Info($"Device: {deviceId}");
// cloud-to-device message
var msg = JsonConvert.SerializeObject(new { temp = 20.5 });
var c2dmsg = new Microsoft.Azure.Devices.Message(Encoding.ASCII.GetBytes(msg));
// send AMQP message
await client.SendAsync(deviceId, c2dmsg);
}
return req.CreateResponse(HttpStatusCode.NoContent);
}
function.json:
{
"bindings": [
{
"authLevel": "function",
"name": "input",
"type": "httpTrigger",
"direction": "in"
},
{
"name": "$return",
"type": "http",
"direction": "out"
}
],
"disabled": false
}
project.json:
{
"frameworks": {
"net46":{
"dependencies": {
"Microsoft.Azure.Devices": "1.3.2"
}
}
}
}
Appendix A
For the test purpose, make the following steps:
Add the setting myIoTHub in your Azure Function App for your
connection string to the Azure IoT Hub.
Create HttpTrigger function, let call it as HttpASA_1
Update run.csx, function.json and project.json files with the above contents
Use the following sample for Test Request Body:
[
{
"time": "2017-11-26T12:52:23.4292501Z",
"counter": 57,
"windSpeed": 8.7358,
"temperature": 16.63,
"humidity": 79.42,
"EventProcessedUtcTime": "2017-11-26T12:52:21.3568252Z",
"PartitionId": 2,
"EventEnqueuedUtcTime": "2017-11-26T12:52:22.435Z",
"IoTHub": {
"MessageId": null,
"CorrelationId": null,
"ConnectionDeviceId": "Device1",
"ConnectionDeviceGenerationId": "636189812948967054",
"EnqueuedTime": "2017-11-26T12:52:21.562Z",
"StreamId": null
}
}
]
Change the value Device1 for your actually deviceId.
Now, the AF is ready to test it. Press the button Run to run the sample and look at the Logs progress.
You should see a C2D Message sent by AF on your device, or you can download a small tester Azure IoT Hub Tester to simulate your MQTT Devices. The following screen snippet shows this tester:
Now, in this step we can go to the ASA job for invoking this HttpASA_1 function. Note, that the ASA job invoking only the HttpTrigger function. The following screen snippet shows adding an output for our Azure Function:
You should see this function in the combobox when you selected your subscription in the Import option combobox. Once you done, press the Save button and watch the notification message on the screen. The ASA will send a validation message to your AF and its response status should be 20x code.
Finally, you can go to the Query to generate an output for your AF. The following screen shows a simple output for all telemetry data to the AF:
Note, the inpsim is my iothub input.
The ASA outputs payload for HttpTrigger Function in the following format, see my example:
[
{
"time": "2017-11-26T12:52:23.4292501Z",
"counter": 57,
"windSpeed": 8.7358,
"temperature": 16.63,
"humidity": 79.42,
"EventProcessedUtcTime": "2017-11-26T12:52:21.3568252Z",
"PartitionId": 2,
"EventEnqueuedUtcTime": "2017-11-26T12:52:22.435Z",
"IoTHub": {
"MessageId": null,
"CorrelationId": null,
"ConnectionDeviceId": "Device1",
"ConnectionDeviceGenerationId": "636189812948967054",
"EnqueuedTime": "2017-11-26T12:52:21.562Z",
"StreamId": null
}
}
]
Note, that my telemetry data (*) are counter, temperature, humidity and timestamp time, so the other properties are created implicitly by ASA job. Based on the query, you can create any business properties for C2D message.

Related

Azure - Debugging Iot Hub - Function trigger

I'm trying to get the payload of Azure IoT Hub telemetry to a Function. I tried using this documentation, but I must be missing something. While I see data coming through, my function is not executed. I tried to put a Service Bus in between, so I created a Message Route in my IoT Hub and used that according to the same documentation, but for Service Bus instead of IoT Hub. I see the messages from a simulated device in the IoT Hub and the Service Bus, but somehow, the function is not executed. I also have no idea how to debug this problem, why the function is not executed. Any help with debugging tips or documentation tips would be much appreciated.
I added the Service Bus parameters in host.json:
...
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
},
"batchOptions": {
"maxMessageCount": 1000,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
...
And set the right trigger binding in functions.json:
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"queueName": "[MyQueueName]",
"connection": "Endpoint=sb://[MyServiceBusName].servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[MyServiceBusSAS]"
}
]
}
So what the instructions lack to tell you, what I did wrong, is that you should give your connection an arbitrary name, say queue_conn_str. Then on the Azure Portal, you go to your function app, and set there the application setting with the actual connection string (Endpoint...), with the same name (queue_conn_str).
With that you can actually connect your IoT hub directly to your function, no need for an Event Hub or Service Bus in between.

Azure Function not working with serviceBusTrigger NodeJs

I have been trying to put a message in azure service bus topic. My azure function test says that the message has been accepted(202). But there is no message on the subscription side, Could you please help me in this POC. Here is my code snippet. It's a sample that got generated from VS code, I am using serviceBusTrigger.
const { ServiceBusClient, ReceiveMode } = require("#azure/service-bus");
module.exports = async function(context, mySbMsg) {
context.log('JavaScript ServiceBus topic trigger function processed message', mySbMsg);
context.done();
};
Is there any way that I can check if the service bus topic is working as expected?
I do not see a connection string and a binding associated with the code, Inorder to save the message to the queue, you need to have the Queue connection strings in the settings file. Follow the docs,
{
"bindings": [
{
"schedule": "0/15 * * * * *",
"name": "myTimer",
"runsOnStartup": true,
"type": "timerTrigger",
"direction": "in"
},
{
"name": "outputSbQueue",
"type": "serviceBus",
"queueName": "testqueue",
"connection": "MyServiceBusConnection",
"direction": "out"
}
],
"disabled": false
}
Okay, So the thing was, I tried calling the function after pushing messages onto the topic. But the purpose of ServiceBusTriger is reverse. The function with this trigger processes the message when there is message on queue. There is no need to Call the function separately. Yeah, so basics, just create the function with your available topic and subscription, and then try putting message onto the topic. You can see the message in the log of the function.

Error calling the azure function from data factory pipeline trigger

I am calling an http triggered Azure function app in data factory pipeline using ADF function activity. It is executing successfully in debug mode, but when I publish that pipeline and run the same code using data factory triggers I get below error-
{
"errorCode": "3600",
"message": "Object reference not set to an instance of an object.",
"failureType": "UserError",
"target": "AzureFunction"
}
Please let me know if I need to make some additional properties changes or I am missing anything here. Also is there any way I can see what URL is getting generated when I call function app through function activity in ADF.
I have tried calling same function app using web activity in ADF and that is working fine in both debug and trigger mode.
Linked service code for Azure function
{
"name": "linkedservicesAzureFunctions",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"typeProperties": {
"functionAppUrl": "https://xyz.azurewebsites.net",
"functionKey": {
"type": "AzureKeyVaultSecret",
"store": {
"type": "LinkedServiceReference",
"referenceName": "linkedservicesKeyVault"
},
"secretName": "functions-host-key-default"
}
},
"type": "AzureFunction"
}
}
There is a known Bug in Azure Data Factory and they are working on that. For now if you are creating the Azure Data Factory using .NET SDK then you'll need to set Headers like this in Azure Function Activity.
new AzureFunctionActivity
{
Name = "CopyFromBlobToSnowFlake",
LinkedServiceName = new LinkedServiceReference(pipelineStructure.AzureFunctionLinkedService),
Method = "POST",
Body = body,
FunctionName = "LoadBlobsIntoSnowFlake",
Headers = new Dictionary<string, string>{ },
DependsOn = new List<ActivityDependency>
{
new ActivityDependency{
Activity = "CopyFromOPSqlServerToBlob",
DependencyConditions= new List<string>{"Succeeded" }
}
}
}
If you are creating Azure Function Activity through UI then just update the description of Activity then Publish and Headers will automatically get Initialized.

How to fix error creating event grid subscription for a Azure Function Core hosted on Linux?

I have an Azure Function Core running on a Linux consumption plan running in WestUS. The function seems to be running OK - I had a previous .NET Standard 2.0 version of the same function running OK in Azure. However, when I try to create a subscription to an Event Grid Topic I get the following error:
Deployment has failed with the following error: {"code":"Url validation","message":"The attempt to validate the provided endpoint https://insysfunctiongetweathercore.azurewebsites.net/runtime/webhooks/EventGrid failed. For more details, visit https://aka.ms/esvalidation."}
I am using an EventGridTrigger so should not have to do anything to handle validation - this should happen automatically:
public static async Task Run([EventGridTrigger] EventGridEvent eventGridEvent, ILogger log)
Any ideas on what I might need to do differently when the function is hosted in Linux consumption plan?
EDIT
Here's a screenshot of the Create Event Subscription Form with error:
and the resulting JSON:
{
"name": "InSysWeatherPull",
"properties": {
"topic": "/subscriptions/xxxxxxxxxxx/resourceGroups/InergySystemsWest/providers/Microsoft.EventGrid/Topics/InSysEventGridWest",
"destination": {
"endpointType": "WebHook",
"properties": {
"endpointUrl": "https://insysfunctiongetweathercore.azurewebsites.net/runtime/webhooks/EventGrid?functionName=ProcessWeatherRequest&code=xxxxxxxxxxxx"
}
},
"filter": {
"includedEventTypes": [
"weather-zip-request"
],
"advancedFilters": []
},
"labels": [],
"eventDeliverySchema": "EventGridSchema"
}
}

Can I add DeviceID in path while storing data from Azure Stream Analytics to Blob storage

I have Data incoming from Different devices to IoT hub from there using Stream Analytics to process it and store it in blob storage.
I know we can add {date}{time} we add in the path according to needed format, in that path can we add deviceId too.
example : For 2018/10/30/01 ( Date/month/day/hour) Can add /deviceId in that path while storing to blob
The following is an example of workaround for your case. It's based on the using an azure function (HttpTrigger) for output ASA job to append a data to the specific blob storage in the push manner.
Note, that the following workaround using the Max batch count for delivering events to the azure function value 1 (one telemetry data at the time).
ASA job query:
SELECT
System.Timestamp as [time], *
INTO outAF
FROM
iot TIMESTAMP BY time
Azure Function (HttpTrigger):
run.csx
#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Microsoft.WindowsAzure.Storage.Blob;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public static async Task<IActionResult> Run(string body, CloudBlobContainer blobContainer, ILogger log)
{
log.LogInformation($"{body}");
var jtoken = JToken.Parse(body);
var jobject = jtoken is JArray ? jtoken.SingleOrDefault<JToken>() : jtoken;
if(jobject != null)
{
var jtext = jobject.ToString(Formatting.None);
var data = JsonConvert.DeserializeAnonymousType(jtext, new {IoTHub = new { ConnectionDeviceId = ""}});
var blobName = $"{DateTime.UtcNow.ToString("yyyy/MM/dd/hh")}/{data.IoTHub.ConnectionDeviceId}";
var blob = blobContainer.GetAppendBlobReference(blobName);
if(!await blob.ExistsAsync())
{
await blob.CreateOrReplaceAsync();
}
await blob.AppendTextAsync(jtext + "\r\n");
}
return new NoContentResult();
}
function.json
{
"bindings": [
{
"authLevel": "function",
"name": "body",
"type": "httpTrigger",
"direction": "in",
"methods": [
"get",
"post"
]
},
{
"name": "blobContainer",
"type": "blob",
"path": "myContainer",
"connection": "mySTORAGE",
"direction": "out"
},
{
"name": "$return",
"type": "http",
"direction": "out"
}
]
}
I know we can add {date}{time} we add in the path according to needed
format, in that path can we add deviceId too.'
As #Peter Bons mentioned in the comment, variable names in output is not supported so far.
As workaround, you could use Blob Trigger Azure Function. You need to pass the deviceId in the output columns then get it in the blob trigger function. Then use blob sdk to create /deviceId directory to copy blob into it and delete the previous blob.

Resources