Is context telemetry "grouped" during the sampling of request telemetry?
For example, the data below contains a request which has a sample count of 10 ("count": 10), meaning that it is being used to represent 9 other "similar" requests; 90% of the telemetry has actually been discarded.
Does Application Insights only sample data together when the context data is exactly the same for the requests? For example, can I assume that the other 9 requests were also from 41.191.204.0 and have a custom field company of value 22f0141f-b3dc-53e1-86b8-dd0727c14497?
{
"request": [
{
"id": "bs6o2dRoL/Q=",
"name": "GET /api/resources",
"count": 10,
"responseCode": 200,
"success": true,
"url": "https://example.com/api/resources",
"urlData": {
"base": "/api/resources",
"host": "example.com",
"hashTag": "",
"protocol": "https"
},
"durationMetric": {
"value": 1073743.0,
"count": 11.0,
"min": 97613.0,
"max": 97613.0,
"stdDev": 0.0,
"sampledValue": 97613.0
}
}
],
"internal": {
"data": {
"id": "8cbd12ec-9780-11e6-b38b-c5e9335e7642",
"documentVersion": "1.61"
}
},
"context": {
"application": {
"version": "1.0.16286.5"
},
"data": {
"eventTime": "2016-10-21T11:21:16.942Z",
"isSynthetic": false,
"samplingRate": 9.09090909090909
},
"device": {
"type": "PC",
"osVersion": "Windows 10",
"roleInstance": "RD0003FF727A10",
"deviceName": "Other",
"deviceModel": "Other",
"browser": "Chrome",
"browserVersion": "Chrome 53.0",
},
"user": {
"isAuthenticated": false
},
"session": {
"isFirst": false
},
"operation": {
"id": "bs6o2dRoL/Q=",
"parentId": "bs6o2dRoL/Q=",
"name": "GET Resources/GetResourceAsync [id]"
},
"location": {
"clientip": "41.191.204.0",
"continent": "Africa",
"country": "South Africa",
"province": "Eastern Cape"
},
"custom": {
"dimensions": [
{
"company": "22f0141f-b3dc-53e1-86b8-dd0727c14497"
},
{
"factor": "100"
}
]
}
}
}
Application Insights does not group telemetry events based on the context, but based on the Operation ID. This is synchronized between the SDK sampling and the server side sampling to make sure you will be able to navigate between related page views and requests.
So if you want to make sure some events are grouped together in sampling, set their OperationId to be the same.
See here for full details on how Application Insights implements it's sampling.
Hope this helps,
Asaf
Related
I have LoRaWAN IoT devices from the TTN network, which I have brought to Azure IoT Central. I would like to get specific data inside the JSON in order to make alert rules. Here is a sample of the JSON I receive from a Milesight PIR & Light Sensor. Suppose I only want to get the "battery", "daylight" and "pir" values to make the alert, how do I go about that?
{
"end_device_ids": {
"device_id": "eui-24e124538c195063",
"application_ids": {
"application_id": "test-1117"
},
"dev_eui": "24E124538C195063",
"join_eui": "24E124C0002A0001",
"dev_addr": "260D82F3"
},
"received_at": "2023-02-01T13:52:47.892042526Z",
"uplink_message": {
"session_key_id": "AYYLj9wg6AeehHWwDLrzLg==",
"f_port": 85,
"f_cnt": 38,
"frm_payload": "AXVkAwAABAAA",
"decoded_payload": {
"battery": 100,
"daylight": "dark",
"pir": "normal"
},
"rx_metadata": [
{
"gateway_ids": {
"gateway_id": "eui-24e124fffef5531c",
"eui": "24E124FFFEF5531C"
},
"time": "2023-02-01T13:52:47.429725885Z",
"timestamp": 2696081442,
"rssi": -64,
"channel_rssi": -64,
"snr": 13,
"uplink_token": "CiIKIAoUZXVpLTI0ZTEyNGZmZmVmNTUzMWMSCCThJP/+9VMcEKLgy4UKGgwIr93pngYQ94rGxQIg0Ino17uJFw==",
"received_at": "2023-02-01T13:52:47.596682842Z"
}
],
"settings": {
"data_rate": {
"lora": {
"bandwidth": 125000,
"spreading_factor": 7,
"coding_rate": "4/5"
}
},
"frequency": "867700000",
"timestamp": 2696081442,
"time": "2023-02-01T13:52:47.429725885Z"
},
"received_at": "2023-02-01T13:52:47.683989041Z",
"confirmed": true,
"consumed_airtime": "0.061696s",
"network_ids": {
"net_id": "000013",
"tenant_id": "ttn",
"cluster_id": "au1",
"cluster_address": "au1.cloud.thethings.network"
}
},
"_eventtype": "Telemetry",
"_timestamp": "2023-02-01T13:52:49.35Z"
}
I was able to get the values by uploading my own DTDL.
Once the device sends data to Azure IOT Central from TTN, the device will have no template.
I first created a template from the raw telemetry sent.
I went to the new device template created, clicked EDIT DTDL, then placed the following JSON:
{
"#id": "dtmi:telavi:eui24e124538c195063;1",
"#type": "Interface",
"contents": [
{
"#type": [
"Telemetry",
"DateTimeValue"
],
"displayName": {
"en": "received_at"
},
"name": "received_at",
"schema": "dateTime"
},
{
"#type": [
"Telemetry",
"NumberValue"
],
"description": {
"en": "battery"
},
"displayName": {
"en": "battery"
},
"name": "battery",
"schema": "integer"
},
{
"#type": [
"Telemetry",
"StringValue"
],
"description": {
"en": "daylight"
},
"displayName": {
"en": "daylight"
},
"name": "daylight",
"schema": "string"
},
{
"#type": [
"Telemetry",
"StringValue"
],
"description": {
"en": "pir"
},
"displayName": {
"en": "pir"
},
"name": "pir",
"schema": "string"
}
],
"displayName": {
"en": "eui-24e124538c195063"
},
"#context": [
"dtmi:iotcentral:context;2",
"dtmi:dtdl:context;2"
]
}
My colleagues and I have been working to fix a reported issue on our Amazon Alexa CBT Test regarding the value “DeepQuery=true”.
Our code has been modified, so that every state change is reported automatically and all the used interfaces have the properties “proactivelyReported” and “retrievable” set to true.
As has been suggested by the WWA-Support we used the Smart Home Debugger of the Developer Console to validate the ReportEvents (e.g. Discovery or ChangeReport) and we checked the state of our device on the “View Device State” page (both pages are referenced on: https://developer.amazon.com/en-US/docs/alexa/smarthome/debug-your-smart-home-skill.html).
For debugging purposes we scaled our device capabilities down to just the PowerController. The AddOrUpdateReport of Alexa.Discovery looks to our eyes now exactly as expected/documented. Same goes for the ChangeReport, which we proactively send right after the AddOrUpdateReport (Two sample-Reports for both are provided at the end).
Unfortunately we are still faced with the issue, that “DeepQuery=true” on the “View Device State” page.
If we set the interface property “retrievable” to false, “DeepQuery=false”, but the Alexa-App does not retain the current state of the device. In this configuration the Alexa-App can only be used to send commands, which unfortunately will lead to other test cases to fail.
Does anyone know how to solve this issue?
How can we set “proactivelyReported” and “retrievable” to true and have “DeepQuery=false”?
Any help would be greatly appreciated and I will gladly provide more informations if needed.
Sample AddOrUpdateReport from Smart Home Debugger
{
"header": {
"namespace": "SkillDebugger",
"name": "CaptureDebuggingInfo",
"messageId": "05b030fb-6393-4ae0-80d0-47fc27876f0e"
},
"payload": {
"skillId": "amzn1.ask.skill.055ca62d-3cf8-4f51-a683-9a98b36f4637",
"timestamp": "2021-09-09T13:28:21.629Z",
"dialogRequestId": null,
"skillRequestId": null,
"type": "SmartHomeAddOrUpdateReportSuccess",
"content": {
"addOrUpdateReport": {
"event": {
"header": {
"namespace": "Alexa.Discovery",
"name": "AddOrUpdateReport",
"messageId": "2458b969-7c3e-47e2-ab0b-6e13a999be76",
"payloadVersion": "3"
},
"payload": {
"endpoints": [
{
"manufacturerName": "Our Company Name",
"description": "Our Product Name",
"endpointId": "device--cb12b420-1171-11ec-81f3-cb34e87ea438",
"friendlyName": "Lampe 1",
"capabilities": [
{
"type": "AlexaInterface",
"version": "3",
"interface": "Alexa.PowerController",
"properties": {
"supported": [
{
"name": "powerState"
}
],
"proactivelyReported": true,
"retrievable": true
}
},
{
"type": "AlexaInterface",
"interface": "Alexa",
"version": "3"
}
],
"displayCategories": [
"LIGHT"
],
"connections": [],
"relationships": {},
"cookie": {}
}
],
"scope": null
}
}
}
}
}
}
Sample ChangeReport from Smart Home Debugger
{
"header": {
"namespace": "SkillDebugger",
"name": "CaptureDebuggingInfo",
"messageId": "194a96a1-6747-46ba-8751-5c9ef715fd34"
},
"payload": {
"skillId": "amzn1.ask.skill.055ca62d-3cf8-4f51-a683-9a98b36f4637",
"timestamp": "2021-09-09T13:28:23.227Z",
"dialogRequestId": null,
"skillRequestId": null,
"type": "SmartHomeChangeReportSuccess",
"content": {
"changeReport": {
"event": {
"header": {
"namespace": "Alexa",
"name": "ChangeReport",
"messageId": "8972e386-9622-40e6-85e7-1a7d81c79c8a",
"payloadVersion": "3"
},
"endpoint": {
"scope": null,
"endpointId": "device--cb12b420-1171-11ec-81f3-cb34e87ea438"
},
"payload": {
"change": {
"cause": {
"type": "APP_INTERACTION"
},
"properties": [
{
"namespace": "Alexa.PowerController",
"name": "powerState",
"value": "ON",
"timeOfSample": "2021-09-09T13:28:18.088Z",
"uncertaintyInMilliseconds": 500
}
]
}
}
},
"context": {
"properties": []
}
}
}
}
}
I am doing some analysis on spark sql query execution plans. the execution plans that explain() api prints are not much readable. If we see spark web UI, a DAG graph is created which is divided into jobs, stages and tasks and much more readable. Is there any way to create that graph from execution plans or any apis in the code? if not, are there any apis that can read that grap from UI?
As close I can see, this project (https://github.com/AbsaOSS/spline-spark-agent) is able to interpret the execution plan and generate it in a readable way.
This spark job is reading a file, convert it to a CSV file, write to local.
A sample output in JSON look like
{
"id": "3861a1a7-ca31-4fab-b0f5-6dbcb53387ca",
"operations": {
"write": {
"outputSource": "file:/output.csv",
"append": false,
"id": 0,
"childIds": [
1
],
"params": {
"path": "output.csv"
},
"extra": {
"name": "InsertIntoHadoopFsRelationCommand",
"destinationType": "csv"
}
},
"reads": [
{
"inputSources": [
"file:/Users/liajiang/Downloads/spark-onboarding-demo-application/src/main/resources/wikidata.csv"
],
"id": 2,
"schema": [
"6742cfd4-d8b6-4827-89f2-4b2f7e060c57",
"62c022d9-c506-4e6e-984a-ee0c48f9df11",
"26f1d7b5-74a4-459c-87f3-46a3df781400",
"6e4063cf-4fd0-465d-a0ee-0e5c53bd52b0",
"2e019926-3adf-4ece-8ea7-0e01befd296b"
],
"params": {
"inferschema": "true",
"header": "true"
},
"extra": {
"name": "LogicalRelation",
"sourceType": "csv"
}
}
],
"other": [
{
"id": 1,
"childIds": [
2
],
"params": {
"name": "`source`"
},
"extra": {
"name": "SubqueryAlias"
}
}
]
},
"systemInfo": {
"name": "spark",
"version": "2.4.2"
},
"agentInfo": {
"name": "spline",
"version": "0.5.5"
},
"extraInfo": {
"appName": "spark-spline-demo-application",
"dataTypes": [
{
"_typeHint": "dt.Simple",
"id": "f0dede5e-8fe1-4c22-ab24-98f7f44a9a5a",
"name": "timestamp",
"nullable": true
},
{
"_typeHint": "dt.Simple",
"id": "dbe1d206-3d87-442c-837d-dfa47c88b9c1",
"name": "string",
"nullable": true
},
{
"_typeHint": "dt.Simple",
"id": "0d786d1e-030b-4997-b005-b4603aa247d7",
"name": "integer",
"nullable": true
}
],
"attributes": [
{
"id": "6742cfd4-d8b6-4827-89f2-4b2f7e060c57",
"name": "date",
"dataTypeId": "f0dede5e-8fe1-4c22-ab24-98f7f44a9a5a"
},
{
"id": "62c022d9-c506-4e6e-984a-ee0c48f9df11",
"name": "domain_code",
"dataTypeId": "dbe1d206-3d87-442c-837d-dfa47c88b9c1"
},
{
"id": "26f1d7b5-74a4-459c-87f3-46a3df781400",
"name": "page_title",
"dataTypeId": "dbe1d206-3d87-442c-837d-dfa47c88b9c1"
},
{
"id": "6e4063cf-4fd0-465d-a0ee-0e5c53bd52b0",
"name": "count_views",
"dataTypeId": "0d786d1e-030b-4997-b005-b4603aa247d7"
},
{
"id": "2e019926-3adf-4ece-8ea7-0e01befd296b",
"name": "total_response_size",
"dataTypeId": "0d786d1e-030b-4997-b005-b4603aa247d7"
}
]
}
}
I am able to create the "message route" in azure portal and able to route messages to servicebusqueue if the query matching, I want to create the message route using the restapi instead of using azure portal, I have seen many documents but unable to find the proper one. Whether creating the message route using restapi is possible or not? if yes,How can I achieve this and please provide the respective links to refer?
I haven't tried this through REST API, but as Roman suggested,
You can check the IotHubResource_CreateOrUpdate which will help you understand how to Create or update the metadata of an Iot hub. The usual pattern to modify a property is to retrieve the IoT hub metadata and security metadata, and then combine them with the modified values in a new body to update the IoT hub.
Sample Request:
PUT https://management.azure.com/subscriptions/91d12660-3dec-467a-be2a-213b5544ddc0/resourceGroups/myResourceGroup/providers/Microsoft.Devices/IotHubs/testHub?api-version=2018-04-01
Request Body:
{
"name": "iot-dps-cit-hub-1",
"type": "Microsoft.Devices/IotHubs",
"location": "centraluseuap",
"tags": {},
"etag": "AAAAAAFD6M4=",
"properties": {
"operationsMonitoringProperties": {
"events": {
"None": "None",
"Connections": "None",
"DeviceTelemetry": "None",
"C2DCommands": "None",
"DeviceIdentityOperations": "None",
"FileUploadOperations": "None",
"Routes": "None"
}
},
"state": "Active",
"provisioningState": "Succeeded",
"ipFilterRules": [],
"hostName": "iot-dps-cit-hub-1.azure-devices.net",
"eventHubEndpoints": {
"events": {
"retentionTimeInDays": 1,
"partitionCount": 2,
"partitionIds": [
"0",
"1"
],
"path": "iot-dps-cit-hub-1",
"endpoint": "sb://iothub-ns-iot-dps-ci-245306-76aca8e13b.servicebus.windows.net/"
},
"operationsMonitoringEvents": {
"retentionTimeInDays": 1,
"partitionCount": 2,
"partitionIds": [
"0",
"1"
],
"path": "iot-dps-cit-hub-1-operationmonitoring",
"endpoint": "sb://iothub-ns-iot-dps-ci-245306-76aca8e13b.servicebus.windows.net/"
}
},
"routing": {
"endpoints": {
"serviceBusQueues": [],
"serviceBusTopics": [],
"eventHubs": [],
"storageContainers": []
},
"routes": [],
"fallbackRoute": {
"name": "$fallback",
"source": "DeviceMessages",
"condition": "true",
"endpointNames": [
"events"
],
"isEnabled": true
}
},
"storageEndpoints": {
"$default": {
"sasTtlAsIso8601": "PT1H",
"connectionString": "",
"containerName": ""
}
},
"messagingEndpoints": {
"fileNotifications": {
"lockDurationAsIso8601": "PT1M",
"ttlAsIso8601": "PT1H",
"maxDeliveryCount": 10
}
},
"enableFileUploadNotifications": false,
"cloudToDevice": {
"maxDeliveryCount": 10,
"defaultTtlAsIso8601": "PT1H",
"feedback": {
"lockDurationAsIso8601": "PT1M",
"ttlAsIso8601": "PT1H",
"maxDeliveryCount": 10
}
},
"features": "None"
},
"sku": {
"name": "S1",
"tier": "Standard",
"capacity": 1
}
}
I'm new to Azure Stream Analytics query. My scenario is using Continuous Export to write Application Insight telemetry to Azure Blob storage and use Stream Analytics job to push data from Blob storage to Power BI. My json file has both Array and Request type as follows:
{
"request": [
{
"id": "|HLHUdGy4c3g=.556f8524_",
"name": "HEAD Todos/Index",
"count": 1,
"responseCode": 200,
"success": true,
"url": "http://todoapp20183001.azurewebsites.net/",
"urlData": {
"base": "/",
"host": "todoapp20183001.azurewebsites.net",
"hashTag": "",
"protocol": "http"
},
"durationMetric": {
"value": 973023,
"count": 1,
"min": 973023,
"max": 973023,
"stdDev": 0,
"sampledValue": 973023
}
}
],
"internal": {
"data": {
"id": "124c5c1c-0820-11e8-a590-d95f25fd3f7f",
"documentVersion": "1.61"
}
},
"context": {
"data": {
"eventTime": "2018-02-02T13:50:39.591Z",
"isSynthetic": false,
"samplingRate": 100
},
"cloud": {},
"device": {
"type": "PC",
"roleName": "todoapp20183001",
"roleInstance": "RD0003FF6D001A",
"screenResolution": {}
},
"user": {
"isAuthenticated": false
},
"session": {
"isFirst": false
},
"operation": {
"id": "HLHUdGy4c3g=",
"parentId": "HLHUdGy4c3g=",
"name": "HEAD Todos/Index"
},
"location": {
"clientip": "35.153.211.0",
"continent": "North America",
"country": "United States",
"province": "Virginia",
"city": "Ashburn"
},
"custom": {
"dimensions": [
{
"_MS.ProcessedByMetricExtractors": "(Name:'Requests', Ver:'1.0')"
}
]
}
}
}
Using the following query I can receive the expected output.
WITH Request AS
(
SELECT
context.location.country as country,
context.location.city as city,
GetArrayElement(request,0) as requests
FROM FromBlob
)
SELECT country, city, requests.name
FROM Request
Now I need to count all the request by city but I cannot seem to get it done with COUNT() and GROUP BY(). Is there a hint or reference to have a look in this case?
Here's a example to count the number of requests every 5 minutes.
Note that I had to add a time component to GROUB BY since your data is streaming data and you want to have the aggregate on a finite time.
WITH Request AS
(
SELECT
context.location.country as country,
context.location.city as city,
GetArrayElement(request,0) as requests
FROM iothub
)
SELECT country, city, count(requests.name)
FROM Request
group by country,city,SlidingWindow(minute,5)
Let me know if it works for you.