I have a question based on FIWARE timescale database design when using STH and grafana for visualization.
**Background**:
I created a test project based on FIWARE using Docker-Compose. The project currently includes:
Orion-LD + Mongo-DB
STH based on Mintaka and Timescale-DB
OPC UA Dummy Car Server and OPC UA IoT-Agent
Grafana
The solutions seems to work quite well: OPC UA dummy car server sends data via OPC UA Iot-Agent and I can also send commands back to OPC dummy car server. I am also able to query historic context for the car entity using orion-ld:
Request
curl --location --request GET 'http://localhost:8080/temporal/entities/?type=Device&pageSize=2&lastN=3&timeproperty=modifiedAt&options=count&timerel=before&timeAt=2022-07-15T15:28:00Z' \
--header 'NGSILD-Tenant: opcua_car' \
--header 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"'\''' \
--header 'NGSILD-Path: /demo'
Response:
[
{
"id": "urn:ngsi-ld:Device:age01_Car",
"type": "Device",
"Acceleration": [
{
"type": "Property",
"value": 0.0,
"modifiedAt": "2022-07-15T13:40:02.960Z",
"instanceId": "urn:ngsi-ld:attribute:instance:a10c6606-0443-11ed-9e31-0242ac120103"
},
{
"type": "Property",
"value": 3.0,
"modifiedAt": "2022-07-15T13:40:01.978Z",
"instanceId": "urn:ngsi-ld:attribute:instance:a0775408-0443-11ed-9cfd-0242ac120103"
},
{
"type": "Property",
"value": 0.0,
"modifiedAt": "2022-07-15T13:25:54.342Z",
"instanceId": "urn:ngsi-ld:attribute:instance:a73b926a-0441-11ed-bca6-0242ac120103"
}
],
"Stop_status": {
"type": "Property",
"value": {
"type": "commandStatus",
"#value": "UNKNOWN"
},
"modifiedAt": "2022-07-15T13:21:24.893Z",
"instanceId": "urn:ngsi-ld:attribute:instance:06a2ed12-0441-11ed-b967-0242ac120103"
},
"Engine_Oxigen": [
{
"type": "Property",
"value": 80.0,
"modifiedAt": "2022-07-15T13:24:41.752Z",
"instanceId": "urn:ngsi-ld:attribute:instance:7bf736f4-0441-11ed-8bcd-0242ac120103"
},
{
"type": "Property",
"value": 78.5,
"modifiedAt": "2022-07-15T13:24:40.734Z",
"instanceId": "urn:ngsi-ld:attribute:instance:7b5c32e4-0441-11ed-ac6e-0242ac120103"
},
{
"type": "Property",
"value": 77.0,
"modifiedAt": "2022-07-15T13:24:39.726Z",
"instanceId": "urn:ngsi-ld:attribute:instance:7ac21740-0441-11ed-b111-0242ac120103"
}
],
"Stop_info": {
"type": "Property",
"value": {
"type": "commandResult",
"#value": " "
},
"modifiedAt": "2022-07-15T13:21:24.893Z",
"instanceId": "urn:ngsi-ld:attribute:instance:06a2ed8a-0441-11ed-b967-0242ac120103"
},
"Accelerate_info": {
"type": "Property",
"value": {
"type": "commandResult",
"#value": " "
},
"modifiedAt": "2022-07-15T13:21:24.893Z",
"instanceId": "urn:ngsi-ld:attribute:instance:06a2ee66-0441-11ed-b967-0242ac120103"
},
"Error_status": {
"type": "Property",
"value": {
"type": "commandStatus",
"#value": "UNKNOWN"
},
"modifiedAt": "2022-07-15T13:21:24.893Z",
"instanceId": "urn:ngsi-ld:attribute:instance:06a2ec0e-0441-11ed-b967-0242ac120103"
},
"Accelerate_status": {
"type": "Property",
"value": {
"type": "commandStatus",
"#value": "UNKNOWN"
},
"modifiedAt": "2022-07-15T13:21:24.893Z",
"instanceId": "urn:ngsi-ld:attribute:instance:06a2edf8-0441-11ed-b967-0242ac120103"
},
"EngineStopped": [
{
"type": "Property",
"value": true,
"modifiedAt": "2022-07-15T13:23:50.922Z",
"instanceId": "urn:ngsi-ld:attribute:instance:5dab3b8c-0441-11ed-9f1d-0242ac120103"
},
{
"type": "Property",
"value": true,
"modifiedAt": "2022-07-15T13:21:25.497Z",
"instanceId": "urn:ngsi-ld:attribute:instance:06fd291c-0441-11ed-9ed9-0242ac120103"
}
],
"Engine_Temperature": [
{
"type": "Property",
"value": 80.0,
"modifiedAt": "2022-07-15T13:24:21.691Z",
"instanceId": "urn:ngsi-ld:attribute:instance:70023236-0441-11ed-8db7-0242ac120103"
},
{
"type": "Property",
"value": 78.0,
"modifiedAt": "2022-07-15T13:24:20.700Z",
"instanceId": "urn:ngsi-ld:attribute:instance:6f6b16c6-0441-11ed-9541-0242ac120103"
},
{
"type": "Property",
"value": 76.0,
"modifiedAt": "2022-07-15T13:24:19.696Z",
"instanceId": "urn:ngsi-ld:attribute:instance:6ed1d326-0441-11ed-81be-0242ac120103"
}
],
"Error_info": {
"type": "Property",
"value": {
"type": "commandResult",
"#value": " "
},
"modifiedAt": "2022-07-15T13:21:24.893Z",
"instanceId": "urn:ngsi-ld:attribute:instance:06a2ec9a-0441-11ed-b967-0242ac120103"
}
}
]
Now I want to visualize time-series data using grafana. For QL + subscription this is documented very well (see here): We will have to use table 'et<entitytype>', time column 'time_index' and metric column 'entity_type'. I tested the scenario successfully using TimescaleDB.
Issue:
I can't find any information on how to visualize data with grafana using STH. TimescaleDB now has different tables for entities, attributes, subattributes and more. Using Grafana I am not sure how to query the entities correctly. Especially I have problems with the example above, where no observedAt but the modifiedAt property is being used as timeproperty.
So actually I am able to query timeseries data with the described STH infrastructure using orion-ld, but not to visualize them with grafana. Do you have recommendations on that?
Update:
to keep it more simple you could also use the sample from FIWARE instead of my custom example from above: https://github.com/FIWARE/tutorials.Short-Term-History
Just clone code, switch to branch NGSI-LD, add Grafana to the example in docker-compose:
...
grafana:
labels:
org.test: 'fiware'
image: grafana/grafana:6.1.6
container_name: grafana
networks:
default:
ipv4_address: 172.18.1.8
ports:
- "3003:3000"
... and start the example as described by ./services orion legacy. Afterwards open farm http://localhost:3000 and do some stuff (refill hay, water on, start tractor, ...)
If you query OCB you get temporal data easily. For example heartRate of cow002:
curl --location --request GET 'http://localhost:8080/temporal/entities/urn:ngsi-ld:Animal:cow002' \
--header 'NGSILD-Tenant: openiot' \
--header 'Link: <http://context/ngsi-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--header 'attrs: heartRate'
Response:
{
"id": "urn:ngsi-ld:Animal:cow002",
"type": "Animal",
"heartRate": [
{
"type": "Property",
"value": 51.0,
"observedAt": "2022-09-26T13:37:25.823Z",
"instanceId": "urn:ngsi-ld:attribute:instance:5ba7d35c-3da0-11ed-a105-0242ac120109",
"unitCode": "5K",
"providedBy": {
"object": "urn:ngsi-ld:Device:cow002",
"type": "Relationship",
"instanceId": "urn:ngsi-ld:attribute:instance:5ba7d3fc-3da0-11ed-a105-0242ac120109"
}
},
{
"type": "Property",
"value": 52.0,
"observedAt": "2022-09-26T13:37:35.523Z",
"instanceId": "urn:ngsi-ld:attribute:instance:6184922e-3da0-11ed-abeb-0242ac120109",
"unitCode": "5K",
"providedBy": {
"object": "urn:ngsi-ld:Device:cow002",
"type": "Relationship",
"instanceId": "urn:ngsi-ld:attribute:instance:618492ce-3da0-11ed-abeb-0242ac120109"
}
},
{
"type": "Property",
"value": 51.0,
"observedAt": "2022-09-26T13:38:00.610Z",
"instanceId": "urn:ngsi-ld:attribute:instance:705e724c-3da0-11ed-a516-0242ac120109",
"unitCode": "5K",
"providedBy": {
"object": "urn:ngsi-ld:Device:cow002",
"type": "Relationship",
"instanceId": "urn:ngsi-ld:attribute:instance:705e72e2-3da0-11ed-a516-0242ac120109"
}
},
{
"type": "Property",
"value": 52.0,
"observedAt": "2022-09-26T13:38:15.597Z",
"instanceId": "urn:ngsi-ld:attribute:instance:794d239e-3da0-11ed-a8c0-0242ac120109",
"unitCode": "5K",
"providedBy": {
"object": "urn:ngsi-ld:Device:cow002",
"type": "Relationship",
"instanceId": "urn:ngsi-ld:attribute:instance:794d242a-3da0-11ed-a8c0-0242ac120109"
}
},
...
In timescale db you now can find tables for attributes, subattributes, entities and more stuff filled with some data. When I want to visualize the same example of cow002 and heartRate with grafana, I am not sure which table or tables I need to query. I am even not able to find the attribute heartRate in any of the tables.
Could you please assist how data can be visualized using grafana and Short Time History? Tutorials / Documentation don't mention how to visualize STH data with grafana with the different table structure. Maybe we can use the example of the heartRate to keep example simple.
Thank you in advance!
Yeah I finally found a solution and described it here in case of Scorpio: https://github.com/FIWARE/tutorials.Short-Term-History/issues/15
It is far more complicated to visualize data when you use STH instead of QL+subscriptions, but it is possible with more advanced Grafana queries.
Related
I am doing some analysis on spark sql query execution plans. the execution plans that explain() api prints are not much readable. If we see spark web UI, a DAG graph is created which is divided into jobs, stages and tasks and much more readable. Is there any way to create that graph from execution plans or any apis in the code? if not, are there any apis that can read that grap from UI?
As close I can see, this project (https://github.com/AbsaOSS/spline-spark-agent) is able to interpret the execution plan and generate it in a readable way.
This spark job is reading a file, convert it to a CSV file, write to local.
A sample output in JSON look like
{
"id": "3861a1a7-ca31-4fab-b0f5-6dbcb53387ca",
"operations": {
"write": {
"outputSource": "file:/output.csv",
"append": false,
"id": 0,
"childIds": [
1
],
"params": {
"path": "output.csv"
},
"extra": {
"name": "InsertIntoHadoopFsRelationCommand",
"destinationType": "csv"
}
},
"reads": [
{
"inputSources": [
"file:/Users/liajiang/Downloads/spark-onboarding-demo-application/src/main/resources/wikidata.csv"
],
"id": 2,
"schema": [
"6742cfd4-d8b6-4827-89f2-4b2f7e060c57",
"62c022d9-c506-4e6e-984a-ee0c48f9df11",
"26f1d7b5-74a4-459c-87f3-46a3df781400",
"6e4063cf-4fd0-465d-a0ee-0e5c53bd52b0",
"2e019926-3adf-4ece-8ea7-0e01befd296b"
],
"params": {
"inferschema": "true",
"header": "true"
},
"extra": {
"name": "LogicalRelation",
"sourceType": "csv"
}
}
],
"other": [
{
"id": 1,
"childIds": [
2
],
"params": {
"name": "`source`"
},
"extra": {
"name": "SubqueryAlias"
}
}
]
},
"systemInfo": {
"name": "spark",
"version": "2.4.2"
},
"agentInfo": {
"name": "spline",
"version": "0.5.5"
},
"extraInfo": {
"appName": "spark-spline-demo-application",
"dataTypes": [
{
"_typeHint": "dt.Simple",
"id": "f0dede5e-8fe1-4c22-ab24-98f7f44a9a5a",
"name": "timestamp",
"nullable": true
},
{
"_typeHint": "dt.Simple",
"id": "dbe1d206-3d87-442c-837d-dfa47c88b9c1",
"name": "string",
"nullable": true
},
{
"_typeHint": "dt.Simple",
"id": "0d786d1e-030b-4997-b005-b4603aa247d7",
"name": "integer",
"nullable": true
}
],
"attributes": [
{
"id": "6742cfd4-d8b6-4827-89f2-4b2f7e060c57",
"name": "date",
"dataTypeId": "f0dede5e-8fe1-4c22-ab24-98f7f44a9a5a"
},
{
"id": "62c022d9-c506-4e6e-984a-ee0c48f9df11",
"name": "domain_code",
"dataTypeId": "dbe1d206-3d87-442c-837d-dfa47c88b9c1"
},
{
"id": "26f1d7b5-74a4-459c-87f3-46a3df781400",
"name": "page_title",
"dataTypeId": "dbe1d206-3d87-442c-837d-dfa47c88b9c1"
},
{
"id": "6e4063cf-4fd0-465d-a0ee-0e5c53bd52b0",
"name": "count_views",
"dataTypeId": "0d786d1e-030b-4997-b005-b4603aa247d7"
},
{
"id": "2e019926-3adf-4ece-8ea7-0e01befd296b",
"name": "total_response_size",
"dataTypeId": "0d786d1e-030b-4997-b005-b4603aa247d7"
}
]
}
}
I am trying to make an ARM Template for a Custom Connector which needs to have enabled the use of on-premises Data Gateway. I am getting everything right except this last requirement, since I create a fully configured Custom Connector, except for the fact that the check-mark for using the on-premises Data Gateway is not active.
Any idea which setting/element I need to add change to get this done?
This is my template up to now:
{
"type": "Microsoft.Web/customApis",
"name": "[parameters('Connection_Name')]",
"apiVersion": "2016-06-01",
"location": "centralus",
"scale": null,
"properties": {
"connectionParameters": {
"authType": {
"type": "string",
"allowedValues": [
{
"value": "none"
}
],
"uiDefinition": {
"displayName": "Authentication Type",
"description": "Authentication type to connect to your API",
"tooltip": "Authentication type to connect to your API",
"constraints": {
"tabIndex": 1,
"required": "true",
"allowedValues": [
{
"text": "none",
"value": "anonymous"
}
],
"capability": [
"gateway"
]
}
}
},
"gateway": {
"type": "gatewaySetting",
"gatewaySettings": {
"dataSourceType": "CustomConnector",
"connectionDetails": []
},
"uiDefinition": {
"constraints": {
"tabIndex": 4,
"required": "true",
"capability": [
"gateway"
]
}
}
}
},
"backendService": {
"serviceUrl": "[parameters('ServiceUrl')]"
},
"apiType": "Soap",
"wsdlDefinition": {
"importMethod": "SoapPassThrough"
},
"swagger": {
"swagger": "2.0",
"info": {
"title": "SOAP pass-through",
"description": "Custom Connector for SOAP Operation",
"version": "1.0"
},
"host": "xxxxxxxxxxxxx",
"basePath": "/xxxxxxxx/xxxxxxxxxx",
"consumes": [],
"produces": [],
"paths": {
"/": {
"post": {
"responses": {
"default": {
"description": "default",
"schema": {
"type": "string",
"title": "",
"x-ms-visibility": "important"
},
"headers": {
"Content-Type": {
"description": "Content-Type",
"type": "string"
}
}
}
},
"summary": "GetOrigins",
"description": "GetOrigins",
"operationId": "GetOrigins",
"parameters": [
{
"name": "Content-Type",
"in": "header",
"required": false,
"type": "string"
},
{
"name": "body",
"in": "body",
"required": false,
"schema": {
"type": "string"
}
}
]
}
}
},
"definitions": {},
"parameters": {},
"responses": {},
"securityDefinitions": {},
"security": [],
"tags": [],
"schemes": [
"http"
]
},
"description": "[concat('Custom Connector for SOAP', parameters('Connection_Name'),' Operation')]",
"displayName": "[parameters('Connection_Name')]",
"iconUri": "/Content/retail/assets/default-connection-icon.6296199fc1d74aa45ca68e1253762025.2.svg"
},
"dependsOn": []
}
To enable the 'On-Premises Data Gateway' option, you need to add gateway to the capabilities array like so:
"properties" {
"capabilities": [
"gateway"
],
}
the On-Premises Data Gateway option is not available from ARM templates, you must to install your template and manually add the check box from the Azure portal.
Thanks.
I have created a REST API with a Swagger/OPEN API specification which I will like to consume trough a Azure API Management tenant in a Logic App.
When I download the specification it looks like this:
{
"swagger": "2.0",
"info": {
"title": "Leasing",
"version": "1.0"
},
"host": "ENDPOINT.azure-api.net",
"basePath": "/leasing",
"schemes": [
"http",
"https"
],
"securityDefinitions": {
"apiKeyHeader": {
"type": "apiKey",
"name": "Ocp-Apim-Subscription-Key",
"in": "header"
},
"apiKeyQuery": {
"type": "apiKey",
"name": "subscription-key",
"in": "query"
}
},
"security": [
{
"apiKeyHeader": []
},
{
"apiKeyQuery": []
}
],
"paths": {
"/{Brand}/groups": {
"get": {
"description": "Get a list of leasing groups on a brand",
"operationId": "GetGroups",
"parameters": [
{
"name": "Brand",
"in": "path",
"description": "Selection of possible brands",
"required": true,
"type": "string",
"enum": [
"Volkswagen",
"Audi",
"Seat",
"Skoda",
"VolkswagenErhverv",
"Porsche",
"Ducati"
]
}
],
"responses": {
"200": {
"description": "Returns a list of leasing groups",
"schema": {
"$ref": "#/definitions/GroupArray"
}
},
"400": {
"description": "If the brand is not valid",
"schema": {
"$ref": "#/definitions/Error"
}
}
},
"produces": [
"application/json"
]
}
}
},
"definitions": {
"Group": {
"type": "object",
"properties": {
"id": {
"format": "int32",
"type": "integer"
},
"name": {
"type": "string"
},
"description": {
"type": "string"
},
"leasingModelCount": {
"format": "int32",
"type": "integer"
},
"lowestMonthlyFee": {
"format": "int32",
"type": "integer"
}
}
},
"Error": {
"type": "object",
"properties": {
"code": {
"enum": [
"NotValidBrand",
"NotValidGroupId"
],
"type": "string",
"x-ms-enum": {
"name": "ErrorCode",
"modelAsString": true
}
},
"message": {
"type": "string"
}
}
},
"GroupArray": {
"type": "array",
"items": {
"$ref": "#/definitions/Group"
}
}
}
}
When I add this in a Logic App with the connector HTTP + Swagger I only get to define the {Brand} query input but not the various ways of using the Subscriptions key (header or query) as defined in SecurityDefiniations.
The whole securityDefinitions and security section are automatically generated in the Azure API Management service, but not recognized in Logic App.
See image of missing subscription key field:
What am I doing wrong?
Update
I have tried the following:
Usage of the 'Authentication' field (but this field is limited to certain types of auths flows https://learn.microsoft.com/en-us/azure/connectors/connectors-native-http#authentication)
Change the Logic App 'Http + Swagger'-action in code to add the header parameter, but this action converts the action to a simple 'Http' action and therfore loosing the automatic schema generation from Swagger.
I think you need to specify this in the Authentication-field in a JSON format. Something like:
{
"apiKeyHeader" : "your Ocp-Apim-Subscription-Key",
"apiKeyQuery" : "your subscription key"
}
I have below config file which pass parameter to ADF pipelines according to activity number. but I want, it should pass according to activity name hence
if I reordered the pipeline I need not to reconfigure my config file beacuse then it will identify according to activity name.
"PL_DATA_IL_Omni_Attend": [
{
"name": "$.properties.start",
"value": "2016-02-08T19:00:00Z"
},
{
"name": "$.properties.end",
"value": "2016-02-08T19:00:00Z"
},
{
"name": "$.properties.activities[0].typeProperties.rootPath",
"value": "container/realtime/"
},
{
"name": "$.properties.activities[0].typeProperties.entryFilePath",
"value": "batchprocessor.jar"
},
{
"name": "$.properties.activities[0].typeProperties.className",
"value": "com.cdp.batch.processor.XMLFilesProcessorToBlob"
},
{
"name": "$.properties.activities[0].typeProperties.arguments[0]",
"value": "dev"
},
{
"name": "$.properties.activities[0].typeProperties.arguments[1]",
"value": "Attend"
},
{
"name": "$.properties.activities[0].typeProperties.arguments[2]",
"value": "DELETE_STG"
},
{
"name": "$.properties.activities[1].typeProperties.rootPath",
"value": "container/realtime/"
},
{
"name": "$.properties.activities[1].typeProperties.entryFilePath",
"value": "batchprocessor.jar"
},
{
"name": "$.properties.activities[1].typeProperties.className",
"value": "com.processor.XMLFilesProcessorToBlob"
},
{
"name": "$.properties.activities[1].typeProperties.arguments[0]",
"value": "dev"
},
{
"name": "$.properties.activities[1].typeProperties.arguments[1]",
"value": "Attend"
},
{
"name": "$.properties.activities[1].typeProperties.arguments[2]",
"value": "READWRITE"
},
{
"name": "$.properties.activities[3].typeProperties.rootPath",
"value": "container/realtime"
},
{
"name": "$.properties.activities[3].typeProperties.entryFilePath",
"value": "batchprocessor.jar"
},
]
I want something like below
{
"name": "$.properties.activities[Name of the activity].typeProperties.rootPath",
"value": "container/realtime/"
},
Sudeep, are you using ADF V1 or ADF V2? We just launched V2 of our product which allows for parameter passing.
An example of an ADF V2 pipeline with parameters here: https://learn.microsoft.com/en-us/azure/data-factory/tutorial-control-flow#create-a-data-factory
Is context telemetry "grouped" during the sampling of request telemetry?
For example, the data below contains a request which has a sample count of 10 ("count": 10), meaning that it is being used to represent 9 other "similar" requests; 90% of the telemetry has actually been discarded.
Does Application Insights only sample data together when the context data is exactly the same for the requests? For example, can I assume that the other 9 requests were also from 41.191.204.0 and have a custom field company of value 22f0141f-b3dc-53e1-86b8-dd0727c14497?
{
"request": [
{
"id": "bs6o2dRoL/Q=",
"name": "GET /api/resources",
"count": 10,
"responseCode": 200,
"success": true,
"url": "https://example.com/api/resources",
"urlData": {
"base": "/api/resources",
"host": "example.com",
"hashTag": "",
"protocol": "https"
},
"durationMetric": {
"value": 1073743.0,
"count": 11.0,
"min": 97613.0,
"max": 97613.0,
"stdDev": 0.0,
"sampledValue": 97613.0
}
}
],
"internal": {
"data": {
"id": "8cbd12ec-9780-11e6-b38b-c5e9335e7642",
"documentVersion": "1.61"
}
},
"context": {
"application": {
"version": "1.0.16286.5"
},
"data": {
"eventTime": "2016-10-21T11:21:16.942Z",
"isSynthetic": false,
"samplingRate": 9.09090909090909
},
"device": {
"type": "PC",
"osVersion": "Windows 10",
"roleInstance": "RD0003FF727A10",
"deviceName": "Other",
"deviceModel": "Other",
"browser": "Chrome",
"browserVersion": "Chrome 53.0",
},
"user": {
"isAuthenticated": false
},
"session": {
"isFirst": false
},
"operation": {
"id": "bs6o2dRoL/Q=",
"parentId": "bs6o2dRoL/Q=",
"name": "GET Resources/GetResourceAsync [id]"
},
"location": {
"clientip": "41.191.204.0",
"continent": "Africa",
"country": "South Africa",
"province": "Eastern Cape"
},
"custom": {
"dimensions": [
{
"company": "22f0141f-b3dc-53e1-86b8-dd0727c14497"
},
{
"factor": "100"
}
]
}
}
}
Application Insights does not group telemetry events based on the context, but based on the Operation ID. This is synchronized between the SDK sampling and the server side sampling to make sure you will be able to navigate between related page views and requests.
So if you want to make sure some events are grouped together in sampling, set their OperationId to be the same.
See here for full details on how Application Insights implements it's sampling.
Hope this helps,
Asaf