I have several IoT devices (code is Java) that are in turn connected to several units locally. Each of these units reports data to the IoT device, and the IoT device publishes all the data in a variable array of nested objects, which also have arrays:
{
"data":{
"version":"1.2.3",
"sensorData":{
"a":18.50733137829912,
"b":8.165982404692084,
"c":20.75894428152493,
},
"units":[
{
"address":"192.168.254.16",
"name":"Unit 1",
"connectors":[
{
"id":1,
"measurement":{
"a":13.44,
"b":0.0,
"c":0.0
},
"status":"running"
}
]
},
{
"address":"192.168.254.17",
"name":"Unit 2",
"connectors":[
{
"id":1,
"measurement":{
"a":0.0,
"b":0.0,
"c":0.0
},
"status":"initialized",
}
]
}
]
},
"notificationType":"Status",
"type":"Notification"
}
Ideally I want to be able to go on some dashboard application like the Contoso example and click on this IoT device, and view a graph showing a,b,c on the sensorData, which is flat and fixed, but also show "Unit 1" and "Unit 2" separately as curves, as for example "Unit 1 - a", "Unit 1 - b", "Unit 1 - c", or aggregate them into a sum as "Unit 1 - total". Is this possible with the data presented?
So, do I
Post this data "as is" and somehow use a notation to get the nested data and aggregate?
Process it on the IoT device to flatten it, like unit1_name, unit1_address, unit1_connector_1_a, unit1_connector_1_b, unit1_connector_1_c etc.
Post this data "as is" and process/flatten the data in azure the same way as I would locally before it reaches the application (through stream analytics?)
All the samples I've seen has "flat" structure like a temperature and pressure on the root json object, so I am not sure what the best way to go forward here.
How does this fit into Application Insights? I also have events like "started", "stopped" etc, should I use the telemetry client for this (in Application Insights for Java)? What about logging? Using log4j2. Everything goes through DeviceClient and then is analyzed and distributed further thru Stream Analytics?
Thanks!
IoT Central currently supports only flat JSON for device measurements. So flatten the JSON on the device when forming the payload to post to IoT Hub. For events like "started", "stopped" you can use State measurement.
Azure IoT Central is a different product than Application Insights. Application Insights' primary focus is on application/user monitoring whereas IoT Central is solely focused on IoT scenarios and has a completely different stack.
Related
I have an Event Hub and Azure Function connected to it. With small amounts of data all works well, but when I tested it with 10 000 events, I got very peculiar results.
For test purposes I send into Event hub numbers from 0 to 9999 and log data in application insights and in service bus. For the first test I see in Azure that hub got exactly 10 000 events, but service bus and AI got all messages between 0 and 4500, and every second message after 4500 (so it lost about 30%). In second test, I got all messages from 0 to 9999, but every second message between 3500 and 3200 was duplicated. I would like to get all messages once, what did I do wrong?
public async Task Run([EventHubTrigger("%EventHubName%", Connection = "AzureEventHubConnectionString")] EventData[] events, ILogger log)
{
int id = _random.Next(1, 100000);
_context.Log.TraceInfo("Started. Count: " + events.Length + ". " + id); //AI log
foreach (var message in events)
{
//log with ASB
var mess = new Message();
mess.Body = message.EventBody.ToArray();
await queueClient.SendAsync(mess);
}
_context.Log.TraceInfo("Completed. " + id); //AI log
}
By using EventData[] events, you are reading events from hub in batch mode, thats why you see X events processing at a time then next seconds you process next batch.
Instead of EventData[] use simply EventData.
When you send events to hub check that all events are sent with the same partition key if you want try batch processing otherwise they can be splitted in several partitions depending on TU (throughput units), PU (Processing Units) and CU (Capacity Units).
Egress: Up to 2 MB per second or 4096 events per second.
Refer to this document.
Throughput limits for Basic, Standard, Premium..:
There are a couple of things likely happening, though I can only speculate with the limited context that we have. Knowing more about the testing methodology, tier of your Event Hubs namespace, and the number of partitions in your Event Hub would help.
The first thing to be aware of is that the timing between when an event is published and when it is available in a partition to be read is non-deterministic. When a publish operation completes, the Event Hubs broker has acknowledged receipt of the events and taken responsibility for ensuring they are persisted to multiple replicas and made available in a specific partition. However, it is not a guarantee that the event can immediately be read.
Depending on how you sent the events, the broker may also need to route events from a gateway by performing a round-robin or applying a hash algorithm. If you're looking to optimize the time from publish to availability, taking ownership of partition distribution and publishing directly to a partition can help, as can ensuring that you're publishing with the right degree of concurrency for your host environment and scenario.
With respect to duplication, it's important to be aware that Event Hubs offers an "at least once" guarantee; your consuming application should expect some duplicates and needs to be able to handle them in the way that is appropriate for your application scenario.
Azure Functions uses a set of event processors in its infrastructure to read events. The processors collaborate with one another to share work and distribute the responsibility for partitions between them. Because collaboration takes place using storage as an intermediary to synchronize, there is an overlap of partition ownership when instances are scaled up or scaled down, during which time the potential for duplication is increased.
Functions makes the decision to scale based on the number of events that it sees waiting in partitions to be read. In the case of your test, if your publication pattern increases rapidly and Functions sees "the event backlog" grow to the point that it feels the need to scale by multiple instances, you'll see more duplication than you otherwise would for a period of 10-30 seconds until partition ownership normalizes. To mitigate this, using an approach of gradually increasing speed of publishing over a 1-2 minute period can help to smooth out the scaling and reduce (but not eliminate) duplication.
Is there any way/trick/workaround to reduce Azure Application Insight cost? I have a very large volume of data (around 20M) ingestion every day. Data sampling set 5%, Even after Daily 5GB of data ingestion in Application Insights.
Application Insights has 90 days default retention period, but I don't need data even after 7 days.
Note: I'm only sending Info logs in application Insights with minimal information.
Few points I want to add here which help me.
Review logging and log data in correct category. (Verbose, Info, Error etc). Most of the time (verbose only for debugging) am using Info and Error log only.
Combined multiple lines of logs in a single line. It will reduce system-generated columns data.
Old
log.Info($"SerialNumber :" serialNumber)
log.Info($"id :" id)
log.Info($"name :" name)
New
log.Info($"SerialNumber, id, name :" serialNumber + id + name);
Check traces collection customDimensions. For me, a lot of system-generated data was there like thread name, logger name and class name. I did some changes in my logger, now my customDimensions column is empty.
Reduce some system-generated logs
{
"logger": {
"categoryFilter": {
"defaultLevel": "Information",
"categoryLevels": {
"Host": "Error",
"Function": "Error",
"Host.Aggregator": "Information"
}
}
}
}
All the above points help me to reduce log data, hope it will help others.
You can do more aggressive sampling. Also review what "info logs" are being sent. Perhaps you can live without sending many of them. Also review all the auto-collected telemetry -if there is something you dont care about - (eg: perf counters), remove them.
Also check this Msdn blog authored by application insights team members: https://msdn.microsoft.com/magazine/mt808502.aspx
I currently have an Azure Application Gateway that is configured for a minimum of 2 instances and a maximum of 10 instances. It's Tier is "WAF V2" and autoscaling is enabled.
If autoscaling is enabled, then theoretically there should be somewhere between 2 and 10 instances. So where can I go to check the current number of instances that the gateway has scaled up to? This seems like important information if you want to figure out if your gateway is overloaded.
I currently have been pointed out by Microsoft to this topic after asking them the same question.
My assumption on this that I take, and may not be acurate, is that I look at the Current Capacity Units metric, to see how many are in use for a certain moment. Since, documentation says one instance uses around 10 capacity units, I do the simple math to know how many instances we are using, and if we need to increase the max, or lower the minimum.
https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-autoscaling-zone-redundant
"Each capacity unit is composed of at most: 1 compute unit, or 2500 persistent connections, or 2.22-Mbps throughput."
"Note
Each instance can currently support approximately 10 capacity units. The number of requests a compute unit can handle depends on various criteria like TLS certificate key size, key exchange algorithm, header rewrites, and in case of WAF incoming request size. We recommend you perform application tests to determine request rate per compute unit. Both capacity unit and compute unit will be made available as a metric before billing starts."
I've tried to obtain this value with Logs analytics.
Enable it and use this query:
AzureDiagnostics
| where TimeGenerated > ago(30m)
| summarize dcount(instanceId_s) by bin(TimeGenerated, 1m)
You will have the different request grouped by the distinct instance name every minute. Consider adding some additional filter to the query, since you may only be interested in certain types of events.
I think it can be a good approximation
I don't think it shows you the current number of instances (if you switch to manual it will show you the instance count under properties blade), because it doesn't make sense. That's what autoscale is for, you don't really care how many instances are running, what you care is request latency\failed requests. If you see those increase, you can increase the number of maximum Application Gateway instances.
Api gives the following response with autoscale enabled:
"sku": {
"name": "Standard_v2",
"tier": "Standard_v2"
},
And this without autoscale enabled:
"sku": {
"name": "Standard_v2",
"tier": "Standard_v2",
"capacity": 4
},
so I guess it hidden from the api, so no way to know it.
As a part of Azure Machine Learning process, I need to continually migrate data from on-premises SQL Db to Azure SQL Db using Data Management Gateway.
This Azure official article describes how to: Move data from an on-premises SQL server to SQL Azure with Azure Data Factory. But the details are a bit confusing to me. If someone to briefly describe the process, how would you do that. What are 2-3 main steps one needs to perform on on-premises and 2-3 steps on Azure Cloud? No details are needed. Note: The solution has to involve using Data Management Gateway
Based on Azure documentation you can use "slices". You can perform a "delta" fetch using a timestamp column as mentioned by this article or using a sequential integer column. To avoid issues about rows not been included on the a slice due to the on-premise server having system date a little behind than Azure system date, is better to use a sequential integer. Below the Input dataset shows how to define slices:
{
"name": "AzureBlobInput",
"properties": {
"type": "AzureBlob",
"linkedServiceName": "StorageLinkedService",
"typeProperties": {
"folderPath": "mycontainer/myfolder/{Year}/{Month}/{Day}/",
"partitionedBy": [
{ "name": "Year", "value": {"type": "DateTime","date": "SliceStart","format": "yyyy"}},
{ "name": "Month","value": {"type": "DateTime","date": "SliceStart","format": "MM"}},
{ "name": "Day","value": {"type": "DateTime","date": "SliceStart","format": "dd"}}
],
"format": {
"type": "TextFormat"
}
},
"external": true,
"availability": {
"frequency": "Hour",
"interval": 1
}
}
}
You can create an activity and use the availability section to specify a schedule for the activity. You can specify "frequency" (minute, hour, day, etc.) and "interval".
"scheduler": {
"frequency": "Hour",
"interval": 1
}
Each unit of data consumed or produced by an activity run is called a data slice. The following diagram shows an example of an activity with one input dataset and one output dataset:
The diagram shows the hourly data slices for the input and output dataset. The diagram shows three input slices that are ready for processing. The 10-11 AM activity is in progress, producing the 10-11 AM output slice.
You can access the time interval associated with the current slice in the dataset JSON by using variables: SliceStart and SliceEnd. You can use these variables in your activity JSON to select data from input dataset representing time series data (for example: 8 AM to 9 AM).
You can also set the start date for the pipeline in the past as shown here. When you do so, Data Factory automatically calculates (back fills) all data slices in the past and begins processing them.
Suppose I am receiving a stream of unordered sequential data in time.
For example, input could be:
[
{id:1, timestamp:1},
{id:2, timestamp:1},
{id:2, timestamp:2},
{id:1, timestamp:2},
{id:3, timestamp:1}
]
Each entity is identified by 'id' field. There could be a large amount of entities and processing for each input could take some time.
The problem is that I need to process each of those events in order it was received for each entity.
I was considering some solutions as to put messages to Kafka topic with partitions and receive parallelism?
Or create local storage of received messages and marking each processed message for each entity after successful processing (on other machine or on the same in Thread pool)?
Questions:
Is it is a good solution?
How can I reach this functionality while scaling data consumers (having fixed number of services/ creating new instances)?
Maybe there is a better way to solve such kind of problem?
"IF" the sequential data you mention just divided by id, 1 2 and 3,
Then Would be the best you make 3 background services as an consumer, just need 1 partition for the case (you can decided this on your own)
Then make 3 topic based on the data
ex :
TOPIC 1
TOPIC 2
TOPIC 3
which mean you need to make 3 kind of consumer, each of consumer would be listen to only 1 topic
Then you would be spawn new process / Thread for every new stream data,
It would work in parallel