Azure Application Insights - values within objects - azure
I'm trying to get my head around writing queries in Azure Application Insights which is capturing interactions with a bot built using Azure Bot Framework.
I have a table with headings such as timestamp, name, customDimensions, customDimensions and within customDimensions are objects such as
{
"conversationData": "{}",
"privateConversationData": "{\"nameForm\":{\"NAME\":\"foo\",\"ICCID\":\"12121212121212121212\"}}",
"userData": "{}",
"conversationId": "878fhiee1k33j5ci",
"userId": "default-user",
"metrics": "92.25833"
}
I can write queries easily to select items by name for example
customEvents
| where name contains "Activity"
but how do I select based on keys within objects such as those within privateConversationData above?
For example "privateConversationData": "{\"nameForm\":{\"NAME\":\"foo\",\"ICCID\":\"12121212121212121212\"}}", refers to one dialog called nameForm, how would I write a query to show the number of times the nameForm was used? Or a query that included the other kinds of dialog (e.g. not just nameForm, but fooForm, barForm) and a count of the times they were used?
Many thanks for any help!
The 'customDimensions' property is a dynamic type and therefore can be treated as a JSON document.
For example - to get the number of times nameForm was used in the last day:
customEvents
| extend conversationData = customDimensions["privateConversationData"]
| where timestamp > ago(1d) and isnotempty(conversationData) and conversationData contains "{\\\"nameForm\\\""
| count
Getting the different dialogs count will be trickier, but possible by parsing the customDimensions JSON document using the parse operator:
customEvents
| where timestamp > ago(1d)
| parse customDimensions with * "privateConversationData\": \"{\\\"" dialogKind "\\\":{\\\"NAME\\\"" *
| where isnotempty(dialogKind) and isnotnull(dialogKind)
| summarize count() by dialogKind
You can read the Analytics Reference to learn more about the language.
Related
How can I ingest data from Apache Avro into the Azure Data Explorer?
for several days I'm trying to ingest Apache Avro formatted data from a blob storage into the Azure Data Explorer.
I'm able to reference the toplevel JSON-keys like $.Body (see red underlined example in the screenshot below), but when it goes to the nested JSON-keys, Azure fails to parse them properly and displays nothing (as seen in the green column: I would expect $.Body.entityId to reference the key "entityId" inside the Body-JSON).
Many thanks in advance for any help!
Here is a screenshot of the azure data explorer web interface
Edit 1
I already tried to increase the "Nested Levels" Option to 2, but all I got is this error message with no further details. The error message won't even disappear when I decrease the Level back to 1. I have to cancel and start the process all over agein.
I just recognize that the auto-generated columns have some strange types. Seems like they add up to the type string... This seems a little odd to me either.
Edit 2
Here is some kql-Code.
This is the schema of my input .avro file, what I get from my Eventhub-Capture:
{
SequenceNumber: ...,
Offset: ...,
EnqueuedTimeUTC: ...,
SystemProperties: ...,
Properties: ...,
Body: {
entityId: ...,
eventTime: ...,
messageId: ...,
data: ...
}
}, ...
And with these ingestion commands I can't reference the inner JSON-keys. The toplevel keys work perfectly fine.
// Create table command
////////////////////////////////////////////////////////////
.create table ['test_table'] (['Body']:dynamic, ['entityId']:string)
// Create mapping command
////////////////////////////////////////////////////////////
.create table ['test_table'] ingestion apacheavro mapping 'test_table_mapping' '[{"column":"Body", "Properties":{"Path":"$.Body"}},{"column":"entityId", "Properties":{"Path":"$.Body.entityId"}}]'
// Ingest data into table command
///////////////////////////////////////////////////////////
.ingest async into table ['test_table'] (h'[SAS URL]') with (format='apacheavro',ingestionMappingReference='test_table_mapping',ingestionMappingType='apacheavro',tags="['503a2cfb-5b81-4c07-8658-639009870862']")
I would love to ingest the inner data fields on separate columns, instead of building any workaround with update policies.
For those having the same issue, here is the workaround we currently use:
First, assume that we want to ingest the contents of the Body field from the avro file to the table avro_destination.
Step 1: Create an ingestion table
.create table avro_ingest(
Body: dynamic
// optional other columns, if you want...
)
Step 2: Create an update policy
.create-or-alter function
with (docstring = 'Convert avro_ingest to avro_destination', folder='ingest')
convert_avro_ingest() {
avro_ingest
| extend entityId = tostring(Body.entityId)
| extend messageId = tostring(Body.messageId)
| extend eventTime = todatetime(Body.eventTime)
| extend data = Body.data
| project entityId, messageId, eventTime, data
}
.alter table avro_destination policy update
#'[{ "IsEnabled": true, "Source": "avro_ingest", "Query": "convert_avro_ingest()", "IsTransactional": false, "PropagateIngestionProperties": true}]'
Step 3: Ingest the .avro files into the avro_ingest table
...as seen in the Question, with one Column containing the whole Body-JSON per entry.
Following the OP updates
Here is the Avro schema of an Event Hubs capture.
As you can see, Body as of type bytes, so there is practically nothing you can do with it at this form, other than ingesting it As Is (as Dynamic).
{
"type":"record",
"name":"EventData",
"namespace":"Microsoft.ServiceBus.Messaging",
"fields":[
{"name":"SequenceNumber","type":"long"},
{"name":"Offset","type":"string"},
{"name":"EnqueuedTimeUtc","type":"string"},
{"name":"SystemProperties","type":{"type":"map","values":["long","double","string","bytes"]}},
{"name":"Properties","type":{"type":"map","values":["long","double","string","bytes"]}},
{"name":"Body","type":["null","bytes"]}
]
}
If you'll take a look on the ingested data, you'll see that the content of Body is arrays of integers.
Those integers are the decimal values of the characters that construct Body.
capture
| project Body
| take 3
Body
[123,34,105,100,34,58,32,34,56,49,55,98,50,99,100,57,45,97,98,48,49,45,52,100,51,53,45,57,48,51,54,45,100,57,55,50,51,55,55,98,54,56,50,57,34,44,32,34,100,116,34,58,32,34,50,48,50,49,45,48,56,45,49,50,84,49,54,58,52,56,58,51,50,46,53,57,54,50,53,52,34,44,32,34,105,34,58,32,48,44,32,34,109,121,105,110,116,34,58,32,50,48,44,32,34,109,121,102,108,111,97,116,34,58,32,48,46,51,57,56,53,52,52,56,55,52,53,57,56,57,48,55,57,55,125]
[123,34,105,100,34,58,32,34,57,53,100,52,100,55,56,48,45,97,99,100,55,45,52,52,57,50,45,98,97,54,100,45,52,56,49,54,97,51,56,100,52,56,56,51,34,44,32,34,100,116,34,58,32,34,50,48,50,49,45,48,56,45,49,50,84,49,54,58,52,56,58,51,50,46,53,57,54,50,53,52,34,44,32,34,105,34,58,32,49,44,32,34,109,121,105,110,116,34,58,32,56,56,44,32,34,109,121,102,108,111,97,116,34,58,32,48,46,54,53,53,51,55,51,51,56,49,57,54,53,50,52,52,49,125]
[123,34,105,100,34,58,32,34,53,50,100,49,102,54,54,53,45,102,57,102,54,45,52,49,50,49,45,97,50,57,99,45,55,55,56,48,102,101,57,53,53,55,48,56,34,44,32,34,100,116,34,58,32,34,50,48,50,49,45,48,56,45,49,50,84,49,54,58,52,56,58,51,50,46,53,57,54,50,53,52,34,44,32,34,105,34,58,32,50,44,32,34,109,121,105,110,116,34,58,32,49,57,44,32,34,109,121,102,108,111,97,116,34,58,32,48,46,52,53,57,54,49,56,54,51,49,51,49,50,50,52,50,50,51,125]
Body can be converted to text using make_string() and then parsed to JSON using todynamic()
capture
| project BodyJSON = todynamic(make_string(Body))
| take 3
BodyJSON
{"id":"817b2cd9-ab01-4d35-9036-d972377b6829","dt":"2021-08-12T16:48:32.5962540Z","i":0,"myint":20,"myfloat":"0.398544874598908"}
{"id":"95d4d780-acd7-4492-ba6d-4816a38d4883","dt":"2021-08-12T16:48:32.5962540Z","i":1,"myint":88,"myfloat":"0.65537338196524408"}
{"id":"52d1f665-f9f6-4121-a29c-7780fe955708","dt":"2021-08-12T16:48:32.5962540Z","i":2,"myint":19,"myfloat":"0.45961863131224223"}
Simply increase "Nested levels" to 2.
Grafana azure log analytics transfer query from logs
I have this query that works in Azure logs when i set the scope to the specific application insights I want to use
let usg_events = dynamic(["*"]);
let mainTable = union pageViews, customEvents, requests
| where timestamp > ago(1d)
| where isempty(operation_SyntheticSource)
| extend name =replace("\n", "", name)
| where '*' in (usg_events) or name in (usg_events)
;
let queryTable = mainTable;
let cohortedTable = queryTable
| extend dimension =tostring(client_CountryOrRegion)
| extend dimension = iif(isempty(dimension), "<undefined>", dimension)
| summarize hll = hll(user_Id) by tostring(dimension)
| extend Users = dcount_hll(hll)
| order by Users desc
| serialize rank = row_number()
| extend dimension = iff(rank > 5, 'Other', dimension)
| summarize merged = hll_merge(hll) by tostring(dimension)
| project ["Country or region"] = dimension, Counts = dcount_hll(merged);
cohortedTable
but trying to use the same in grafana just gives an error.
"'union' operator: Failed to resolve table expression named 'pageViews'"
Which is the same i get in azure logs if i dont set the scope to the specific application insights resource. So my question is. how do i make it so grafana targets this specific scope inside the logs? The query jsut gets the countries of the users that log in
As far as I know, Currently, there is no option/feature to add Scope in Grafana. The Scope is available only in the Azure Log Analytics Workspace. If you want the Feature/Resolution, please raise a ticket in Grafana Community where all the issues are officially addressed.
How to create an Azure Kusto query to group by client OS name only (OS version removed) on App insights?
What I want The number of page views grouped by client OS (no OS version = only OS name) and week. What I have - Kusto query: pageViews |where timestamp > ago(90d) |summarize Browser_hits = count() by Date = startofweek(timestamp), client_Browser |sort by Date |render timechart The problem with this query is that the client OS name is coming with the version in it and that ends up in different versions grouping separately (see picture below). Update This is close to what I need but it won't work for any names, I'm just posting this as an example that helps to understand the actual question. pageViews |where timestamp > ago(90d) |summarize Browser_hits = count() by Date = startofweek(timestamp), BrowserNameTrimed = substring(client_Browser,0,5) | sort by Date | render timechart With the previous query I get this (kind of cheating):
would this work? (parsing the browser name out of the "browser name + browser version" combination using the parse operator): pageViews | where timestamp > ago(90d) | summarize Browser_hits = count() by Date = startofweek(timestamp), client_Browser | parse kind=regex client_Browser with client_Browser #" \d+" * | render timechart
Search Query should contain 'AggregatedValue' and 'bin(timestamp, [roundTo])' for Metric alert type
I'm trying to create a custom metric alert based on some metrics in my Application Insights logs. Below is the query I'm using; let start = customEvents | where customDimensions.configName == "configName" | where name == "name" | extend timestamp, correlationId = tostring(customDimensions.correlationId), configName = tostring(customDimensions.configName); let ending = customEvents | where customDimensions.configName == configName" | where name == "anotherName" | where customDimensions.taskName == "taskName" | extend timestamp, correlationId = tostring(customDimensions.correlationId), configName = tostring(customDimensions.configName), name= name, nameTimeStamp= timestamp ; let timeDiffs = start | join (ending) on correlationId | extend timeDiff = nameTimeStamp- timestamp | project timeDiff, timestamp, nameTimeStamp, name, anotherName, correlationId; timeDiffs | summarize AggregatedValue=avg(timeDiff) by bin(timestamp, 1m) When I run this query in Analytics page, I get results, however when I try to create a custom metric alert, I got the error Search Query should contain 'AggregatedValue' and 'bin(timestamp, [roundTo])' for Metric alert type The only response I found was adding AggregatedValue which I already have, I'm not sure why custom metric alert page is giving me this error.
I found what was wrong with my query. Essentially, aggregated value needs to be numeric, however AggregatedValue=avg(timeDiff) produces time value, but it was in seconds, so it was a bit hard to notice. Converting it to int solves the problem, I have just updated last bit as follows timeDiffs | summarize AggregatedValue=toint(avg(timeDiff)/time(1ms)) by bin(timestamp, 5m) This brings another challenge on Aggregate On while creating the alert as AggregatedValue is not part of the grouping that is coming after by statement.
Azure Log Analytics - Search REST API - How to Paginate through results
When grabbing search result using Azure Log Analytics Search REST API I'm able to receive only the first 5000 results (as by the specs, at the top of the document), but know there are many more (by the "total" attribute in the metadata in the response). Is there a way to paginate so to get the entire result set? One hacky way would be to attempt to break down the desired time-range iteratively until the "total" is less than 5000 for that timeframe, and do this process iteratively for the entire desired time-range - but this is guesswork that will cost many redundant requests.
While it doesn't appear to be a way to paginate using the REST API itself, you can use your query to perform the pagination. The two key operators here are TOP and SKIP: Suppose you want page n with pagesize x (starting at page 1), then append to your query: query | skip (n-1) * x | top x. For a full reference list, see https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-search-reference
Yes, skip operation is not available anymore but if you want create pagination there is still an option. You need to count total count of entries, use a simple math and two opposite sortings.
Prerequisites for this query are values: ContainerName, Namespace, Page, PageSize.
I'm using it in Workbook where these values are set by fields.
let containers = KubePodInventory
| where ContainerName matches regex '^.*{ContainerName}$' and Namespace == '{Namespace}'
| distinct ContainerID
| project ContainerID;
let TotalCount = toscalar(ContainerLog
| where ContainerID in (containers)
| where LogEntry contains '{SearchText}'
| summarize CountOfLogs = count()
| project CountOfLogs);
ContainerLog
| where ContainerID in (containers)
| where LogEntry contains '{SearchText}'
| extend Log=replace(#'(\x1b\[[0-9]*m|\x1b\[0 [0-9]*m)','', LogEntry)
| project TimeGenerated, Log
| sort by TimeGenerated asc
| take {PageSize}*{Page}
| top iff({PageSize}*{Page} > TotalCount, TotalCount - ({PageSize}*({Page} - 1)) , {PageSize}) by TimeGenerated desc;
// The '| extend' is not needed if in logs are not the annoying special characters