We are using Azure application Insights for error logging. I am new to KQL and trying to fetch custom properties from inbuilt "customDimensions" column in the following format,
Value as is from "customDimensions" column
exceptions
| project customDimensions
{
"File Name":"Sample File 1",
"Correlation ID":"e33a8d45-1234-1234-1223-54a6fec30356",
"Error List":"[
{\"Function Name\":\"Sample Function 1\",\"Code\":\"12345\"},
{\"Function Name\":\"Sample-Function-2\",\"Code\":\"12343\"}]"
}
Expected Output
File Name
Correlation ID
Function Name
Code
Sample File 1
e33a8d45-1234-1234-1223-54a6fec30356
Sample Function 1
12345
Sample File 1
e33a8d45-1234-1234-1223-54a6fec30356
Sample-Function-2
12343
How can I achieve the above output using KQL?
Thank You.
This might seem a little bit tricky, but bear with me :-)
Every sub-element extracted from a dynamic element, is dynamic.
parse_json() / todynamic() when given a dynamic argument, returns it, As Is.
So first, we use tostring() and only then we use todynamic() so the string would be parsed as json, to dynamic type.
datatable(ErrorDetails:dynamic)
[
dynamic({
"File Name":"Sample File 1",
"Correlation ID":"e33a8d45-0566-4bf2-94f8-54a6fec29bff",
"Error List":"[{\"Function Name\":\"Sample Function 1\",\"Code\":\"12345\"},{\"Function Name\":\"Sample-Function-2\",\"Code\":\"12343\"}]"
})
]
| mv-expand EL = todynamic(tostring(ErrorDetails["Error List"]))
| project ["File Name"] = ErrorDetails["File Name"], ["Correlation ID"] = ErrorDetails["Correlation ID"], ["Function Name"] = EL["Function Name"], ["Code"] = EL["Code"]
File Name
Correlation ID
Function Name
Code
Sample File 1
e33a8d45-0566-4bf2-94f8-54a6fec29bff
Sample Function 1
12345
Sample File 1
e33a8d45-0566-4bf2-94f8-54a6fec29bff
Sample-Function-2
12343
Fiddle
Related
I have a Kusto query taken from this example that looks like this:
Resources
| where type =~ 'microsoft.compute/virtualmachines'
| extend vmPowerState = tostring(properties.extended.instanceView.powerState.code)
| summarize count() by vmPowerState
I would like to create an weekly alert that send the result through an e-mail in a CSV file.
The Logic App is organized in 5 steps:
One:
Two:
With
URL: https://management.azure.com/providers/Microsoft.ResourceGraph/resources
Body:
{
"query": "Resources | where type =~ 'microsoft.compute/virtualmachines' | extend vmPowerState = tostring(properties.extended.instanceView.powerState.code) | summarize count() by vmPowerState"
}
Three:
Where I parse the Body and I give an extract of the JSON Schema:
{
"count": 3,
"data": [
{
"count_": 3,
"vmPowerState": "PowerState/stopped"
},
{
"count_": 29,
"vmPowerState": "PowerState/deallocated"
},
{
"count_": 118,
"vmPowerState": "PowerState/running"
}
],
"skip_token": null,
"total_records": 3
}
Here I have a few doubt because I found a guide that says that I should use array formula instead. I'm not very sure about that because I cannot see the details in the example. Anyway this is what I do:
Four:
Five:
Where I create the attachment from the CSV
The e-mail in the end arrives but the attachment is not a CSV, it's a JSON file:
What the hack am I doing wrong?
if you want to use "Create CSV table" with Columns set to "Automatic", do pass the "body" of "parse Json".
you don't need to use the array variable but whatever you use need to return an array like this:
The body of the json parser on your example has many other json nodes enveloping that. You should have the option "data" as there is an array there called "data"
if you want to cut it short, try "data"
you can change to "custom". that would allow you to remove redundant data or format data (like the "PowerState" in "PowerState/stopped"):
you can also add the .csv to the file name:
The above worked for me but it can be enhanced
The suggestoin posted by #BrunoLucasAzure really helped me understand how Logic Apps works.
However I would like to reply to my own question with the right solution: I had to paste a sample of the JSON output pressing on the button Use sample payload to generate schema.
Then follow the workflow and everything will be fine.
The next problem I need to fix is pagination but apparently there is a solution for that too: https://techcommunity.microsoft.com/t5/integrations-on-azure-blog/logic-app-http-pagination-deeper-look-build-custom-paging/ba-p/2907605
I need to get the category name from category id using kusto query.
First i have got the most searched url from the website using the below kusto query and ran it in app insights logs
Requests
| where resultCode==200
| where url contains "bCatID"
| summarize count=sum(itemCount) by url
| sort by count
| take 1
From the above query i got the result like
https://www.test.com/item.aspx?idItem=123456789&bCatID=1282
So for corresponding categoryid=1282 i need to get the category name using kusto
you can use the parse operator.
for example:
print input = 'https://www.test.com/item.aspx?idItem=123456789&bCatID=1282'
| parse input with 'https://www.test.com/item.aspx?idItem=123456789&bCatID='category_id:long
parse_urlquery
print url ="https://www.test.com/item.aspx?idItem=123456789&bCatID=1282"
| extend tolong(parse_urlquery(url)["Query Parameters"]["bCatID"])
url
Query Parameters_bCatID
https://www.test.com/item.aspx?idItem=123456789&bCatID=1282
1282
Fiddle
Feedback to the OP query
Not an answer
KQL is case sensitive. The name of the table in Azure Application Insights is requests (and not Requests).
resultCode is of type string (and not integer/long) and should be compared to "200" (and not 200)
bCatID is a token and therefore can be searched using has or even has_cs, which should be preferred over contains due to performance reasons.
URLs can be used with different parameters. It might make more sense to summarize only by the Host & Path parts + the bCatID query parameter.
count is a reserved word. It can be used as alias only if qualified: ["count"] or ['count'] (or better not used at all).
sort followed by take 1 can be replaced with the more elegant top 1 by ...
requests
| where resultCode == "200"
| project url = parse_url(url), itemCount
| summarize sum(itemCount) by tostring(url.Host), tostring(url.Path), tolong(url["Query Parameters"]["bCatID"])
| top 1 by sum_itemCount
Json text isn't parsing in KQL correctly. I tried using parse_json as well but that didn't work either. I did confirm the extend AllProperties is holding the correct data.
DeviceInfo
| where RegistryDeviceTag == "Standard"
| extend AllProperties = todynamic(LoggedOnUsers)
| project DeviceName, Users = AllProperties["Username"]
Output gives me the correct DeviceName but doesn't give any data in the Username field.
(based on the sample input you provided in the comment)
if the array that is "LoggedOnUsers" includes exactly one entry, you can do this:
print input = '[{"UserName":"TheUserName","DomainName":"TheDomainName","Sid":"TheSID#"}]'
| project UserName = parse_json(input)[0].UserName
otherwise, you can use mv-expand or mv-apply:
print input = '[{"UserName":"TheUserName","DomainName":"TheDomainName","Sid":"TheSID#"}]'
| project parse_json(input)
| mv-apply input on (
project UserName = input.UserName
)
I have below JSON output from an API, in Office Excel I am importing data via Web from API.
[{
"level": 1,
"children": [{
"level": 2,
"children": [{
"level": 3,
"name": "Chandni Chowk",
"data": ["Data 1", "Data 2"]
}],
"name": "Delhi",
"data": ["Delhi Area"]
}],
"name": "Country",
"data": ["India", "Bangladesh"]
}]
https://learn.microsoft.com/en-us/powerquery-m/quick-tour-of-the-power-query-m-formula-language
I have above document.
let
Source = Json.Document(Web.Contents("MY API URL GOES HERE")),
AsTable = Table.FromRecords(Source)
----
----
in
#"Renamed Column2"
In the power query editor I have this for now.
As a out put in Excel file I need like this.
Country Delhi Chandni Chowk
India Delhi Area Data 1
Bangladesh Data 2
Can I get this data from this JSON or I need to change my JSON output format which matches power query?
Power Query interprets JSON as a hierarchy of records and lists. My goal is to flatten the JSON into a record like this and then convert it into a table:
Country : {"India", "Bangladesh"}
Delhi : {"Delhi Area"}
Chandni Chowk : {"Data 1", "Data 2"}
At any particular level, we can pull the name and data value using Record.FromList:
Record.FromList({CurrentLevel[data]}, {CurrentLevel[name]})
For the first level, this is
Record.FromList({{"India","Bangladesh"}}, {"Country"})
which corresponds to the first field in the goal record.
At any level, we can navigate to the next level like this:
NextLevel = CurrentLevel[children]{0}
Using these to building blocks, we can now write a custom function Expand to flatten the record:
1 | (R as record) as record =>
2 | let
3 | ThisLevel = Record.FromList({R[data]}, {R[name]}),
4 | CombLevel = if Record.HasFields(R, {"children"})
5 | then Record.Combine({ThisLevel, #Expand(R[children]{0})})
6 | else ThisLevel
7 | in
8 | CombLevel
Line 1: The syntax for defining a function. It takes a record R and returns a record after doing some transformations.
Line 3: How to deal with the current level, as mentioned earlier.
Line 4: Check if the record has another level to expand down to.
Line 5: If it does, then Record.Combine the result of the current level with the result of the next level, where the result of the next level is calculated by navigating to the next level and recursively applying the function we're defining. With three levels this looks like:
Record.Combine({Level1, Record.Combine({Level2, Level3})})
Line 6: Recursion stops when there are no more levels to expand. No more combinations, just the last level is returned.
All that's left is to transform it into the shape we want. Here's what my query looks like using the Expand function we just defined:
let
Source = Json.Document( < JSON Source > ),
ExpandRecord = Expand(Source{0}),
ToTable = Table.FromColumns(
Record.FieldValues(ExpandRecord),
Record.FieldNames(ExpandRecord)
)
in
ToTable
This uses Record.FieldValues and Record.FieldName as arguments in Table.FromColumns.
The step after using the Expand custom function looks like this in the query editor if you select the first list cell:
The final result is what you asked for:
I have telemetry in my code that creates two custom events(longNameHere_event_success/longNameHere_event_error). I have a small AI Analytics query that looks for my events, gets the count of each, and renders a pie chart for a percentage metric.
As my event names are rather long, I'd like to create a variable for each to make the pie chart more legible.
customEvents
| where name contains "event"
|summarize count() by name
| render piechart
Current result is a pie chart with percentages for "longNameHere_event_success" and "longNameHere_event_error"
Desired result would be renaming "longNameHere_event_success" to "Success" and "longNameHere_event_error" to "Failure".
You can rename the column value as follows (I used an inline function for readability, but you can replace the function call with the case statement). Is this what you were looking for?
let rename = (original:string)
{
case(original == "longNameHere_event_success", "success",
original == "longNameHere_event_error", "error",
"unknown")
};
let customEvents = datatable(name:string)
[
"longNameHere_event_success",
"longNameHere_event_success",
"longNameHere_event_error"
];
customEvents
| where name contains "event"
| summarize count() by name
| project name = rename(name), count_
| render piechart