I am trying to write an App Insights query that will report back the timespan between two known events, specifically circuit breaker open and close events. The assumption is that these events always occur in pairs, so we need to know the time between the two for every occurrence in a time period.
My first attempt was to use an inline function. Simplified version below.
let timeOpened = (timeClosed:datetime)
{
let result = customEvents
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Open'
| where timestamp < timeClosed
| order by timestamp desc
| take 1
| project timestamp;
let scalar = toscalar(result);
scalar
};
customEvents
| where timestamp > ago(4h)
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Closed'
| extend timeOpen = timestamp - timeOpened(timestamp)
There may be a better way to do this. If so your ideas are welcome! But in this particular attempt the only feedback I get from Azure when running this is "Syntax error". However, I don't believe there's a syntax error here because if I just change the return value of the function from scalar to now() it runs successfully. Also I can run the body of the function in isolation successfully. Any idea what's wrong here?
I think you are getting syntax error because query language does not allow possibly recursive constructs. Now() worked because it was statically (not dynamically) retrieved at the query time.
I think you may achieve the desired outcome with serialize and prev() operators:
Table | order by timestamp asc | serialize
| extend previousTime = prev(timestamp,1)
| extend Diff = iff(customDimensions['State'] == 'Closed', timestamp - previousTime, 0)
| where Diff > 0
Note: I haven't tested the example above and it may need some additional thought to make it work (e.g. making sure that the previous record is actually "Opened" before doing previousTime calculation).
Related
I need to get the category name from category id using kusto query.
First i have got the most searched url from the website using the below kusto query and ran it in app insights logs
Requests
| where resultCode==200
| where url contains "bCatID"
| summarize count=sum(itemCount) by url
| sort by count
| take 1
From the above query i got the result like
https://www.test.com/item.aspx?idItem=123456789&bCatID=1282
So for corresponding categoryid=1282 i need to get the category name using kusto
you can use the parse operator.
for example:
print input = 'https://www.test.com/item.aspx?idItem=123456789&bCatID=1282'
| parse input with 'https://www.test.com/item.aspx?idItem=123456789&bCatID='category_id:long
parse_urlquery
print url ="https://www.test.com/item.aspx?idItem=123456789&bCatID=1282"
| extend tolong(parse_urlquery(url)["Query Parameters"]["bCatID"])
url
Query Parameters_bCatID
https://www.test.com/item.aspx?idItem=123456789&bCatID=1282
1282
Fiddle
Feedback to the OP query
Not an answer
KQL is case sensitive. The name of the table in Azure Application Insights is requests (and not Requests).
resultCode is of type string (and not integer/long) and should be compared to "200" (and not 200)
bCatID is a token and therefore can be searched using has or even has_cs, which should be preferred over contains due to performance reasons.
URLs can be used with different parameters. It might make more sense to summarize only by the Host & Path parts + the bCatID query parameter.
count is a reserved word. It can be used as alias only if qualified: ["count"] or ['count'] (or better not used at all).
sort followed by take 1 can be replaced with the more elegant top 1 by ...
requests
| where resultCode == "200"
| project url = parse_url(url), itemCount
| summarize sum(itemCount) by tostring(url.Host), tostring(url.Path), tolong(url["Query Parameters"]["bCatID"])
| top 1 by sum_itemCount
I know that this is a really long post but I'm not sure of what part of my process is making my file crash, so I tried to detail everything about what I did to get to the error messages.
So, first of all, I created a query on Kusto, which looks something similar to this but in reality is 160 lines of code, this is just a summarized version of what my code might do just to show my working process.
First, what I do in Session_Id_List is create a list of all distinct Session Id's from the past day.
Then on treatment_alarms1 I count the amount of alarms for each type of alarm that was active during each session.
Then, on treatment_alarms2 I create a list which might look something like this
1x Alarm_Type_Number1
30x Alarm_Type_Number2
7x Alarm_Type_Number3
and like that for each treatment, so I have a list of all alarms that were active for that treatment.
Lastly, I create a left outer join with Session_Id_List and treatment_alarms2. This means that I will get shown all of the treatment ID's, even the ones that did not have any active alarms.
let _StartTime = ago(1d);
let _EndTime = ago(0d);
let Session_Id_List = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime
| summarize by SessionId, SerialNumber, StartTime
| distinct SessionId, StartTime, SerialNumber;
let treatment_alarms1 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| summarize number_alarms = count() by TranslatedData_Value, SessionId
| project final_Value = strcat(number_alarms, "x ", TranslatedData_Value), SessionId;
let treatment_alarms2 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| join kind=inner treatment_alarms1 on SessionId
| summarize list_of_alarms_all = make_set(final_Value) by SessionId
| project SessionId, list_of_alarms_all;
let final_join = Session_Id_List
| join kind=leftouter treatment_alarms2 on SessionId;
final_join
| project SessionId, list_of_alarms_all
Then I put this query into Excel, by using the following method
I go to Tools -> Query to Power BI on Kusto Explorer
I go to Data -> Get Data -> From Other Sources -> Blank Query
I go to advanced editor
I copy and paste my query and press "Done" at the bottom
If you see now, the preview of my data will show "List" on the list_of_alarms_all column, rather than showing me the actual values of the list.
To fix this issue I first press the arrows on the header of the column
I press on "Extract Values"
I select Custom -> Concatenate using special characters -> Line Feed -> Press OK
That works fine for all of the ID's that do have alarms on them, it shows them as a list and tells me how many there are, the issue is with the ID's that did not have any treatments where I get "Error" on the Excel preview. Once I press "Close & Load" the data is put on the worksheet and it looks fine, the "Error" are all gone and instead I get empty cells where the "Error" would be at.
The problem now starts when I close the file and try to open it again.
First I get this message. So I press yes to try and enter the file.
Then I get this other message. The problem with this message is that it says that I have the file open when that is not true. I even tried to restart my laptop and open the file again and I would still get the message when in reality I don't have that file open.
Then I get this message, telling me that the connection to the query was removed.
So my problem here is that 1) I can't edit the file anymore unless I make a copy because I keep getting the message saying that I already have the file opened and it is locked for editing and 2) I would like to refresh this query with VBA maybe once a week from now on but I can't because when I save the file the connection to the query is deleted by excel itself.
I'm not sure of why this is happening, I'm guessing it's because of the "Error" I get on the empty cells when I try to extract the values from the lists. If anybody has any information on how I can fix this so I don't get these error messages please let me know.
I was not able to reproduce your issue, however there are some things you might want to try.
Within ADX, you could wrap you query with a function, so you won't have to copy a large piece of code into your Excel.
You could deal with null values (this is what gives you the Error values) already in your query. Note the use of coalesce.
// I used datatable to mimic your query results
.create-or-alter function SessionAlarms()
{
datatable (SessionId:int,list_of_alarms_all:dynamic)
[
1, dynamic([10,20,30])
,2, dynamic([])
,3, dynamic(null)
]
| extend list_of_alarms_all = coalesce(list_of_alarms_all, dynamic([]))
}
You can use Power Query ADX connector and copy your query/function As Is
If you haven't dealt with null values in you KQL you can take care of the error in Excel by using Replace Errors
I'd like to render a cumulative histogram of the duration of various workflows.
This is the query I use to compute the duration of various workflows, and the timeElapsedInMs result can be used to do things like compute percentiles (as shown). The durations span a fairly sizable range, from under a second to several minutes. What I would like to do is render a cumulative histogram of the data. My best guess at this point is that the accumulate property of one of the render types might work, but I can't seem to wrap my head around how to do that.
let start = fluentbit_CL
| where log_s has_cs "<starting event>"
| project trackingId = trackingId_g, timestampStart= _timestamp_t;
let finish = fluentbit_CL
| where log_s contains "<ending event>"
| project trackingId = trackingId_g, timestampFinish = _timestamp_t;
start
| join finish on trackingId
| extend timeElapsedInMs = datetime_diff('millisecond', todatetime(timestampFinish ), todatetime(timestampStart))
| summarize min=min(timeElapsedInMs), avg=avg(timeElapsedInMs), percentiles_array(timeElapsedInMs, 50, 95), max=max(timeElapsedInMs)
I'd also be happy with another sensible way to render the data to get a look at the percentiles, but my current best approaches don't get a very clear impression of the data.
There is no need to write one query for starting event and one for ending event and then join them. This can be done with a single query, using summarize.
Histogram can be easily created using bin (doc)
Accumulated histogram can be easily rendered using with (accumulate=True) (doc)
// This part of the code is just for data sample generation and is not part of the solution
let start = materialize (range trackingId_g from 1 to 100 step 1 | extend _timestamp_t = ago(24h * rand()), log_s = "<starting event>");
let finish = materialize (start | extend _timestamp_t = _timestamp_t + 5m * rand(), log_s = "<ending event>");
let fluentbit_CL = (union start, finish);
// The solution starts from here
fluentbit_CL
| summarize duration = anyif(_timestamp_t, log_s has_cs "<ending event>") - anyif(_timestamp_t, log_s has_cs "<starting event>")
by trackingId_g
| summarize count() by bin(duration, 10s)
| order by duration asc
| render timechart with (accumulate=True)
Fiddle
I need to app insights traces with following pattern for messages:
"D1 connected"
"D2 connected"
"D3 connected"
"D1 disconnected"
"D3 disconnected"
"D1 connected"
"D2 disconnected"
etc.
I'm basically monitoring some devices and the connection time. How can I write a query that "pairs" events (D1 connected/disconnected, D2 connected/disconnected, etc.) and evaluates how long the "sessions" are?
I'd need to get information like:
total connection time for a day
distribution of the connection for a specific device on a day
etc.
Doing this just based on the text of the trace will be hard. I suggest using custom properties to assist in this.
By far the easiest option is to send some additional properties along with the disconnected event that have all the info required. Like:
// Start of session
var tt = new TraceTelemetry("D1 connected");
tt.Properties.Add("Event", "SessionStart");
telemetryClient.TrackTrace(tt);
var startTime = DateTime.Now;
// Do your thing
....
tt = new TraceTelemetry("D1 disconnected");
tt.Properties.Add("Event", "SessionEnd");
tt.Properties.Add("SessionLength", (startTime - DateTime.Now).TotalMilliseconds.ToString());
telemetryClient.TrackTrace(tt);
Custom properties are stored in the customDimensions field of an event.
Now in AI analytics you can query these values like this:
Count:
traces
| where customDimensions.Event == "SessionEnd"
| summarize count()
Session lengths:
traces
| where customDimensions.Event == "SessionEnd"
| project message, customDimensions.Length
Total duration of all sessions:
traces
| where customDimensions.Event == "SessionEnd"
| extend duration = tolong(customDimensions.SessionLength)
| summarize sum(duration)
I would also suggest adding the device Id as a custom property for all emitted events. It will make querying easier. You can then calculate min, max and average session lengths per device, for example:
traces
| where customDimensions.Event == "SessionEnd"
| extend duration = tolong(customDimensions.SessionLength)
| extend device = tostring(customDimensions.DeviceName)
| summarize sum(duration) by device
If you want to join the start events as well or cannot or will not do the above you have to join the start events with the end events to make these queries. You will still need to use some custom properties since querying on text alone will be hard because you will then need to analyze the text to determine what event and what device is involved.
take a look here azure AI QUERY combine start and response to calculate average to see how joins work in AI Analytics.
In AfterScenario method, I want to get the rows from table "Examples" in Scenario Outline, to get the values in it, and search that specific values in the database
I know that this can be achieved by using Context.Scenario.Current...
Context.Scenario.Current[key]=value;
...but for some reason I'd like to be able to get it in a simpler way
like this:
ScenarioContext.Current.Examples();
----------- SCENARIO --------------------------------
Scenario Outline: Create a Matter
Given I create matter "< matterName >"
Examples:
| matterName |
| TAXABLE |
----------AFTER SCENARIO -----------------------------------
[AfterScenario()]
public void After()
{
string table = ScenarioContext.Current.Examples();
}
So if you look at the code for ScenarioContext you can see it inherits from SpecflowContext which is itself a Dictionary<string, object>. This means that you can simply use Values to get the collection of values, but I have no idea if they are Examples or not.
The best solution I came up with was to infer the examples by keeping my own static singleton object, then counting how many times the same scenario ran.
MyContext.Current.Counts[ScenarioContext.Current.ScenarioInfo.Title]++;
Of course, this doesn't work very well if you don't run all the tests at the same time or run them in random order. Having a table with the examples themselves would be more ideal, but if you combine my technique along with using ScenarioStepContext you could extract the parameters of the Examples table out of the rendered step definition text itself.
Feature
Scenario Outline: The system shall do something!
Given some input <input>
When something happens
Then something should have happened
Examples:
| input |
| 1 |
| 2 |
| 3 |
SpecFlow Hook
[BeforeStep]
public void BeforeStep()
{
var text = ScenarioStepContext.Current.StepInfo.Text;
var stepType = ScenarioStepContext.Current.StepInfo.StepDefinitionType;
if (text.StartsWith("some input ") && stepType == StepDefinitionType.Given)
{
var input = text.Split(' ').Last();
}
}