Pair events in Application Insights Analytics - azure

I need to app insights traces with following pattern for messages:
"D1 connected"
"D2 connected"
"D3 connected"
"D1 disconnected"
"D3 disconnected"
"D1 connected"
"D2 disconnected"
etc.
I'm basically monitoring some devices and the connection time. How can I write a query that "pairs" events (D1 connected/disconnected, D2 connected/disconnected, etc.) and evaluates how long the "sessions" are?
I'd need to get information like:
total connection time for a day
distribution of the connection for a specific device on a day
etc.

Doing this just based on the text of the trace will be hard. I suggest using custom properties to assist in this.
By far the easiest option is to send some additional properties along with the disconnected event that have all the info required. Like:
// Start of session
var tt = new TraceTelemetry("D1 connected");
tt.Properties.Add("Event", "SessionStart");
telemetryClient.TrackTrace(tt);
var startTime = DateTime.Now;
// Do your thing
....
tt = new TraceTelemetry("D1 disconnected");
tt.Properties.Add("Event", "SessionEnd");
tt.Properties.Add("SessionLength", (startTime - DateTime.Now).TotalMilliseconds.ToString());
telemetryClient.TrackTrace(tt);
Custom properties are stored in the customDimensions field of an event.
Now in AI analytics you can query these values like this:
Count:
traces
| where customDimensions.Event == "SessionEnd"
| summarize count()
Session lengths:
traces
| where customDimensions.Event == "SessionEnd"
| project message, customDimensions.Length
Total duration of all sessions:
traces
| where customDimensions.Event == "SessionEnd"
| extend duration = tolong(customDimensions.SessionLength)
| summarize sum(duration)
I would also suggest adding the device Id as a custom property for all emitted events. It will make querying easier. You can then calculate min, max and average session lengths per device, for example:
traces
| where customDimensions.Event == "SessionEnd"
| extend duration = tolong(customDimensions.SessionLength)
| extend device = tostring(customDimensions.DeviceName)
| summarize sum(duration) by device
If you want to join the start events as well or cannot or will not do the above you have to join the start events with the end events to make these queries. You will still need to use some custom properties since querying on text alone will be hard because you will then need to analyze the text to determine what event and what device is involved.
take a look here azure AI QUERY combine start and response to calculate average to see how joins work in AI Analytics.

Related

Excel removes my query connection on it's own and gives me several error messages

I know that this is a really long post but I'm not sure of what part of my process is making my file crash, so I tried to detail everything about what I did to get to the error messages.
So, first of all, I created a query on Kusto, which looks something similar to this but in reality is 160 lines of code, this is just a summarized version of what my code might do just to show my working process.
First, what I do in Session_Id_List is create a list of all distinct Session Id's from the past day.
Then on treatment_alarms1 I count the amount of alarms for each type of alarm that was active during each session.
Then, on treatment_alarms2 I create a list which might look something like this
1x Alarm_Type_Number1
30x Alarm_Type_Number2
7x Alarm_Type_Number3
and like that for each treatment, so I have a list of all alarms that were active for that treatment.
Lastly, I create a left outer join with Session_Id_List and treatment_alarms2. This means that I will get shown all of the treatment ID's, even the ones that did not have any active alarms.
let _StartTime = ago(1d);
let _EndTime = ago(0d);
let Session_Id_List = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime
| summarize by SessionId, SerialNumber, StartTime
| distinct SessionId, StartTime, SerialNumber;
let treatment_alarms1 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| summarize number_alarms = count() by TranslatedData_Value, SessionId
| project final_Value = strcat(number_alarms, "x ", TranslatedData_Value), SessionId;
let treatment_alarms2 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| join kind=inner treatment_alarms1 on SessionId
| summarize list_of_alarms_all = make_set(final_Value) by SessionId
| project SessionId, list_of_alarms_all;
let final_join = Session_Id_List
| join kind=leftouter treatment_alarms2 on SessionId;
final_join
| project SessionId, list_of_alarms_all
Then I put this query into Excel, by using the following method
I go to Tools -> Query to Power BI on Kusto Explorer
I go to Data -> Get Data -> From Other Sources -> Blank Query
I go to advanced editor
I copy and paste my query and press "Done" at the bottom
If you see now, the preview of my data will show "List" on the list_of_alarms_all column, rather than showing me the actual values of the list.
To fix this issue I first press the arrows on the header of the column
I press on "Extract Values"
I select Custom -> Concatenate using special characters -> Line Feed -> Press OK
That works fine for all of the ID's that do have alarms on them, it shows them as a list and tells me how many there are, the issue is with the ID's that did not have any treatments where I get "Error" on the Excel preview. Once I press "Close & Load" the data is put on the worksheet and it looks fine, the "Error" are all gone and instead I get empty cells where the "Error" would be at.
The problem now starts when I close the file and try to open it again.
First I get this message. So I press yes to try and enter the file.
Then I get this other message. The problem with this message is that it says that I have the file open when that is not true. I even tried to restart my laptop and open the file again and I would still get the message when in reality I don't have that file open.
Then I get this message, telling me that the connection to the query was removed.
So my problem here is that 1) I can't edit the file anymore unless I make a copy because I keep getting the message saying that I already have the file opened and it is locked for editing and 2) I would like to refresh this query with VBA maybe once a week from now on but I can't because when I save the file the connection to the query is deleted by excel itself.
I'm not sure of why this is happening, I'm guessing it's because of the "Error" I get on the empty cells when I try to extract the values from the lists. If anybody has any information on how I can fix this so I don't get these error messages please let me know.
I was not able to reproduce your issue, however there are some things you might want to try.
Within ADX, you could wrap you query with a function, so you won't have to copy a large piece of code into your Excel.
You could deal with null values (this is what gives you the Error values) already in your query. Note the use of coalesce.
// I used datatable to mimic your query results
.create-or-alter function SessionAlarms()
{
datatable (SessionId:int,list_of_alarms_all:dynamic)
[
1, dynamic([10,20,30])
,2, dynamic([])
,3, dynamic(null)
]
| extend list_of_alarms_all = coalesce(list_of_alarms_all, dynamic([]))
}
You can use Power Query ADX connector and copy your query/function As Is
If you haven't dealt with null values in you KQL you can take care of the error in Excel by using Replace Errors

Having Trouble with creation of an Azure Sentinel Stealthwatch Log Analytics Query

I'm working on a project to create a Stealthwatch (Cisco) Log Analytics query from a canned workbook query available from the Azure Sentinel portal (Data Connectors > Cisco Stealthwatch). I've added the function to my LA query section and it will run (depending on the time parameters - more on that later).
When I run the query within the Log Query window, it runs great. However, when trying to use the Analytics Rule Wizard, the same query errors out with the following error:
'extend' operator: Failed to resolve scalar expression named 'start'.
Included here is the simple query I'm running:
StealthwatchEvent
| where EventSeverity == "Critical"
And here is the complete function as supplied by the Data Connector from the Azure Sentinel workbook for Cisco Stealthwatch:
Syslog
| where SyslogMessage has 'Stealthwatch'
| extend EventVendor = 'Cisco'
| extend EventProduct = 'Stealthwatch'
| extend EventProductVersion = extract(#'Stealthwatch\|([0-9\.]+)\|', 1, SyslogMessage)
| mv-apply ExtractedFields = extract_all(#'\|(?P<key>[a-zA-Z0-9-_]+)=(?P<value>[a-zA-Z0-9-_:/#.,#{}>< ]+)', dynamic(["key","value"]), SyslogMessage) on (
project packed = pack(tostring(ExtractedFields[0]), tostring(ExtractedFields[1]))
| summarize bag = make_bag(packed)
)
| evaluate bag_unpack(bag)
| extend EventStartTime=todatetime(start)
| extend EventEndTime=todatetime(end)
| project-rename EventOriginalUid=alarmID
, EventSeverity=alarmSev
, EventStatus=alarmStatus
, EventType=cat
, SrcDvcDomain=domain
, DstIpAddr=dst
, DstPortNumber=dstPort
, EventMessage=msg
, Protocol=proto
, SrcGeoCountry=sourceHG
, SrcHostSnapshot=sourceHostSnapshot
, SrcIpAddr=src
, DvcIpAddr=flowCollectorIP
, DvcHostname=flowCollectorName
, SrcUserName=sourceUser
, DstUserName=targetUser
, DstGeoCountry=argetHG
, DstDvcHostname=targetHostname
, DstHostSnapshot=targetHostSnapshot
| project-away start
, end
, SyslogMessage
I've narrowed it down to the "start" value in the one extend statement. Note when I adjust the time on the Rule wizard for the query period (I run the query every 15 minutes and lookup data from the last 15 minutes) it should pass that to the query time parameters - but I'm just not sure why it's failing. Sometimes when I adjust the time, it fails on the "end" value instead as reported by the error. I have also tried removing the two extend lines related to time altogether, but then the query dies on AlarmID in the next section.
(And yes, I am fairly new to this - I'm learning Kusto/Sentinel and Stealthwatch tuning and reporting on the fly here)
:)
Any help or pointers would be greatly appreciated.

Active users and views in azure workbooks

Can someone briefly explain the differences between active users and page views in an azure workbook? I would like to get the stats of active users of a particular azure event, but the counts shown are not realistic for the provided scenario.
As per the attached image, there is only one active user, which has to be at least 3-4 and can go up to 15-16. The value displayed in Views is unrealistic though.
Your valuable inputs are highly appreciated. Thank you
if you edit the template/workbook and look at the queries you can see exactly what is going on there:
let start = startofday(ago({Metric}));
union customEvents, pageViews
| where timestamp >= start
| where name in ({Activities}) or '*' in ({Activities}) or ('%' in ({Activities}) and itemType == 'pageView') or ('#' in ({Activities}) and itemType == 'customEvent')
{OtherFilters}
| summarize Users = dcount(user_Id), Sessions = dcount(session_Id), Views = count()
| evaluate narrow()
| project Title = case(Column == 'Users', 'Active Users', Column == 'Sessions', 'Unique Sessions', 'Views'), Metric = Value, SubTitle = '━━'
views in this case, is a count of items in the tables; Views = count(),
Users is dcount(user_Id) and Sessions = dcount(session_Id)
seeing just 1 user/session implies that there's nothing automatically generating "distinct" ids for those values, so the user_Id, session_Id fields are completely empty across all rows (the one unique id being undefined in both columns)
how user and session id values get populated is very specific to what application insights sdks you are using, and how you've configured them.

Kusto regex query for email

I am using the below Kusto query for exporting logs from App insight log
traces
| extend request = parse_json(tostring(parse_json(customDimensions).PayloadMessage))
| extend message = request.message
| extend SecondaryInfo = request.SecondaryInfo
| extend Logtype = request.Logtype
| extend File = request.File
| extend LineNo = request.LineNo
| extend MemberName = request.MemberName
| extend User = split(split(message,'User(').[1],')').[0]
| project timestamp,message,SecondaryInfo,Logtype,File,LineNo,MemberName,User
I want to extract the email address from the message column. So I am using split operation. However, I could not find the correct logic to extract the email address. Could someone please share some regex.
Below are some values in the messages column
-MMS.Core.Logging.MmsException: 1012 - Error while fetching Room Booking Requests for (User: TestAccount100#fr.domain.com) at
-MMS.Core.Logging.MmsException: 1011 - Error while fetching User's Locations (TestAccount100#fr.domain.com) at"
RbacRepository.GetUserCityBuildings-correlationId-11111fd6-e111-4d11-1111-11e101fbe111--DT-04162021T084110893-- START: (Email:TestAccount100#fr.domain.com), Cities mapped 4"
RoomBookingDetailsRepository.GetRequestDetailsAsync: Getting the BookingDetails data for User(TestAccount100#fr.domain.com), Role(Admin), IsAdmin(True), PPED() Status(), AssignedTo ()"
Below is an example that uses a naive regex.
You can go over the documentation for RE2 to adjust it, if required: https://github.com/google/re2/wiki/Syntax
datatable(s:string)
[
"-MMS.Core.Logging.MmsException: 1012 - Error while fetching Room Booking Requests for (User: TestAccount100#fr.domain.com) at",
"-MMS.Core.Logging.MmsException: 1011 - Error while fetching User's Locations (TestAccount100#fr.domain.com) at",
"RbacRepository.GetUserCityBuildings-correlationId-11111fd6-e111-4d11-1111-11e101fbe111--DT-04162021T084110893-- START: (Email:TestAccount100#fr.domain.com), Cities mapped 4",
"RoomBookingDetailsRepository.GetRequestDetailsAsync: Getting the BookingDetails data for User(TestAccount100#fr.domain.com), Role(Admin), IsAdmin(True), PPED() Status(), AssignedTo ()",
]
| extend email = extract(#"(\w+#\w+\.\w+)", 1, s)

Is it possible to get query results in an App Insights inline function?

I am trying to write an App Insights query that will report back the timespan between two known events, specifically circuit breaker open and close events. The assumption is that these events always occur in pairs, so we need to know the time between the two for every occurrence in a time period.
My first attempt was to use an inline function. Simplified version below.
let timeOpened = (timeClosed:datetime)
{
let result = customEvents
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Open'
| where timestamp < timeClosed
| order by timestamp desc
| take 1
| project timestamp;
let scalar = toscalar(result);
scalar
};
customEvents
| where timestamp > ago(4h)
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Closed'
| extend timeOpen = timestamp - timeOpened(timestamp)
There may be a better way to do this. If so your ideas are welcome! But in this particular attempt the only feedback I get from Azure when running this is "Syntax error". However, I don't believe there's a syntax error here because if I just change the return value of the function from scalar to now() it runs successfully. Also I can run the body of the function in isolation successfully. Any idea what's wrong here?
I think you are getting syntax error because query language does not allow possibly recursive constructs. Now() worked because it was statically (not dynamically) retrieved at the query time.
I think you may achieve the desired outcome with serialize and prev() operators:
Table | order by timestamp asc | serialize
| extend previousTime = prev(timestamp,1)
| extend Diff = iff(customDimensions['State'] == 'Closed', timestamp - previousTime, 0)
| where Diff > 0
Note: I haven't tested the example above and it may need some additional thought to make it work (e.g. making sure that the previous record is actually "Opened" before doing previousTime calculation).

Resources