I'm working on a project to create a Stealthwatch (Cisco) Log Analytics query from a canned workbook query available from the Azure Sentinel portal (Data Connectors > Cisco Stealthwatch). I've added the function to my LA query section and it will run (depending on the time parameters - more on that later).
When I run the query within the Log Query window, it runs great. However, when trying to use the Analytics Rule Wizard, the same query errors out with the following error:
'extend' operator: Failed to resolve scalar expression named 'start'.
Included here is the simple query I'm running:
StealthwatchEvent
| where EventSeverity == "Critical"
And here is the complete function as supplied by the Data Connector from the Azure Sentinel workbook for Cisco Stealthwatch:
Syslog
| where SyslogMessage has 'Stealthwatch'
| extend EventVendor = 'Cisco'
| extend EventProduct = 'Stealthwatch'
| extend EventProductVersion = extract(#'Stealthwatch\|([0-9\.]+)\|', 1, SyslogMessage)
| mv-apply ExtractedFields = extract_all(#'\|(?P<key>[a-zA-Z0-9-_]+)=(?P<value>[a-zA-Z0-9-_:/#.,#{}>< ]+)', dynamic(["key","value"]), SyslogMessage) on (
project packed = pack(tostring(ExtractedFields[0]), tostring(ExtractedFields[1]))
| summarize bag = make_bag(packed)
)
| evaluate bag_unpack(bag)
| extend EventStartTime=todatetime(start)
| extend EventEndTime=todatetime(end)
| project-rename EventOriginalUid=alarmID
, EventSeverity=alarmSev
, EventStatus=alarmStatus
, EventType=cat
, SrcDvcDomain=domain
, DstIpAddr=dst
, DstPortNumber=dstPort
, EventMessage=msg
, Protocol=proto
, SrcGeoCountry=sourceHG
, SrcHostSnapshot=sourceHostSnapshot
, SrcIpAddr=src
, DvcIpAddr=flowCollectorIP
, DvcHostname=flowCollectorName
, SrcUserName=sourceUser
, DstUserName=targetUser
, DstGeoCountry=argetHG
, DstDvcHostname=targetHostname
, DstHostSnapshot=targetHostSnapshot
| project-away start
, end
, SyslogMessage
I've narrowed it down to the "start" value in the one extend statement. Note when I adjust the time on the Rule wizard for the query period (I run the query every 15 minutes and lookup data from the last 15 minutes) it should pass that to the query time parameters - but I'm just not sure why it's failing. Sometimes when I adjust the time, it fails on the "end" value instead as reported by the error. I have also tried removing the two extend lines related to time altogether, but then the query dies on AlarmID in the next section.
(And yes, I am fairly new to this - I'm learning Kusto/Sentinel and Stealthwatch tuning and reporting on the fly here)
:)
Any help or pointers would be greatly appreciated.
Related
I need to get the category name from category id using kusto query.
First i have got the most searched url from the website using the below kusto query and ran it in app insights logs
Requests
| where resultCode==200
| where url contains "bCatID"
| summarize count=sum(itemCount) by url
| sort by count
| take 1
From the above query i got the result like
https://www.test.com/item.aspx?idItem=123456789&bCatID=1282
So for corresponding categoryid=1282 i need to get the category name using kusto
you can use the parse operator.
for example:
print input = 'https://www.test.com/item.aspx?idItem=123456789&bCatID=1282'
| parse input with 'https://www.test.com/item.aspx?idItem=123456789&bCatID='category_id:long
parse_urlquery
print url ="https://www.test.com/item.aspx?idItem=123456789&bCatID=1282"
| extend tolong(parse_urlquery(url)["Query Parameters"]["bCatID"])
url
Query Parameters_bCatID
https://www.test.com/item.aspx?idItem=123456789&bCatID=1282
1282
Fiddle
Feedback to the OP query
Not an answer
KQL is case sensitive. The name of the table in Azure Application Insights is requests (and not Requests).
resultCode is of type string (and not integer/long) and should be compared to "200" (and not 200)
bCatID is a token and therefore can be searched using has or even has_cs, which should be preferred over contains due to performance reasons.
URLs can be used with different parameters. It might make more sense to summarize only by the Host & Path parts + the bCatID query parameter.
count is a reserved word. It can be used as alias only if qualified: ["count"] or ['count'] (or better not used at all).
sort followed by take 1 can be replaced with the more elegant top 1 by ...
requests
| where resultCode == "200"
| project url = parse_url(url), itemCount
| summarize sum(itemCount) by tostring(url.Host), tostring(url.Path), tolong(url["Query Parameters"]["bCatID"])
| top 1 by sum_itemCount
I know that this is a really long post but I'm not sure of what part of my process is making my file crash, so I tried to detail everything about what I did to get to the error messages.
So, first of all, I created a query on Kusto, which looks something similar to this but in reality is 160 lines of code, this is just a summarized version of what my code might do just to show my working process.
First, what I do in Session_Id_List is create a list of all distinct Session Id's from the past day.
Then on treatment_alarms1 I count the amount of alarms for each type of alarm that was active during each session.
Then, on treatment_alarms2 I create a list which might look something like this
1x Alarm_Type_Number1
30x Alarm_Type_Number2
7x Alarm_Type_Number3
and like that for each treatment, so I have a list of all alarms that were active for that treatment.
Lastly, I create a left outer join with Session_Id_List and treatment_alarms2. This means that I will get shown all of the treatment ID's, even the ones that did not have any active alarms.
let _StartTime = ago(1d);
let _EndTime = ago(0d);
let Session_Id_List = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime
| summarize by SessionId, SerialNumber, StartTime
| distinct SessionId, StartTime, SerialNumber;
let treatment_alarms1 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| summarize number_alarms = count() by TranslatedData_Value, SessionId
| project final_Value = strcat(number_alarms, "x ", TranslatedData_Value), SessionId;
let treatment_alarms2 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| join kind=inner treatment_alarms1 on SessionId
| summarize list_of_alarms_all = make_set(final_Value) by SessionId
| project SessionId, list_of_alarms_all;
let final_join = Session_Id_List
| join kind=leftouter treatment_alarms2 on SessionId;
final_join
| project SessionId, list_of_alarms_all
Then I put this query into Excel, by using the following method
I go to Tools -> Query to Power BI on Kusto Explorer
I go to Data -> Get Data -> From Other Sources -> Blank Query
I go to advanced editor
I copy and paste my query and press "Done" at the bottom
If you see now, the preview of my data will show "List" on the list_of_alarms_all column, rather than showing me the actual values of the list.
To fix this issue I first press the arrows on the header of the column
I press on "Extract Values"
I select Custom -> Concatenate using special characters -> Line Feed -> Press OK
That works fine for all of the ID's that do have alarms on them, it shows them as a list and tells me how many there are, the issue is with the ID's that did not have any treatments where I get "Error" on the Excel preview. Once I press "Close & Load" the data is put on the worksheet and it looks fine, the "Error" are all gone and instead I get empty cells where the "Error" would be at.
The problem now starts when I close the file and try to open it again.
First I get this message. So I press yes to try and enter the file.
Then I get this other message. The problem with this message is that it says that I have the file open when that is not true. I even tried to restart my laptop and open the file again and I would still get the message when in reality I don't have that file open.
Then I get this message, telling me that the connection to the query was removed.
So my problem here is that 1) I can't edit the file anymore unless I make a copy because I keep getting the message saying that I already have the file opened and it is locked for editing and 2) I would like to refresh this query with VBA maybe once a week from now on but I can't because when I save the file the connection to the query is deleted by excel itself.
I'm not sure of why this is happening, I'm guessing it's because of the "Error" I get on the empty cells when I try to extract the values from the lists. If anybody has any information on how I can fix this so I don't get these error messages please let me know.
I was not able to reproduce your issue, however there are some things you might want to try.
Within ADX, you could wrap you query with a function, so you won't have to copy a large piece of code into your Excel.
You could deal with null values (this is what gives you the Error values) already in your query. Note the use of coalesce.
// I used datatable to mimic your query results
.create-or-alter function SessionAlarms()
{
datatable (SessionId:int,list_of_alarms_all:dynamic)
[
1, dynamic([10,20,30])
,2, dynamic([])
,3, dynamic(null)
]
| extend list_of_alarms_all = coalesce(list_of_alarms_all, dynamic([]))
}
You can use Power Query ADX connector and copy your query/function As Is
If you haven't dealt with null values in you KQL you can take care of the error in Excel by using Replace Errors
I want to be able to toggle a filter on my query via a parameter in dashboard. How can I turn the "where" operator off?
E.g. the parameter in the dashboard is "_toggle"
let _filter = dynamic(["A", "B"]);
Table
| where id in (_filter) // execute this line only if _toggle == true
| project id
I already tried creating a second list containing all the ids and toggle between the small an the complete list via iff() but this is too resource intensive.
you could try something like this:
let _filter = dynamic(["A", "B"]);
Table
| where _toggle == false or id in (_filter)
| project id
I am trying to write an App Insights query that will report back the timespan between two known events, specifically circuit breaker open and close events. The assumption is that these events always occur in pairs, so we need to know the time between the two for every occurrence in a time period.
My first attempt was to use an inline function. Simplified version below.
let timeOpened = (timeClosed:datetime)
{
let result = customEvents
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Open'
| where timestamp < timeClosed
| order by timestamp desc
| take 1
| project timestamp;
let scalar = toscalar(result);
scalar
};
customEvents
| where timestamp > ago(4h)
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Closed'
| extend timeOpen = timestamp - timeOpened(timestamp)
There may be a better way to do this. If so your ideas are welcome! But in this particular attempt the only feedback I get from Azure when running this is "Syntax error". However, I don't believe there's a syntax error here because if I just change the return value of the function from scalar to now() it runs successfully. Also I can run the body of the function in isolation successfully. Any idea what's wrong here?
I think you are getting syntax error because query language does not allow possibly recursive constructs. Now() worked because it was statically (not dynamically) retrieved at the query time.
I think you may achieve the desired outcome with serialize and prev() operators:
Table | order by timestamp asc | serialize
| extend previousTime = prev(timestamp,1)
| extend Diff = iff(customDimensions['State'] == 'Closed', timestamp - previousTime, 0)
| where Diff > 0
Note: I haven't tested the example above and it may need some additional thought to make it work (e.g. making sure that the previous record is actually "Opened" before doing previousTime calculation).
I'm working on a code that will have a list of items in a specific order and I'd like to reorder them at will. The setup isn't really that important, but to summarize it, it's a node server with MSSQL database.
For the sake of the demonstration lets say we're discussing forum categories that show in a specific order.
Id | OrderNumber | Name
------------------------
1 | 1 | Rules
2 | 3 | Off-topic
5 | 2 | General
8 | 4 | Global
I've already handled the front end that will allow me to reorder them as I like and the problem is what should happen when I press the save button on the database.
Ideally I'd like to send a JavaScript object containing item IDs in the right order to the API endpoint on the server that will execute a stored procedure. Something like:
Data = {
IDs:"5,2,8,1"
}
Is there a way that I can program a that stored procedure that it's only parameter is the list of Ids but that it can go through that list and do something I can only describe as the following pseudo code:
var Order = 1;
foreach ID in Data.IDs
UPDATE Categories SET OrderNum = Order WHERE Id = ID
Order = Order + 1
My biggest problem is that I'm not very experienced with advanced SQL commands, but that's the only part I need help with, I handled everything else already. Thank you for your help.
Example
Declare #IDs varchar(max) = '5,2,8,1'
Update A
set OrderNumber=B.RetSeq
From YourTable A
Join (
Select RetSeq = row_number() over (order by (select null))
,RetVal = B.n.value('(./text())[1]', 'int')
From ( values (cast('<x>' + replace(#IDs,',','</x><x>')+'</x>' as xml) )) A(xmldata)
Cross Apply xmldata.nodes('x') B(n)
) B on A.ID=B.RetVal
Updated Table
Id OrderNumber Name
1 4 Rules
2 2 Off-topic
5 1 General
8 3 Global