problem:
for each row in a table (from analytics table) I am trying to run a subquery to find the corresponding row in a second table (from externaldata). I think I want a subquery but there maybe a better option. there is no column linking each table so I cant use join, the only relationship is that the numbers from the analytics table may be between a start and end number in an externaldata table.
let IDlist = datatable(value:long)
[
45,
76,
150,
202,
2156,
3004,
5001,
];
// imported from externaldata
let idlookup = datatable(start:long, end:long, name:string)
[
1,100,"bob",
101,105,"susan",
200,1000,"henry",
5000,5004,"clair",
];
//using between doesnt work
idlookup
| where idlist between (start .. end)
| project idlist, name
expected output
45 - bob
76 - bob
150 - no match
202 - henry
2156 - no match
3004 - no match
5001 - clair
The query above doesn't work as expression before the between statement is not a scalar expression. I hoped a subquery would solve this but I cant get figure out the syntax.
Any help is much appreciated.
Here is one way:
let IDlist = datatable(value:long)
[
45,
76,
150,
202,
2156,
3004,
5001,
];
// imported from externaldata
let idlookup = datatable(start:long, end:long, name:string)
[
1, 100, "bob",
101, 105, "susan",
200, 1000, "henry",
5000, 5004, "clair",
];
IDlist
| extend dummy = 1
| lookup (idlookup | extend dummy = 1) on dummy
| where value >=start and value <= end
| join kind=rightouter (IDlist) on value
| extend name = iff(isempty( name), "no match", name)
| project name, value = value1
Related
I have a problem with my kusto query. this kusto query is running inside the alert call. I tried to send a notification to our client via email.
Scenario :
I am trying to send messages between 7 am and 13 am on Saturday. (only Saturday) But I am getting messages also Sunday. There is nothing here below the query. I think it is related to app insight alert.
requests
| extend Customer= trim_end('/', tostring(split(customDimensions.source, '//')[1]))
| extend alarmOK=iif(datetime_diff('minute', now(), timestamp) > 20, 1, 0)
| extend issaturday=iif(dayofweek(timestamp) == 6d, 1, 0)
| extend workinghour = hourofday(timestamp)
| extend
sendnotify1=iif(workinghour >= 7 and workinghour < 13, 1, 0),
sendnotify2=iif(hourofday(now()) >= 7 and hourofday(now()) < 13, 1, 0)
| extend alarmmessage = "alert message"
| where timestamp > ago(24h) and Customer == "mycustomer"
| where issaturday == 1
| where workinghour >= 7 and workinghour < 13
| top 1 by timestamp desc
All datetimes in Kusto are stored as UTC.
Use datetime_utc_to_local to get the timestamp in your local time zone, e.g.:
let timestamp = now();
print datetime_utc_to_local(timestamp, "Australia/Melbourne")
Fiddle
print_0
2023-01-10T21:10:08.0645922Z
P.S.
Your query could be simplified significantly.
KQL supports the Boolean data type (bool).
KQL supports datetime & timespan arithmetic.
Even if for some reason you wanted to add a column named issaturday and then filter by it, it could easily be done like this:
| extend issaturday = dayofweek(timestamp) == 6d | where issaturday
// Sample data generation. Not part of the solution.
let requests = materialize(
range i from 1 to 100 step 1
| extend timestamp = ago(7d * rand())
);
// Solution starts here
requests
| where dayofweek(timestamp) == 6d
and timestamp % 1d between (7h .. 13h)
and now() - timestamp > 20m
i
timestamp
5
2023-01-07T08:37:39.3449345Z
80
2023-01-07T09:07:36.4794478Z
83
2023-01-07T10:51:19.4051319Z
Fiddle
I have a table that consists of one row and number of columns. One of the columns is named EventProperties which is a JSON of properties of this format:
{
"Success":true,
"Counters":{
"Counter1":1,
"Counter2":-1,
"Counter3":5,
"Counter4":4,
}
}
I want to convert the Counters from this JSON to a two-column table of keys and values, where the first column is the name of the counter (e.g. Counter3) and the second column is the value of the counter (e.g. 5).
I've tried this:
let eventPropertiesCell = materialize(MyTable
| project EventProperties
);
let countersStr = extractjson("$.Counters", tostring(toscalar(eventPropertiesCell)), typeof(string));
let countersJson = parse_json(countersStr);
let result =
print mydynamicvalue = todynamic(countersJson)
| mvexpand mydynamicvalue
| evaluate bag_unpack(mydynamicvalue);
result
But I get a table with a column for each counter from the JSON, and number of rows that is equal to the number of counters, while only one random row is filled with the counter value. For example, with the JSON from the example above, I get:
But I want something like this:
Any help will be appreciated!
you could try using mv-apply as follows:
datatable(event_properties:dynamic)
[
dynamic({
"Success":true,
"Counters":{
"Counter1":1,
"Counter2":-1,
"Counter3":5,
"Counter4":4
}
}),
dynamic({
"Success":false,
"Counters":{
"Counter1":1,
"Counter2":2,
"Counter3":3,
"Counter4":4
}
})
]
| mv-apply event_properties.Counters on (
extend key = tostring(bag_keys(event_properties_Counters)[0])
| project key, value = event_properties_Counters[key]
)
| project-away event_properties
I'm racking my brain with this and would like some help. :)
I want to know how to use wildcard(*) for join union parameter.
I need to join two tables with the same names in the fields, however, some fields may come with the wildcard(*), since for this field I want all to be validated.
My exceptions table:
let table_excep= datatable (Computer:string,Event_id:string, logon_type:string)
[
"Pc_01","*","4",
"Pc_02","4648","*",
"*","*","60"
];
My data table:
let table_windows= datatable (Computer:string,Event_id:string, logon_type:string)
[
"Pc_01","5059","4",
"Pc_02","4648","1",
"Pc_03","61","60"
];
When running, it doesn't bring anything in the result.
For this union, I want the 3 union fields to be considered, ie based on the exceptions table, if computer_name is Pc_01 and logon_type is 4, no matter what event_id is, this log should be displayed, since the field of eventi_id in the exception list is wildcard(*).
I'm not finding a way to solve this problem since the join condition only allows "==" and "and".
cross join (inner join on 1=1) + where
let table_excep= datatable (Computer:string,Event_id:string, logon_type:string)
[
"Pc_01","*","4",
"Pc_02","4648","*",
"*","*","60"
];
let table_windows= datatable (Computer:string,Event_id:string, logon_type:string)
[
"Pc_01","5059","4",
"Pc_02","4648","1",
"Pc_03","61","60"
];
table_excep | extend dummy = 1
| join kind=inner (table_windows | extend dummy = 1) on dummy
| where (Computer == Computer1 or Computer == '*')
and (Event_id == Event_id1 or Event_id == '*')
and (logon_type == logon_type1 or logon_type == '*')
Computer
Event_id
logon_type
dummy
Computer1
Event_id1
logon_type1
dummy1
Pc_01
*
4
1
Pc_01
5059
4
1
Pc_02
4648
*
1
Pc_02
4648
1
1
*
*
60
1
Pc_03
61
60
1
Fiddle
I am trying to simply run a subquery in Azure application insights, using Kusto, so that I can get some information from two tables displayed as one.
The query I'm trying is
table1
| extend progressLog = toscalar(
table2
| where common_Id == table1.common_Id // errors saying Ensure that expression: table1.common_Id is indeed a simple name
| summarize makelist(stringColumn)
)
I have attempted to alias this id, and even join the two tables, as such:
requests
| extend aliased_id = common_Id
| join traces on operation_Id, $left.operation_Id == $right.operation_Id
| extend test_id = operation_Id
| extend progressLog = toscalar(
traces
| where operation_Id == aliased_id // Failed to resolve column or scalar expression named 'aliased_id'
| summarize makelist(message)
)
Failed to resolve column or scalar expression named 'aliased_id'.
I am simply trying to do the equivalent of the T-SQL query:
SELECT
... ,
STRING_AGG(table2.stringColumn, ',')
FROM
table1
INNER JOIN
table2
ON table1.common_Id = table2.common_Id
GROUP BY
table.<props>
My main question is - how do I reference "common_Id" in the kusto language inside a subquery
Please see if the next query provides what you're looking for. If not, please share sample input using datatable, as I did below, and expected output:
let requests = datatable(common_Id:string, operation_Id:string)
[
"A", "X",
"B", "Y",
"C", "Z"
];
let traces = datatable(operation_Id:string, message:string)
[
"X", "m1",
"X", "m2",
"Y", "m3"
];
let messagesByOperationId = traces | summarize makelist(message) by operation_Id;
requests
| join kind=leftouter messagesByOperationId on operation_Id
| project common_Id, operation_Id, progressLog = list_message
I'm trying to implement some kind of pagination feature for my app that using cassandra in the backend.
CREATE TABLE sample (
some_pk int,
some_id int,
name1 txt,
name2 text,
value text,
PRIMARY KEY (some_pk, some_id, name1, name2)
)
WITH CLUSTERING ORDER BY(some_id DESC)
I want to query 100 records, then store the last records keys in memory to use them later.
+---------+---------+-------+-------+-------+
| sample_pk| some_id | name1 | name2 | value |
+---------+---------+-------+-------+-------+
| 1 | 125 | x | '' | '' |
+---------+---------+-------+-------+-------+
| 1 | 124 | a | '' | '' |
+---------+---------+-------+-------+-------+
| 1 | 124 | b | '' | '' |
+---------+---------+-------+-------+-------+
| 1 | 123 | y | '' | '' |
+---------+---------+-------+-------+-------+
(for simplicity, i left some columns empty. partition key(sample_pk) is not important)
let's assume my page size is 2.
select * from sample where sample_pk=1 limit 2;
returns first 2 rows. now i store the last record in my query result and run query again to get next 2 rows;
this is the query that does not work because of restriction of a single non-EQ relation
select * from where sample_pk=1 and some_id <= 124 and name1>='a' and name2>='' limit 2;
and this one returns wrong results because some_id is in descending order and name columns are in ascending order.
select * from where sample_pk=1 and (some_id, name1, name2) <= (124, 'a', '') limit 2;
So I'm stuck. How can I implement pagination?
You can run your second query like,
select * from sample where some_pk =1 and some_id <= 124 limit x;
Now after fetching the records ignore the record(s) which you have already read (this can be done because you are storing the last record from the previous select query).
And after ignoring those records if you are end up with empty list of rows/records that means you have iterated over all the records else continue doing this for your pagination task.
You don't have to store any keys in memory, also you don't need to use limit in your cqlsh query. Just use the capabilities of datastax driver in your application code for doing pagination like the following code:
public Response getFromCassandra(Integer itemsPerPage, String pageIndex) {
Response response = new Response();
String query = "select * from sample where sample_pk=1";
Statement statement = new SimpleStatement(query).setFetchSize(itemsPerPage); // set the number of items we want per page (fetch size)
// imagine page '0' indicates the first page, so if pageIndex = '0' then there is no paging state
if (!pageIndex.equals("0")) {
statement.setPagingState(PagingState.fromString(pageIndex));
}
ResultSet rows = session.execute(statement); // execute the query
Integer numberOfRows = rows.getAvailableWithoutFetching(); // this should get only number of rows = fetchSize (itemsPerPage)
Iterator<Row> iterator = rows.iterator();
while (numberOfRows-- != 0) {
response.getRows.add(iterator.next());
}
PagingState pagingState = rows.getExecutionInfo().getPagingState();
if(pagingState != null) { // there is still remaining pages
response.setNextPageIndex(pagingState.toString());
}
return response;
}
note that if you make the while loop like the following:
while(iterator.hasNext()) {
response.getRows.add(iterator.next());
}
it will first fetch number of rows as equal as the fetch size we set, then as long as the query still matches some rows in Cassandra it will go fetch again from cassandra till it fetches all rows matching the query from cassandra which may not be intended if you want to implement a pagination feature
source: https://docs.datastax.com/en/developer/java-driver/3.2/manual/paging/