I am trying to create a sentinel query using KQL which would only search for information on certain dates such as bank holidays. I have seen this can be done on other query languages however best I have created so far is having multiple dates in a row such as
let startdate = (datetime(23/09/2022));
let enddate = (datetime(26/09/2022));
where timegenerated between (startdate .. enddate)
However this does not allow multiple different date slots. I have tried below creating an array of dates however '!contains' is invalid with timegenerated field. Does anyone know a fix for this?
let HolidayDates = datatable(HDates: string)
[
'24/12/2022',
'25/12/2022',
//first holiday
'01/01/2023',
'02/01/2023',
'03/01/2023',
//New years
'07/04/2023',
'08/04/2023',
'09/04/2023',
'10/04/2023'
//easter holiday
];
| where TimeGenerated !contain (HolidayDates)
You can do this like this:
let HolidayDateTimes = datatable(HDates: datetime )
[
datetime(2022-12-24),
datetime(2022-09-27),
datetime(2022-12-25)];
traces | where startofday(timestamp) in (HolidayDateTimes)
Intead of using contains, which is a string operator, use the in operation to check whether an item is in a collection. Then I use startofday to strip away the time information.
Related
I'm having problems with writing a simple KQL query in Log Analytics Workspace. Basically I want to filter out all Application Registrations that have secret expiry date longer than 3 years.
I have made a following query, but I think my custom made datetime is still considered a string, instead of datetime type.
Here is the query:
//create a new date which is 3 years later than today
let newyear = format_datetime(datetime_add('year',3,datetime(now)), 'M/d/y, h:m:s.fff tt');
//Query the AppRegs for SecretEndDate ending at least in 2025
AppRegExp_CL
|project AppRegName_s, AppID_g, SecretEndDate_t
|where SecretEndDate_t >= datetime(newyear)
I'm always getting the same error:
Query could not be parsed at 'datetime(newyear)'on line [6,0]
I've tried using makedatetime(), to_datetime() but the result was the same.
Normally I'm not working on KQL queries, so maybe I'm missing something very obvious. I hope that you can help me out. Thanks
Since SecretEndDate_t is of datetime type, no conversion is needed.
//create a new date which is 3 years later than today
let newyear = datetime_add('year', 3, now());
//Query the AppRegs for SecretEndDate ending at least in 2025
AppRegExp_CL
|project AppRegName_s, AppID_g, SecretEndDate_t
|where SecretEndDate_t >= newyear
I want to find events that are a on now and upcoming (next 30 days) but that are also not in the past.
When i run this as a cloud function, I get "Cannot have inequality filters on multiple properties". How am I meant to get this type of data.
(ignore the fact that the date stuff is a bit messy, am still playing around).
// Create date 30 days in future
const searchData: Date = new Date();
searchData.setDate(searchData.getDate() + 30);
// Load data and handle empty respoonse
const response: admin.firestore.QuerySnapshot = await admin
.firestore()
.collection(Collections.EVENTS)
.where("startDate", "<=", admin.firestore.Timestamp.fromMillis(searchData.valueOf()))
.where("endDate", ">=", admin.firestore.Timestamp.fromMillis(new Date().valueOf()))
.where("public", "==", true)
.limit(NUMBER_OF_EVENTS)
.get();
Edit:
I would like to know the data structure/query method that will allow me to return all events in the events collection that are on now or that will start in the next month. Additionally, I would like the query to exclude events that have already finished. Each document (event) has a startDate and endDate timestamp on it.
Since you have two fields to check with ranges, I'm not sure this is doable with a single query. What you can do instead is perform two queries, merge the results on the client, and perform a final filter to get the exact matches.
Make an assumption about the maximum duration for an event. Call that amount of time "X".
Query for all documents where startDate is greater than now - X, and also less than now + 30 days. Call this result set "A".
Query for all documents where endDate is greater than now, and also less than now + 30 days. Call this result set "B".
On the client, iterate all the results from A and B, checking to see if the start and end dates fit the criteria you want.
I can't think of a way to structure your data that will do this with a single query.
I know this is kind a old thread, but the answer might be good for others.
What you can do as I at the end ended up doing, is that you have a $start_date and a $target_date.
You then do like this:
<?php
$start = strtotime('2021-11-22');
$target = strtotime('2022-01-01 0:00:00');
$limit = (($target - $start) / 86400);
$query = $col_data->where('date.day_start', '>=', $start);
$query = $query->limit($limit);
?>
Not bad, eh? You welcome!
From the docs:
You can only perform range comparisons (<, <=, >, >=) on a single field, and you can include at most one array-contains or array-contains-any clause in a compound query:
citiesRef.where("state", ">=", "CA").where("state", "<=", "IN");
citiesRef.where("state", "==", "CA").where("population", ">", 1000000);
https://firebase.google.com/docs/firestore/query-data/queries
I am trying to write an App Insights query that will report back the timespan between two known events, specifically circuit breaker open and close events. The assumption is that these events always occur in pairs, so we need to know the time between the two for every occurrence in a time period.
My first attempt was to use an inline function. Simplified version below.
let timeOpened = (timeClosed:datetime)
{
let result = customEvents
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Open'
| where timestamp < timeClosed
| order by timestamp desc
| take 1
| project timestamp;
let scalar = toscalar(result);
scalar
};
customEvents
| where timestamp > ago(4h)
| where name == 'CircuitBreakerStatusChange'
| where customDimensions['State'] == 'Closed'
| extend timeOpen = timestamp - timeOpened(timestamp)
There may be a better way to do this. If so your ideas are welcome! But in this particular attempt the only feedback I get from Azure when running this is "Syntax error". However, I don't believe there's a syntax error here because if I just change the return value of the function from scalar to now() it runs successfully. Also I can run the body of the function in isolation successfully. Any idea what's wrong here?
I think you are getting syntax error because query language does not allow possibly recursive constructs. Now() worked because it was statically (not dynamically) retrieved at the query time.
I think you may achieve the desired outcome with serialize and prev() operators:
Table | order by timestamp asc | serialize
| extend previousTime = prev(timestamp,1)
| extend Diff = iff(customDimensions['State'] == 'Closed', timestamp - previousTime, 0)
| where Diff > 0
Note: I haven't tested the example above and it may need some additional thought to make it work (e.g. making sure that the previous record is actually "Opened" before doing previousTime calculation).
I have document contains properties like this
{
"id":"1bd13f8f-b56a-48cb-9b49-7fc4d88beeac",
"name":"Sam",
"createdOnDateTime": "2018-07-23T12:47:42.6407069Z"
}
I want to query a document on basis of createdOnDateTime which is stored as string.
query e.g. -
SELECT * FROM c where c.createdOnDateTime>='2018-07-23' AND c.createdOnDateTime<='2018-07-23'
This will return all documents which are created on that day.
I am providing date value from date selector which gives only date without time so, it gives me problem while comparing date.
Is there any way to remove time from createdOnDateTime property or is there any other way to achieve this?
CosmosDB clients are storing timestamps in ISO8601 format and one of the good reasons to do so is that its lexicographical order matches the flow of time. Meaning - you can sort and compare those strings and get them ordered by time they represent.
So in this case you don't need to remove time components just modify the passed in parameters to get the result you need. If you want all entries from entire date of 2018-07-23 then you can use query:
SELECT * FROM c
WHERE c.createdOnDateTime >= '2018-07-23'
AND c.createdOnDateTime < '2018-07-24'
Please note that this query can use a RANGE index on createdOnDateTime.
Please use User Defined Function to implement your requirement, no need to update createdOnDateTime property.
UDF：
function con(date){
var myDate = new Date(date);
var month = myDate.getMonth()+1;
if(month<10){
month = "0"+month;
}
return myDate.getFullYear()+"-"+month+"-"+myDate.getDate();
}
SQL：
SELECT c.id,c.createdOnDateTime FROM c where udf.con(c.createdOnDateTime)>='2018-07-23' AND udf.con(c.createdOnDateTime)<='2018-07-23'
Output :
Hope it helps you.
I am trying to return search results for every document that was created on a day. Below is the query I use.
var query = Document.find({}).populate('contacts');
var gte = moment(req.query.date, 'DD-MM-YYYY').startOf('Day');
var lte = moment(req.query.date, 'DD-MM-YYYY').endOf('Day');
query.where('dates.createdAt').gte(gte).lt(lte);
This query works for some days but not all. I can't seem to understand the behaviour. Please help out.
The format of the date in the query string is DD/MM/YYYY.
Works for : 2016-04-16T00:02:30.065Z
Does not work for: 2016-04-15T19:02:59.758Z
It's wrong because you don't initialize as .utc(), and MongoDB dates are stored in UTC:
var gte = moment.utc(req.query.date, 'DD-MM-YYYY');
var lte = moment.utc(req.query.date, 'DD-MM-YYYY').endOf('Day');
And there is no need for the startOf() either.
If you don't construct like that, then the resulting Date object is skewed by the difference in the local timezone. Hence why you don't see the selection working where the hours would cross over dates.
Also if dates are coming in as 01/01/2016 then the format string would be 'DD/MM/YYYY', but one or the other is likely a typo in your question.