I have a problem with my kusto query. this kusto query is running inside the alert call. I tried to send a notification to our client via email.
Scenario :
I am trying to send messages between 7 am and 13 am on Saturday. (only Saturday) But I am getting messages also Sunday. There is nothing here below the query. I think it is related to app insight alert.
requests
| extend Customer= trim_end('/', tostring(split(customDimensions.source, '//')[1]))
| extend alarmOK=iif(datetime_diff('minute', now(), timestamp) > 20, 1, 0)
| extend issaturday=iif(dayofweek(timestamp) == 6d, 1, 0)
| extend workinghour = hourofday(timestamp)
| extend
sendnotify1=iif(workinghour >= 7 and workinghour < 13, 1, 0),
sendnotify2=iif(hourofday(now()) >= 7 and hourofday(now()) < 13, 1, 0)
| extend alarmmessage = "alert message"
| where timestamp > ago(24h) and Customer == "mycustomer"
| where issaturday == 1
| where workinghour >= 7 and workinghour < 13
| top 1 by timestamp desc
All datetimes in Kusto are stored as UTC.
Use datetime_utc_to_local to get the timestamp in your local time zone, e.g.:
let timestamp = now();
print datetime_utc_to_local(timestamp, "Australia/Melbourne")
Fiddle
print_0
2023-01-10T21:10:08.0645922Z
P.S.
Your query could be simplified significantly.
KQL supports the Boolean data type (bool).
KQL supports datetime & timespan arithmetic.
Even if for some reason you wanted to add a column named issaturday and then filter by it, it could easily be done like this:
| extend issaturday = dayofweek(timestamp) == 6d | where issaturday
// Sample data generation. Not part of the solution.
let requests = materialize(
range i from 1 to 100 step 1
| extend timestamp = ago(7d * rand())
);
// Solution starts here
requests
| where dayofweek(timestamp) == 6d
and timestamp % 1d between (7h .. 13h)
and now() - timestamp > 20m
i
timestamp
5
2023-01-07T08:37:39.3449345Z
80
2023-01-07T09:07:36.4794478Z
83
2023-01-07T10:51:19.4051319Z
Fiddle
Related
i have a Question about an KQL Query.
I would like to restrict the Query only of the Work Days from Monday to Friday.
How can i build this in KQL Query?
Thanks a lot.
A short Edit:
How can i build this in a existing Query with a Timestamp "timestamp between (datetime(06:00) .. datetime(15:00))" ??
For Example here the Code:
availabilityResults
| where timestamp between (datetime(06:00) .. datetime(15:00))
| where true
| where name == "warmup" and true
| extend percentage = toint(success) * 100
| render timechart
And now i would like only have an Result when is an Work Day
Best Regards,
Phil
dayofweek()
// Sample data generation. Not part of the solution.
let t = range Date from startofday(ago(28d)) to now() step 1d;
// Solution starts here.
t
| where dayofweek(Date)/1d between (1 .. 5)
Date
2022-10-05T00:00:00Z
2022-10-06T00:00:00Z
2022-10-07T00:00:00Z
2022-10-10T00:00:00Z
2022-10-11T00:00:00Z
2022-10-12T00:00:00Z
2022-10-13T00:00:00Z
2022-10-14T00:00:00Z
2022-10-17T00:00:00Z
2022-10-18T00:00:00Z
2022-10-19T00:00:00Z
2022-10-20T00:00:00Z
2022-10-21T00:00:00Z
2022-10-24T00:00:00Z
2022-10-25T00:00:00Z
2022-10-26T00:00:00Z
2022-10-27T00:00:00Z
2022-10-28T00:00:00Z
2022-10-31T00:00:00Z
2022-11-01T00:00:00Z
2022-11-02T00:00:00Z
Fiddle
Here The Solution of my Specific Answer:
Table
| where timestamp between (datetime(06:00) .. datetime(15:00))
| where dayofweek(timestamp) != time(6.00:00:00)
| where dayofweek(timestamp) != time(0.00:00:00)
| render timechart
This is my current query that i have in azure logs:
let numberOfBuckets = 24;
let interval = toscalar(requests | where url matches regex "courses.*"
| summarize interval = (max(timestamp)-min(timestamp)) / numberOfBuckets
| project floor(interval, 1m));
requests | where url matches regex "courses.*"
| summarize count() by url
It doesn't quite work and I've tried a lot of different ways to do this
like this...
let under400_course = requests | where url matches regex "/courses.*" | where duration < 400 | count;
let total_req_course = requests | where url matches regex "/courses.*" | count;
print under400_apt_SLI = toscalar(under400_course) * 100/toscalar(total_req_course);
just as a query to get information...
how do I actually get each response time for every connection in the last 24 hours for this endpoint?
how do I actually get each response time for every connection in the
last 24 hours for this endpoint?
I think the query is simpler for this request. Have you tried this?
requests
| where timestamp > ago(24h)
| where url matches regex "courses.*"
| project timestamp, url, resultCode, duration
The query timestamp > ago(24h) will filter all requests in the last 24 hours.
And the response time for request is already presented in requests table.
You can refer to Kusto guideline by MS here: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/
let fastResponseTimeMaxMs = 800;
let errorBudgetThresholdForFastResponseTime = 90.0;
//
let startTime = ago(7days);
let endTime = now();
let timeStep = 300m;
//
requests
| where timestamp > startTime and timestamp < endTime
| where success == 'True' | where url matches regex "<URL>.*"
| summarize TotalCount = count(), ActualCount = countif(duration <= fastResponseTimeMaxMs) by bin(timestamp, timeStep)
| extend Percentage = round(todecimal(ActualCount * 100) / todecimal(TotalCount), 2)
| extend ErrorBudgetMinPercent = errorBudgetThresholdForFastResponseTime
| extend InBudget = case(Percentage >= ErrorBudgetMinPercent, 1, 0)
This works... took a bit but I got it!
| where TimeGenerated > ago(30d)
only gives me the last 30 days logs and I'm searching for a query to get previous month logs from a table, so I can export it directly into Power BI.
Here is how you can do it below. I am showing two ways. The 'easy' way is to just hand jam the dates in for the month. The harder way requires you to use the make_datetime function.
// The Easy 'Manual' Way
AuditLogs
| where TimeGenerated >= datetime('2021-08-01') and TimeGenerated <= datetime('2021-08-31')
// Automated Way
let lastmonth = getmonth(datetime(now)) -1;
let year = getyear(datetime(now));
let monthEnd = endofmonth(datetime(now),-1);
AuditLogs
| where TimeGenerated >= make_datetime(year,lastmonth,01) and TimeGenerated <= monthEnd
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/make-datetimefunction
Just wanted to add on to #Ken W MSFT's great query, by suggesting this for the automation
let time_start = startofmonth(datetime(now), -1);
let time_end = endofmonth(datetime(now),-1);
AuditLogs
| where TimeGenerated between (time_start .. time_end)
This question already has answers here:
pyspark: count distinct over a window
(2 answers)
Closed 1 year ago.
Let's imagine we have the following dataframe :
port | flag | timestamp
---------------------------------------
20 | S | 2009-04-24T17:13:14+00:00
30 | R | 2009-04-24T17:14:14+00:00
32 | S | 2009-04-24T17:15:14+00:00
21 | R | 2009-04-24T17:16:14+00:00
54 | R | 2009-04-24T17:17:14+00:00
24 | R | 2009-04-24T17:18:14+00:00
I would like to calculate the number of distinct port, flag over the 3 hours in Pyspark.
The result will be something like :
port | flag | timestamp | distinct_port_flag_overs_3h
---------------------------------------
20 | S | 2009-04-24T17:13:14+00:00 | 1
30 | R | 2009-04-24T17:14:14+00:00 | 1
32 | S | 2009-04-24T17:15:14+00:00 | 2
21 | R | 2009-04-24T17:16:14+00:00 | 2
54 | R | 2009-04-24T17:17:14+00:00 | 2
24 | R | 2009-04-24T17:18:14+00:00 | 3
The SQL request looks like :
SELECT
COUNT(DISTINCT port) OVER my_window AS distinct_port_flag_overs_3h
FROM my_table
WINDOW my_window AS (
PARTITION BY flag
ORDER BY CAST(timestamp AS timestamp)
RANGE BETWEEN INTERVAL 3 HOUR PRECEDING AND CURRENT
)
I found this topic that solves the problem but only if we want to count distinct elements over one field.
Do someone has any idea of how to achieve that in :
python 3.7
pyspark 2.4.4
Just collect set of structs (port, flag) and get its size. Something like this:
w = Window.partitionBy("flag").orderBy("timestamp").rangeBetween(-10800, Window.currentRow)
df.withColumn("timestamp", to_timestamp("timestamp").cast("long"))\
.withColumn("distinct_port_flag_overs_3h", size(collect_set(struct("port", "flag")).over(w)))\
.orderBy(col("timestamp"))\
.show()
I've just code something like that that works to :
def hive_time(time:str)->int:
"""
Convert string time to number of seconds
time : str : must be in the following format, numberType
For exemple 1hour, 4day, 3month
"""
match = re.match(r"([0-9]+)([a-z]+)", time, re.I)
if match:
items = match.groups()
nb, kind = items[0], items[1]
try :
nb = int(nb)
except ValueError as e:
print(e, traceback.format_exc())
print("The format of {} which is your time aggregaation is not recognize. Please read the doc".format(time))
if kind == "second":
return nb
if kind == "minute":
return 60*nb
if kind == "hour":
return 3600*nb
if kind == "day":
return 24*3600*nb
assert False, "The format of {} which is your time aggregaation is not recognize. \
Please read the doc".format(time)
# Rolling window in spark
def distinct_count_over(data, window_size:str, out_column:str, *input_columns, time_column:str='timestamp'):
"""
data : pyspark dataframe
window_size : Size of the rolling window, check the doc for format information
out_column : name of the column where you want to stock the results
input_columns : the columns where you want to count distinct
time_column : the name of the columns where the timefield is stocked (must be in ISO8601)
return : a new dataframe whith the stocked result
"""
concatenated_columns = F.concat(*input_columns)
w = (Window.orderBy(F.col("timestampGMT").cast('long')).rangeBetween(-hive_time(window_size), 0))
return data \
.withColumn('timestampGMT', data.timestampGMT.cast(time_column)) \
.withColumn(out_column, F.size(F.collect_set(concatenated_columns).over(w)))
Works well, didn't check yet for performance monitoring.
I have such a query:
let start=datetime("2019-06-22T01:44:00.000");
let end=datetime("2019-06-22T07:44:00.000");
let timeGrain=5m;
let dataset1= requests
| where timestamp > start and timestamp < end ;
dataset1
| summarize Gesamt=sum(itemCount) , Durchschnittsdauer=round(avg(duration /1000),2), Instanz=dcount(cloud_RoleInstance) by Funktionsname=name
| join kind= inner
(
exceptions
| where timestamp > start and timestamp < end
| summarize Fehler=count() by Funktionsname=operation_Name
) on Funktionsname
| project Funktionsname ,Gesamt , Erfolgreich=Gesamt - Fehler, Fehler, Durchschnittsdauer
If I test it in Application insight query manager, I get data. But after I pin it to the share dashboard, and changing the Time (local and UTC) the dashboard shows me no results. Do you know how can I solve this problem?
I got it
I should change starttime and endtime to:
let start=datetime("2019-06-24 13:44:00.000Z");
let end=datetime("2019-06-24 19:44:00.000Z");