Hi Am working on logstash, and i have come to an scenario where i need to select final values from logs and showcase in Kibana.
For eg:
uniq_reference 01 : Status = "Processing" uniq_reference 01 : Status = "Completed" uniq_reference 02 : Status = "Processing"
now in Kibana if i use data filter visual am getting count as Processing - 2 and completed - 1
whereas the actual output should be Processing - 1 and completed - 1 earlier value of uniq_reference 01 should be ignored
Related
Hi Kusto Query Language(KQL) lovers,
I am trying to write a query in Kusto Query Language (KQL), that can compare count of APIs that failed today in a specific time (lets say 2:30 p.m. to 3 p.m.) with respect to count of APIs that failed yesterday in the same timeframe (2:30 p.m. to 3 p.m.).
For instance, if today, in last 30 min operation X was failed 10 times with failure code 400, I need to see count with which operation X failed today in last 30 minutes (Same time frame).
For this purpose, I used Scalar function bin() and wrote following query that extracts data from request table:
requests
|where timestamp > ago(1d)
| where client_Type != "Browser"
| where (cloud_RoleName == 'X')
| where name != 'HTTP GET'
| where success == false
| summarize count() by bin (timestamp, 1d), name, resultCode
|sort by timestamp
Here is the output I got when using timestamp > ago(1d). This way, I was shown APIs that failed today and yesterday but there is no clear comparison between both dates.
Is there any way I can display count of APIs that failed yesterday on separate Column adjacent to the count_ Column that has count of corresponding APIs that failed today?
I know of project operator that adds extra column but I don't know how to incorporate and assign count of APIs that failed yesterday to project operator.
Kindly add to my knowledge of any relevant function or operator in KQL that can achieve this task.
The other way I tried was to define two variables, startDateTime and endDateTime to get the data of specific time as shown below.
Blank Output when I defined variables to define selected time frame:
let startDateTime = todatetime("2023-02-07 06:35:00.0");
let endDateTime = todatetime("2023-02-07 06:35:00.0");
requests
|where timestamp > startDateTime and timestamp < endDateTime
| where client_Type != "Browser"
| where (cloud_RoleName == 'web-memberappservice-weu-prod')
| where name != 'HTTP GET'
| where success == false
| summarize count() by bin (timestamp, 1d), name, resultCode
|sort by timestamp
I searched about KQL query to compare count of failed APIs for today with respect to count of APIs that failed yesterday and checked some results from Stack overflow which are not helping me in achieving desired result.
I tried these links but queries on these links do not reflect what I want to achieve:
Best way to show today Vs yesterday Vs week in KQL azure monitror
kql query to compare the hour which has minimum number of TriggersStarted from last week to today past hour TriggersStarted
What am I expecting?
I want a query that can display count of APIs that failed yesterday on separate Column adjacent to the count_ Column that has count of corresponding APIs that failed today.
I know of project operator that adds extra column but I don't know how to incorporate and assign count of APIs that failed yesterday to project operator.
Kindly identify any relevant function or operation that can help in this regard.
* The where clause was added for performance reasons.
// Sample data generation. Not part of the solution.
let requests = materialize(range i from 1 to 100000 step 1 | extend timestamp = ago(2d * rand()), name = tostring(dynamic(["PUT", "POST", "PATCH", "GET"])[toint(rand(4))]), resultCode = 400 + toint(rand(3)));
// Solution starts here.
let _period = 30m;
requests
| where timestamp between (ago(_period) .. _period)
or timestamp between (ago(_period + 1d) .. _period)
| summarize todayCount = countif(timestamp between (ago(_period) .. _period))
,YesterdayCount = countif(timestamp between (ago(_period + 1d) .. _period))
by name, resultCode
|sort by name asc, resultCode asc
name
resultCode
todayCount
YesterdayCount
GET
400
91
100
GET
401
98
98
GET
402
109
89
PATCH
400
93
77
PATCH
401
84
85
PATCH
402
74
82
POST
400
78
85
POST
401
96
77
POST
402
85
102
PUT
400
98
81
PUT
401
97
85
PUT
402
77
83
Fiddle
So I'm creating an app to go through one table to collect and change some data, then move the updated information to the new table. The issue I'm running into is the types of columns. I ran into copying a timestamp that has Infinity instead of timestamp. When I try to enter the data in the file, I get the following message:
column "infinity" does not exist
Any ideas on what I should be using here?
Query: insert into schema.table (id,time,processed,in,out) values ('828h29s2838','Tue, 01 Feb 2020 08:58:22 GMT',false,'Tue, 01 Feb 2020 08:58:22 GMT',infinity);
data in table:
828h29s2838 | 2020-02-01 00:58:22.418 | false | 2020-02-0 00:58:22.455 | infinity
Oh, I got it now. I should have placed apostrophes around infinity in my query.
corrected query: insert into schema.table (id,time,processed,in,out) values ('828h29s2838','Tue, 01 Feb 2020 08:58:22 GMT',false,'Tue, 01 Feb 2020 08:58:22 GMT','infinity');
I have a DataFrame, df, that has a range of values such as:
ID
Code
01
AB001
02
AB002
02
BC123
02
CD576
03
AB444
03
CD332
04
BC434
04
CD894
I want to remove all duplicates in the ID column and keep the row that has a certain value in Code. Let's suppose that if the Code that starts with BC is available, I want to keep that row. Otherwise, I want to take the first row with the ID. My desired output would look like:
ID
Code
01
AB001
02
BC123
03
AB444
04
BC434
I want to do something like:
# 'x' denotes a list of rows per unique ID
def keep_row(x):
# determine if 'BC' is even an available Code
if any([True for row in x if row['Code'].startswith('BC') else False]):
return the row that has Code that starts with BC
else:
# return the first row with the unique ID if there is no Code that begins with BC
return x[0]
df.groupby('ID', group_keys=False).apply(lambda x: keep_row(x))
I'd appreciate any help - thanks.
You can sort your dataframe by ID and boolean value (False when Code starts with "BC"), then .groupby() and take first item:
df["tmp"] = ~df.Code.str.startswith("BC")
df = df.sort_values(by=["ID", "tmp"])
print(df.groupby("ID", as_index=False)["Code"].first())
Prints:
ID Code
0 1 AB001
1 2 BC123
2 3 AB444
3 4 BC434
I have query data that should be inserted in the structure. While looping over data each value should be assigned to matching column. Here is example of query data:
REC_ID NAME STATUS TYPE YEAR
1 01 PARIS Approved 1 2019
2 06 AUSTIN Inactive 3 2017
3 48 LONDON Approved 2 2018
4 43 ROME Inactive 5 2019
5 61 DUBLIN Inactive 4 2019
This data should be organized to look like this:
1
REC_ID 01
NAME PARIS
STATUS Approved
TYPE 1
YEAR 2019
2
REC_ID 06
NAME AUSTIN
STATUS Inactive
TYPE 3
YEAR 2017
3
REC_ID 48
NAME LONDON
STATUS Approved
TYPE 2
YEAR 2018
4
REC_ID 43
NAME ROME
STATUS Inactive
TYPE 5
YEAR 2019
5
REC_ID 61
NAME DUBLIN
STATUS Inactive
TYPE 4
YEAR 2019
I tried to get desired data format with this function:
function formatData(qryData) {
local.fnResult = structNew();
local.fnData = structNew();
if(qryData.recordcount){
for(row in qryData) {
for(column in qryData.columnList) {
local.strRec = structNew();
structInsert(strRec, column, row[column]);
local.fnData[qryData.currentrow] = strRec;
}
}
}
writeDump(fnData);
}
Here is how my result looks:
1
REC_ID 01
2
REC_ID 06
3
REC_ID 48
4
REC_ID 43
5
REC_ID 61
I use ColdFusion 11 cfscript syntax. If anyone can help me solve this problem please let me know. Thank you.
There is an issue with your column list loop, you are creating a new structure for each column and that get reset in the next loop.
This loop should be changed
for(column in qryData.columnList) {
local.strRec = structNew();
structInsert(strRec, column, row[column]);
local.fnData[qryData.currentrow] = strRec;
}
to look like
local.strRec = structNew();
for(column in qryData.columnList) {
structInsert(strRec, column, row[column]);
}
local.fnData[qryData.currentrow] = strRec;
#espresso_coffee, Here I've go through your problem. I hope you need to create a structure with key as row count and each key value should be in a structure format. Here I've provide some sample code for with my query. I hope it's useful to you.
<cfquery name="readStudent" datasource="student">
select * from user
</cfquery>
In my user table having 6 records with firstname, lastName & id. Here I've loop over the query and convert it in to structure key values. I've used script syntax because your code having script style. :)
<cfscript>
myStr = {};
for (row in readStudent) {
structInsert(myStr, #readStudent.currentrow#, row);
}
writeDump(myStr);
</cfscript>
Here I've create my first structure name as myStr, and loop over query data and insert a structure value for key with current row that is readStudent.currentRow and value is in row . The row having firstname,lastname & id as structure key & it's value.
FYR : I've attached my sample query and converted to structure value. Correct me if I'm wrong understood your problem
I hope it's helpful to you. Thanks.
If I understand this correctly, it looks like you could just
array function formatData(qryData) {
return DeserializeJSON(SerializeJSON(arguments.qryData, "struct"));
}
Runnable example on trycf.com
Result:
Alternative
array function formatData(qryData) {
var result = [];
for (var row in qryData) {
result.append(row);
}
return result;
}
I am trying to create a Power View report tiled by KPIs, is this possible?
An example of my raw data:
Company ID Employee ID Measure numerator denominator
1 01 1 2 5
2 04 1 3 6
3 02 1 0 5
4 03 1 1 2
1 01 2 4 4
2 04 2 2 3
2 06 2 0 6
4 01 2 1 4
I have created a calculated column in Power Pivot using the following DAX function:
RATE:=[numerator]/[denominator]
From this, I want to create KPIs for each measure (each measure has different targets), and use these KPIs as tiles in a Power View report filtered by Company ID and/or Employee ID.
Can this be done?
You cannot tile by KPI. You KPI has to be a measure and you cannot tile by a measure. You can tile by a calculated column or by any original field in your dataset. So you can drop RATE into a TILE BY box.
To use the Measure column or a KPI type icon as the Tile field the best approach might be to use an image for your measure column. See more on including images in Power View here: https://support.office.com/en-ca/article/Images-in-Power-View-84e91b90-b1da-4913-83d6-beada5591fce?ui=en-US&rs=en-CA&ad=CA