I need some help with a SQL query in Cosmos. I have to find the average time it takes a message to complete during a load test from the events stored in our DB.
So far I can get the start and end times like this:
SELECT
(SELECT c.TimestampUTC WHERE c.Event = 'message-accepted') as StartTime,
(SELECT c.TimestampUTC WHERE c.Event = 'message-completed') as EndTime
FROM c
WHERE c.TrackingId = 'LoadTest' AND (c.Event = 'message-accepted' OR c.Event = 'message-completed')
But I get an error when I try to get the DateTimeDiff like this:
SELECT
(SELECT c.TimestampUTC WHERE c.Event = 'message-accepted') as StartTime,
(SELECT c.TimestampUTC WHERE c.Event = 'message-completed') as EndTime,
DateTimeDiff("second", StartTime, EndTime) as TotalTime
FROM c
WHERE c.TrackingId = 'LoadTest' AND (c.Event = 'message-accepted' OR c.Event = 'message-completed')
I am stuck here because I need the difference to use the AVG function. Any help would be appreciated.
EDIT
Here is a sample of the data stored in Cosmos
{
"PartitionKey": "LoadTest",
"RowKey": "4ee9709f-c826-4a88-9d6f-240ba439eb1d",
"TrackingId": "LoadTest",
"Event": "message-accepted",
"TimestampUTC": "2022-09-14T19:12:18.8358914Z"
}
And this is the error I am getting when trying DateTimeDiff:
"Failed to query item for container enginelog:
One of the input values is invalid."
It is not giving much info which is why I am looking for help, I am following the format for the function in the documentation here
As per sample data shared in question, the assumption is that there are two different documents with Event property value "message-accepted" and "message-completed". As David mentioned, functions can't be used on data/properties available across separate document until unless property names are same.
In order to achieve what is required, you may need to write client-side code to recursively fetch the properties value from separate documents. Please refer the link
Related
I know that this is a really long post but I'm not sure of what part of my process is making my file crash, so I tried to detail everything about what I did to get to the error messages.
So, first of all, I created a query on Kusto, which looks something similar to this but in reality is 160 lines of code, this is just a summarized version of what my code might do just to show my working process.
First, what I do in Session_Id_List is create a list of all distinct Session Id's from the past day.
Then on treatment_alarms1 I count the amount of alarms for each type of alarm that was active during each session.
Then, on treatment_alarms2 I create a list which might look something like this
1x Alarm_Type_Number1
30x Alarm_Type_Number2
7x Alarm_Type_Number3
and like that for each treatment, so I have a list of all alarms that were active for that treatment.
Lastly, I create a left outer join with Session_Id_List and treatment_alarms2. This means that I will get shown all of the treatment ID's, even the ones that did not have any active alarms.
let _StartTime = ago(1d);
let _EndTime = ago(0d);
let Session_Id_List = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime
| summarize by SessionId, SerialNumber, StartTime
| distinct SessionId, StartTime, SerialNumber;
let treatment_alarms1 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| summarize number_alarms = count() by TranslatedData_Value, SessionId
| project final_Value = strcat(number_alarms, "x ", TranslatedData_Value), SessionId;
let treatment_alarms2 = Database1
| where StartTime >= _StartTime and StartTime <= _EndTime and TranslatedData_Status == "ALARM_ACTIVE"
| join kind=inner treatment_alarms1 on SessionId
| summarize list_of_alarms_all = make_set(final_Value) by SessionId
| project SessionId, list_of_alarms_all;
let final_join = Session_Id_List
| join kind=leftouter treatment_alarms2 on SessionId;
final_join
| project SessionId, list_of_alarms_all
Then I put this query into Excel, by using the following method
I go to Tools -> Query to Power BI on Kusto Explorer
I go to Data -> Get Data -> From Other Sources -> Blank Query
I go to advanced editor
I copy and paste my query and press "Done" at the bottom
If you see now, the preview of my data will show "List" on the list_of_alarms_all column, rather than showing me the actual values of the list.
To fix this issue I first press the arrows on the header of the column
I press on "Extract Values"
I select Custom -> Concatenate using special characters -> Line Feed -> Press OK
That works fine for all of the ID's that do have alarms on them, it shows them as a list and tells me how many there are, the issue is with the ID's that did not have any treatments where I get "Error" on the Excel preview. Once I press "Close & Load" the data is put on the worksheet and it looks fine, the "Error" are all gone and instead I get empty cells where the "Error" would be at.
The problem now starts when I close the file and try to open it again.
First I get this message. So I press yes to try and enter the file.
Then I get this other message. The problem with this message is that it says that I have the file open when that is not true. I even tried to restart my laptop and open the file again and I would still get the message when in reality I don't have that file open.
Then I get this message, telling me that the connection to the query was removed.
So my problem here is that 1) I can't edit the file anymore unless I make a copy because I keep getting the message saying that I already have the file opened and it is locked for editing and 2) I would like to refresh this query with VBA maybe once a week from now on but I can't because when I save the file the connection to the query is deleted by excel itself.
I'm not sure of why this is happening, I'm guessing it's because of the "Error" I get on the empty cells when I try to extract the values from the lists. If anybody has any information on how I can fix this so I don't get these error messages please let me know.
I was not able to reproduce your issue, however there are some things you might want to try.
Within ADX, you could wrap you query with a function, so you won't have to copy a large piece of code into your Excel.
You could deal with null values (this is what gives you the Error values) already in your query. Note the use of coalesce.
// I used datatable to mimic your query results
.create-or-alter function SessionAlarms()
{
datatable (SessionId:int,list_of_alarms_all:dynamic)
[
1, dynamic([10,20,30])
,2, dynamic([])
,3, dynamic(null)
]
| extend list_of_alarms_all = coalesce(list_of_alarms_all, dynamic([]))
}
You can use Power Query ADX connector and copy your query/function As Is
If you haven't dealt with null values in you KQL you can take care of the error in Excel by using Replace Errors
This is my first try at using the Power Query... I've build a "dynamic" query in which I can change the retrieved fields as well as the filtering fields and values to be used by the query.
It's working perfectly on my computer but as soon as I try to execute it on another computer, I get the "Please rebuild this data combination" error. I saw some post saying I'll have to kind of split my query but I have not been able to figure it out.
Here is what my 2 tables look like:
Condition and fields selection
and here is my Query with the error:
Query
This might not be very elegant, but it allow me, thru a VBA script, to generate the list of fields to be retrieved and to generate the condition to be used by the SQL.
Any idea why it's not working on the other computers or how to improved the solution I'm using?
Thank you!
Notes:
Hi, all my Privacy Level are already set to 'None'.
I've tried to parametrize my code but I can't figure how. The Where condition is dynamic: it could be Where Number = "1234" but in other condition, the where might be like: 'Where Assignee = "xyz"'.
Here is a simplified example of my code:
let
Source = Sql.Database("xxxx", "yyyy", [Query=
"Select network, testid
from CM3T1M1 "
& paramConditions[Conditions]{0} &
" "])
in
Source
rebuild query, Formula.Firewall
That's a feature to prevent prevent accidentally leaking data. You can change the privacy level to ignore it
See also: docs.microsoft/dataprivacyfirewall
Is the dynamic query inserting those cells into the SQL query ? Report Parameters are nice for letting the user change variables without having to re-edit the query.
Parameterized native SQL queries
from: https://blog.crossjoin.co.uk/2016/12/11/passing-parameters-to-sql-queries-with-value-nativequery-in-power-query-and-power-bi/
let
Source = Sql.Database("localhost", "Adventure Works DW"),
Test = Value.NativeQuery(
Source,
"SELECT * FROM DimDate
WHERE EnglishMonthName=#MonthName AND
EnglishDayNameOfWeek=#DayName",
[
MonthName = "March",
DayName = "Tuesday"
]
)
in
Test
Dynamic Power Query version of SQL Query
To dynamically generate this SQL Query
select NUMBER, REQUESTED_BY from SourceTable
where NUMBER = 404115
Table.SelectRows is your Where.
SelectColumns is your select
let
Source = ...,
filterByNum = 404115,
columnNames = {"NUMBER", "REQUESTED_BY"},
removedColumns = Table.SelectColumns(
Source, columnNames, MissingField.Error
),
// I used 'MissingField.Error' so you know right away
// if there's a typo or bug
// assuming you are comparing Source[NUMBER]
filteredTable = Table.SelectRows(
Source, each [NUMBER] = filterByNum
)
in
filteredTable
I want to find events that are a on now and upcoming (next 30 days) but that are also not in the past.
When i run this as a cloud function, I get "Cannot have inequality filters on multiple properties". How am I meant to get this type of data.
(ignore the fact that the date stuff is a bit messy, am still playing around).
// Create date 30 days in future
const searchData: Date = new Date();
searchData.setDate(searchData.getDate() + 30);
// Load data and handle empty respoonse
const response: admin.firestore.QuerySnapshot = await admin
.firestore()
.collection(Collections.EVENTS)
.where("startDate", "<=", admin.firestore.Timestamp.fromMillis(searchData.valueOf()))
.where("endDate", ">=", admin.firestore.Timestamp.fromMillis(new Date().valueOf()))
.where("public", "==", true)
.limit(NUMBER_OF_EVENTS)
.get();
Edit:
I would like to know the data structure/query method that will allow me to return all events in the events collection that are on now or that will start in the next month. Additionally, I would like the query to exclude events that have already finished. Each document (event) has a startDate and endDate timestamp on it.
Since you have two fields to check with ranges, I'm not sure this is doable with a single query. What you can do instead is perform two queries, merge the results on the client, and perform a final filter to get the exact matches.
Make an assumption about the maximum duration for an event. Call that amount of time "X".
Query for all documents where startDate is greater than now - X, and also less than now + 30 days. Call this result set "A".
Query for all documents where endDate is greater than now, and also less than now + 30 days. Call this result set "B".
On the client, iterate all the results from A and B, checking to see if the start and end dates fit the criteria you want.
I can't think of a way to structure your data that will do this with a single query.
I know this is kind a old thread, but the answer might be good for others.
What you can do as I at the end ended up doing, is that you have a $start_date and a $target_date.
You then do like this:
<?php
$start = strtotime('2021-11-22');
$target = strtotime('2022-01-01 0:00:00');
$limit = (($target - $start) / 86400);
$query = $col_data->where('date.day_start', '>=', $start);
$query = $query->limit($limit);
?>
Not bad, eh? You welcome!
From the docs:
You can only perform range comparisons (<, <=, >, >=) on a single field, and you can include at most one array-contains or array-contains-any clause in a compound query:
citiesRef.where("state", ">=", "CA").where("state", "<=", "IN");
citiesRef.where("state", "==", "CA").where("population", ">", 1000000);
https://firebase.google.com/docs/firestore/query-data/queries
I have document contains properties like this
{
"id":"1bd13f8f-b56a-48cb-9b49-7fc4d88beeac",
"name":"Sam",
"createdOnDateTime": "2018-07-23T12:47:42.6407069Z"
}
I want to query a document on basis of createdOnDateTime which is stored as string.
query e.g. -
SELECT * FROM c where c.createdOnDateTime>='2018-07-23' AND c.createdOnDateTime<='2018-07-23'
This will return all documents which are created on that day.
I am providing date value from date selector which gives only date without time so, it gives me problem while comparing date.
Is there any way to remove time from createdOnDateTime property or is there any other way to achieve this?
CosmosDB clients are storing timestamps in ISO8601 format and one of the good reasons to do so is that its lexicographical order matches the flow of time. Meaning - you can sort and compare those strings and get them ordered by time they represent.
So in this case you don't need to remove time components just modify the passed in parameters to get the result you need. If you want all entries from entire date of 2018-07-23 then you can use query:
SELECT * FROM c
WHERE c.createdOnDateTime >= '2018-07-23'
AND c.createdOnDateTime < '2018-07-24'
Please note that this query can use a RANGE index on createdOnDateTime.
Please use User Defined Function to implement your requirement, no need to update createdOnDateTime property.
UDF：
function con(date){
var myDate = new Date(date);
var month = myDate.getMonth()+1;
if(month<10){
month = "0"+month;
}
return myDate.getFullYear()+"-"+month+"-"+myDate.getDate();
}
SQL：
SELECT c.id,c.createdOnDateTime FROM c where udf.con(c.createdOnDateTime)>='2018-07-23' AND udf.con(c.createdOnDateTime)<='2018-07-23'
Output :
Hope it helps you.
I'm relatively new to Apex, but I have some questions about a batch job that I am creating. I want to make a query with a subquery (please see the code). Every Portal_c can have more than 200 Exporte_r.
global Database.QueryLocator start(Database.BatchableContext BC) {
String query = 'SELECT Id, Name, (SELECT Id FROM Exporte__r) FROM Portal__c';
return Database.getQueryLocator(query);
}
global void execute(Database.BatchableContext BC, List<Portal__c> scope) {
for (Portal__c portal : scope) {
// doesn't work -> First error: Aggregate query has too many rows for direct assignment, use FOR loop
// when using FOR loop -> System.QueryException: invalid query locator
//List<Export__c> relatedExports = portal.Exporte__r;
// grab all the related Export__c records using 'getSObjects' to avoid errors described above
Export__c[] relatedExports = portal.getSObjects('Exporte__r');
if (relatedExports != null) {
for (Export__c exp : relatedExports) {
// do something
}
}
}
}
I have the following questions:
If I use List<Export__c> relatedExports = portal.Exporte__r (which I commented out) to get the sub query records then I will receive the error message: “Aggregate query has too many rows for direct assignment, use FOR loop”. The error message makes no sense for me as the SOQL is done already before. Is there any explaination?
With the solution above the maximal amount of records from type Exporte_r received per Portal_c with the sub query is 199 though I have more than 200 for some records of Portal__c, why is it limited to that number? It seems all records above 199 are ignored in this case.
Is there any possibility to receive more than 199 records from a sub query? I have tried to change the batch size but it seems it is independent of the number of records receivable by the sub query. Any idea?
Many thanks!
As per the salesforce doc http://www.salesforce.com/us/developer/docs/apexcode/Content/langCon_apex_loops_for_SOQL.htm
You might get a QueryException in a SOQLfor loop with the message
Aggregate query has too many rows for direct assignment, use FOR loop.
This exception is sometimes thrown when accessing a large set of child
records of a retrieved sObject inside the loop, or when getting the
size of such a record set. To avoid getting this exception, use a for
loop to iterate over the child records, as follows.
Integer count=0;
for (Contact c : returnedAccount.Contacts) {
count++;
// Do some other processing
}