Why the generated Queries by ormLite to load related references using IN do not use parameterized queries? - servicestack
I have an issue related to loading reference/child tables when loading the parent table records from the database. My DB engine is MS SQL Server. I'm using ServiceStack.OrmLite v5.8.0. My application is written in C# .NET v4.6.2.
I have three entities, 1 parent and 2 children. They are marked properly using the required annotations by OrmLite. I noticed when running a query from C# to fetch the parent table including the children in one shot using an IN Condition, the queries that get generated to load the children records are not parameterized. Please have a look at the queries I got from SQL Profiler below:
This is the first query that fetches the records from the parent table. Notice the usage of the parameterized query:
exec sp_executesql N'SELECT "Id", "MeetingDateAndTime", "Location", "IsHeld", "ScheduledForDate", "ContactMeBy", "IsCanceled", "IncidentId", "IncidentNumber", "IncidentDateTime"
FROM "HomePageSearch_ResolutionMeetings"
WHERE "Id" IN (#0,#1,#2,#3,#4,#5,#6,#7,#8,#9,#10)
ORDER BY "IncidentDateTime" DESC',N'#0 int,#1 int,#2 int,#3 int,#4 int,#5 int,#6 int,#7 int,#8 int,#9 int,#10 int',#0=16,#1=15,#2=13,#3=14,#4=12,#5=8,#6=5,#7=6,#8=4,#9=1,#10=2
This is the second query that fetches the records from the first child table. Notice the usage of the hard-coded values while the parameters are being generated:
exec sp_executesql N'SELECT "Id", "ResolutionSessionIdFk", "Type", "RespondentId", "FirstName", "LastName" FROM "HomePageSearch_ResolutionMeetingsInvitees" WHERE "ResolutionSessionIdFk" IN (SELECT "HomePageSearch_ResolutionMeetings"."Id"
FROM "HomePageSearch_ResolutionMeetings"
WHERE "Id" IN (16,15,13,14,12,8,5,6,4,1,2))',N'#0 int,#1 int,#2 int,#3 int,#4 int,#5 int,#6 int,#7 int,#8 int,#9 int,#10 int',#0=16,#1=15,#2=13,#3=14,#4=12,#5=8,#6=5,#7=6,#8=4,#9=1,#10=2
This is the third query that fetches the records from the second child table. Notice the usage of the hard-coded values while the parameters are being generated:
exec sp_executesql N'SELECT "Id", "ResolutionMeetingIdFk", "Type", "FirstName", "LastName" FROM "HomePageSearch_ResolutionMeetingsOfficers" WHERE "ResolutionMeetingIdFk" IN (SELECT "HomePageSearch_ResolutionMeetings"."Id"
FROM "HomePageSearch_ResolutionMeetings"
WHERE "Id" IN (16,15,13,14,12,8,5,6,4,1,2))',N'#0 int,#1 int,#2 int,#3 int,#4 int,#5 int,#6 int,#7 int,#8 int,#9 int,#10 int',#0=16,#1=15,#2=13,#3=14,#4=12,#5=8,#6=5,#7=6,#8=4,#9=1,#10=2
Any idea why?
Related
great expectation with delta table
I am trying to run a great expectation suite on a delta table in Databricks. But I would want to run this on part of the table with a query. Though the validation is running fine, it's running on full table data.
I know that I can load a Dataframe and pass it to Batch Request but I would like to load the data directly with query.
batch_request = RuntimeBatchRequest(
datasource_name="datasource",
data_connector_name="data_quality_run",
data_asset_name="Input Data",
runtime_parameters={"path": "/delta table path"},
batch_identifiers={"data_quality_check": f"data_quality_check_{datetime.date.today().strftime('%Y%m%d')}"},
batch_spec_passthrough={"reader_method": "delta", "reader_options": {"header": True}, "query" : {"name":"John"}},
)
Above batch request loading the data ignoring the query option. Is there any way to pass the query for delta table in the batch request
You can try to put query inside of runtime_parameters.
This works for me when I am querying data in SQL Server:
batch_request = RuntimeBatchRequest(
datasource_name="my_mssql_datasource",
data_connector_name="default_runtime_data_connector_name",
data_asset_name="default_name",
runtime_parameters={
"query": "SELECT * from dbo.MyTable WHERE Created = GETDATE()"
},
batch_identifiers={"default_identifier_name": "default_identifier"},
)
Cosmos db null value
I have two kind of record mention below in my table staudentdetail of cosmosDb.In below example previousSchooldetail is nullable filed and it can be present for student or not.
sample record below :-
{
"empid": "1234",
"empname": "ram",
"schoolname": "high school ,bankur",
"class": "10",
"previousSchooldetail": {
"prevSchoolName": "1763440",
"YearLeft": "2001"
} --(Nullable)
}
{
"empid": "12345",
"empname": "shyam",
"schoolname": "high school",
"class": "10"
}
I am trying to access the above record from azure databricks using pyspark or scala code .But when we are building the dataframe reading it from cosmos db it does not bring previousSchooldetail detail in the data frame.But when we change the query including id for which the previousSchooldetail show in the data frame .
Case 1:-
val Query = "SELECT * FROM c "
Result when query fired directly
empid
empname
schoolname
class
Case2:-
val Query = "SELECT * FROM c where c.empid=1234"
Result when query fired with where clause.
empid
empname
school name
class
previousSchooldetail
prevSchoolName
YearLeft
Could you please tell me why i am not able to get previousSchooldetail in case 1 and how should i proceed.
As #Jayendran, mentioned in the comments, the first query will give you the previouschooldetail document wherever they are available. Else, the column would not be present. You can have this column present for all the scenarios by using the IS_DEFINED function. Try tweaking your query as below: SELECT c.empid, c.empname, IS_DEFINED(c.previousSchooldetail) ? c.previousSchooldetail : null as previousSchooldetail, c.schoolname, c.class FROM c If you are looking to get the result as a flat structure, it can be tricky and would need to use two separate queries such as: Query 1 SELECT c.empid, c.empname, c.schoolname, c.class, p.prevSchoolName, p.YearLeft FROM c JOIN c.previousSchooldetail p Query 2 SELECT c.empid, c.empname, c.schoolname, c.class, null as prevSchoolName, null as YearLeft FROM c WHERE not IS_DEFINED (c.previousSchooldetail) or c.previousSchooldetail = null Unfortunately, Cosmos DB does not support LEFT JOIN or UNION. Hence, I'm not sure if you can achieve this in a single query. Alternatively, you can create a stored procedure to return the desired result.
Query to get all Cosmos DB documents referenced by another
Assume I have the following Cosmos DB container with the possible doc type partitions:
{
"id": <string>,
"partitionKey": <string>, // Always "item"
"name": <string>
}
{
"id": <string>,
"partitionKey": <string>, // Always "group"
"items": <array[string]> // Always an array of ids for items in the "item" partition
}
I have the id of a "group" document, but I do not have the document itself. What I would like to do is perform a query which gives me all "item" documents referenced by the "group" document.
I know I can perform two queries: 1) Retrieve the "group" document, 2) Perform a query with IN clause on the "item" partition.
As I don't care about the "group" document other than getting the list of ids, is it possible to construct a single query to get me all the "item" documents I want with just the "group" document id?
You'll need to perform two queries, as there are no joins between separate documents. Even though there is support for subqueries, only correlated subqueries are currently supported (meaning, the inner subquery is referencing values from the outer query). Non-correlated subqueries are what you'd need. Note that, even though you don't want all of the group document, you don't need to retrieve the entire document. You can project just the items property, which can then be used in your 2nd query, with something like array_contains(). Something like: SELECT VALUE g.items FROM g WHERE g.id="1" AND g.partitionKey="group" SELECT VALUE i.name FROM i WHERE array_contains(<items-from-prior-query>,i.id) AND i.partitionKey="item" This documentation page clarifies the two subquery types and support for only correlated subqueries.
Presto subqueries: Key not present in map
I have been banging my head a while to Superset -> Presto (PrestoSQL) -> Prometheus combination (as Superset does not yet support Prometheus) and got stymied with an issue when trying to extract columns from Presto's map type column containing Prometheus labels.
In order to get necessary labels mapped as columns from Superset's point of view, I create extra table (or I guess a view in this case) in Superset on top of existing table which had following SQL for creating the necessary columns:
SELECT labels['system_name'] AS "system",labels['instance'] AS "instance","timestamp" AS "timestamp","value" AS "value" FROM "up"
This table is then used as a data source in Superset's chart which treats it as a subquery. The resulting SQL query created by Superset and then sent to Presto looks e.g. like this:
SELECT "system" AS "system",
"instance" AS "instance",
"timestamp" AS "timestamp",
"value" AS "value"
FROM
(SELECT labels['system_name'] AS "system",
labels['instance'] AS "instance",
"timestamp" AS "timestamp",
"value" AS "value"
FROM "up") AS "expr_qry"
WHERE "timestamp" >= from_iso8601_timestamp('2020-10-19T12:00:00.000000')
AND "timestamp" < from_iso8601_timestamp('2020-10-19T13:00:00.000000')
ORDER BY "timestamp" ASC
LIMIT 250;
However, what I get out from above is an error:
io.prestosql.spi.PrestoException: Key not present in map: system_name
at io.prestosql.operator.scalar.MapSubscriptOperator$MissingKeyExceptionFactory.create(MapSubscriptOperator.java:173)
at io.prestosql.operator.scalar.MapSubscriptOperator.subscript(MapSubscriptOperator.java:143)
at io.prestosql.$gen.CursorProcessor_20201019_165636_32.filter(Unknown Source)
After reading a bit about queries from Presto's user guide, I tried a modified query from command line by using WITH:
WITH x AS (SELECT labels['system_name'] AS "system",labels['instance'] AS "instance","timestamp" AS "timestamp","value" AS "value" FROM "up")
SELECT system, timestamp, value FROM x
WHERE "timestamp" >= from_iso8601_timestamp('2020-10-19T12:00:00.000000')
AND "timestamp" < from_iso8601_timestamp('2020-10-19T13:00:00.000000')
LIMIT 250;
And that went throught without any issues. But it seems that I have no way to define how Superset executes its queries, so I'm stuck with the first option. The question is, is there anything wrong with it which could be fixed?
I guess that one option (if everything else fails) would be defining extra tables in Presto side which would do the same trick for mapping the columns, thus hopefully avoiding above issue.
The map subscript operator in Presto requires that the key be present in the map. Otherwise, you get the failure you described. If some keys can be missing, you can use the element_at function instead, which will return a NULL result: Returns value for given key, or NULL if the key is not contained in the map.
Possible to add dynamic WHERE clause with a QueryFile?
I have a complex query stored in an SQL file and I would like to reuse it for various routes but change up the WHERE clause depending on the route. This would be instead of having a large complex query in multiple files with the only difference being the WHERE statement.
Is it possible to dynamically add a WHERE when using QueryFile? Simplified example below:
SELECT "id", "userId", "locationId", "title", "body",
(
SELECT row_to_json(sqUser)
FROM (
SELECT "id", "firstname", "lastname"
FROM "users"
WHERE "users"."id" = "todos"."userId"
) sqUser
) as "user"
FROM "todos"
const queryIndex = new pgp.QueryFile('sql/todos/index.pgsql', queryOptions);
// 1. Use as is to get a list of all todos
// 2. OR Append WHERE "locationId" = $1 to get list filtered by location
// 3. OR Append WHERE "id" = $1 to get a specific item
// without having three separate SQL files?
It seems like (maybe?) you could get away with adding the below in the query file but that still feels limiting (would still need two files for = and LIKE and it still limits to only one WHERE condition). It also also feels weird to do something like WHERE 1 = 1 to get all records to return.
WHERE $1 = $2
I would be interested in hearing peoples' thoughts on this or if there is a better approach.
You can inject dynamic condition into a query-file as Raw Text:
SELECT "id", "userId", "locationId", "title", "body",
(
SELECT row_to_json(sqUser)
FROM (
SELECT "id", "firstname", "lastname"
FROM "users"
${condition:raw}
) sqUser
) as "user"
FROM "todos"
Pre-formatted parameters, based on the condition:
// Generate a condition, based on the business logic:
const condition = pgp.as.format('WHERE col_1 = $1 AND col_2 = $2', [111, 222]);
Executing your query-file:
await db.any(myQueryFile, {condition});
Advanced
Above is for the scenario when your have a simple dynamic condition that you want to generate in the code. But sometimes you may have complex static conditions that you want to alternate. In this case, you can have your master query file refer to the condition from a slave query file (nested query files are supported right out of the box). And in this case you do not even need to use :raw filter, because query files are injected as raw text by default:
Master query:
SELECT * FROM table ${condition}
Load your slave query files with complex conditions (when the app starts):
const conditionQueryFile1 = new QueryFile(...);
const conditionQueryFile2 = new QueryFile(...);
Selecting the right slave query, based on the business logic:
const condition = conditionQueryFile1; // some business logic here;
Executing master query with a slave as parameter:
await db.any(myQueryFile, {condition});