I have an Azure Stream Analytics job that combines the results of multiple queries and outputs them to the same sink. To do this, I define my queries within a WITH statement, then combine them using UNION and then write them to my sink. However, unfortunately I only get an output to my sink whenever all of my queries actually have an output, and this is where it goes wrong.
I have some queries that continuously (every 5 minutes) give an output, but I also have some queries that rare give an output (maybe a few times per day). This causes the output to not get any results, until the queries all have something to return. Does anyone know why this is and how I can fix it? Shouldn't UNION also give results when set A has results, but set B doesn't? I'm running this locally in VS Code, with a live connection to Event Hub by the way.
Here is a simplified example of 2 queries (one with frequent output, one with infrequent output) that goes wrong:
WITH
HarmonizedMeasurements AS (
SELECT
CAST(EHHARM.TimeStamp AS datetime) AS "TimeStamp",
CAST(EHHARM.ValueNumber AS float) AS "ValueNumber",
EHHARM.ValueBit AS "ValueBit",
EHHARM.MeasurementName,
EHHARM.PartName,
EHHARM.ElementId,
EHHARM.ElementName,
EHHARM.ObjectName,
EHHARM.TranslationTableId
FROM EventHubHarmonizedMeasurements AS EHHARM TIMESTAMP BY "TimeStamp"
PARTITION BY TranslationTableId
),
ToerenAandrijvingCategoriesMeasurements AS (
SELECT
AANDRCAT.TimeStamp AS "TimeStamp",
AANDRCAT.ValueNumber AS "ValueNumber",
AANDRCAT.ValueBit AS "ValueBit",
AANDRCAT.MeasurementName AS "MeasurementName",
AANDRCAT.PartName AS "PartName",
AANDRCAT.ElementId AS "ElementId",
AANDRCAT.ElementName AS "ElementName",
AANDRCAT.ObjectName AS "ObjectName",
AANDRCAT.TranslationTableId AS "TranslationTableId",
CASE
WHEN (-5000 < AANDRCAT.ValueNumber AND AANDRCAT.ValueNumber <= -1000) THEN 'Dalen'
WHEN (-1000 < AANDRCAT.ValueNumber AND AANDRCAT.ValueNumber <= -200) THEN 'Dalen Retarderen'
WHEN (-200 < AANDRCAT.ValueNumber AND AANDRCAT.ValueNumber <= 0) THEN 'Stilstand'
WHEN (0 < AANDRCAT.ValueNumber AND AANDRCAT.ValueNumber <= 250) THEN 'Nivelleren'
WHEN (250 < AANDRCAT.ValueNumber AND AANDRCAT.ValueNumber <= 400) THEN 'Heffen Retarderen'
WHEN (400 < AANDRCAT.ValueNumber AND AANDRCAT.ValueNumber <= 5000) THEN 'Heffen'
ELSE 'NoCategory'
END AS "Category"
FROM HarmonizedMeasurements AS AANDRCAT
WHERE
AANDRCAT.ObjectName LIKE 'Schutsluis%' AND
AANDRCAT.MeasurementName = 'Motortoerental terugkoppeling' AND
AANDRCAT.ValueNumber <> 0
),
AandrijvingCatStartMeasurements AS (
SELECT
AANDRCAT.TimeStamp AS "StartTime",
AANDRCAT.Category AS "Category",
AANDRCAT.ElementId AS "ElementId",
AANDRCAT.TranslationTableId AS "TranslationTableId"
FROM ToerenAandrijvingCategoriesMeasurements AS AANDRCAT
WHERE
LAG(Category, 1) OVER (PARTITION BY ElementId LIMIT DURATION(day, 1)) <> Category
),
AandrijvingCatEndMeasurements AS (
SELECT
AANDRST.StartTime AS "EndTime",
LAG(AANDRST.StartTime, 1) OVER (PARTITION BY ElementId LIMIT DURATION(day, 1)) AS "StartTime",
LAG(AANDRST.Category, 1) OVER (PARTITION BY ElementId LIMIT DURATION(day, 1)) AS "Category",
AANDRST.ElementId AS "ElementId",
AANDRST.TranslationTableId AS "TranslationTableId"
FROM AandrijvingCatStartMeasurements AS AANDRST
),
VermogenAandrijvingMeasurements AS (
SELECT
AANDRVER.TimeStamp AS "TimeStamp",
AANDRVER.ValueNumber AS "ValueNumber",
AANDRVER.ValueBit AS "ValueBit",
CONCAT(AANDRVER.MeasurementName, ' ', AANDREN.Category) AS "MeasurementName",
AANDRVER.PartName AS "PartName",
AANDRVER.ElementId AS "ElementId",
AANDRVER.ElementName AS "ElementName",
AANDRVER.ObjectName AS "ObjectName",
AANDRVER.TranslationTableId AS "TranslationTableId"
FROM HarmonizedMeasurements AS AANDRVER
LEFT JOIN AandrijvingCatEndMeasurements AS AANDREN
ON DATEDIFF(minute, AANDRVER, AANDREN) BETWEEN 0 AND 30 AND
AANDRVER.TimeStamp >= AANDREN.StartTime AND
AANDRVER.Timestamp < AANDREN.EndTime AND
AANDRVER.ElementId = AANDREN.ElementId AND
AANDRVER.TranslationTableId = AANDREN.TranslationTableId
INNER JOIN SQLMeasurementType AS MEAS
ON MEAS.Name = CONCAT(AANDRVER.MeasurementName, ' ', AANDREN.Category)
WHERE
AANDRVER.ObjectName LIKE 'Schutsluis%' AND
AANDRVER.MeasurementName = 'Vermogen'
),
LockDoorTop AS (
SELECT
Lock.TimeStamp AS "TimeStamp",
Lock.ValueNumber AS "ValueNumber",
Lock.ValueBit AS "ValueBit",
Lock.MeasurementName,
Lock.PartName,
Lock.ElementId,
Lock.ElementName,
Lock.ObjectName,
Lock.TranslationTableId
FROM HarmonizedMeasurements AS Lock
WHERE
Lock.MeasurementName = 'Sluisdeur open' AND
Lock.ElementName = 'Deur sluiskolk 1' AND
Lock.PartName = 'Bovenhoofd' AND
Lock.ValueBit = '1'
),
WaterLTop AS (
SELECT
WaterTop.TimeStamp AS "TimeStamp",
WaterTop.ValueNumber AS "ValueNumber",
WaterTop.ValueBit AS "ValueBit",
WaterTop.MeasurementName,
WaterTop.PartName,
WaterTop.ElementId,
WaterTop.ElementName,
WaterTop.ObjectName,
WaterTop.TranslationTableId
FROM HarmonizedMeasurements AS WaterTop
WHERE
WaterTop.MeasurementName = 'Waterniveaumeting' AND
WaterTop.ElementName = 'Sluiskolk 1' AND
WaterTop.PartName = 'Opvaartzijde'
),
WaterLLock AS (
SELECT
WaterLock.TimeStamp AS "TimeStamp",
WaterLock.ValueNumber AS "ValueNumber",
WaterLock.ValueBit AS "ValueBit",
WaterLock.MeasurementName,
WaterLock.PartName,
WaterLock.ElementId,
WaterLock.ElementName,
WaterLock.ObjectName,
WaterLock.TranslationTableId
FROM HarmonizedMeasurements AS WaterLock
WHERE
WaterLock.MeasurementName = 'Waterniveaumeting' AND
WaterLock.ElementName = 'Sluiskolk 1' AND
WaterLock.PartName = 'Sluiskamer'
),
WaterLevelTopMeasurements AS (
SELECT
LockDoor.TimeStamp AS "TimeStamp",
CAST(ROUND((WaterLevelLock.ValueNumber - WaterLevelTop.ValueNumber), 2) AS float) AS "ValueNumber",
null AS "ValueBit",
MEAS.Name AS "MeasurementName",
LockDoor.PartName AS "PartName",
LockDoor.ElementId AS "ElementId",
LockDoor.ElementName AS "ElementName",
LockDoor.ObjectName AS "ObjectName",
LockDoor.TranslationTableId AS "TranslationTableId"
FROM LockDoorTop AS LockDoor
JOIN WaterLTop AS WaterLevelTop
ON DATEDIFF(minute, LockDoor, WaterLevelTop) BETWEEN 0 AND 1 AND
LockDoor.ObjectName = WaterLevelTop.ObjectName
JOIN WaterLLock AS WaterLevelLock
ON DATEDIFF(minute, LockDoor, WaterLevelLock) BETWEEN 0 AND 1 AND
LockDoor.ObjectName = WaterLevelLock.ObjectName
INNER JOIN SQLMeasurementType AS MEAS
ON MEAS.Name = 'Waterniveauverschil'
),
-- Combine queries
DatalakeCombinedMeasurements AS (
SELECT * FROM VermogenAandrijvingMeasurements
UNION
SELECT * FROM WaterLevelTopMeasurements
)
-- Output data
SELECT *
INTO DatalakeHarmonizedMeasurements
FROM DatalakeCombinedMeasurements
PARTITION BY TranslationTableId
Related
I have this SQL query that I confirmed works in SQLite. It updates two columns in the Table. I have 144 columns that need to be updated using the same query. How can I, using Python, pass along variables so I can use the same query to update all of them?
Here is my query to update one column:
UPDATE GBPAUD_TA AS t1
SET _1m_L3_Time = COALESCE(
(
SELECT
MIN(
CASE t1.Action
WHEN 'Buy' THEN CASE WHEN (t2._1M_55 >= t2.Low AND t2._1M_55 < t2.Open) THEN t2.Date_Time END
WHEN 'Sell' THEN CASE WHEN (t2._1M_55 <= t2.High AND t2._1M_55 < t2.Open) THEN t2.Date_Time END
END
)
FROM GBPAUD_DATA t2
WHERE t2.Date_Time >= t1.Open_Date AND t2.Date_Time <= t1.New_Closing_Time
),
t1._1m_L3_Time
);
UPDATE GBPAUD_TA
SET _1m_L3_Price = (SELECT _1M_55
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA._1m_L3_Time)
where EXISTS (SELECT _1M_55
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA._1m_L3_Time)
Here is my query showing the variables that I would need to automatically insert:
UPDATE GBPAUD_TA AS t1
SET Variable1 = COALESCE(
(
SELECT
MIN(
CASE t1.Action
WHEN 'Buy' THEN CASE WHEN (t2.Variable2 >= t2.Low AND t2.Variable2< t2.Open) THEN t2.Date_Time END
WHEN 'Sell' THEN CASE WHEN (t2.Variable2 <= t2.High AND t2.Variable2< t2.Open) THEN t2.Date_Time END
END
)
FROM GBPAUD_DATA t2
WHERE t2.Date_Time >= t1.Open_Date AND t2.Date_Time <= t1.New_Closing_Time
),
t1.Variable1
);
UPDATE GBPAUD_TA
SET Variable3 = (SELECT Variable2
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA.Variable1)
where EXISTS (SELECT Variable2
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA.Variable1)
I have a total of 3 Variables.
Based upon googling and reading, I found a possible way by using host variables: I use the "?" in place of the variable, combine the variables into a tuple, and then use "executemany()"?
I tried this, but it did not work. It gave me an error:
"cursor.executemany(sql_update_query, SLTuple)
OperationalError: near "?": syntax error"
So what should I do? Any guidance is much appreciated!
Found the answer after I figured out the proper terminology: string formatting and interloping. Found the answer here.
Error given when adding query that runs in SQL developer but not in MS Query. Seems to not like my nested query.
Code I am using:
SELECT ORDER_DATE
,SALES_ORDER_NO
,CUSTOMER_PO_NUMBER
,DELIVER_TO
,STATUS
,ITEM_NUMBER
,DESCRIPTION
,ORD_QTY
,SUM(QUANTITY) AS ON_HAND
,PACKAGE_ID
,PACKAGE_STATUS
,MAX(TRAN_DATE) AS LAST_TRANSACTION
,MIN(DAYS) AS DAYS
FROM (
SELECT TRUNC(SH.ORDER_DATE) AS ORDER_DATE
,SH.SALES_ORDER_NO
,SH.CUSTOMER_PO_NUMBER
,SH.SHIP_CODE AS DELIVER_TO
,SH.STATUS
,SB.ITEM_NUMBER
,IM.DESCRIPTION
,SB.ORD_QTY
,BID.QUANTITY
,SPM.PACKAGE_ID
,CASE
WHEN SPM.SHIPPED = 'Y'
THEN 'SHIPPED'
WHEN SPM.STATUS = 'C'
THEN 'PACKED'
WHEN BID.QUANTITY IS NOT NULL
THEN 'AVAILABLE'
WHEN BID.QUANTITY IS NULL
THEN 'UNAVAILABLE'
END AS PACKAGE_STATUS
,CASE
WHEN SPM.SHIPPED = 'Y'
THEN TRUNC(SPM.BILLING_DATE)
WHEN SPM.STATUS = 'C'
THEN TRUNC(SPM.END_TIME)
WHEN BID.QUANTITY IS NOT NULL
THEN TRUNC(BID.ACTIVATION_TIME)
END AS TRAN_DATE
,CASE
WHEN SPM.SHIPPED = 'Y'
THEN ROUND(SYSDATE - SPM.BILLING_DATE, 0)
WHEN SPM.STATUS = 'C'
THEN ROUND(SYSDATE - SPM.END_TIME, 0)
WHEN BID.QUANTITY IS NOT NULL
THEN ROUND(SYSDATE - BID.ACTIVATION_TIME, 0)
END AS DAYS
FROM SO_HEADER SH
LEFT JOIN SO_BODY SB ON SB.SO_HEADER_TAG = SH.SO_HEADER_TAG
LEFT JOIN SO_PACKAGE_MASTER SPM ON SPM.PACKAGE_ID = SB.PACKAGE_ID
LEFT JOIN ITEM_MASTER IM ON IM.ITEM_NUMBER = SB.ITEM_NUMBER
LEFT JOIN V_BIN_ITEM_DETAIL BID ON BID.ITEM_NUMBER = SB.ITEM_NUMBER
WHERE SH.ORDER_TYPE = 'MSR'
AND (
SB.REASON_CODE IS NULL
OR SB.REASON_CODE NOT LIKE 'CANCEL%'
)
AND (
IM.DESCRIPTION NOT LIKE '%CKV%'
OR IM.DESCRIPTION IS NULL
AND IM.ITEM_NUMBER IS NOT NULL
)
)
WHERE PACKAGE_STATUS <> 'SHIPPED'
GROUP BY ORDER_DATE
,SALES_ORDER_NO
,CUSTOMER_PO_NUMBER
,DELIVER_TO
,STATUS
,ITEM_NUMBER
,DESCRIPTION
,ORD_QTY
,PACKAGE_ID
,PACKAGE_STATUS
ORDER BY (
CASE PACKAGE_STATUS
WHEN 'AVAILABLE'
THEN 1
WHEN 'UNAVAILABLE'
THEN 2
WHEN 'PACKED'
THEN 3
END
)
,LAST_TRANSACTION;
Is there any option I can select that will allow me to run this query?
Hi I am getting an error like this: Operand type clash: date is incompatible with int.
Below is my query which I am running on a SQL Server:
CREATE TABLE val.census_last_month
WITH(
DISTRIBUTION = ROUND_ROBIN
, CLUSTERED COLUMNSTORE INDEX
)
AS
SELECT
dt_mydate AS dt_census,
(SELECT count(DISTINCT encounter_id)
FROM prod.encounter
WHERE encounter_type = 'Inpatient' AND (ts_admit BETWEEN dt_mydate - 30 AND dt_mydate) AND
(ts_discharge IS NULL OR ts_discharge > dt_mydate)) AS census,
(SELECT count(DISTINCT encounter_id)
FROM prod.encounter
WHERE encounter_type = 'Inpatient' AND cast(ts_admit AS DATE) = dt_mydate) AS admits,
(SELECT count(DISTINCT encounter_id)
FROM prod.encounter
WHERE encounter_type = 'Inpatient' AND cast(ts_discharge AS DATE) = dt_mydate) AS discharges
FROM ref.calendar_day
WHERE ref.calendar_day.dt_mydate BETWEEN (cast(getdate() as date) - 30) AND cast(getdate() as date);
You need to use dateadd function. See details here. https://learn.microsoft.com/en-us/sql/t-sql/functions/dateadd-transact-sql?view=sql-server-2017
There are multiple issues with this script, however can you confirm the datatypes starting "ts_" are dates stored as integers in the format yyyyMMdd and datatypes starting "dt_" are DATE?
Based on these assumptions, this is my attempted rewrite:
SELECT dt_mydate AS dt_census,
(
SELECT COUNT( DISTINCT encounter_id )
FROM prod.encounter
WHERE encounter_type = 'Inpatient' AND ( CAST( CAST( ts_admit AS CHAR(8) ) AS DATE ) BETWEEN DATEADD( day, -30, dt_mydate ) AND dt_mydate )
AND ( ts_discharge IS NULL OR CAST( CAST( ts_discharge AS CHAR(8) ) AS DATE ) > dt_mydate)
) AS census,
(
SELECT COUNT( DISTINCT encounter_id )
FROM prod.encounter
WHERE encounter_type = 'Inpatient' AND CAST( CAST( ts_admit AS CHAR(8) ) AS DATE ) = dt_mydate
) AS admits,
(
SELECT COUNT( DISTINCT encounter_id )
FROM prod.encounter
WHERE encounter_type = 'Inpatient'
AND CAST( CAST( ts_discharge AS CHAR(8) ) AS DATE ) = dt_mydate
) AS discharges
FROM ref.calendar_day
WHERE ref.calendar_day.dt_mydate BETWEEN CAST ( DATEADD( day, -30, GETDATE() ) AS DATE ) AND CAST( GETDATE() AS DATE );
If either of my assumptions are incorrect, please let me know and I will update the script.
i have problems with this query. It shall return the max temperature of the day but it returns empty. what have i done wrong?
SELECT *
FROM tempdat
WHERE date( tdate ) = curdate( )
AND temperature = (
SELECT max( temperature )
FROM tempdat ) ;
Rather use:
SELECT
convert(datetime, convert(varchar(12), tdate )) AS DayOfTemp
,max(temperature) AS MaxTemp
FROM
tempdat
WHERE
convert(datetime, convert(varchar(12), tdate )) = convert(datetime, convert(varchar(12, curdate( ) ))
group by
convert(datetime, convert(varchar(12), tdate ))
Your subquery returns the overall maximum of the data - not of the day, you need something like:
select max(temperature) from tempdat group by tdate having tdate = curdate()
I am trying to use CASE in a SQL Select statement that will allow me to get results where I can utilize the length of one string to produce the resutls of another string. These are for non-matched records from two data sets that share a common ID, but variant Data Source.
Case statement is below:
Select Column1, Column2,
Case
When Column1 = 'Something" and Len(Column2) = '35' Then Column1 = "Something Else" and substring(Column2, 1, 35)
End as Column3
From dbo.xxx
When I run it I get the following error:
Msg 102, Level 15, State 1, Line 5 Incorrect syntax near '='.
You need to have a value for each WHEN, and ought to have an ELSE:
Select Data_Source, CustomerID,
CASE
WHEN Data_Source = 'Test1' and Len(CustomerName) = 35 THEN 'First Value'
WHEN Data_Source = 'Test2' THEN substring(CustomerName, 1, 35)
ELSE 'Sorry, no match.'
END AS CustomerName
From dbo.xx
FYI: Len() doesn't return a string.
EDIT:
A SQL Server answer that addresses some of the comments might be:
declare #DataSource as Table ( Id Int Identity, CustomerName VarChar(64) )
declare #VariantDataSource as Table ( Id Int Identity, CostumerName VarChar(64) )
insert into #DataSource ( CustomerName ) values ( 'Alice B.' ), ( 'Bob C.' ), ( 'Charles D.' )
insert into #VariantDataSource ( CostumerName ) values ( 'Blush' ), ( 'Dye' ), ( 'Pancake Base' )
select *,
-- Output the CostumerName padded or trimmed to the same length as CustomerName. NULLs are not handled gracefully.
Substring( CostumerName + Replicate( '.', Len( CustomerName ) ), 1, Len( CustomerName ) ) as Clustermere,
-- Output the CostumerName padded or trimmed to the same length as CustomerName. NULLs in CustomerName are explicitly handled.
case
when CustomerName is NULL then ''
when Len( CustomerName ) > Len( CostumerName ) then Substring( CostumerName, 1, Len( CustomerName ) )
else Substring( CostumerName + Replicate( '.', Len( CustomerName ) ), 1, Len( CustomerName ) )
end as 'Crustymore'
from #DataSource as DS inner join
#VariantDataSource as VDS on VDS.Id = DS.Id
Select
Column1,
Column2,
Case
When Column1 = 'Something' and Len(Column2) = 35
Then 'Something Else' + substring(Column2, 1, 35)
End as Column3
From dbo.xxx
Update your query on
use '+' for string concat
len() returns int, no need to use ''
remove "Column1 =" in the case when condition
replace "" with ''
Hope this help.