Following on from a question I posted a couple of weeks back, the original question has been resolved but I've got some follow up questions relating to performance.
Firstly, here's an example of the stored procedure that addresses the original question, there are 23 values in the INSERT statement but I hope below paints a clear enough picture:
DELIMITER ;;
CREATE PROCEDURE myprocedure()
BEGIN
DECLARE n INT DEFAULT 0;
DECLARE i INT DEFAULT 1;
SELECT COUNT(*) FROM temp_table INTO n;
SET i=1;
WHILE i<=n DO
SET #dataId :=IFNULL((SELECT rowId FROM perm_table WHERE CONCAT(ID, somedate) = (SELECT CONCAT(ID, somedate) FROM temp_table WHERE temp_rowId = i)), 0);
INSERT INTO perm_table (val1,
val2,
val3,
...
)
SELECT #dataId,
val2,
val3,
...
FROM temp_table
WHERE temp_rowId = i
ON DUPLICATE KEY UPDATE val5 = (SELECT val5 FROM temp_table WHERE temp_rowId = i),
val6 = (SELECT val6 FROM temp_table WHERE temp_rowId = i),
val7 = (SELECT val7 FROM temp_table WHERE temp_rowId = i);
SET i = i + 1;
END WHILE;
TRUNCATE TABLE temp_table;
End;
;;
DELIMITER ;
This procedure produces the expected result in terms of row count, but the throughput is steady at 10 rows per second which results in a runtime of ~3.5 hours for an update across 120k rows which I'd really like to optimize further.
Here's a few things I've already tried to improve performance, unfortunately none of these attempts seem to have resulted in a meaningful drop in runtime:
temp_table and perm_table are both indexed on all relevant fields, also added composite index on (ID, somedate) to each
RDS (db.m5.large) storage using provisioned iops
innodb_flush_log_at_trx_commit=0
Just wondering if there are any other suggestions as to what I might be able to look at to improve the runtime?
This is now solved, removing the CONCAT statements and using variables instead has brought the runtime down dramatically to around 5 minutes.
SET #dataId :=IFNULL((SELECT rowId FROM perm_table WHERE CONCAT(ID, somedate) = (SELECT CONCAT(ID, somedate) FROM temp_table WHERE temp_rowId = i)), 0);
Became
SET #id := (SELECT ID FROM temp_table WHERE remp_rowId = i);
SET #somedate := (SELECT somedate FROM temp_table WHERE temp_rowId = i);
SET #dataId :=IFNULL((SELECT rowId FROM perm_table WHERE id = #id AND somedate = #somedate), 0);
Related
I have this SQL query that I confirmed works in SQLite. It updates two columns in the Table. I have 144 columns that need to be updated using the same query. How can I, using Python, pass along variables so I can use the same query to update all of them?
Here is my query to update one column:
UPDATE GBPAUD_TA AS t1
SET _1m_L3_Time = COALESCE(
(
SELECT
MIN(
CASE t1.Action
WHEN 'Buy' THEN CASE WHEN (t2._1M_55 >= t2.Low AND t2._1M_55 < t2.Open) THEN t2.Date_Time END
WHEN 'Sell' THEN CASE WHEN (t2._1M_55 <= t2.High AND t2._1M_55 < t2.Open) THEN t2.Date_Time END
END
)
FROM GBPAUD_DATA t2
WHERE t2.Date_Time >= t1.Open_Date AND t2.Date_Time <= t1.New_Closing_Time
),
t1._1m_L3_Time
);
UPDATE GBPAUD_TA
SET _1m_L3_Price = (SELECT _1M_55
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA._1m_L3_Time)
where EXISTS (SELECT _1M_55
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA._1m_L3_Time)
Here is my query showing the variables that I would need to automatically insert:
UPDATE GBPAUD_TA AS t1
SET Variable1 = COALESCE(
(
SELECT
MIN(
CASE t1.Action
WHEN 'Buy' THEN CASE WHEN (t2.Variable2 >= t2.Low AND t2.Variable2< t2.Open) THEN t2.Date_Time END
WHEN 'Sell' THEN CASE WHEN (t2.Variable2 <= t2.High AND t2.Variable2< t2.Open) THEN t2.Date_Time END
END
)
FROM GBPAUD_DATA t2
WHERE t2.Date_Time >= t1.Open_Date AND t2.Date_Time <= t1.New_Closing_Time
),
t1.Variable1
);
UPDATE GBPAUD_TA
SET Variable3 = (SELECT Variable2
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA.Variable1)
where EXISTS (SELECT Variable2
FROM GBPAUD_DATA
WHERE Date_Time = GBPAUD_TA.Variable1)
I have a total of 3 Variables.
Based upon googling and reading, I found a possible way by using host variables: I use the "?" in place of the variable, combine the variables into a tuple, and then use "executemany()"?
I tried this, but it did not work. It gave me an error:
"cursor.executemany(sql_update_query, SLTuple)
OperationalError: near "?": syntax error"
So what should I do? Any guidance is much appreciated!
Found the answer after I figured out the proper terminology: string formatting and interloping. Found the answer here.
I have a stored procedure in AZURE SQL database.In that there is a requirement to insert the records into the remote table from #temp table.
As xxxx_table is in the remote database used sp_execute_remote.
below is the scenario:
Create Procedure SP1 parameter1, Parameter2
As
select Distinct B.column1, B.Column2
into #A
from (Query1
Union
Query2) B
if (select count(1) from #A) > 0
Begin
Exec sp_execute_remote #data_source_name = N'Remotedatabase',
#stmt = N'INSERT INTO [dbo].[xxxx_table]
SELECT DISTINCT
'xxx' AS 'column1',
'xxx as 'Column2',
'xxx' AS 'Column3',
'xxx' AS 'Column4',
'xxx' AS Column4
FROM #A A INNER JOIN table1 on A.Column1 = Table1.Column2'
End
)
Getting the syntax error as below:
Incorrect syntax near 'xxx'.
Where am i going wrong? or let me know if there is another way to achieve this.
If you need to dynamically build a string in SQL single-quote the whole sentence, or use 'some text' + 'another text' to concat sentences. If you must add single quote use a double single quote ''
Example:
DECLARE #param1 int;
DECLARE #param1 VARCHAR(10);
SET #param1 = 10;
SET #param2 = 'CCDOS87'
#Stmt = 'SELECT Field1 FROM TableName WHERE Field1 = '
+ CAST(#param1 AS VARCHAR(100))
+ ' AND Field1 = '''
+ param2
+ ''''; <- This is a single '
#stmt = N'INSERT INTO [dbo].[Error_table]
SELECT DISTINCT
xxx AS column1,
xxx as Column2,
xxx AS Column3,
xxx AS Column4,
xxx AS Environment
FROM #A A INNER JOIN table1 on A.Column1 = Table1.Column2'
update
If your tables are in different databases but in the same server use:
INSERT INTO SERVER.SCHEMA.TABLE_NAME
SELECT Something
FROM SERVER.SCHEMA.TABLE_NAME
I'm using Teradata. I'd like rename a table with a script sql and not using bteq, if a specific conditions is satisfied.
In particular:
if TABLE_A exists => rename table TABLE_B to TABLE_B_OLD
In Sql Server:
IF OBJECT_ID('TABLE_A', 'U') IS NULL
EXEC sp_rename 'TABLE_B', 'TABLE_B_OLD';
In Oracle:
DECLARE
cnt NUMBER;
BEGIN
select COUNT(*) INTO cnt from sys.user_tables where table_name = 'TABLE_A'
IF cnt>0 THEN
execute immediate 'rename table TABLE_B to TABLE_B_OLD';
END IF;
END;
How can I made it with Teradata,
Thanks
Fabio
How about this?
REPLACE PROCEDURE IF_EXISTS_RENAME
(
IN table_name VARCHAR(30),
IN new_table_name VARCHAR(30)
)
BEGIN
IF EXISTS(SELECT 1 FROM dbc.tables WHERE 1=1 AND tablename = table_name and databasename=DATABASE) THEN
CALL DBC.SysExecSQL('RENAME TABLE ' || table_name ||' to '|| new_table_name);
END IF;
END;
Changed code provided by #access_granted to include Database Name as variable
REPLACE PROCEDURE FAR.RENAME_TABLE
(
IN table_name VARCHAR(30),
IN new_table_name VARCHAR(30),
IN db_name VARCHAR(50)
)
BEGIN
declare my_sql VARCHAR(1000);
IF EXISTS(SELECT 1
FROM dbc.tables
WHERE 1=1 AND tablename = table_name and databasename= db_name)
THEN
set my_sql ='RENAME TABLE ' || table_name ||' to '|| new_table_name||';' ;
EXECUTE IMMEDIATE my_sql;
END IF;
END;
Calling the procedure with three arguments:
Old Table Name
New Table Name
Database Name
call FAR.RENAME_TABLE('TEST_ABC','TEST_11','FAR')
Assuming you're on a relatively modern version of Teradata, you can do this in SQL Assistant (or BTEQ):
select
count (*)
from
dbc.tablesv where tablename = '<your table>'
and databasename = '<your db>'
having count (*) > 0;
.if activitycount = 1 then .GOTO RenameTable;
.if activitycount <> 1 then .quit;
.LABEL RenameTable
rename table <your table> <your new name;
How do I increment field of table using varchar. Here an example of what I want to have:
Mark 00001
Mark 00002
Mark 00003
Jaques 00001
Jaques 00002
Jaques 00003
Here is my example It can be useful for other people thanks to those who have helped me a lot thank you
The table is
`CREATE TABLE [dbo].[TAG_Sinistre](
[ID] [int] IDENTITY(1,1) NOT NULL,
[ref_ag] [varchar](7) NULL,
[ref_sinistre] [varchar](7) NULL,
)`
The stored procedure is
create PROC [dbo].[sp_Add_AgSinistre]
#ref_ag varchar (7)
AS BEGIN
declare #id int
DECLARE #ref_sin VARCHAR
SET #id = (SELECT ISNULL(MAX(CAST(ID AS INT)), 0) + 1
FROM TAG_Sinistre where ref_ag=#ref_ag
)
select #ref_sin=right('000000'+CAST(#ref_sin AS VARCHAR(6)),6)
BEGIN
INSERT into TAG_Sinistre(ref_ag,ref_sinistre)
VALUES (#ref_ag,#ref_sin)
Assuming the CompanyName field is UNIQUE, you could get the MAX added value, and increment that?
DECLARE #CurrentSequence INT
SET #CurrentSequence = (SELECT MAX(SequenceId)
FROM TableName WHERE CompanyName = #CompanyName)
INSERT INTO TableName
(CompanyName, SequenceId)
VALUES
(#CompanyName, #CurrentSequence+1)
Your StoredProc would pass in the Company Name as #CompanyName. Obviously, this is a naive approach, and I'm not including what would happen if multiple attempts to update the company would happen simultaneously, but there's no mention of that ion the question.
EDIT: Based on comments
DECLARE #maxSeq INT
SET #maxSeq = (SELECT ISNULL(MAX(CAST(SequenceId AS INT)), 0) + 1 FROM
TableName WHERE CompanyName = #CompanyName)
INSERT INTO TableName
(CompanyName, SequenceId)
VALUES
(#CompanyName, right('000000'+CAST(#maxSeq AS VARCHAR(7)),7))
As I said, I would look to use an INT, and get the UI to present the sequence as you want, instead of casting etc in the DB.
EDIT 2: Added Fiddle
SQL Fiddle
Assuming that your columns always look like those specified above.
DB Trigger (MS-SQL) --> Pseudocode/Untested:
CREATE TRIGGER [dbo].[YourTrigger]
ON [dbo].[YourTable]
INSTEAD OF INSERT
AS
BEGIN
DECLARE #maxVal int
-- PSEUDOCODE!!!
SELECT #maxVal = MAX(CAST(dbo.RegexReplace( '.*?(\d+)', myColumn, '$1') AS INT ))
FROM YourTable
INSERT INTO YourTable
SELECT myColumn + ' ' + #maxVal FROM inserted
END
Use insert now like this:
INSERT INTO YourTable values('Jaques')
CF schema:
CREATE TABLE mytable (
upperId int,
lowerId int,
hour timestamp,
counter text,
succ int,
fail int,
PRIMARY KEY ((upperId, lowerId), hour, counter));
each record is keyed by composite id upperId:lowerid, how can I do multiget with CQL3?
This is not valid:
select * from mytable where (upperid, lowerid) in ((10000, 1), (10000, 2), (20000, 1));
I can't do this either:
select * from mytable where (upperid = 10000 and lowerid in (1, 2)) or (upperid = 20000 and lowerid = 1);
I got error: missing EOF at ')'.
Please help point to effective way to do multiget for composite row key in CQL3.
Thanks,
William
CQL does not yet support a logical "or" in select statements.
Instead, in your application your could combine the result sets from the two queries:
select * from mytable where upperid = 10000 and lowerid in (1, 2);
select * from mytable where upperid = 20000 and lowerid = 1;
Reference:
SO question: Alternative for OR condition after where clause in select statement Cassandra
Latest CQL docs