I am using DataFrame to read data from each postgres table and using to_sql() method to insert data into the oracle. The problem I am facing is that It gets stuck after copying a few records to oracle. Jupyter Notebook gets busy but does nothing.
def duplicateData(conn, conn2, session):
query1 = "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'"
all_tables = session.execute(query1)
count = 0
for index,tables in enumerate(all_tables):
count += 1
# getting rid of comma and parathesis
for i, table in enumerate(tables):
print("\n"+table+" - NO: "+str(count)+"\n")
query2 = "SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '" + table + "'"
columns = session.execute(query2)
cols = []
for col in columns:
cols.append(col[0])
query3 = "SELECT * FROM " + table
df = pd.read_sql(query3, conn)
alias = (table[:30] + '') if len(table) > 30 else table
df.to_sql(alias, conn2, index=False, schema="PMS")
print("\nDONE\n")
I have two tables, one in ms sql and other one in sqlite3
I need to update a field in mssql table to the related value from sqlite3 table.
here is my code:
import pypyodbc
import sqlite3
connection = pypyodbc.connect('Driver={SQL Server};'
'Server=****'
'Database=****;'
'uid=****;pwd=****')
cursor = connection.cursor()
SQLCommand = ("select ObjectId, ObjectColumnName,Label from LocalizedLabel "
"where LanguageId = 1065")
sqlUpdate = ("UPDATE LocalizedLabel "
"SET Label = ? "
"WHERE ObjectId = ? and ObjectColumnName = ? and LanguageId = 1065")
cursor.execute(SQLCommand)
results = cursor.fetchall()
srcDB = sqlite3.connect('crmLLDB')
for result in results:
val = (result[0], result[1])
print('SELECT "1065" FROM crm '
'WHERE UPPER(ObjectID)= UPPER(%s) AND ObjectColumnName = %s' % val)
srcCursor = srcDB.execute('SELECT "1065" FROM crm '
'WHERE UPPER(ObjectID)= UPPER(?) AND ObjectColumnName = ?', val)
if srcCursor.fetchone() is None:
print("No translation found for " + result[2])
else:
translation = srcCursor.fetchone()[0]
updateVals = (translation, result[0], result[1])
cursor.execute(sqlUpdate, updateVals)
connection.close()
srcDB.close()
objectId is a GuID and other fields are all strings
the print function returns:
SELECT "1065" FROM crm WHERE UPPER(ObjectID)= UPPER(b'0340B506-2341-DB11-898A-0007E9E17EBD') AND ObjectColumnName = DisplayName
while the watcher gives:
result[0] = 'b\\'0340B506-2341-DB11-898A-0007E9E17EBD\\''
result[1] = 'DisplayName'
this query of course returns this error:
[1] [SQLITE_ERROR] SQL error or missing database (near "'0340B506-2341-DB11-898A-0007E9E17EBD'": syntax error)
while this query :
SELECT "1065" FROM crm WHERE UPPER(ObjectID)= UPPER('0340B506-2341-DB11-898A-0007E9E17EBD') AND ObjectColumnName = 'DisplayName'
return the perfect answer,
can someone point out my problem, please?
appearaently the problem was with pypyodbc and its problem with GuId,
I used pyodbc and everything works fine now!
my final code, if some googler in the future passes by:
import pyodbc
import sqlite3
startTme = datetime.now()
connection = pyodbc.connect('Driver={SQL Server};'
'Server=*****;'
'Database=****;'
'uid=sa;pwd=*****')
cursor = connection.cursor()
SQLCommand = ("select ObjectId, ObjectColumnName,Label from LocalizedLabel "
"where LanguageId = 1065")
sqlUpdate_p = ("UPDATE LocalizedLabel "
"SET Label = ? "
"WHERE ObjectId = ? and ObjectColumnName = ? and LanguageId = 1065")
cursor.execute(SQLCommand)
results = cursor.fetchall()
srcDB = sqlite3.connect('crmLLDB')
jobLength = str(len(results))
i = 0
for result in results:
val = (result[0], result[1])
srcCursor = srcDB.execute('SELECT "1065" FROM crm '
'WHERE UPPER(ObjectID)= UPPER(?) AND ObjectColumnName = ?', val)
trans = srcCursor.fetchall()
for tr in trans:
updateVals = (tr[0], result[0], result[1])
cursor.execute(sqlUpdate_p, updateVals)
i += 1
connection.commit()
connection.close()
srcDB.close()
string tablename = DropDownList1.SelectedValue.ToString()
string id = TextBox1.Text;
adp = new SqlDataAdapter("select * from '" + tablename + "' where id='" + id + "'", obj.connect());
DataSet ds = new DataSet();
adp.Fill(ds);
string.Format("select * from {0} Where id = {1}" , tablename , id)
Similar Question may have been asked on this forum regarding exporting stored procedure result to excel file,I've tried few attempts to this but the file does not export to the folder that I expect,Your help is appreciated
Stored Procedure that I created
/****** Object: StoredProcedure [dbo].[Copy_DataDump_SpecificQuery] Script Date: 10/28/2014 15:59:03 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROC [dbo].[Copy_DataDump_SpecificQuery]
#SQLScript VARCHAR(MAX)
,#OutPut_Number INT OUTPUT
,#Output_FIleName VARCHAR(MAX) OUTPUT
AS
SET NOCOUNT ON
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
BEGIN
BEGIN TRY
DECLARE #server Varchar (100)
SET #server = 'MyServerName'
DECLARE #FileName VARCHAR(255)
DECLARE #Date VARCHAR(12)
SELECT #Date = CONVERT(VARCHAR(10),GETDATE(),120)
SET #Output_FIleName = CAST(NEWID() AS VARCHAR(MAX))+ '.xls'
DECLARE #FilePath VARCHAR (4000)
SET #FilePath = '\\ComputerName\Users\MyUserName\Downloads\Exel_File' + #Output_FIleName ;
IF OBJECT_ID('tempDB..##temp_Query', 'U') IS NOT NULL
DROP TABLE ##temp_Query;
----------------------------------------------------------------------------------------------
DECLARE #Query AS VARCHAR(MAX)
DECLARE #FROM_Start INT
SELECT #FROM_Start = CHARINDEX(' FROM ',#SQLScript)
SELECT #Query = SUBSTRING(#SQLScript,0,#FROM_Start) + ' INTO ##temp_Query FROM ' + SUBSTRING(#SQLScript,#FROM_Start+6,LEN(#SQLScript))
PRINT (#Query)
Exec(#Query)
------------------------------------------------------------------------------------------------------------------
DECLARE #dbName VARCHAR (100)
SET #dbName = 'master'
DECLARE #sql VARCHAR (5000)
SET #sql = 'Select * from tempdb.dbo.##temp_Query'
DECLARE #dbName1 VARCHAR (100)
IF OBJECT_ID ('tempDB..##TempExport_Query1','U') is not null
DROP TABLE ##TempExport_Query1
IF OBJECT_ID ('tempDB..##TempExport_Query2','U') is not null
DROP TABLE ##TempExport_Query2
SELECT #dbName1 = #dbName
SELECT #dbName = 'use ' + #dbName + ';'
DECLARE #columnNames VARCHAR (8000), #columnConvert VARCHAR (8000), #tempSQL VARCHAR (8000)
SELECT
#tempSQL = LEFT(#sql, CHARINDEX('from', #sql) - 1) + ' into ##TempExport_Query1 ' +
SUBSTRING(#sql, CHARINDEX('from', #sql) - 1, LEN(#sql))
EXEC (#dbName + #tempSQL)
--SELECT *
--FROM tempdb.INFORMATION_SCHEMA.Columns
--WHERE table_name = '##TempExport_Query1'
SELECT #columnNames = COALESCE(#columnNames + ',', '') + '"' + column_name + '"',
#columnConvert = COALESCE(#columnConvert + ',', '') + 'convert(nvarchar(4000),'
+ '[' + column_name + ']' + CASE
WHEN data_type IN ('datetime', 'smalldatetime') THEN ',121'
WHEN data_type IN ('numeric', 'decimal') THEN ',128'
WHEN data_type IN ('float', 'real', 'money', 'smallmoney') THEN ',2'
WHEN data_type IN ('datetime', 'smalldatetime') THEN ',120' ELSE ''
END + ') as ' + '[' + column_name + ']'
FROM tempdb.INFORMATION_SCHEMA.Columns
WHERE table_name = '##TempExport_Query1'
-- execute select query to insert data and column names into new temp table
SELECT #sql = 'select ' + #columnNames + 'temp##SortID into ##TempExport_Query2 from (select ' + #columnConvert + ',
''2'' as temp##SortID
from ##TempExport_Query1 union all select ''' + REPLACE(#columnNames, ',', ''', ''') + ''',
''1'') t '
EXEC (#sql)
SET #sql = 'bcp " select * from ##TempExport_Query2 ORDER BY temp##SortID " queryout "' + #FilePath +
'" -U uno -P uno -c -T -S ' + #server
EXEC master..xp_cmdshell #sql
SET #OutPut_Number = 1
END TRY
BEGIN CATCH
--DECLARE #ErrorDescription VARCHAR(1000) = 'ERROR - ' + ERROR_MESSAGE()
-- RAISERROR(#ErrorDescription,16,1)
SET #OutPut_Number = 2
--SET #OutPut_Message = ##ERROR
END CATCH
End
Parameters
DECLARE #OutPut_Number INT;
DECLARE #OutPut_Message VARCHAR(1000);
EXECUTE dbo.Copy_DataDump_SpecificQuery 'SELECT O.ID AS SystemNumber
,'' AS ManualNumber
,O.Date AS InvoiceDate
,OT.ID AS CustomerCode
,OT.Name AS CustomerName
,I.ID AS ItemID
,I.Description AS Item
,OL.UnitQty
,OL.FreeQty
,OL.UnitPrice
,OL.GrossValue
,NetSaleValue
,DueDate = ''
,A.ID AS SalesRepCode
,ItemCostPrice = OL.UnitPrice
FROM TxnOrder O
INNER JOIN TxnOrderLine OL ON O.UID = OL.TxnOrderUID AND O.SiteUID = Ol.TxnOrder_SiteUID
INNER JOIN Outlet OT ON O.OutletUID = OT.UID
INNER JOIN Item I ON OL.ItemUID = I.UID
INNER JOIN Agent A ON O.AgentUID = A.UID
WHERE O.mpt_TypeEnum = 1
AND( O.IsPrinted = 1 OR O.mpt_SalesmodelEnum = 2)
--AND O.Date >= #StartDate AND O.Date <= #EndDate
--AND (#DistributorUID IS NULL OR O.DistributorUID = #DistributorUID)
--AND (#AgentUID IS NULL OR O.AgentUID = #AgentUID)'
,#OutPut_Number OUTPUT
,#OutPut_Message OUTPUT
SELECT #OutPut_Message
After I execute I receive the following Output on the SQL Output window
69111E97-BEDE-4BDE-9EBE-C063DB690E9F.xls
I have created the permission for the folder as per below screenshot
This Problem was resolved ,I Logged in to the server via RDP and I executed the SP as per below and it worked out..
DECLARE #OutPut_Number INT;
DECLARE #OutPut_Message VARCHAR(1000);
EXECUTE dbo.Copy_DataDump_SpecificQuery 'SELECT O.ID AS SystemNumber
,O.Date AS InvoiceDate
,OT.ID AS CustomerCode
,OT.Name AS CustomerName
,I.ID AS ItemID
,I.Description AS Item
,OL.UnitQty
,OL.FreeQty
,OL.UnitPrice
,OL.GrossValue
,NetSaleValue
,A.ID AS SalesRepCode
,ItemCostPrice = OL.UnitPrice
FROM TxnOrder O
INNER JOIN TxnOrderLine OL ON O.UID = OL.TxnOrderUID AND O.SiteUID = Ol.TxnOrder_SiteUID
INNER JOIN Outlet OT ON O.OutletUID = OT.UID
INNER JOIN Item I ON OL.ItemUID = I.UID
INNER JOIN Agent A ON O.AgentUID = A.UID
WHERE O.mpt_TypeEnum = 1
AND( O.IsPrinted = 1 OR O.mpt_SalesmodelEnum = 2)'
,#OutPut_Number OUTPUT
,#OutPut_Message OUTPUT
SELECT #OutPut_Message
AND In My select Query I removed the Following Columns
'' AS ManualNumber
DueDate = ''
AND as Im already have the permission to access the server via RDP,I replaced the following path as follows
SET #FilePath = '\\ComputerName\Users\MyUserName\Downloads\Exel_File' + #Output_FIleName
C:\Users\Pathuma\Downloads\Excel
SET #FilePath = '\C:\Users\MyUsername\Downloads\Excel' + #Output_FIleName
We just updated to sequelize version 1.7.0-rc1. All of a sudden the following sequelize code stopped working:
exports.invalidate = function (partnerId, importStartTime, fn) {
console.log('\r\n Import start time: ' + JSON.stringify(importStartTime));
var sql = '';
sql += 'UPDATE "Product" p SET "status" = 0 FROM "Brand" b WHERE b."id" = p."BrandId" AND b."PartnerId" = ' + partnerId + ' AND p."status" <> 0 AND \
p."validatedAt" < \'' + importStartTime + '\'; ';
sql += 'UPDATE "Variant" v SET "status" = 0 FROM "Product" p INNER JOIN "Brand" b ON b."id" = p."BrandId" WHERE \
p."id" = v."ProductId" AND b."PartnerId" = ' + partnerId + ' AND v."status" <> 0 AND v."validatedAt" < \'' + importStartTime + '\'; ';
sql += 'UPDATE "ProductCategory" pc SET "status" = 0 FROM "Product" p INNER JOIN "Brand" b ON b."id" = p."BrandId" WHERE \
p."id" = pc."ProductId" AND b."PartnerId" = ' + partnerId + ' AND pc."status" <> 0 AND pc."validatedAt" < \'' + importStartTime + '\'; ';
sql += 'UPDATE "VariantImg" vi SET "status" = 0 FROM "Variant" v INNER JOIN "Product" p ON p."id" = v."ProductId" INNER JOIN "Brand" b ON b."id" = p."BrandId" WHERE \
v."id" = vi."VariantId" AND b."PartnerId" = ' + partnerId + ' AND vi."status" <> 0 AND vi."validatedAt" < \'' + importStartTime + '\'; ';
console.log('\r\n ' + sql);
sequelize.query(sql).success(function () {
console.log('\r\n Invalidation completed :)');
fn(null);
}).error(function (err) {
logExceptOnTest('\r\n Invalidation failed :(');
logExceptOnTest('\r\n Sequelize failed to set status to 0 for items that have been invalidated :(');
logExceptOnTest('\r\n err: ' + JSON.stringify(err));
fn(err);
});
}
We are using PostgreSql v9.3 for our datastore and it appears that the Update (sql) is not being executed. If I execute the same sql directly in PostgreSql, everything works as expected. The funny thing is, if I enable sequelize logging, I can see the Update statments being logged.
Update: 1/23/2014
I decided to move the Update SQL to a PosgreSql function. Strangely, the result is the same . If I call the function within PosgreSql, everything works. If I call the function from Sequelize, via the sequelize.query method, the same problem occurs.
Here is the code for the function:
CREATE OR REPLACE FUNCTION invalidate(partnerId INTEGER, importStartTime TIMESTAMP)
RETURNS TABLE (
validatedAt TIMESTAMP,
importStartTime2 TIMESTAMP
)
AS
$$
BEGIN
UPDATE "Product" p SET "status" = 0 FROM "Brand" b WHERE b."id" = p."BrandId" AND b."PartnerId" = $1 AND p."status" <> 0 AND
p."validatedAt"::TIMESTAMP WITHOUT TIME ZONE < $2;
UPDATE "Variant" v SET "status" = 0 FROM "Product" p INNER JOIN "Brand" b ON b."id" = p."BrandId" WHERE
p."id" = v."ProductId" AND b."PartnerId" = $1 AND v."status" <> 0 AND
v."validatedAt"::TIMESTAMP WITHOUT TIME ZONE < $2;
UPDATE "ProductCategory" pc SET "status" = 0 FROM "Product" p INNER JOIN "Brand" b ON b."id" = p."BrandId" WHERE
p."id" = pc."ProductId" AND b."PartnerId" = $1 AND pc."status" <> 0 AND
pc."validatedAt"::TIMESTAMP WITHOUT TIME ZONE < $2;
UPDATE "VariantImg" vi SET "status" = 0 FROM "Variant" v INNER JOIN "Product" p ON p."id" = v."ProductId" INNER JOIN "Brand" b ON b."id" = p."BrandId" WHERE
v."id" = vi."VariantId" AND b."PartnerId" = $1 AND vi."status" <> 0 AND
vi."validatedAt"::TIMESTAMP WITHOUT TIME ZONE < $2;
RETURN QUERY select pc."validatedAt"::TIMESTAMP WITHOUT TIME ZONE as "validatedAt", $2 as "importStartTime"
from "ProductCategory" pc
inner join "Product" p on pc."ProductId" = p."id"
inner join "Brand" b on b."id" = p."BrandId"
where pc."id" = 34;
END
$$
LANGUAGE plpgsql;
If I call the function like so within PosgreSql, all is fine:
SELECT invalidate(31, '2014-01-22 22:27:53');
If I call the function like so from node.js using Sequelize, it doesn't work:
var sql = 'SELECT invalidate(' + partnerId + ', \'' + importStartTime + '\');';
console.log('\r\n ' + sql);
sequelize.query(sql).success(function (data) {
//console.log('\r\n Invalidation completed :)');
console.log('\r\n data: ' + JSON.stringify(data));
fn(null);
}).error(function (err) {
logExceptOnTest('\r\n Invalidation failed :(');
logExceptOnTest('\r\n Sequelize failed to set status to 0 for items that have been invalidated :(');
logExceptOnTest('\r\n err: ' + JSON.stringify(err));
fn(err);
});
I have narrowed down the problem to this part of the sql:
p."validatedAt"::TIMESTAMP WITHOUT TIME ZONE < $2
I believe the problem is with how dates are stored and passed by both PostgreSql and Sequelize. Somthing changed with Sequelize 1.7.0-rc1, that is converting the date to a format that doesn't match what PosgreSql is reading from the validatedAt timestamp column.
I played around for a while, by attempting to message the dates into the right UTC format. I tried both with and without time zone, without any success. I even added a select at the end of the function to return me the date which is stored in the validatedAt column and the date I pass into the function (importStartTime). Both dates are clearly the same when looking at them visually in the code (via console.log) and in the table (when selected directly in PostgreSql), but when the function call returns the results of it's select statement, the dates are not in the same format. So, I believe the Update statment's date comparison part is also seeing the dates in different format and thus making it appear that the query is not working, even though it actually is.
I updated the title of this post to reflect the new findings.