I am using DataFrame to read data from each postgres table and using to_sql() method to insert data into the oracle. The problem I am facing is that It gets stuck after copying a few records to oracle. Jupyter Notebook gets busy but does nothing.
def duplicateData(conn, conn2, session):
query1 = "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'"
all_tables = session.execute(query1)
count = 0
for index,tables in enumerate(all_tables):
count += 1
# getting rid of comma and parathesis
for i, table in enumerate(tables):
print("\n"+table+" - NO: "+str(count)+"\n")
query2 = "SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '" + table + "'"
columns = session.execute(query2)
cols = []
for col in columns:
cols.append(col[0])
query3 = "SELECT * FROM " + table
df = pd.read_sql(query3, conn)
alias = (table[:30] + '') if len(table) > 30 else table
df.to_sql(alias, conn2, index=False, schema="PMS")
print("\nDONE\n")
I'm trying to implement my first SQLite Database in an Android App regarding obtaining location coordinates to keep track of where the user has been.
I'm trying to add information from my entry into two tables:
a Location table that contains information of the places name, id, latitude, and longitude information &
a CheckIn table that contains information of the places address, corresponding location_id to know which location it corresponds to, latitude, longitude, and time of check in.
Whenever I try to do this, my entry is never updated for the Locations table, solely the CheckIn table, despite using the insert() function to insert into the Locations table as well the id is not updating for the Location table.
I've went through my app in a debugger and I can't figure out what's causing the problem here, as there's no error and the program proceeds just fine to add in the necessary info for the CheckIn table.
I've tried checking StackOverFlow but I can't quite find anything that has been able to help fix my problem. If there's anyone who could help me, it'd be greatly appreciated
My add function:
fun addLoc_CheckIn(Entry: Locations)
{
val selectQuery = "SELECT * FROM $LOCATIONS ORDER BY ID"
val db = this.readableDatabase
val cursor = db.rawQuery(selectQuery, null)
var con = 0
if (cursor.moveToFirst())
{
while (cursor.moveToNext())
{
val pSLong = cursor.getDouble(cursor.getColumnIndex(SLONG))
val pCLong = cursor.getDouble(cursor.getColumnIndex(CLONG))
val pSLat = cursor.getDouble(cursor.getColumnIndex(SLAT))
val pCLat = cursor.getDouble(cursor.getColumnIndex(CLAT))
val Theta = (pCLong * Entry.cLong) + (pSLong * Entry.sLong)
var dist = (pSLat * Entry.sLat) + (pCLat * Entry.cLat * Theta)
// dist = (Math.acos(dist) * 180.00 / Math.PI) * (60 * 1.1516 * 1.609344) / 1000
dist = Math.acos(dist) * 6380000
if (dist <= 30)
{
con = 1
val db1 = this.writableDatabase
val values = ContentValues()
values.put(LOC_ID, cursor.getInt(cursor.getColumnIndex(ID)))
values.put(ADDRESS, Entry.Checks[0].Address)
values.put(LATI, Entry.Lat)
values.put(LONGI, Entry.Long)
values.put(TIME, Entry.Checks[0].Date_Time)
db1.insert(CHECKINS, null, values)
break
}
}
}
if (con == 0)
{
val db1 = this.writableDatabase
val values = ContentValues()
values.put(LOC_NAME, Entry.Name)
values.put(LAT, Entry.Lat)
values.put(LONG, Entry.Long)
values.put(CLAT, Entry.cLat)
values.put(SLAT, Entry.sLat)
values.put(CLONG, Entry.cLong)
values.put(SLONG, Entry.sLong)
Entry.Id = db1.insert(LOCATIONS, null, values)
val cvalues = ContentValues()
cvalues.put(LOC_ID, Entry.Id)
cvalues.put(ADDRESS, Entry.Checks[0].Address)
cvalues.put(LATI, Entry.Lat)
cvalues.put(LONGI, Entry.Long)
cvalues.put(TIME, Entry.Checks[0].Date_Time)
db1.insert(CHECKINS, null, cvalues)
}
}
My OnCreate function with the corresponding companion object:
companion object {
private val DATABASE_NAME = "LocationsDB"
private val DATABASE_VERSION = 1
// 1st Table - Unique Check Ins
private val LOCATIONS = "LOCATIONS"
private val ID = "ID"
private val LOC_NAME = "LOC NAME"
private val LAT = "LAT"
private val LONG = "LONG"
private val CLAT = "CLAT"
private val SLAT = "SLAT"
private val CLONG = "CLONG"
private val SLONG = "SLONG"
// 2nd Table - Repeated Check Ins
private val CHECKINS = "CHECKINS"
private val CHECKIN_ID = "CHECKIN_ID"
private val LOC_ID = "LOC_ID"
private val ADDRESS = "ADDRESS"
private val TIME = "TIME"
private val LATI = "LAT"
private val LONGI = "LONG"
}
override fun onCreate(p0: SQLiteDatabase?) {
val LOCATION_QUERY = "CREATE TABLE " + LOCATIONS + "(" + ID +
" INTEGER PRIMARY KEY AUTOINCREMENT, " + LOC_NAME +
" TEXT, " + LAT + " INTEGER, " + LONG + " INTEGER, " +
CLAT + " INTEGER, "+ SLAT + " INTEGER, " + CLONG + " INTEGER, "+ SLONG + " INTEGER " + ")"
val CHECKIN_QUERY = "CREATE TABLE " + CHECKINS + "(" +
LOC_ID + " INTEGER, " + CHECKIN_ID + " INTEGER PRIMARY KEY AUTOINCREMENT, " + LATI + " INTEGER, " + LONGI + " INTEGER, " + ADDRESS +
" TEXT, " + TIME + " TEXT " + ")"
p0!!.execSQL(LOCATION_QUERY)
p0.execSQL(CHECKIN_QUERY)
}
Now, in my constructor for the Location class and the CheckIns class, I have the id's set to -1, which is what the id for the location remains, even after using the insert() function. Now, this doesn't cause me any issues with regards to adding in my CheckIns as well incrementing the ids in my CheckIns table and I doubt it's causing an issue but I figured it'd be best to include the information, just in case.
I believe that you have an issue with the name of the column due to using
private val LOC_NAME = "LOC NAME"
A column name cannot have a space unless it is enclosed in special characters as per SQL As Understood By SQLite - SQLite Keywords.
This isn't an issue when the table is create (the column name will be LOC). However, when you attempt to insert you will get a syntax error, the row will not be inserted but as you are using the SQLiteDatabase insert method, the error is trapped and processing continues.
However, in the log you would see something similar to :-
2019-10-29 15:47:35.119 12189-12189/aso.so58600930insert E/SQLiteLog: (1) near "NAME": syntax error
2019-10-29 15:47:35.121 12189-12189/aso.so58600930insert E/SQLiteDatabase: Error inserting LOC NAME=MyLoc LAT=100 CLAT=120 LONG=110 SLAT=140 CLONG=130 SLONG=150
android.database.sqlite.SQLiteException: near "NAME": syntax error (code 1 SQLITE_ERROR): , while compiling: INSERT INTO LOCATIONS(LOC NAME,LAT,CLAT,LONG,SLAT,CLONG,SLONG) VALUES (?,?,?,?,?,?,?)
You could circumvent the above by using :-
val db1 = this.writableDatabase
val values = ContentValues()
values.put("LOC", Entry.Name)
values.put(LAT, Entry.Lat)
values.put(LONG, Entry.Long)
values.put(CLAT, Entry.cLat)
values.put(SLAT, Entry.sLat)
values.put(CLONG, Entry.cLong)
values.put(SLONG, Entry.sLong)
Entry.Id = db1.insert(LOCATIONS, null, values)
However, it is not suggested that you use the above BUT that instead you correct the name, e.g. using :-
private val LOC_NAME = "LOC_NAME"
then clear the App's data or uninstall the App and then rerun the App.
This fix assumes that you are developing the App and can afford to lose any existing data. You could retain data but this is a little more complicated as you basically have to create a new table with the appropriate column name, copy the data from the original table, rename or drop the original table and then rename the new table to be the original name.
Not able to extract table names used within with clause, I'm using presto-parser version 0.226.
SqlParser sqlParser = new SqlParser();
String sql = "WITH dataset AS ( SELECT ROW('Bob', 38) AS users from tabb ) SELECT * FROM dataset";
Query query = (Query)sqlParser.createStatement(sql, ParsingOptions.builder().build());
QuerySpecification body = (QuerySpecification)query.getQueryBody();
System.out.println("From = " + body.getFrom().get());
/* Output
From = Table{dataset}
*/
Expected output
From = Table{dataset, tabb}
conn = pyodbc.connect("DRIVER={SQL Server};"
"SERVER="+server+";"
"UID="+username+";"
"PWD="+password,
autocommit=True)
cursor = conn.cursor()
database= "abcd"
sql_create = (
"DECLARE #sql AS NVARCHAR(MAX);"
"SET #sql = 'if not exists(select * from sys.databases where name = ' + QUOTENAME(?) + ')' + ' CREATE DATABASE ' + QUOTENAME(?);"
"EXEC sp_executesql #sql")
cursor.execute(sql_create,database,database)
Getting error msg like pyodbc.ProgrammingError: ('42S22', u"[42S22] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid column name 'abcd'. (207) (SQLExecDirectW)")"
Don't use QUOTENAME and concatenation for the WHERE clause parameter. Also, avoid using the legacy SQL Server ODBC driver that ships with Windows to access Azure SQL Database. Instead, download and use a newer ODBC driver. Below is an example with these changes.
conn = pyodbc.connect("DRIVER={ODBC Driver 17 for SQL Server};"
"SERVER="+server+";"
"UID="+username+";"
"PWD="+password,
autocommit=True)
cursor = conn.cursor()
database= "abcd"
sql_create = (
"DECLARE #sql AS NVARCHAR(MAX);"
"SET #sql = N'if not exists(select * from sys.databases where name = #DatabaseName)' + N' CREATE DATABASE ' + QUOTENAME(?) + N';';"
"EXEC sp_executesql #sql, N'#DatabaseName sysname', #DatabaseName = ?;")
cursor.execute(sql_create,database,database)
You could also declare a T-SQL variable for the database name and assign it to the parameter value so that you only need to pass a single parameter:
sql_create = (
"DECLARE #sql AS NVARCHAR(MAX);"
"DECLARE #DatabaseName sysname = ?;"
"SET #sql = N'if not exists(select * from sys.databases where name = #DatabaseName)' + N' CREATE DATABASE ' + QUOTENAME(#DatabaseName) + N';';"
"EXEC sp_executesql #sql, N'#DatabaseName sysname', #DatabaseName = #DatabaseName;")
cursor.execute(sql_create,database)
I have two tables, one in ms sql and other one in sqlite3
I need to update a field in mssql table to the related value from sqlite3 table.
here is my code:
import pypyodbc
import sqlite3
connection = pypyodbc.connect('Driver={SQL Server};'
'Server=****'
'Database=****;'
'uid=****;pwd=****')
cursor = connection.cursor()
SQLCommand = ("select ObjectId, ObjectColumnName,Label from LocalizedLabel "
"where LanguageId = 1065")
sqlUpdate = ("UPDATE LocalizedLabel "
"SET Label = ? "
"WHERE ObjectId = ? and ObjectColumnName = ? and LanguageId = 1065")
cursor.execute(SQLCommand)
results = cursor.fetchall()
srcDB = sqlite3.connect('crmLLDB')
for result in results:
val = (result[0], result[1])
print('SELECT "1065" FROM crm '
'WHERE UPPER(ObjectID)= UPPER(%s) AND ObjectColumnName = %s' % val)
srcCursor = srcDB.execute('SELECT "1065" FROM crm '
'WHERE UPPER(ObjectID)= UPPER(?) AND ObjectColumnName = ?', val)
if srcCursor.fetchone() is None:
print("No translation found for " + result[2])
else:
translation = srcCursor.fetchone()[0]
updateVals = (translation, result[0], result[1])
cursor.execute(sqlUpdate, updateVals)
connection.close()
srcDB.close()
objectId is a GuID and other fields are all strings
the print function returns:
SELECT "1065" FROM crm WHERE UPPER(ObjectID)= UPPER(b'0340B506-2341-DB11-898A-0007E9E17EBD') AND ObjectColumnName = DisplayName
while the watcher gives:
result[0] = 'b\\'0340B506-2341-DB11-898A-0007E9E17EBD\\''
result[1] = 'DisplayName'
this query of course returns this error:
[1] [SQLITE_ERROR] SQL error or missing database (near "'0340B506-2341-DB11-898A-0007E9E17EBD'": syntax error)
while this query :
SELECT "1065" FROM crm WHERE UPPER(ObjectID)= UPPER('0340B506-2341-DB11-898A-0007E9E17EBD') AND ObjectColumnName = 'DisplayName'
return the perfect answer,
can someone point out my problem, please?
appearaently the problem was with pypyodbc and its problem with GuId,
I used pyodbc and everything works fine now!
my final code, if some googler in the future passes by:
import pyodbc
import sqlite3
startTme = datetime.now()
connection = pyodbc.connect('Driver={SQL Server};'
'Server=*****;'
'Database=****;'
'uid=sa;pwd=*****')
cursor = connection.cursor()
SQLCommand = ("select ObjectId, ObjectColumnName,Label from LocalizedLabel "
"where LanguageId = 1065")
sqlUpdate_p = ("UPDATE LocalizedLabel "
"SET Label = ? "
"WHERE ObjectId = ? and ObjectColumnName = ? and LanguageId = 1065")
cursor.execute(SQLCommand)
results = cursor.fetchall()
srcDB = sqlite3.connect('crmLLDB')
jobLength = str(len(results))
i = 0
for result in results:
val = (result[0], result[1])
srcCursor = srcDB.execute('SELECT "1065" FROM crm '
'WHERE UPPER(ObjectID)= UPPER(?) AND ObjectColumnName = ?', val)
trans = srcCursor.fetchall()
for tr in trans:
updateVals = (tr[0], result[0], result[1])
cursor.execute(sqlUpdate_p, updateVals)
i += 1
connection.commit()
connection.close()
srcDB.close()