I tried using the py4j referred Connecting and testing a JDBC driver from Python
from py4j.JavaGateway import java_gateway
# Open JVM interface with the JDBC Jar
jdbc_jar_path = 'C:\Program Files\CData\CData JDBC Driver for MongoDB 2019\lib\cdata.jdbc.mongodb.jar'
gateway = java_gateway(classpath=jdbc_jar_path)
# Load the JDBC Jar
jdbc_class = "cdata.jdbc.mongodb.MongoDBDriver"
gateway.jvm.class.forName(jdbc_class)
# Initiate connection
jdbc_uri = "jdbc:mongodb:Server=127.0.0.1;Port=27017;Database=EmployeeDB;"
con = gateway.jvm.DriverManager.getConnection(jdbc_uri)
# Run a query
sql = "select * from Employees"
stmt = con.createStatement(sql)
rs = stmt.executeQuery()
while rs.next():
rs.getInt(1)
rs.getFloat(2)
.
.
rs.close()
stmt.close()
Getting error as
File "assignment.py", line 9
gateway.jvm.class.forName(jdbc_class)
^
SyntaxError: invalid syntax
Try replacing
gateway.jvm.class.forName(jdbc_class)
with
gateway.jvm.Class.forName(jdbc_class)
(i.e. capitalise the c in class.)
Class.forName is the Java method you want to call here. (Note also how the D in DriverManager is capitalised in gateway.jvm.DriverManager.getConnection(...).) However, the syntax error is caused because class is a Python keyword. You can't have a local variable, or a function or method, named class.
Related
I use Glue 3.0 - Supports Spark 3.1 and Python 3 from an infrastructure perspective. I am trying to do MERGE INTO target USING source operation in spark sql for a table UPSERT operation. However, I am getting the below error for the same:
An error occurred while calling o91.sql. MERGE INTO TABLE is not supported temporarily.
I am not using any Delta Table, I read directly from a postgreSQL - AuroraDb using spark dataframe reader which is my target. The source here is another dataframe read from parquet file using spark dataframe reader.
I have tried changing the Glue Version but it did not help. When I looked for answers in internet I get links to Iceberg and DeltaTable. Is my approach to the problem is correct. Please share you inputs.
The code is provided as below:
def changeDataCapture(inputDf, currDf, spark):
inputDf.createOrReplaceTempView('inputDf')
currDf.createOrReplaceTempView('currDf')
currDf = spark.sql("""
MERGE INTO currDf USING inputDf
ON currDf.REG_NB = inputDf.registerNumber
AND currDf.ANN_RTN_DT = inputDf.annual_return_date
WHEN MATCHED
THEN UPDATE SET
currDf.LAST_SEEN_DT = inputDf.LAST_SEEN_DT,
currDf.TO_DB_DT = inputDf.TO_DB_DT,
currDf.TO_DB_TM = inputDf.TO_DB_TM,
currDf.BATCH_ID = inputDf.BATCH_ID,
currDf.DATA_PROC_ID = inputDf.DATA_PROC_ID,
currDf.FIRST_SEEN_DT = CASE
WHEN currDf.CO_REG_DEBT = inputDf.registered_indebtedness
AND currDf.HLDR_LIST_CD = inputDf.holder_list_indicator
AND currDf.HLDR_LEGAL_STAT = inputDf.holder_legal_status
AND currDf.HLDR_REFRESH_CD = inputDf.holder_refresh_flag
AND currDf.HLDR_SUPRESS_IN = inputDf.HLDR_SUPRESS_IN
AND currDf.BULK_LIST_ID = inputDf.Bulk_List_In
THEN currDf.FIRST_SEEN_DT
ELSE inputDf.FIRST_SEEN_DT
END,
currDf.SUPERSEDED_DT = CASE
WHEN currDf.CO_REG_DEBT = inputDf.registered_indebtedness
AND currDf.HLDR_LIST_CD = inputDf.holder_list_indicator
AND currDf.HLDR_LEGAL_STAT = inputDf.holder_legal_status
AND currDf.HLDR_REFRESH_CD = inputDf.holder_refresh_flag
AND currDf.HLDR_SUPRESS_IN = inputDf.HLDR_SUPRESS_IN
AND currDf.BULK_LIST_ID = inputDf.Bulk_List_In
THEN currDf.SUPERSEDED_DT
ELSE inputDf.SUPERSEDED_DT
END
WHEN NOT MATCHED
THEN INSERT
(REG_NB, ANN_RTN_DT, SUPERSEDED_DT, TO_DB_DT, TO_DB_TM, FIRST_SEEN_DT, LAST_SEEN_DT, BATCH_ID,
DATA_PROC_ID, CO_REG_DEBT, HLDR_LIST_CD, HLDR_LIST_DT, HLDR_LEGAL_STAT,
HLDR_REFRESH_CD, HLDR_SUPRESS_IN, BULK_LIST_ID, DOC_TYPE_CD)
VALUES
(registerNumber, annual_return_date, SUPERSEDED_DT, TO_DB_DT, TO_DB_TM, FIRST_SEEN_DT, LAST_SEEN_DT,
BATCH_ID, DATA_PROC_ID, registered_indebtedness, holder_list_indicator,
holder_list_date, holder_legal_status, holder_refresh_flag, HLDR_SUPRESS_IN,
Bulk_List_In, DOC_TYPE_CD)
""")
return currDf
Thanks
I confirm that I have gone through multiple posts in StackOverflow with respect to similar problem, still stuck with the below problem, hence posting to seek guidance/pointers.
Following is the code
import pypyodbc as pyodbc
import configparser
config = configparser.ConfigParser()
config.read('config.ini')
conn_str = 'DRIVER={' + config['db2']['driver'] + '};' \
+ 'SERVER=' + config['db2']['server'] + ';' \
+ 'DATABASE=' + config['db2']['database'] + ';' \
+ 'UID=' + config['db2']['uid'] + ';' \
+ 'PWD=' + config['db2']['password']
print(conn_str)
connection = pyodbc.connect(
conn_str
)
cur = connection.cursor()
cur.execute('SELECT col_1, col_2 FROM schema.table_name LIMIT 2')
for row in cur:
print (row)
Output from code execution
[connect string output]
DRIVER={'IBM i Access ODBC Driver 64-bit'};SERVER='hostname';DATABASE='database';UID='userid';PWD='password'
[error from executing the code]
raise Error(state,err_text)
pypyodbc.Error: ('IM002', '[IM002] [unixODBC][Driver Manager]Data source name not found, and no default driver specified')
Configuration file
$ cat config.ini
[db2]
driver = 'IBM i Access ODBC Driver 64-bit'
server = 'hostname'
database = 'database'
uid = 'userid'
password = 'password'
Output of ODBC installer and uninstaller command
odbcinst -j
unixODBC 2.3.1
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /home/useradmin/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
cat /etc/odbcinst.ini
[PostgreSQL]
Description=ODBC for PostgreSQL
Driver=/usr/lib/psqlodbcw.so
Setup=/usr/lib/libodbcpsqlS.so
Driver64=/usr/lib64/psqlodbcw.so
Setup64=/usr/lib64/libodbcpsqlS.so
FileUsage=1
[MySQL]
Description=ODBC for MySQL
Driver=/usr/lib/libmyodbc5.so
Setup=/usr/lib/libodbcmyS.so
Driver64=/usr/lib64/libmyodbc5.so
Setup64=/usr/lib64/libodbcmyS.so
FileUsage=1
[IBM i Access ODBC Driver]
Description=IBM i Access for Linux ODBC Driver
Driver=/opt/ibm/iaccess/lib/libcwbodbc.so
Setup=/opt/ibm/iaccess/lib/libcwbodbcs.so
Driver64=/opt/ibm/iaccess/lib64/libcwbodbc.so
Setup64=/opt/ibm/iaccess/lib64/libcwbodbcs.so
Threading=0
DontDLClose=1
UsageCount=1
[IBM i Access ODBC Driver 64-bit]
Description=IBM i Access for Linux 64-bit ODBC Driver
Driver=/opt/ibm/iaccess/lib64/libcwbodbc.so
Setup=/opt/ibm/iaccess/lib64/libcwbodbcs.so
Threading=0
DontDLClose=1
UsageCount=1
$ cat ~/.odbc.ini
[db2]
Driver = IBM i Access ODBC Driver 64-bit
DATABASE = 'database'
SYSTEM = hostname
HOSTNAME = hostname
PORT = 446
PROTOCOL = TCPIP
$ isql db2 $username $password -v
[08001][unixODBC][IBM][System i Access ODBC Driver]The specified database can not be accessed at this time.
[ISQL]ERROR: Could not SQLConnect
I have double checked and confirm that there is no typo with driver name "IBM i Access ODBC Driver 64-bit"
OS information
x86_64 GNU/Linux
Any pointers/guidance on how to debug the issue, please?
I think you are confusing the schema name with the database name.
Odds are you can omit the database name completely (or leave it empty string and let it default to *SYSBAS). Instead, you can specify the DefaultLibraries argument.
See the IBM doc here for info on the valid connection string (and odbc.ini) keywords.
Similarly, you can omit the PORT, PROTOCOL, and HOSTNAME keywords, as they're not supported by this driver.
That will leave you with an odbc.ini that looks as simple as this:
[db2]
Driver = IBM i Access ODBC Driver 64-bit
DefaultLibraries = 'database'
SYSTEM = hostname
I'm trying to pull a SELECT query from within the python 3.6.5 code using sqlite3 in PyCharm.
The query has the double dot operator. Here is a sample code:
#!/usr/bin/python
import sqlite3
connection = sqlite3.connect('ddbserver')
c = connection.cursor()
sql_query = """ select * from database..tablename """
res = c.execute(sql_query)
When I run the script, it throws the following error:
sqlite3.OperationalError: near ".": syntax error
The same query runs fine on SQL Server Management Studio, as well as in an xml file.
I've tried using the full path to the table as well i.e.
select * from database.schemaname.tablename
But it throws the same error.
Any sort of help would be appreciated. Thank you!
I'm trying to run plain SQL with slick 3.1.0.
The following works:
val q = sql"select name from users".as[String]
however if my sql is in a variable:
val string2 : String = "select name from users"
how do I execute string2 using sql prefix? This doesn't work:
sql+string2
Use interpolation within the string:
val q = sql"#$string2"
The #$ interpolator will use the literal string you're interpolating, so don't use it for user input - it won't quote or anything.
See this section of the docs for more details.
I'm giving my first steps into HDBC using ODBC to connect to a local SQL Server.
After a quickQuery on the connection, I can't close it. I need to perform a commit first.
Is this the way it is supposed to be? Why is the commit necessary when I'm only performing a query?
On GHCi:
m + Database.HDBC Database.HDBC.ODBC
conn <- connectODBC "Driver={SQL Server};Server=thiagon\\sqlserver2012;Database=senior;UID=framework;PWD=framework;"
vals <- quickQuery conn "SELECT TOP 5 * FROM whatever;" []
print vals
commit conn
disconnect conn
If I remove the commit conn line, I get an exception:
*** Exception: SqlError {seState = "[\"25000\"]", seNativeError = -1, seErrorMsg = "disconnect: [\"0: [Microsoft][ODBC SQL Server Driver]Estado de transa\\65533\\65533o inv\\65533lido\"]"}
The message is in portuguese, it means "invalid transaction state".
A quickQuery could modify the table. I don't think the API analyses the string itself, or checks the database, to see whether or not the table was modified. And HDBC doesn't support autocommit.
You could use withTransaction, which will automatically handle this detail for you.
EDIT: Try using quickQuery', which is the strict version of quickQuery. In an example on http://book.realworldhaskell.org/read/using-databases.html (scroll down to ch21/query.hs), they didn't need a commit after a plain SELECT statement, but they were using quickQuery'.