PySpark is raising error (ORA-00933) when fetching data from Oracle Database - apache-spark

Question

Context
I am using Databricks to connect to an Oracle Database and fetch data daily. We are using the following example code in PySpark to authenticate and access the database:
testDF = (spark.read
.format("jdbc")
.option("url", "jdbc:oracle:thin:#10.80.40.5:1526:D301")
.option("dbtable", "SCHEMA_TEST.TABLE_TEST")
.option("user", jdbcUsername)
.option("password", jdbcPassword)
.option("driver", "oracle.jdbc.driver.OracleDriver")
.option("oracle.jdbc.timezoneAsRegion", "false")
.option("fetchSize", "256")
.load()
.createOrReplaceTempView("test")
)
Then we used to access the information via SQL code using the command below:
SELECT * FROM test
The problem
Today we realized this approach is raising an error like the one below:
SQLSyntaxErrorException: ORA-00933: SQL command not properly ended
Caused by: OracleDatabaseException: ORA-00933: SQL command not
properly ended
Upon further investigation into the error log, I saw the following message:
Caused by: Error : 933, Position : 2735, Sql = SELECT *
SCHEMA_TEST.TABLE_TEST LIMIT 10 , Error Msg = ORA-00933: SQL command
not properly ended
So I believe that PySpark/DBSQL translates the instruction SELECT * FROM test into the instruction SELECT * SCHEMA_TEST.TABLE_TEST LIMIT 10 to fetch information from the Oracle database.
The problem is that LIMIT does not exist in the ORACLE SQL Clause, therefore raising the error.
So I tried to implement an instruction with a WHERE clause. Because I thought that the translation would prevent to use of the LIMIT clause in the backend. The following code then was executed:
SELECT * FROM test WHERE column_example is not null
Well, IT WORKED. So for some reason, the PySpark/DBSQL is translating a simple select star instruction (SELECT * FROM test) WRONGLY.
Does someone know what might be the reason behind it?
Additional info:
Databricks Runtime: 12.0
Spark: 3.3.1
Oracle Driver: oracle.jdbc.driver.OracleDriver
Maven Oracle Driver: com.oracle.ojdbc:ojdbc8:19.3.0.0

Answer 1

Try an alternate approach by closing query in parentheses and passing query instead of calling entire table.
queryString = "(SELECT * SCHEMA_TEST.TABLE_TEST LIMIT 10)"
print("Query to fetch data from Oracle:: ", query)
# Fetch data from query
oracleDataDf = spark.read
.format("jdbc")
.option("url", oracleUrl)
.option("dbtable", queryString)
.option("user", oracleDbUser)
.option("password", oracleDbPass)
.option("driver", oracleDriver)
.load()
oracleDataDf.show()