How to call oracle stored procedure in queries? - apache-spark

Question

In my spark project , I am using spark-sql-2.4.1v.
As part of my code , I need to call oracle stored procs in my spark job.
We are converting an old project into spark that has got lot of logic based on oracle stored procs. The middleware logic we are converting to spark ... so want to keep the procs logic as is , as there are other application uses them...hence need to call existing procs in spark code.
how to call oracle stored procs?

Answer 1

cx_Oracle module in python can be used to call a oracle stored procedure from python / pyspark scripts.
Documentation is here - https://cx-oracle.readthedocs.io/en/latest/user_guide/plsql_execution.html
If for any reason, cx_Oracle does not work in the hadoop environment (as it requires oracle client binaries installed), we can use below Spark JDBC method.
In PySpark JDBC option, there is a property called sessionInitStatement which can be used to execute a custom SQL statement or a PL/SQL block before the JDBC process starts reading the data. This option in a spark JDBC read can be used to call a stored proc as below.
Here first we execute the PL/SQL proc using sessionInitStatement and then read the final data set from stored proc using spark/jdbc read.
from pyspark.sql import SparkSession, HiveContext
spark = (SparkSession.builder.enableHiveSupport().getOrCreate())
# Provide PL/SQL code here - call the stored proc within BEGIN and END block.
plsql_block = """
BEGIN
SCHEMA.STORED_PROC_NAME;
END;
"""
# Read the final table that is created / updated within the stored proc.
count_query = """
(
select count(*) from SCHMA.TABLE_NAME
) t1
"""
df = spark.read \
.format("jdbc") \
.option("url", "JDBC_URL") \
.option('driver', 'oracle.jdbc.driver.OracleDriver') \
.option("oracle.jdbc.timezoneAsRegion", "false") \
.option("sessionInitStatement", plsql_block) \
.option("dbtable", count_query) \
.option("user", "USER_ID") \
.option("password", "PASSWORD") \
.load()
print("Total Records")
df.show(10, False)

Answer 2

Spark SQL can be used to read and write from/to a Oracle table, in other words you can do select,insert and delete. You can do update by doing delete+insert. You can call a function as part of the SQL too. But I don't think you can call a stored procedure using Spark SQL but you can do that using plain old java/scala/python syntax. The strategy I use is to use Spark SQL to populate a table then run a stored procedure based on that table using standard JDBC connection and java code. The driver will execute this stored procedure in a single thread and obviously you wouldn't expect scalability here.
spark.write.mode(SaveMode.append).jdbc(jdbcURL, tableName, jdbcProperties)
val con = DriverManager.getConnection(jdbcURL)
//execute the stored procedure using JDBC connection
con.close()

Answer 3

You can try doing something like this, though I have never tried this personally in any implementation
query = "exec SP_NAME"
empDF = spark.read \
.format("jdbc") \
.option("url", "jdbc:oracle:thin:username/password#//hostname:portnumber/SID") \
.option("dbtable", query) \
.option("user", "db_user_name") \
.option("password", "password") \
.option("driver", "oracle.jdbc.driver.OracleDriver") \
.load()