How to make Spark JDBC driver show full exception? - apache-spark

Question

I want to get an appropriate exception message from Spark JDBC driver.
My test case:
Using Spark ver2.0.2 to access DB through JDBC Driver
At first locked DB, then try write to DB (write mode are append and overwrite)
Get exception message
Exception message I got:
case1 append: Cause already initialized <-Not good
case2 overwrite: DB2 SQL Error: SQLCODE=-913, SQLSTATE=57033, SQLERRMC=00C9008E;00000210;DSN00009.DRITEMRI.00000001, DRIVER=4.19.56 <-Good
My question:
How can I get an exception message like DB2 SQL Error: SQLCODE=-913, SQLSTATE=57033~ in 'case1 append'?
I guss a reason is function savePartitions (this function is called when Spark execute saveTable) doesn't show good exception. But I don't know how to fix it.
Here is detail exception message of Spark shell
case1 append:
scala> prodtbl.write.mode("Append").jdbc(url3,"DB2.D_ITEM_INFO",prop1)
17/04/03 17:50:26 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.IllegalStateException: Cause already initialized
at java.lang.Throwable.setCause(Throwable.java:365)
at java.lang.Throwable.initCause(Throwable.java:341)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:241)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:300)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:299)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)
case2 overwrite:
scala> prodtbl.write.mode("Overwrite").jdbc(url3,"DB2.D_ITEM_INFO",prop1)
com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-913, SQLSTATE=57033, SQLERRMC=00C9008E;00000210;DSN00009.DRITEMRI.00000001, DRIVER=4.19.56
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.fp.c(Unknown Source)
at com.ibm.db2.jcc.am.fp.d(Unknown Source)
at com.ibm.db2.jcc.am.fp.b(Unknown Source)
at com.ibm.db2.jcc.t4.bb.i(Unknown Source)
at com.ibm.db2.jcc.t4.bb.c(Unknown Source)
at com.ibm.db2.jcc.t4.p.b(Unknown Source)
at com.ibm.db2.jcc.t4.vb.h(Unknown Source)
at com.ibm.db2.jcc.am.fp.jb(Unknown Source)
at com.ibm.db2.jcc.am.fp.a(Unknown Source)
at com.ibm.db2.jcc.am.fp.c(Unknown Source)
at com.ibm.db2.jcc.am.fp.executeUpdate(Unknown Source)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.dropTable(JdbcUtils.scala:94)
at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:422)
... 48 elided

Answer 1

So initCause described as below:
public Throwable initCause(Throwable cause)
Initializes the cause of this throwable to the specified value. (The cause is the throwable that caused this throwable to get thrown.)
This method can be called at most once. It is generally called from within the constructor, or immediately after creating the throwable. If this throwable was created with Throwable(Throwable) or Throwable(String,Throwable), this method cannot be called even once.
An example of using this method on a legacy throwable type without other support for setting the cause is:
try {
lowLevelOp();
} catch (LowLevelException le) {
throw (HighLevelException)
new HighLevelException().initCause(le); // Legacy constructor
}
Parameters:
cause - the cause (which is saved for later retrieval by the getCause() method). (A null value is permitted, and indicates that the cause is nonexistent or unknown.)
Returns:
a reference to this Throwable instance.
Throws:
IllegalArgumentException - if cause is this throwable. (A throwable cannot be its own cause.)
IllegalStateException - if this throwable was created with Throwable(Throwable) or Throwable(String,Throwable), or this method has already been called on this throwable.
And it means the cause was something not originated to initialize with initCase method and you haven't received what you waited for.