How to convert java Resultset into Spark dataframe - apache-spark

Question

I am trying to use preparestatement with JDBC. It results ResultSet object. I want to convert it into spark dataframe.
object JDBCRead {
val tableName:String = "TABLENAME"
val url :String = "jdbc:teradata://TERADATA_URL/user=USERNAME,password=PWD,charset=UTF8,TYPE=FASTEXPORT,SESSIONS=10"
val selectTable:String = "SELECT * FROM " + tableName +" sample 10";
val con : Connection = DriverManager.getConnection(url);
val pstmt2: PreparedStatement = con.prepareStatement(selectTable)
import java.sql.ResultSet
val rs: ResultSet = pstmt2.executeQuery
val rsmd: ResultSetMetaData = rs.getMetaData
while(rs.next()!=null)
{
val k: Boolean = rs.next()
for(i<-1 to rsmd.getColumnCount) {
print(" " + rs.getObject(i))
}
println()
}
}
I want to call above code from Spark Dataframe so that I can load the data into dataframe and get the results faster distributedly.
I must use PreparedStatement. I can not use spark.jdbc.load since FASTEXPORT of Teradata does not work with jdbc load. It has to be used with PreparedStatement
How to achieve this? How can I user preparestatement along with SELECT statement to load into Spark Dataframe.

Answer 1

-
AFAIK there are 2 options available for this kind of requirements
1. DataFrame 2. JdbcRDD
I'd offer JdbcRDD (since you are so specific to preparedstatement)
Which used prepareStatement internally in compute method. Therefore, you don't need to create connection and maintain it explicitly(error prone).
Later you can convert result in to dataframe
For speed you can configure other parameters.
Example code usage of JdbcRDD is below..
import org.apache.log4j.{Level, Logger}
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext.__
import org.apache.spark.SparkConf
import org.apache.spark.rdd.JdbcRDD
import java.sql.{connection, DriverManager,ResultSet}
object jdbcRddExample {
def main(args: Array[String]) {
// Connection String
VAL URL = "jdbc:teradata://SERVER/demo"
val username = "demo"
val password = "Spark"
Class.forName("com.teradata.jdbc.Driver").newInstance
// Creating & Configuring Spark Context
val conf = new SparkConf().setAppName("App1").setMaster("local[2]").set("spark.executor.memory",1)
val sc = new SparkContext(conf)
println("Start...")
// Fetching data from Database
val myRDD = new JdbcRDD(sc,() => DriverManager.getConnection(url,username,password),
"select first_name, last_name, gender from person limit ?,?",
3,5,1,r => r.getString("last_name") + "," +r.getString("first_name"))
// Displaying the content
myRDD.foreach(println)
// Saving the content inside Text File
myRDD.saveAsTextFile("c://jdbcrdd")
println("End...")
}
}