Spark not saving all data to redshift - apache-spark

Question

The following code loads data from S3, cleans and removes duplicates using SparkSQL, and then saves data using JDBC to Redshift. I have also tried using spark-redshift maven dependency and getting same result. I am using Spark 2.0.
What I cannot understand is how when showing the result loaded in memory, the sum is the expected number, however when Spark saves to Redshift, it's always less. Somehow not all records are saved, and I do not see any errors in STL_LOAD_ERRORS either. Anybody encountered this or have any ideas to why this happens?
// Load files that were loaded into firehose on this day
var s3Files = spark.sqlContext.read.schema(schema).json("s3://" + job.getAWSAccessKey + ":" + job.getAWSSecretKey + "#" + job.getBucketName + "/"+ job.getAWSS3RawFileExpression + "/" + year+ "/" + monthCheck+ "/" + dayCheck + "/*/" ).rdd
// Apply the schema to the RDD, here we will have duplicates
val usersDataFrame = spark.createDataFrame(s3Files, schema)
usersDataFrame.createOrReplaceTempView("results")
// Clean and use partition by the keys to eliminate duplicates and get latest record
var results = spark.sql(buildCleaningQuery(job,"results"))
results.createOrReplaceTempView("filteredResults")
// This returns the correct result!
var check = spark.sql("select sum(Reward) from filteredResults where period=1706")
check.show()
var path = UUID.randomUUID().toString()
println("s3://" + job.getAWSAccessKey + ":" + job.getAWSSecretKey + "#" + job.getAWSS3TemporaryDirectory + "/" + path)
val prop = new Properties()
results.write.jdbc(job.getRedshiftJDBC,"work.\"" + path + "\"",prop)

Answer 1

Using jdbc means that Spark will be trying to do repeated INSERT INTO statements - which is massively slow in Redshift. That's why you're not seeing entries in stl_load_errors.
I'd suggest that you use the spark-redshift library instead. It's well tested and will perform better. https://github.com/databricks/spark-redshift
Example (showing many options):
my_dataframe.write
.format("com.databricks.spark.redshift")
.option("url", "jdbc:redshift://my_cluster.qwertyuiop.eu-west-1.redshift.amazonaws.com:5439/my_database?user=my_user&password=my_password")
.option("dbtable", "my_table")
.option("tempdir", "s3://my-bucket")
.option("diststyle", "KEY")
.option("distkey", "dist_key")
.option("sortkeyspec", "COMPOUND SORTKEY(key_1, key_2)")
.option("extracopyoptions", "TRUNCATECOLUMNS COMPUPDATE OFF STATUPDATE OFF")
.mode("overwrite") // "append" / "error"
.save()