NullPointerException while writing from Spark to Cassandra - cassandra

Question

I am using spark-cassandra-connector-2.4.0-s_2.11 to write data from spark to Cassandra on Databricks cluster.
I am getting java.lang.NullPointerException while writing data from Spark to Cassandra. This is working fine with few records.
But getting issue when I try to load ~150 Million records.
Can someone help me in finding the root cause?
Here is the code snippet:
val paymentExtractCsvDF = spark
.read
.format("csv")
.option("header", "true")
.load(/home/otl/extract/csvout/Payment)
paymentExtractCsvDF.printSchema()
root
|-- BAN: string (nullable = true)
|-- ENT_SEQ_NO: string (nullable = true)
|-- PYM_METHOD: string (nullable = true)
case class Payment(account_number: String, entity_sequence_number: String, payment_type: String)
val paymentResultDf = paymentExtractCsvDF.map(row => Payment(row.getAs("BAN"),
row.getAs("ENT_SEQ_NO"),
row.getAs("PYM_METHOD"))).toDF()
var paymentResultFilterDf = paymentResultDf
.filter($"account_number".isNotNull || $"account_number" != "")
.filter($"entity_sequence_number".isNotNull || $"entity_sequence_number" != "")
paymentResultFilterDf
.write
.format("org.apache.spark.sql.cassandra")
.mode("append")
.options(Map( "table" -> "cassandratable", "keyspace" -> "cassandrakeyspace"))
.save()
here is the exception I am getting:
Failed to write statements to cassandrakeyspace.cassandratable. The
latest exception was
An unexpected error occurred server side on /10.18.15.198:9042: java.lang.NullPointerException
Please check the executor logs for more exceptions and information
at com.datastax.spark.connector.writer.TableWriter$$anonfun$writeInternal$1$$anonfun$apply$3.apply(TableWriter.scala:243)
at com.datastax.spark.connector.writer.TableWriter$$anonfun$writeInternal$1$$anonfun$apply$3.apply(TableWriter.scala:241)
at scala.Option.map(Option.scala:146)
at com.datastax.spark.connector.writer.TableWriter$$anonfun$writeInternal$1.apply(TableWriter.scala:241)
at com.datastax.spark.connector.writer.TableWriter$$anonfun$writeInternal$1.apply(TableWriter.scala:210)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:112)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:111)
at com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:145)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:210)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:197)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:183)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139)
at org.apache.spark.scheduler.Task.run(Task.scala:112)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1526)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/11/22 01:12:17 INFO CoarseGrainedExecutorBackend: Got assigned task 1095
19/11/22 01:12:17 INFO Executor: Running task 39.1 in stage 21.0 (TID 1095)
19/11/22 01:12:17 INFO ShuffleBlockFetcherIterator: Getting 77 non-empty blocks including 10 local blocks and 67 remote blocks
19/11/22 01:12:17 INFO ShuffleBlockFetcherIterator: Started 7 remote fetches in 3 ms
19/11/22 01:12:17 INFO ShuffleBlockFetcherIterator: Getting 64 non-empty blocks including 8 local blocks and 56 remote blocks
19/11/22 01:12:17 INFO ShuffleBlockFetcherIterator: Started 7 remote fetches in 1 ms
19/11/22 01:12:17 ERROR Executor: Exception in task 7.0 in stage 21.0 (TID 1012)

Answer 1

It seems your dataframe has key fields with null values. The problem might be in you filter condition. I think you want to do something like this:
var paymentResultFilterDf = paymentResultDf
.filter($"account_number".isNotNull && $"account_number" != "")
.filter($"entity_sequence_number".isNotNull && $"entity_sequence_number" != "")