DataStax Enterprise: Spark Cassandra Batch Size - cassandra

Question

I set the parameter spark.cassandra.output.batch.size.rows in my SparkConf as following:
val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "host")
.set("spark.cassandra.auth.username", "cassandra")
.set("spark.cassandra.auth.password", "cassandra")
.set("spark.cassandra.output.batch.size.rows", "5120")
.set("spark.cassandra.output.concurrent.writes", "10")
but when i perform
saveToCassandra("data","ten_days")
I continue to see warning in my system.log
NFO [FlushWriter:7] 2014-11-20 11:11:16,498 Memtable.java (line 395) Completed flushing /var/lib/cassandra/data/system/hints/system-hints-jb-76-Data.db (5747287 bytes) for commitlog position ReplayPosition(segmentId=1416480663951, position=44882909)
INFO [FlushWriter:7] 2014-11-20 11:11:16,499 Memtable.java (line 355) Writing Memtable-ten_days#1656582530(32979978/329799780 serialized/live bytes, 551793 ops)
WARN [Native-Transport-Requests:761] 2014-11-20 11:11:16,499 BatchStatement.java (line 226) Batch of prepared statements for [data.ten_days] is of size 36825, exceeding specified threshold of 5120 by 31705.
WARN [Native-Transport-Requests:777] 2014-11-20 11:11:16,500 BatchStatement.java (line 226) Batch of prepared statements for [data.ten_days] is of size 36813, exceeding specified threshold of 5120 by 31693.
WARN [Native-Transport-Requests:822] 2014-11-20 11:11:16,501 BatchStatement.java (line 226) Batch of prepared statements for [data.ten_days] is of size 36823, exceeding specified threshold of 5120 by 31703.
WARN [Native-Transport-Requests:835] 2014-11-20 11:11:16,500 BatchStatement.java (line 226) Batch of prepared statements for [data.ten_days] is of size 36817, exceeding specified threshold of 5120 by 31697.
WARN [Native-Transport-Requests:781] 2014-11-20 11:11:16,501 BatchStatement.java (line 226) Batch of prepared statements for [data.ten_days] is of size 36817, exceeding specified threshold of 5120 by 31697.
WARN [Native-Transport-Requests:755] 2014-11-20 11:11:16,501 BatchStatement.java (line 226) Batch of prepared statements for [data.ten_days] is of size 36822, exceeding specified threshold of 5120 by 31702.
I know that are only warnings, but I would like to understand why my settings aren't working as expected. Then I can see a lot of hints in my cluster. Could the batch size affects the number of hints in the cluster ?
Thanks

Answer 1

You have set batch size rows instead of batch size bytes. This means the connector is limiting on the amount of rows rather than the memory size of the batch.
spark.cassandra.output.batch.size.rows: number of rows per single
batch; default is 'auto' which means the connector will adjust the
number of rows based on the amount of data in each row
spark.cassandra.output.batch.size.bytes: maximum total size of the
batch in bytes; defaults to 64 kB.
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md
On a more important note you are most likely going to be better off with a larger batch size (64kb) and changing the warn limit in the cassandra.yaml file.
Edit:
Recently we've seen that larger batches can cause instability with certain C* configurations so lower the value if the system becomes unstable.