Caused by: kafka.common.OffsetOutOfRangeException - apache-spark

Question

I'm using Kafka and Spark to stream my data updates into my Hbase tables.
But I keep getting the OffsetOutOfRangeException, here's my code:
new KafkaStreamBuilder()
.setStreamingContext(streamingContext)
.setTopics(topics)
.setDataSourceId(dataSourceId)
.setOffsetManager(offsetManager)
.setConsumerParameters(
ImmutableMap
.<String, String>builder()
.putAll(kafkaConsumerParams)
.put("group.id", groupId)
.put("metadata.broker.list", kafkaBroker))
.build()
)
.build()
.foreachRDD(
rdd -> {
rdd.foreachPartition(
iter -> {
final Table hTable = createHbaseTable(settings);
try {
while (iter.hasNext()) {
String json = new String(iter.next());
try {
putRow(
hTable,
json,
settings,
barrier);
} catch (Exception e) {
throw new RuntimeException("hbase write failure", e);
}
}
} catch (OffsetOutOfRangeException e) {throw new RuntimeException(
"encountered OffsetOutOfRangeException: ", e);
}
});
});
I set my streaming job to run every 5 mins, and every time, after my consumers finish one batch of streaming, it'll write the latest markers and checkpoints to S3. And when next time, before the streaming job runs, it'll read the previous checkpoints and markers from S3, and start from there.
Here's the exception stacktrace:
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: kafka.common.OffsetOutOfRangeException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:86)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.handleFetchErr(KafkaRDD.scala:188)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:197)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:212)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
What I had done:
I've checked and markers and checkpoints are both working as expected.
So, I'm a bit lost here, how could this exception happen and what the possible/reasonable fix could it be?
Thanks!