Structured Streaming from Kafka to Hbase - need to set custom timestamps - apache-spark

Question

I am using
Spark 2.2 (from HDP 2.6.3)
Kafka 1.0.1
HBase 1.1.2 (from HDP 2.6.3)
shc-core-1.1.2-2.2-s_2.11-SNAPSHOT.jar built manually with an additional scala class HBaseSinkProvider (link to a github issue)
I want to use this stack to create a kind of a realtime ETL:
I have millions objects of one type (for example "customers")
These object have different fields (like Name, Surname, Status, Email etc) and fields Version and LastUpdateDate
When an object changes, its full description with all fields and values is published to Kafka topic
With Spark I have a Structured Streaming app, which consumes the stream of data and save object to HBase
Everything above works well now, the code is like this (Python):
spark \
.readStream \
.format('kafka') \
.option('kafka.bootstrap.servers', 'worker01:9092,worker02:9092,worker03:9092') \
.option('subscribe', 'youdo') \
.option('group.id', 'spark') \
.option('maxOffsetsPerTrigger', 100) \
.option('startingOffsets', 'earliest') \
.load() \
.withColumn(
'decoded',
from_json(
col('value').cast('string'),
schema
)
) \
.select(
'decoded.body.*',
'timestamp'
) \
.na.fill('null') \
.writeStream \
.outputMode("append") \
.format('HBase.HBaseSinkProvider') \
.option('hbasecat', catalog) \
.option('checkpointLocation', '/tmp/checkpoint') \
.start() \
.awaitTermination()
Now I have a problem that I have no guarantee, that messages in different partitions are in proper order. I mean for example, that O may have a version 8 of an object in HBase, and I can consume a message from another partition with version 7 - in this case I must not update the data in HBase.
First I tried to join the input stream with HBase and filter out rows with version lower, than in HBase:
<...>
.select(
'decoded.body.*',
'timestamp'
) \
.join(
sqlc.read \
.format('org.apache.spark.sql.execution.datasources.hbase') \
.options(catalog=catalog_hbase) \
.load() \
.select('id', col('hbase_version').cast('integer')),
['id'],
'left'
) \
.na.fill({'hbase_version': 0}) \
.filter(col('version').cast('integer') > col('hbase_version'))
<...>
Doing like this I always get streamingQueryException: key not found: hbase_version - I think this doesn't work due to shc, it looks like a not supported feature.
Another way, that I see, is to set HBase row timestamp manually for each row (i have the LastUpdateDate attribute along with Version) - how can I do that?
Are there any other ways?
The resulting code must be in Python, but it is OK to compile some jars and pass them to script with spark-submit.