How to share state between runs of streaming jobs? - apache-spark

Question

I have a Spark streaming job triggered every day using Trigger.Once method due to business requirements.
StreamingQuery query = processed
.writeStream()
.outputMode("append")
.format("parquet")
.option("path", resultPath)
.option("checkpointLocation", checkpointLocationPathForDate)
.trigger(Trigger.Once())
.start();
I am using map flatMapGroupsWithState so that we can store state (GroupState) for grouped data.
Somewhere I read checkpointLocation should be different for every StreamingQuery. Therefore I use a checkpointLocation like this: /path/to/nfs/checkpoint/<current date in format: yyyyMMdd>
Every day, Spark job processes files in the folder /path/to/data/<current date in format: yyyyMMdd>
I want to access the state of the yesterday's Spark job since yesterday's data may contain relevant state that is needed in today's data.
However, Spark stores state data in checkpointLocation i.e /path/to/nfs/checkpoint/<current date in format: yyyyMMdd>/<queryName>/state so when different checkpointLocation is used, it is not possible to access it.
So, how can I access the GroupState data stored at checkpointLocation of previous Spark job? Is it OK to use same checkpointLocation for different StreamingQueries?
Edit:
I tried to use same checkpointLocation for yesterday's StreamingQuery and today's StreamingQuery and Spark restored state of yesterday's batch which is I want however is this documented anywhere? Is this expected behaviour or is misbehaving possible when same checkpointLocation is used between daily batches?
Edit2:
Data is stored at S3, in parquet format, path: s3a://bucket/batchdata/year=2022/month=01/day=19/
Sample data for 2022-01-19:
s3a://bucket/batchdata/year=2022/month=01/day=19/a.parquet
s3a://bucket/batchdata/year=2022/month=01/day=19/b.parquet
s3a://bucket/batchdata/year=2022/month=01/day=19/c.parquet
Data is read using Spark parquet readStream method:
// .parquet(...) is called for 2022-01-19
Dataset<Row> dataset = spark
.readStream()
.schema(PARQUET_SCHEMA)
.parquet("s3a://bucket/batchdata/year=2022/month=01/day=19/");
Dataset<Row> processed = dataset.groupByKey(keyFuncion,encoder)
.flatMapGroupsWithState(flatMapStateFunc,
OutputMode.Append(),
stateEncoder,
outputEncoder,
GroupStateTimeout.ProcessingTimeTimeout());
StreamingQuery query = processed.writeStream()
.outputMode("append")
.format("parquet")
.option("path", resultPath)
.option("checkpointLocation", checkpointLocation)
.trigger(Trigger.Once())
.start();
query.awaitTermination();
Next day same job is run with next day's parquet files stored under:
s3a://bucket/batchdata/year=2022/month=01/day=20/
Sample data for 2022-01-20:
s3a://bucket/batchdata/year=2022/month=01/day=20/d.parquet
s3a://bucket/batchdata/year=2022/month=01/day=20/e.parquet
// .parquet(...) is called for 2022-01-20
Dataset<Row> dataset = spark
.readStream()
.schema(PARQUET_SCHEMA)
.parquet("s3a://bucket/batchdata/year=2022/month=01/day=20/");
Dataset<Row> processed = dataset.groupByKey(keyFuncion,encoder)
.flatMapGroupsWithState(flatMapStateFunc,
OutputMode.Append(),
stateEncoder,
outputEncoder,
GroupStateTimeout.ProcessingTimeTimeout());
StreamingQuery query = processed.writeStream()
.outputMode("append")
.format("parquet")
.option("path", resultPath)
.option("checkpointLocation", checkpointLocation)
.trigger(Trigger.Once())
.start();
query.awaitTermination();

Answer 1

how can I access the GroupState data stored at checkpointLocation of previous Spark job?
You should not. Technically, you could (with some extra coding) but there are so many things specific to the other query (e.g., stateful operator IDs) that you should take into account. Use at your own risk.
Is it OK to use same checkpointLocation for different StreamingQueries?
No. You should not share the same checkpointLocation between different streaming queries. One is that they different with their operators so the numbers may not match and, even if they did, the sinks could be different and hence some data could get skipped (as already processed).
I tried to use same checkpointLocation for yesterday's StreamingQuery and today's StreamingQuery and Spark restored state of yesterday's batch which is I want however is this documented anywhere? Is this expected behaviour or is misbehaving possible when same checkpointLocation is used between daily batches?
That's documented and that's exactly how checkpointLocation is supposed to work. It's the directory with the state of a streaming query at a given time.
Quoting Recovering from Failures with Checkpointing:
In case of a failure or intentional shutdown, you can recover the previous progress and state of a previous query, and continue where it left off. This is done using checkpointing and write-ahead logs. You can configure a query with a checkpoint location, and the query will save all the progress information (i.e. range of offsets processed in each trigger) and the running aggregates (e.g. word counts in the quick example) to the checkpoint location. This checkpoint location has to be a path in an HDFS compatible file system, and can be set as an option in the DataStreamWriter when starting a query.