Delta lake stream-stream left outer join using watermarks - apache-spark

Question

My aim here is to:
Read two delta lake tables as streams.
Add a timestamp column to the data frames.
Create a watermark using the timestamp columns.
Use the watermarks for an Outer left join.
Write the join as a stream to a delta lake table.
Here is my code:
from delta import *
from pyspark.sql.functions import *
import pyspark
my_working_dir = "/working"
spark = (
pyspark.sql.SparkSession.builder.appName("test_delta_stream").master("yarn").getOrCreate()
)
df_stream_1 = spark.readStream.format("delta").load(f"{my_working_dir}/table_1").alias("table_1")
df_stream_2 = spark.readStream.format("delta").load(f"{my_working_dir}/table_2").alias("table_2")
df_stream_1 = df_stream_1.withColumn("inserted_stream_1", current_timestamp())
df_stream_2 = df_stream_2.withColumn("inserted_stream_2", current_timestamp())
stream_1_watermark = df_stream_1.selectExpr(
"ID AS STREAM_1_ID", "inserted_stream_1"
).withWatermark("inserted_stream_1", "10 seconds")
stream_2_watermark = df_stream_2.selectExpr(
"ID AS STREAM_2_ID", "inserted_stream_2"
).withWatermark("inserted_stream_2", "10 seconds")
left_join_df = stream_1_watermark.join(
stream_2_watermark,
expr(
"""
STREAM_1_ID = STREAM_2_ID AND
inserted_stream_1 <= inserted_stream_2 + interval 1 hour
"""
),
"leftOuter",
).select("*")
left_join_df.writeStream.format("delta").outputMode("append").option(
"checkpointLocation", f"{my_working_dir}/output_delta_table/mycheckpoint"
).start(f"{my_working_dir}/output_delta_table")
Unfortunately, I get the error:
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "/some-dir/python/pyspark/sql/streaming.py", line 1493, in start
return self._sq(self._jwrite.start(path))
File "/some-dir/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
File "/some-dir/python/pyspark/sql/utils.py", line 117, in deco
raise converted from None
pyspark.sql.utils.AnalysisException: Stream-stream LeftOuter join between two streaming DataFrame/Datasets is not supported without a watermark in the join keys, or a watermark on the nullable side and an appropriate range condition;
It seems that even when I create watermarks, I still cannot do a left outer join. Any ideas why I am getting that error even though I clearly have defined a watermark on the joining columns?