How to access the SparkSession from the worker? - apache-spark

Question

I'm using spark structured streaming in databricks. In that, I'm using foreach operation to perform some operations on every record of the data. But the function which I'm passing in foreach uses SparkSession but it's throwing an error: _pickle.PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers.
So, Is there any way to use SparkSession inside foreach?
EDIT #1:
One example of function passed in foreach would be something like:
def process_data(row):
df = spark.createDataFrame([row])
df.write.mode("overwrite").saveAsTable("T2")
spark.sql("""
MERGE INTO T1
USING T2 ON T2.type="OrderReplace" AND T1.ReferenceNumber=T2.originalReferenceNumber
WHEN MATCHED THEN
UPDATE SET
shares = T2.shares,
price = T2.price,
ReferenceNumber = T2.newReferenceNumber
""")
So, I need SparkSession here which is not available inside foreach.

Answer 1

From the description of the tasks you want to perform row by row on your streaming dataset, you can try using foreachBatch.
A streaming dataset may have thousands to millions of records and if you are hitting external systems on each record, it is a massive overhead. From your example, it seems you are updating a base data set T1 based on events in T2.
From the link above, you can do something like below,
def process_data(df, epoch_id):
df.write.mode("overwrite").saveAsTable("T2")
spark.sql("""
MERGE INTO T1
USING T2 ON T2.type="OrderReplace" AND T1.ReferenceNumber=T2.originalReferenceNumber
WHEN MATCHED THEN
UPDATE SET
shares = T2.shares,
price = T2.price,
ReferenceNumber = T2.newReferenceNumber
""")
streamingDF.writeStream.foreachBatch(process_data).start()
Some additional points regarding what you are doing,
You are trying to do a foreach on a streaming data set and overwriting T2 hive table. Since records are processed parallel in clusters-managers like yarn, there might be a corrupted T2 since multiple tasks maybe updating it.
If you are persisting records in T2 (even temporarily), why don't you try making the merge/ update logic as a separate batch process? This will be similar to the foreachBatch solution I wrote above.
Updating/ Creating Hive tables fire a map/reduce job ultimately which requires resource negotiations, etc., etc. and may be painfully slow if done on each record, you may want to consider a different type of destination like a JDBC based one for example.