java.io.FileNotFoundException: File does not exist even if I cache - apache-spark

Question

I have a long pyspark script.
At the start of the script, I read a user_table, and need to use it many times all along.
Sometimes the underlying files in the relevant partitions get updated (by an outside script of data team), and the script will fail with java.io.FileNotFoundException.
I would totally understand if I didn't cache: Spark always goes back to the source and gets it from there. But I explicitly cache, and show 1, to initiate the caching.
a.user_profile_df = user_profile_df.cache()
a.user_profile_df.show(1)
So wondering, how could the update of underlying files cause this error, if the data is already cached? It would mean it wants to read the data from the source files, but then what is the point of caching?
# Set the default spark-shell log level to WARN. When running the spark-shell, the21/07/24 12:57:31 ERROR TaskSetManager: Task 9 in stage 98.0 failed 4 times; aborting job
# Set the default spark-shell log level to WARN. When running the spark-shell, the21/07/24 12:57:31 WARN TaskSetManager: Lost task 16.1 in stage 98.0 (TID 10131, ip-12-345-67-8.id-element.io, executor 47): java.io.FileNotFoundException: File does not exist: hdfs://R2/projects/data_usermart/hive/user_tables/user_table_micro/region=AB/date=2021-07-23/part-00005-5ed5074f-3199-487d-9d3f-20dce24f4a59.c000
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.scan_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:1124)
at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:1130)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.writeIteratorToStream(PythonUDFRunner.scala:50)
at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:346)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:195)
My solution now is to
save down the relevant data I need from that table as CSV on the HDFS and
read it in and then refer on that all along the script,
but there has to be a better way.

Answer 1

#Chris, I am not clear or convinced about caching. In my case, like you mentinoed as a temporary solution, after I copied the file from local linux file system to hdfs, the java.io.FileNotFoundException was gone and it started working. This is what I did -
hdfs dfs -cp file:///<path_to_local_file> /<hdfs_file_dir>