How do I refresh a HDFS path? - apache-spark

Question

I am runing a sparksession in jupyter notebook .
I would got error sometime on a dataframe which is initial by spark.read.parquet(some_path) when files under that path have changed, even if I cache the dataframe .
For example
reading code is
sp = spark.read.parquet(TB.STORE_PRODUCT)
sp.cache()
sometimes, sp can't not be access anymore, complain :
Py4JJavaError: An error occurred while calling o3274.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 10 in stage 326.0 failed 4 times, most recent failure: Lost task 10.3 in stage 326.0 (TID 111818, dc38, executor 7): java.io.FileNotFoundException: File does not exist: hdfs://xxxx/data/dm/sales/store_product/part-00000-169428df-a9ee-431e-918b-75477c073d71-c000.snappy.parquet
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
The problem
'REFRESH TABLE tableName' doesn't work, because
I don't have a hive table, it is only a hdfs path
Restart sparksession and read that path again can solve this problem , but
I don't want to restart sparksession, it would waste much time
One more
execute sp = spark.read.parquet(TB.STORE_PRODUCT) again doesn't work , I can understand why, spark should scan the path again or there must be a option/setting to force it scan . Keep whole path location in memory is not smart .
spark.read.parquet doesn't have a force scan option
Signature: spark.read.parquet(*paths)
Docstring:
Loads Parquet files, returning the result as a :class:`DataFrame`.
You can set the following Parquet-specific option(s) for reading Parquet files:
* ``mergeSchema``: sets whether we should merge schemas collected from all Parquet part-files. This will override ``spark.sql.parquet.mergeSchema``. The default value is specified in ``spark.sql.parquet.mergeSchema``.
>>> df = spark.read.parquet('python/test_support/sql/parquet_partitioned')
>>> df.dtypes
[('name', 'string'), ('year', 'int'), ('month', 'int'), ('day', 'int')]
.. versionadded:: 1.4
Source:
#since(1.4)
def parquet(self, *paths):
"""Loads Parquet files, returning the result as a :class:`DataFrame`.
You can set the following Parquet-specific option(s) for reading Parquet files:
* ``mergeSchema``: sets whether we should merge schemas collected from all \
Parquet part-files. This will override ``spark.sql.parquet.mergeSchema``. \
The default value is specified in ``spark.sql.parquet.mergeSchema``.
>>> df = spark.read.parquet('python/test_support/sql/parquet_partitioned')
>>> df.dtypes
[('name', 'string'), ('year', 'int'), ('month', 'int'), ('day', 'int')]
"""
return self._df(self._jreader.parquet(_to_seq(self._spark._sc, paths)))
File: /opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/readwriter.py
Type: method
Is there a proper way to solve my problem ?

Answer 1

The problem is caused by Dataframe.cache .
I need clear that cache at first , then read again would solve the problem
code :
try:
sp.unpersist()
except:
pass
sp = spark.read.parquet(TB.STORE_PRODUCT)
sp.cache()

Answer 2

You can try two solutions
one is to unpersist the dataframe before reading everytime as suggested by #Mithril
or just create a temp view and trigger the refresh command
sp.createOrReplaceTempView('sp_table')
spark.sql('''REFRESH TABLE sp_table''')
df=spark.sql('''select * from sp_table''')