How to check path existence in spark? - apache-spark

Question

I have datetime paths as follow:
/data/2020-04-01/data.csv
/data/2020-04-02/data.csv
For some days, the file may not exist and If I call spark.read.load() function it fails. Is there a way to avoid this?

Answer 1

You can check like below,
import os.path
from os import path
if path.exists('/data/2020-04-01/data.csv'):
spark.read.load('/data/2020-04-01/data.csv')

Answer 2

In databricks you can use dbutils:
dbutils.fs.ls(path)
Using this function, you will get all the valid paths that exist. You can also use following hadoop library to get valid paths from hdfs:
org.apache.hadoop.fs

Answer 3

I assume you have a list of data paths and want to load data for the paths which exists on HDFS. You can pass your path to the get method in FileSystem. If the path doesn't exist it will throw IOException, and then you can proceed with the next set of paths.
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
// method to get files which exists
public static List<String> listFilesForBasePath(String path, Configuration conf) throws IOException {
FileSystem fs = FileSystem.get(new Path(path).toUri(), conf);
FileStatus[] items = fs.listStatus(new Path(path));
List<String> files = new ArrayList<>();
for (FileStatus item : items) {
files.add(item.getPath().toString());
}
return files;
}
// read files as :
Configuration configuration = sparkSession.sparkContext().hadoopConfiguration();
List<String> inputPaths = listFilesForBasePath(basePath, configuration)
Dataset<String> dataSet = sparkSession.read().textFile(inputPaths.toArray(new String[0]));

Answer 4

To fix issue in Simple way, Define your schema & apply that before loading. It will work without files in specified directory.
Check below code.
scala> import sys.process._
import sys.process._
scala> "ls -ltr D:\\tmp\\data\\customers".! // I don't have files inside directory.
total 0
Defining schema
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val schema = DataType.fromJson("""{"type":"struct","fields":[{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"age","type":"integer","nullable":false,"metadata":{}}]}""").asInstanceOf[StructType] // Definining schema
Reading files from empty directory without schema will fail.
scala> spark.read.load("D:\\tmp\\data\\customers") // Reading file without schema will fail.
org.apache.spark.sql.AnalysisException: Unable to infer schema for Parquet. It must be specified manually.;
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:185)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:185)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:184)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
... 61 elided
Reading files from empty directory with schema.
scala> spark.read.schema(schema).load("D:\\tmp\\data\\customers") // Reading files from directory with predefined schema.
res80: org.apache.spark.sql.DataFrame = [name: string, age: int]
scala> val df = spark.read.schema(schema).load("D:\\tmp\\data\\customers") // Reading files from directory with predefined schema.
df: org.apache.spark.sql.DataFrame = [name: string, age: int]
scala> df.count
res81: Long = 0