how can i overcome the file not foundexception - apache-spark

Question

I am rying to read multiple excel files which under one directory, but i am encountered an error java.io.FileNotFoundException: File path/** does not exist
object example {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().appName("Excel to
DataFrame").master("local[2]").getOrCreate()
val path = "C:\\excel\\files"
val df = spark.read.format("com.crealytics.spark.excel")
.option("location", "true")
.option("useHeader", "true")
.option("treatEmptyValuesAsNulls", "true")
.option("inferSchema","true")
.option("addColorColumns", "true")
.option("timestampFormat", "MM-dd-yyyy HH:mm:ss")
.load("path")

Answer 1

Try this:
def readExcel(file: String): DataFrame = sqlContext.read
.format("com.crealytics.spark.excel")
.option("location", file)
.option("useHeader", "true")
.option("treatEmptyValuesAsNulls", "true")
.option("inferSchema", "true")
.option("addColorColumns", "False")
.load()
val data = readExcel("path to your excel file")
data.show(false)
If you want to read a particular sheet:
.option("sheetName", "Sheet2")
EDIT: To read multiple excel files into one dataframe. (provided the columns in the excel file are consistent)
For this I have used spark-excel package. It can be added to build.sbt file as:
libraryDependencies += "com.crealytics" %% "spark-excel" % "0.8.2"
The code is as follows:
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.{SparkSession, DataFrame}
import java.io.File
val conf = new SparkConf().setAppName("Excel to DataFrame").setMaster("local[*]")
val sc = new SparkContext(conf)
sc.setLogLevel("WARN")
val spark = SparkSession.builder().getOrCreate()
// Function to read xlsx file using spark-excel.
// This code format with "trailing dots" can be sent to Scala Console as a block.
def readExcel(file: String): DataFrame = spark.read.
format("com.crealytics.spark.excel").
option("location", file).
option("useHeader", "true").
option("treatEmptyValuesAsNulls", "true").
option("inferSchema", "true").
option("addColorColumns", "False").
load()
val dir = new File("path to your excel file")
val excelFiles = dir.listFiles.sorted.map(f => f.toString) // Array[String]
val dfs = excelFiles.map(f => readExcel(f)) // Array[DataFrame]
val ppdf = dfs.reduce(_.union(_)) // DataFrame
ppdf.count()
ppdf.show(5)
Hope this helps. Good luck.