Input string error after a spark-submit - string

Question

I'm trying to run some Spark Scala code :
import org.apache.spark.{SparkConf, SparkContext}
import scala.collection.mutable.ListBuffer
object EzRecoMRjobs {
def main(args: Array[String]) {
val conf = new SparkConf()
conf.setMaster("local")
conf.setAppName("Product Cardinalities")
val sc = new SparkContext(conf)
val dataset = sc.textFile(args(0))
// Load parameters
val customerIndex = args(1).toInt - 1
val ProductIndex = args(2).toInt - 1
val outputPath = args(3).toString
val resu = dataset.map( line => { val orderId = line.split("\t")(0)
val cols = line.split("\t")(1).split(";")
cols(ProductIndex)
})
.map( x => (x,1) )
.reduceByKey(_ + _)
.saveAsTextFile(outputPath)
sc.stop()
}
}
This code works in Intellij and write the result in the "outputPath" folder.
From my Intellij project I have generate a .jar file and I want to run this code with a spark-submit. So in my terminal I launch :
spark-submit \
--jars /Users/users/Documents/TestScala/ezRecoPreBuild/target/ezRecoPreBuild-1.0-SNAPSHOT.jar \
--class com.np6.scala.EzRecoMRjobs \
--master local \
/Users/users/Documents/DATA/data.txt 1 2 /Users/users/Documents/DATA/dossier
But I got this error :
Exception in thread "main" java.lang.NumberFormatException: For input string: "/Users/users/Documents/DATA/dossier"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
at com.np6.scala.EzRecoMRjobs$.main(ezRecoMRjobs.scala:51)
at com.np6.scala.EzRecoMRjobs.main(ezRecoMRjobs.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
What's the reason of this error ? Thank's

Answer 1

Check out the documentation: https://spark.apache.org/docs/latest/submitting-applications.html
The first application argument is expected to be the jar file path, so obviously your are getting a NumberFormatException, because your code is parsing the last argument (which is a String) as a number.
The --jars flag is used to specify additional jars that will be used in your application.
You must run the spark-submit command in this way:
spark-submit \
--class com.np6.scala.EzRecoMRjobs \
--master local[*] \
/Users/users/Documents/TestScala/ezRecoPreBuild/target/ezRecoPreBuild-1.0-SNAPSHOT.jar /Users/users/Documents/DATA/data.txt 1 2 /Users/users/Documents/DATA/dossier
Hope it helps.