I am newbie on spark and was trying to execute my jar on the cluster. while i run the jar, the job fails with error Error: invalid log directory /usr/share/spark-2.1.1/work//7/
In another thread where it was suggested to remove flag : SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true". I also changed the permission of the log directory on all the workers.
Nothing helped. can someone help me understanding why it fails.
Related
I'm running a spark job using spark-submit. At the end of the day, I'm also getting some output but the log file shows
WARN SparkEnv:87 - Exception while deleting Spark temp dir: ......
java.io.IOException: Failed to delete: ...
I've created a temp directory and have pointed spark.local.dir to this new path with the spark-submit command
would this exception affect my output.
I'll be using the same tmp directory while running other spark jobs. Will it affect them
and is there anyway I can avoid this?
This is expected on windows, you can ignore it and you can disable the warning using this in your log4j.properties file:
log4j.logger.org.apache.spark.util.ShutdownHookManager=OFF
log4j.logger.org.apache.spark.SparkEnv=ERROR
No it won't affect your output
It won't affect other output
Either ignore the message using the log4j.properties or troubleshoot why the tmp directory can't be deleted.
I am getting this error, how can I run these Spark jobs (in Scala)?
Command:
bin/run-example /home/datadotz/streaming/wc_str.scala localhost 9999
Error:
Failed to load org.apache.spark.examples./home/datadotz/streami
java.lang.ClassNotFoundException: org.apache.spark.examples.
Start with the documentation -- https://spark.apache.org/docs/latest/#running-the-examples-and-shell
To run one of the Java or Scala sample programs, use bin/run-example [params] in the top-level Spark directory
It also mentions you can use spark-submit to run programs, which seems to take a path. Try that script instead.
I am trying to run an R script buy piping out an RDD to an Rscript using Spark's pipe(). I am using sc.addFile() to copy the Rscript to the executor's memory.
sc.addFile(rScript) and using SparkFiles.get(rName) to get the file name.
While running the job cluster mode, I am getting below error
Cannot run program "/data/tmp/spark-b8b8053e-0110-4ddb-91a3-
ae6f0f633c68/userFiles-78ed11c0-483b-4615-88eb-
8d1c97571997/RSCRIPT_NAME.R": error=2, No such file
or directory
But the file is getting copied to /data/tmp/spark-b8b8053e-0110-4ddb-91a3-
ae6f0f633c68/userFiles-78ed11c0-483b-4615-88eb-
8d1c97571997 location.
Not sure how to fix this issue.
I think you are trying to execute it as
rdd.pipe("scriptName.R")
Please add "./" before you call the script.
rdd.pipe("./scriptName.R")
I'm using spark in HDInsight with Jupyter notebook. I'm using the %%configure "magic" to import packages. Every time there is a problem with the package, spark crashes with the error:
The code failed because of a fatal error: Status 'shutting_down' not
supported by session..
or
The code failed because of a fatal error: Session 28 unexpectedly
reached final status 'dead'. See logs:
Usually the problem was with me mistyping the name of the package, so after a few attempts I could solve it. Now I'm trying to import spark-streaming-eventhubs_2.11 and I think I got the name right, but I still receive the error. I looked at all kinds of logs but still couldn't find the one which shows any relevant info. Any idea how to troubleshoot similar errors?
%%configure -f
{ "conf": {"spark.jars.packages": "com.microsoft.azure:spark-streaming-eventhubs_2.11:2.0.5" }}
Additional info: when I run
spark-shell --conf spark.jars.packages=com.microsoft.azure:spark-streaming-eventhubs_2.11:2.0.5
The shell starts fine, and downloads the package
I finally was able to find the log files which contain the error. There are two log files which could be interesting
Livy log: livy-livy-server.out
Yarn log
On my HDInsight cluster, I found the livy log by connecting to one of the Head nodes with SSH and downloading a a file at this path (this log didn't contain useful info):
/var/log/livy/livy-livy-server.out
The actual error was in the yarn log file accessible from YarnUI. In HDInsight Azure Portal, go to "Cluster dashboard" -> "Yarn", find your session (KILLED status), click on "Logs" in the table, find "Log Type: stderr", click "click here for full log".
The problem in my case was Scala version incompatibility between one of the dependencies of spark-streaming_2.11 and Livy. This is supposed to be fixed Livy 0.4. More info here
I have been following this tutorial in order to set up Zeppelin on a Spark cluster (version 1.5.2) in HDInsight, on Linux. Everything worked fine, I have managed to successfully connect to the Zeppelin notebook through the SSH tunnel. However, when I try to run any kind of paragraph, the first time I get the following error:
java.io.IOException: No FileSystem for scheme: wasb
After getting this error, if I try to rerun the paragraph, I get another error:
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
These errors occur regardless of the code I enter, even if there is no reference to the hdfs. What I'm saying is that I get the "No FileSystem" error even for a trivial scala expression, such as parallelize.
Is there a missing configuration step?
I am download the tar ball that the script that you pointed to as I type. But want I am guessing is that your zeppelin install and spark install are not complete to work with wasb. In order to get spark to work with wasb you need to add some jars to the Class path. To do this you need to add something like this to your spark-defaults.conf (the paths might be different in HDInsights, this is from HDP on IaaS)
spark.driver.extraClassPath /usr/hdp/2.3.0.0-2557/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.3.0.0-2557/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.0.0-2557/hadoop/hadoop-azure-2.7.1.2.3.0.0-2557.jar
spark.executor.extraClassPath /usr/hdp/2.3.0.0-2557/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.3.0.0-2557/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.0.0-2557/hadoop/hadoop-azure-2.7.1.2.3.0.0-2557.jar
Once you have spark working with wasb, or next step is make those sames jar in zeppelin class path. A good way to test your setup is make a notebook that prints your env vars and class path.
sys.env.foreach(println(_))
val cl = ClassLoader.getSystemClassLoader
cl.asInstanceOf[java.net.URLClassLoader].getURLs.foreach(println)
Also looking at the install script, it trying to pull the zeppelin jar from wasb, you might want to change that config to somewhere else while you try some of these changes out. (zeppelin.sh)
export SPARK_YARN_JAR=wasb:///apps/zeppelin/zeppelin-spark-0.5.5-SNAPSHOT.jar
I hope this helps, if you are still have problems I have some other ideas, but would start with these first.