I need to modify LD_LIBRARY_PATH JAVA_LIBRARY_PATH and CLASSPATH before running hadoop job at cluster. In LD_LIBRARY_PATH and JAVA_LIBRARY_PATH i need to add location of some jars which are required while running the job, As these jars are available at my cluster, similar with CLASSPATH.
I have a 3 NODE cluster, I need to modify this LD_LIBRARY_PATH and CLASSPATH for all the 3 data nodes in such a way that jars available at my cluster node at added to classpath, so that the following jar are available while running the job as i am avoiding jar distribution while running the job to use all ready available jar on cluster nodes. I have tried the given below options
1.I have tried modifying hadoop-env.sh to modify CLASSPATH
export HADOOP_TASKTRACKER_OPTS="-classpath:/opt/oracle/oraloader-2.0.0-2/jlib/
but the above thing modify HADOOP_CLASSPATH not the CLASSPATH
For LD_LIBRARY_PATH and JAVA_LIBRARY_PATH i have tired adding given below property in mapred-site.xml as suggested a my place but that didn't work.
< property >
< name >mapred.child.env< /name >
< value >JAVA_LIBRARY_PATH=/opt/oracle/oraloader-2.0.0-2/lib/< /value >
< value >LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/oracle/oraloader-2.0.0-2/lib/< /value >
< description>User added environment variables for the task tracker child processes. Example : 1) A=foo This will set the env variable A to foo 2) B=$B:c This is inherit tasktracker's B env variable.
< /description>
< /property>
I have also restarted my all 3 data nodes,all tasktrakers and 2 NAMENOdes. Still these variables are not set and my hadoop job is not able to find all those jar files required for running the test.
ERROR LOG::
Error: java.lang.ClassNotFoundException:
oracle.i18n.text.OraDateFormat at
java.net.URLClassLoader$1.run(URLClassLoader.java:202)
When i do echo HADOOP_CLASSPATH at my cluster nodes, all the required for running hadoop job are coming. But i think the following jars need to be added in JAVA_LIBRARY_PATH but it not coming.
Do not reinvent the wheel.
If your implementation uses ToolRunner (which you really should use if you implement map-reduce in Java) then you can use -libjars jar1,jar2 to ship your jars to the cluster.
Check out "Side Data Distribution" section in "Hadoop: The definitive guide" by Tom White.
Related
I am trying to run the spark-submit command on my Hadoop cluster Here is a summary of my Hadoop Cluster:
The cluster is built using 5 VirtualBox VM's connected on an internal network
There is 1 namenode and 4 datanodes created.
All the VM's were built from the Bitnami Hadoop Stack VirtualBox image
I am trying to run one of the spark examples using the following spark-submit command
spark-submit --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/jars/spark-examples_2.12-3.0.3.jar 10
I get the following error:
[2022-07-25 13:32:39.253]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher
I get the same error when trying to run a script with PySpark.
I have tried/verified the following:
environment variables: HADOOP_HOME, SPARK_HOME and HADOOP_CONF_DIR have been set in my .bashrc file
SPARK_DIST_CLASSPATH and HADOOP_CONF_DIR have been defined in spark-env.sh
Added spark.master yarn, spark.yarn.stagingDir hdfs://hadoop-namenode:8020/user/bitnami/sparkStaging and spark.yarn.jars hdfs://hadoop-namenode:8020/user/bitnami/spark/jars/ in spark-defaults.conf
I have uploaded the jars into hdfs (i.e. hadoop fs -put $SPARK_HOME/jars/* hdfs://hadoop-namenode:8020/user/bitnami/spark/jars/ )
The logs accessible via the web interface (i.e. http://hadoop-namenode:8042 ) do not provide any further details about the error.
This section of the Spark documentation seems relevant to the error since the YARN libraries should be included, by default, but only if you've installed the appropriate Spark version
For with-hadoop Spark distribution, since it contains a built-in Hadoop runtime already, by default, when a job is submitted to Hadoop Yarn cluster, to prevent jar conflict, it will not populate Yarn’s classpath into Spark. To override this behavior, you can set spark.yarn.populateHadoopClasspath=true. For no-hadoop Spark distribution, Spark will populate Yarn’s classpath by default in order to get Hadoop runtime. For with-hadoop Spark distribution, if your application depends on certain library that is only available in the cluster, you can try to populate the Yarn classpath by setting the property mentioned above. If you run into jar conflict issue by doing so, you will need to turn it off and include this library in your application jar.
https://spark.apache.org/docs/latest/running-on-yarn.html#preparations
Otherwise, yarn.application.classpath in yarn-site.xml refers to local filesystem paths in each of ResourceManager servers where JARs are available for all YARN applications (spark.yarn.jars or extra packages should get layered onto this)
Another problem could be file permissions. You probably shouldn't put Spark jars into an HDFS user folder if they're meant to be used by all users. Typically, I'd put it under hdfs:///apps/spark/<version>, then give that 744 HDFS permissions
In the Spark / YARN UI, it should show the complete classpath of the application for further debugging
I figured out why I was getting this error. It turns out that I made an error while specifying spark.yarn.jars in spark-defaults.conf
The value of this property must be
hdfs://hadoop-namenode:8020/user/bitnami/spark/jars/*
instead of
hdfs://hadoop-namenode:8020/user/bitnami/spark/jars/
i.e. Basically, we need to specify the jar files as the value to this property and not the folder containing the jar files.
I'm trying to submit a cluster-mode spark 2 application from a Java Spring app using InProcessLauncher. I was previously using the SparkLauncher class, which worked, but it fires up a long-lived SparkSubmit java process for each job, which was eating up too many resources with lots of jobs in play.
My code sets sparkLauncher.setMaster("yarn") and sparkLauncher.setDeployMode("cluster")
I set the HADOOP_CONF_DIR env variable to the directory containing my config (yarn-site.xml etc) before starting my Spring app, and it logs that it is picking up this variable:
INFO System Environment - HADOOP_CONF_DIR = /etc/hadoop/conf
Yet when it comes to submitting, I see INFO o.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032 - i.e. it is using the default 0.0.0.0 rather than the actual ResourceManager IP, and of course it fails. It seems not to be picking up the Hadoop config.
I can submit jobs from the same shell directly using spark-submit, and even by directly invoking java -cp /usr/hdp/current/spark2-client/conf/:/usr/hdp/current/spark2-client/jars/*:/etc/hadoop/conf/ org.apache.spark.deploy.SparkSubmit .... So I'm not sure why my Spring App isn't picking up the same config.
I managed to get my app to pick up the hadoop config by adding the conf folders to the classpath. This is something spark-submit does for you when launching as a separate process, but doesn't happen when using InProcessLauncher.
Because my Spring Boot app is launched using -jar xxx.jar, I couldn't use -cp on the command line (cannot be combined with -jar), but had to add it to the manifest in the jar. I did this by adding the following to build.gradle (which is using the Spring Boot gradle plugin):
bootJar {
manifest {
attributes 'Class-Path': '/usr/hdp/current/spark2-client/conf/ /etc/hadoop/conf/'
}
}
While analyzing the yarn launch_container.sh logs for a spark job, I got confused by some part of log.
I will point out those asks step by step here
When you will submit a spark job with spark-submit having --pyfiles and --files on cluster mode on YARN:
The config files passed in --files , executable python files passed in --pyfiles are getting uploaded into .sparkStaging directory created under user hadoop home directory.
Along with these files pyspark.zip and py4j-version_number.zip from $SPARK_HOME/python/lib is also getting copied
into .sparkStaging directory created under user hadoop home directory
After this launch_container.sh is getting triggered by yarn and this will export all env variables required.
If we have exported anything explicitly such as PYSPARK_PYTHON in .bash_profile or at the time of building the spark-submit job in a shell script or in spark_env.sh , the default value will be replaced by the value which we
are providing
This PYSPARK_PYTHON is a path in my edge node.
Then how a container launched in another node will be able to use this python version ?
The default python version in data nodes of my cluster is 2.7.5.
So without setting this pyspark_python , containers are using 2.7.5.
But when I will set pyspark_python to 3.5.x , they are using what I have given.
It is defining PWD='/data/complete-path'
Where this PWD directory resides ?
This directory is getting cleaned up after job completion.
I have even tried to run the job in one session of putty
and kept the /data folder opened in another session of putty to see
if any directories are getting created on run time. but couldn't find any?
It is also setting the PYTHONPATH to $PWD/pyspark.zip:$PWD/py4j-version.zip
When ever I am doing a python specific operation
in spark code , its using PYSPARK_PYTHON. So for what purpose this PYTHONPATH is being used?
3.After this yarn is creating softlinks using ln -sf for all the files in step 1
soft links are created for for pyspark.zip , py4j-<version>.zip,
all python files mentioned in step 1.
Now these links are again pointing to '/data/different_directories'
directory (which I am not sure where they are present).
I know soft links can be used for accessing remote nodes ,
but here why the soft links are created ?
Last but not the least , whether this launch_container.sh will run for each container launch ?
Then how a container launched in another node will be able to use this python version ?
First of all, when we submit a Spark application, there are several ways to set the configurations for the Spark application.
Such as:
Setting spark-defaults.conf
Setting environment variables
Setting spark-submit options (spark-submit —help and —conf)
Setting a custom properties file (—properties-file)
Setting values in code (exposed in both SparkConf and SparkContext APIs)
Setting Hadoop configurations (HADOOP_CONF_DIR and spark.hadoop.*)
In my environment, the Hadoop configurations are placed in /etc/spark/conf/yarn-conf/, and the spark-defaults.conf and spark-env.sh is in /etc/spark/conf/.
As the order of precedence for configurations, this is the order that Spark will use:
Properties set on SparkConf or SparkContext in code
Arguments passed to spark-submit, spark-shell, or pyspark at run time
Properties set in /etc/spark/conf/spark-defaults.conf, a specified properties file
Environment variables exported or set in scripts
So broadly speaking:
For properties that apply to all jobs, use spark-defaults.conf,
for properties that are constant and specific to a single or a few applications use SparkConf or --properties-file,
for properties that change between runs use command line arguments.
Now, regarding the question:
In Cluster mode of Spark, the Spark driver is running in container in YARN, the Spark executors are running in container in YARN.
In Client mode of Spark, the Spark driver is running outside of the Hadoop cluster(out of YARN), and the executors are always in YARN.
So for your question, it is mostly relative with YARN.
When an application is submitted to YARN, first there will be an ApplicationMaster container, which nigotiates with NodeManager, and is responsible to control the application containers(in your case, they are Spark executors).
NodeManager will then create a local temporary directory for each of the Spark executors, to prepare to launch the containers(that's why the launch_container.sh has such a name).
We can find the location of the local temporary directory is set by NodeManager's ${yarn.nodemanager.local-dirs} defined in yarn-site.xml.
And we can set yarn.nodemanager.delete.debug-delay-sec to 10 minutes and review the launch_container.sh script.
In my environment, the ${yarn.nodemanager.local-dirs} is /yarn/nm, so in this directory, I can find the tempory directories of Spark executor containers, they looks like:
/yarn/nm/nm-local-dir/container_1603853670569_0001_01_000001.
And in this directory, I can find the launch_container.sh for this specific container and other stuffs for running this container.
Where this PWD directory resides ?
I think this is a special Environment Variable in Linux OS, so better not to modify it unless you know how it works percisely in your application.
As per above, if you export this PWD environment at the runtime, I think it is passed to Spark as same as any other Environment Variables.
I'm not sure how the PYSPARK_PYTHON Environment Variable is used in Spark's launch scripts chain, but here you can find the instruction in the official documentation, showing how to set Python binary executable while you are using spark-submit:
spark-submit --conf spark.pyspark.python=/<PATH>/<TO>/<FILE>
As for the last question, yes, YARN will create a temp dir for each of the containers, and the launch_container.sh is included in the dir.
I am following the instruction here for setting up dynamic allocation for YARN resource manager.
However, I am confused by step 3: Add this jar to the classpath of all NodeManagers in your cluster.
Does this mean go to each node server and add the path to shuffle.jar to PATH environment variable? export=$PATH:<loc-to-shuffle.jar>?
Yarn classpath means that on all node managers, either set the yarn.application.classpath in yarn-site.xml which contains comma-separated list of CLASSPATH entries.
When this value is empty, the following default CLASSPATH for YARN applications would be used.
For Linux:
$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, $HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
For Windows:
%HADOOP_CONF_DIR%, %HADOOP_COMMON_HOME%/share/hadoop/common/*, %HADOOP_COMMON_HOME%/share/hadoop/common/lib/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*
So put spark-<version>-yarn-shuffle.jar in one of the listed classpath directories defined in yarn.application.classpath or the default classpath directories.
You can also create the soft link of spark-<version>-yarn-shuffle.jar in one of the yarn classpath directories
Hope this helps...
I am able to do a spark-submit to my cloudera cluster. the job dies after a few minutes with exceptions complaining it can not find various classes. These are classes that are in the spark dependency path. I keep adding the jars one at a time using command line args --jars, the yarn log keeps dumping out the next jar it can't find.
What setting allows the spark/yarn job to find all the dependent jars?
I already set the "spark.home" attribute to the correct path - /opt/cloudera/parcels/CDH/lib/spark
I found it!
remove
.set("spark.driver.host", "driver computer ip address")
from your driver code.