JAVA_HOME is not set, cannot proceed. (Installing Apache Spark on Linux) - linux

Question

How I am here.
Running Command:
opt/develop/spark-3.3.1$ sudo ./dev/make-distribution.sh -pHADOOP-3 Dhadoop.version=3.2.4 -Pyarn --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes
The script (make-distribution.sh), apparently, is not reading the value of the variable i.e., $JAVA_HOME.
Running Command:
/opt/develop/spark-3.3.1$ echo $JAVA_HOME
Shows expected result: /usr/lib/jvm/jdk-19.0.1
Attempted:
Well, added to the file /opt/develop/spark-3.3.1/sbin/spark-config.sh
export JAVA_HOME="/usr/lib/jvm/jdk-19.0.1" export PATH=$PATH:$JAVA_HOME/bin
did not seem to change the result of Running command.
Logging-out and logging back in is also of no avail.
Note-1:
My JAVA_HOME is also set in /etc/profile.d/my-envvars file as:
export JAVA_HOME="/usr/lib/jvm/jdk-19.0.1" export PATH=$PATH:$JAVA_HOME/bin
Note-2: commands java --version and javac --version appears to be working okay.
The file: /opt/develop/dev/make-distribution.sh
Reads:
`
set -o pipefail
set -e
set -x
# Figure out where the Spark framework is installed
SPARK_HOME="$(cd "`dirname "$0"`/.."; pwd)"
DISTDIR="$SPARK_HOME/dist"
MAKE_TGZ=false
MAKE_PIP=false
MAKE_R=false
NAME=none
MVN="$SPARK_HOME/build/mvn"
function exit_with_usage {
set +x
echo "make-distribution.sh - tool for making binary distributions of Spark"
echo ""
echo "usage:"
cl_options="[--name] [--tgz] [--pip] [--r] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
echo "See Spark's \"Building Spark\" doc for correct Maven options."
echo ""
exit 1
}
# Parse arguments
while (( "$#" )); do
case $1 in
--tgz)
MAKE_TGZ=true
;;
--pip)
MAKE_PIP=true
;;
--r)
MAKE_R=true
;;
--mvn)
MVN="$2"
shift
;;
--name)
NAME="$2"
shift
;;
--help)
exit_with_usage
;;
--*)
echo "Error: $1 is not supported"
exit_with_usage
;;
-*)
break
;;
*)
echo "Error: $1 is not supported"
exit_with_usage
;;
esac
shift
done
if [ -z "$JAVA_HOME" ]; then # THIS SHOULD EVALUATE FALSE.
# Fall back on JAVA_HOME from rpm, if found
if [ $(command -v rpm) ]; then
RPM_JAVA_HOME="$(rpm -E %java_home 2>/dev/null)"
if [ "$RPM_JAVA_HOME" != "%java_home" ]; then
JAVA_HOME="$RPM_JAVA_HOME"
echo "No JAVA_HOME set, proceeding with '$JAVA_HOME' learned from rpm"
fi
fi
if [ -z "$JAVA_HOME" ]; then
if [ `command -v java` ]; then
# If java is in /usr/bin/java, we want /usr
JAVA_HOME="$(dirname $(dirname $(which java)))"
fi
fi
fi
if [ -z "$JAVA_HOME" ]; then # This SHOULD EVALUATE FALSE. APPARENTLY, IT DOES NOT
echo "Error: JAVA_HOME is not set, cannot proceed."
# I Should not be here, but I am!
exit -1
fi
`

Answer 1

Your problem lies within the fact that you're executing your script with sudo, which does not preserve environment variables by default.
As a small demonstration, consider the following:
someone#somewhere:~/my-path$ export SOMEVAR="testmyvar"
someone#somewhere:~/my-path$ echo $SOMEVAR
testmyvar
someone#somewhere:~/my-path$ sudo bash -c 'echo $SOMEVAR'
As you can see, when executing a command with sudo, you don't preserve all of your environment variables.
There is luckily quite an easy fix for that: use sudo -E (that does preserve environment variables, you can find that option in the man page of sudo)
So in our little example a bit higher:
someone#somewhere:~/my-path$ sudo -E bash -c 'echo $SOMEVAR'
testmyvar
You see that the -E option does what you want it to do!
Hope this fixes your problem :)