I am not able to launch Spark web UI using the docker container. I am using this link
I am running docker-compose up -d.Post which I am trying to access the Spark UI in local host,its getting exited in Docker desktop
On further digging,I see the logs
: No such file or directorypt/spark/bin/load-spark-env.sh
/start-spark.sh: line 12: syntax error near unexpected token `elif'
'start-spark.sh: line 12: `elif [ "$SPARK_WORKLOAD" == "worker" ];
I double-checked the start-spark.sh script but didn't find any issues. Is it due to CRLF? Contents of start-spark.sh is below
#!/bin/bash
. "/opt/spark/bin/load-spark-env.sh"
if [ "$SPARK_WORKLOAD" == "master" ];
then
export SPARK_MASTER_HOST=`hostname`
cd /opt/spark/bin && ./spark-class org.apache.spark.deploy.master.Master --ip $SPARK_MASTER_HOST --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT >> $SPARK_MASTER_LOG
elif [ "$SPARK_WORKLOAD" == "worker" ];then
cd /opt/spark/bin && ./spark-class org.apache.spark.deploy.worker.Worker --webui-port $SPARK_WORKER_WEBUI_PORT $SPARK_MASTER >> $SPARK_WORKER_LOG
elif [ "$SPARK_WORKLOAD" == "submit" ];
then
echo "SPARK SUBMIT"
else
echo "Undefined Workload Type $SPARK_WORKLOAD, must specify: master, worker, submit"
fi
Related
Hi, all
I am trying to implement automated test running from Makefile target. As my test dependent on running docker container, I need to check that container is up and running during whole test execution, and restart it, if it's down. I am trying do it with bash script, running in background mode.
At glance code looks like this:
run-tests:
./check_container.sh & \
docker-compose run --rm tests; \
#Need to finish check_container.sh right here, after tests execution
RESULT=$$?; \
docker-compose logs test-container; \
docker-compose kill; \
docker-compose rm -fv; \
exit $$RESULT
Tests has vary execution time (from 20min to 2hrs), so I don't know before, how much time it will take. So, I try to poll it within script longer, than the longest test suite. Script looks like:
#!/bin/bash
time=0
while [ $time -le 5000 ]; do
num=$(docker ps | grep selenium--standalone-chrome -c)
if [[ "$num" -eq 0 ]]; then
echo 'selenium--standalone-chrome container is down!';
echo 'try to recreate'
docker-compose up -d selenium
elif [[ "$num" -eq 1 ]]; then
echo 'selenium--standalone-chrome container is up and running'
else
docker ps | grep selenium--standalone-chrome
echo 'more than one selenium--standalone-chrome containers is up and running'
fi
time=$(($time + 1))
sleep 30
done
So, I am looking for how to exit script exactly after test running is finished, its meant, after command docker-compose run --rm tests is finished?
P.S. It is also ok, if background process can be finished on Makefile target finish
Docker (and Compose) can restart containers automatically when they exit. If your docker-compose.yml file has:
version: '3.8'
services:
selenium:
restart: unless-stopped
Then Docker will do everything your shell script does. If it also has
services:
tests:
depends_on:
- selenium
then the docker-compose run tests line will also cause the selenium container to be started, and you don't need a script to start it at all.
When you launch a command in the background, the special parameter $! contains its process ID. You can save this in a variable and later kill(1) it.
In plain shell-script syntax, without Make-related escaping:
./check_container.sh &
CHECK_CONTAINER_PID=$!
docker-compose run --rm tests
RESULT=$?
kill "$CHECK_CONTAINER_PID"
I am writing a script which will check docker service but I want check services which are inside the docker-compose without getting into it.
For ex: we have custom services like tass and inception basically we check it's status by this command "service tass status"
Is there any way to check this services in Docker-compose
You can use:
docker-compose ps <name>
docker-compose top <name>
docker-compose logs <name>
To "check" the service with name <name>. You can learn many more commands by doing `docker-compose --help.
Finally, you can run docker-compose exec <name> <shell> and get an interactive shell inside the cont inaner, which with some unix-utilities will allow you to "check" the container further with ease.
Finally, you can extract the name of the running container as in How do I retrieve the exact container name started by docker-compose.yml , and use any of the docker commands mentioned in the other answer to "check". From docker inspect <the container name> you can get the cgroup name and mounted filesystem, which you can "check".
Docker compose is only a tool to build docker images.
You should rely on docker commands in order to check each service health, for example:
docker ps
docker stat
docker inspect
docker container ls
In this How to check if the docker engine and a docker container are running? thread you can find a lot of alternatives about container checking.
Checking for .State.Status, .State.Running, etc. will tell you if it's running, but it's better to ensure that the health of your containers. Below is a script that you can run that will make sure that two containers are in good health before executing a command in the 2nd container. It prints out the docker logs if the wait time/attempts threshold has been reached.
Example taken from npm sql-mdb.
#!/bin/bash
# Wait for two docker healthchecks to be in a "healthy" state before executing a "docker exec -it $2 bash $3"
##############################################################################################################################
# $1 Docker container name that will wait for a "healthy" healthcheck (required)
# $2 Docker container name that will wait for a "healthy" healthcheck and will be used to run the execution command (required)
# $3 The actual execution command that will be ran (required)
attempt=0
health1=checking
health2=checking
while [ $attempt -le 79 ]; do
attempt=$(( $attempt + 1 ))
echo "Waiting for docker healthcheck on services $1 ($health1) and $2 ($health2): attempt: $attempt..."
if [[ health1 != "healthy" ]]; then
health1=$(docker inspect -f {{.State.Health.Status}} $1)
fi
if [[ $health2 != "healthy" ]]; then
health2=$(docker inspect -f {{.State.Health.Status}} $2)
fi
if [[ $health1 == "healthy" && $health2 == "healthy" ]]; then
echo "Docker healthcheck on services $1 ($health1) and $2 ($health2) - executing: $3"
docker exec -it $2 bash -c "$3"
[[ $? != 0 ]] && { echo "Failed to execute \"$3\" in docker container \"$2\"" >&2; exit 1; }
break
fi
sleep 2
done
if [[ $health1 != "healthy" || $health2 != "healthy" ]]; then
echo "Failed to wait for docker healthcheck on services $1 ($health1) and $2 ($health2) after $attempt attempts"
docker logs --details $1
docker logs --details $2
exit 1
fi
I wrote a bash script to build a docker container (a NODE JS app) at the end so that the application runs on a DEV server. The thing is that the bash script has to be run in stages across multiple servers i.e., we need to build the docker container in local, ssh to a jump box and rsync it over there and then rsync it from the jumpbox to the dev server and then ssh to it again (if this process is done manually).
If you look at step 5 in the BASH script below, the rsync gets completed and the script runs to SSH to the jumpbox server, however, the script stops over there and I am just logged in to the jumpbox.
Could someone let me know what is wrong with this bash script and how should I fix it?
Thank you in advance.
Best,
R
#!/bin/bash
#This script allows users to deploy the application with minimal work
echo -n "Shall we begin the deployment process? Are you ready to rule the world with QA Hero's next version? `\n` If you are, then type YES and press [ENTER]?: "
read begin
echo "Wohoooo! You did the right thing! We are now ready to roll and you can actually see your terminal scroll! Lol, what a troll!"
echo -n "Yo mate! Can you enter your enumber and press [ENTER]?: "
read enumber
docker build --build-arg NODE_ENV=staging -t course-hero-x -f Dockerfile .
echo "The file has been built! Step 1 completed"
termdown 3
echo "BOOM! Get ready for step 2."
docker save -o course-hero.tar course-hero-x
termdown 3
echo "BOOM! Get ready for step 3."
rsync -avzh --progress --stats course-hero.tar `echo ${enumber}`#10.188.129.25:/home/`echo ${enumber}`
termdown 3
echo "BOOM! Get ready for step 4."
ssh `echo ${enumber}`#10.188.129.25
termdown 3
echo "BOOM! Get ready for step 5."
ls -la
termdown 3
echo "BOOM! Get ready for step 6."
if ["$enumber" == "e30157" || "$enumber" == "E30167"]; then
rsync -avzh --progress --stats course-hero.tar `echo ${enumber^^}`#10.80.63.65:/home/eh7/`echo ${enumber^^}`
elif ["$enumber" == "e32398" || "$enumber" == "E32398"]; then
rsync -avzh --progress --stats course-hero.tar `echo ${enumber^^}`#10.80.63.65:/home/eh8/`echo ${enumber^^}`
else
echo "You cannot access this system."
fi
termdown 3
echo "BOOM! Get ready for step 7."
ssh `echo ${enumber^^}`#10.80.63.65
echo "BOOM! Get ready for step 8."
ls -la
echo "BOOM! Get ready for step 9."
sudo docker load -i course-hero.tar
termdown 3
echo "BOOM! Get ready for step 10."
sudo docker ps
termdown 3
echo "BOOM! Get ready for step 11."
sudo docker stop $(sudo docker ps -a -q)
termdown 3
echo "BOOM! Get ready for step 12."
sudo docker rm $(sudo docker ps -a -q)
termdown 3
echo "BOOM! Get ready for step 13."
sudo docker load -i course-hero.tar
termdown 3
echo "BOOM! Get ready for step 14."
sudo docker run -d -e TZ=Australia/Melbourne --net=courseHero -p 0.0.0.0:80:3000 course-hero-x
echo "QA Hero is up and running. Go to http://cohero-dev.rmit.edu.au to checkout the latest version!"
termdown 3
echo "Step 15 completed! We are done here! See you next time homey!"
Once you've open an ssh session to the other box your local script stops because it's waiting for the ssh session to end. You're just left sitting at a command prompt on the remote box.
It looks like what you're intending to have happen is the commands following your ssh run on the remote box. To send the commands to the remote machine redirect them:
ssh ${enumber}#10.188.129.25 <<'ENDSSH'
# series of commands to
# run on remote host
ENDSSH
Consider running your script through Shellcheck — you've got several problems with it beyond the one you asked about.
I want to run a bunch of spark jobs in parallel through yarn, then wait for all of them to finish before launching another set of jobs. How can I find out when my first set of jobs has finished? Thank you.
Sample Work Around;
Give your spark job a unique name in spark-submit command.
spark-submit --master yarn-cluster --name spark_job_name job1.jar
Check on yarn, spark job is running or not. If not running run your second job. Bash script below
JOB="spark_job_name"
applicationId=$(yarn application -list -appStates RUNNING | awk -v tmpJob=$JOB '{ if( $2 == tmpJob) print $1 }')
if [ ! -z $applicationId ]
then
echo " "
echo "JOB: ${JOB} is already running. ApplicationId : ${applicationId}"
echo " "
else
printf "first job is not running. Starting the spark job. ${JOB}\n"
echo " "
spark-submit --master yarn-cluster --name spark_job_name2 job2.jar
echo " "
fi
I have a node server running within a vagrant. The script that starts node 'start.sh' can pick up a debug flag from a file, which I named debug.mode
On the local side I have a script 'startdebug.sh' which logs into vagrant over ssh, writes to the debug.mode file, restarts the script, waits till that is done, then tunnels 5858.
If I run the start.sh file with debug.mode containing '--debug' node opens 5858 and the port is available (I'm checking within vagrant using telnet).
If I do the same using startdebug.sh node says it's opened the debugging port, however the 5858 port it unavailable when I try telnet'ing within the VM.
Any idea? :)
startdebug.sh
/usr/bin/vagrant ssh-config > /tmp/vagrant-ssh-config
ssh -F /tmp/vagrant-ssh-config nodejs "cd /var/www/sportsbook-api && echo $mode > debug.mode && export TERM=linux && sudo ./scripts/restart.sh"
sleep 2.5s
ssh -N -F /tmp/vagrant-ssh-config -L 5858:127.0.0.1:5858 nodejs &
start.sh
if [ -e "debug.mode" ]; then
debug=$(cat "debug.mode")
echo "\nNode $debug mode activated."
fi
nohup node src/main/apps/api & echo $! > run.pid &