I'm on CentOS 6.9 running slurm 17.11.7. I've modified my /gpfs0/export/slurm/conf/epilog script. I'm ultimately would like to print out job resource utilization information to the stdout file used be each users' job.
I've been testing it within the conditional at the end of the script for myself before I roll it out to other users. Below is my modified epilog script:
#!/bin/bash
# Clear out TMPDIR on the shared file system after job completes
exec >> /var/log/epilog.log
exec 2>> /var/log/epilog.log
if [ -z $SLURM_JOB_ID ]
then
echo -e " This script should be executed from slurm."
exit 1
fi
TMPDIR="/gpfs0/scratch/${SLURM_JOB_ID}"
rm -rf $TMPDIR
### My additions to the existing script ###
if [ "$USER" == "myuserid" ]
then
STDOUT=`scontrol show jobid ${SLURM_JOB_ID} | grep StdOut | awk 'BEGIN{FS="="}{print $2}'`
# Regular stdout/stderr is not respected, must use python.
python -c "import sys; stdout=sys.argv[1]; f=open(stdout, 'a'); f.write('sticks\n'); f.close();" ${STDOUT}
fi
exit 0
From the Prolog and Epilog section of the slurm.conf user manual it seems that stdout/stderr are not respected. Hence I modify the stdout file with python.
I've picked the compute node node21 to run this job, so I logged into node21 and tried several things to get it to notice my changes to the epilog script.
Reconfiguring slurmd:
sudo scontrol reconfigure
Restart slurm daemon:
sudo service slurm stop
sudo service slurm start
Neither of which seems to get the changes to the epilog script when I submit jobs. When put the same conditional in a batch script it runs flawlessly:
#!/bin/bash
#SBATCH --nodelist=node21
echo "Hello you!"
echo $HOSTNAME
if [ "$USER" == "myuserid" ]
then
STDOUT=`scontrol show jobid ${SLURM_JOB_ID} | grep StdOut | awk 'BEGIN{FS="="}{print $2}'`
python -c "import sys; stdout=sys.argv[1]; f=open(stdout, 'a'); f.write('sticks\n'); f.close();" ${STDOUT}
#echo "HELLO! ${USER}"
fi
QUESTION : Where am I going wrong?
EDIT : This is a MWE from within the context of trying to print resource utilization of jobs at the end of the output.
To get this, append the end of the epilog.log script with
# writing job statistics into job output
OUT=`scontrol show jobid ${SLURM_JOB_ID} | grep StdOut | awk 'BEGIN{FS="="}{print $2}'`
echo -e "sticks" >> ${OUT} 2>&1
There was no need to restart the slurm daemons. Additional commands can be added to it to get resource utilization, e.g.
sleep 5s ### Sleep to give chance for job to be written to slurm database for job statistics.
sacct --units M --format=jobid,user%5,state%7,CPUTime,ExitCode%4,MaxRSS,NodeList,Partition,ReqTRES%25,Submit,Start,End,Elapsed -j $SLURM_JOBID >> $OUT 2>&1
Basically, you can still append the output file using >>. Evidently, it did not occur to me that regular output redirection still works. It is still unclear why the python statement to this did not work.
According to this page, you can print to stdout from the Slurm prolog by prefacing your output with the 'print' command.
For example, instead of
echo "Starting prolog"
You need to do
echo "print Starting Prolog"
Unfortunately this only seems to work for the prolog, not the epilog.
Related
I need to write a shell script that starts a process in background and parse its output till it checks the output doesn't contains any Error in its output. The process will keep on running in the background as it needs to listen on ports. If the process output contained an error exit the script.
Based on the output of the previous process (it didn't contain any errors, the process was able to establish connection to DB) run the next command.
I have tried many approches suggested on Stack overflow, which includes:
https://unix.stackexchange.com/questions/12075/best-way-to-follow-a-log-and-execute-a-command-when-some-text-appears-in-the-log
https://unix.stackexchange.com/questions/45941/tail-f-until-text-is-seen
https://unix.stackexchange.com/questions/137030/how-do-i-extract-the-content-of-quoted-strings-from-the-output-of-a-command
/home/build/a_process 2>&1 | tee "output_$(date +"%Y_%m_%d").log"
tail -fn0 "output_$(date +"%Y_%m_%d").log" | \
while read line ; do
if [ echo "$line" | grep "Listening" ]
then
/home/build/b_process 2>&1 | tee "output_script_$(date +"%Y_%m_%d").log"
elif [ echo "$line" | grep "error occurred in load configuration" ] || [ echo "$line" | grep "Binding Failure" ]
then
sl -e
fi
done
The problem is since the process keep running despite it contains the text i was searching for it gets stuck in parsing the staring and never able to exit watching the output or tailing. As a result it's not able to execute next command.
On surface, the issue is with "tee" command (a_process ... | tee).
Recall that a pipeline will result in the shell
Creating the pipeline between the command
Waiting for the LAST command the finish.
Since the tee will not finish until a_process is done, and since a_process is a daemon, your script may wait forever (at least, until a_process exit).
In this case, consider sending the whole pipeline to the background.
log_file=output_$(date +"%Y_%m_%d").log
( /home/build/a_process 2>&1 | tee "$logfile" ) &
tail -fn0 "$logfile" |
...
Side note: consider setting the log file into a variable. This will make it easier to maintain (and understand) the script.
With SBATCH you can use the job-id in automatically generated output files using the following syntax with %j:
#!/bin/bash
# omitting some other sbatch commands here ...
#SBATCH -o slurm-%j.out-%N # name of the stdout, using the job number (%j) and the first node (%N)
#SBATCH -e slurm-%j.err-%N # name of the stderr, using job and first node values
I've been looking for a similar syntax for using the job-name instead of the job-id. Does anyone have a reference for what other slurm/sbatch values can be referenced in the %j style syntax?
In the newest versions of SLURM there is an option %x that represents job name.
See the "Changes in Slurm 17.02.1" section on the github:
https://github.com/SchedMD/slurm/blob/master/NEWS
However on many current clusters the slurm version is older than that and this option is not implemented. You can view the version of the slurm scheduler on your system:
sbatch --version
However there is a workaround.
You can create your own bash script, that can take a name as an argument, create a submission script that uses that name for the job name and output files and then submit it. For example,
You can create a script submit.sh:
#!/bin/bash
echo "#!/bin/bash" > jobscript.sh
echo "#SBATCH -o $1-%j.out-%N" >> jobscript.sh
echo "#SBATCH -e $1-%j.err-%N" >> jobscript.sh
echo "#SBATCH -J $1" >> jobscript.sh
#other echo commands with SBATCH options
echo "srun mycommand" >> jobscript.sh
#submit the job
sbatch jobscript.sh
And then execute it with an argument that correspond to the job name you want to give to your job:
bash ./submit.sh myJobName
I am having great difficulty in understanding what shows up on ps command. To test my understanding I created below dummy script
#!/bin/bash
for i in {1..100000}
do
date -u
date -u
date -u
done
while running this script I opened a new terminal and executed repeatedly
ps -eaf | grep date | grep -v grep
and I was able to date process in the output.
I later changed dummy script by replacing date -u with echo "what is going on"
#!/bin/bash
for i in {1..100000}
do
echo "What is going on"
echo "What is going on"
echo "What is going on"
done
while running the updated dummy script, I opened a new terminal and executed repeatedly
ps -eaf | grep echo | grep -v grep
and echo was never shown in output. Why is this? I suspect the reason is the script being a bash script, may be it is using builtin echo therefore it was not displayed in ps output. Am I correct? What am I missing here?
echo is a builtin in bash:
$ type echo
echo is a shell builtin
That means that a new process is not created when echo is run. All the work is done by the bash process instead, which is way more efficient.
You can run the non-builtin echo explicitly:
command echo "What is going on"
This forks and execs /bin/echo instead, letting it show up in ps.
I am using a bash script that calls multiple processes which have to start up in a particular order, and certain actions have to be completed (they then print out certain messages to the logs) before the next one can be started. The bash script has the following code which works really well for most cases:
tail -Fn +1 "$log_file" | while read line; do
if echo "$line" | grep -qEi "$search_text"; then
echo "[INFO] $process_name process started up successfully"
pkill -9 -P $$ tail
return 0
elif echo "$line" | grep -qEi '^error\b'; then
echo "[INFO] ERROR or Exception is thrown listed below. $process_name process startup aborted"
echo " ($line) "
echo "[INFO] Please check $process_name process log file=$log_file for problems"
pkill -9 -P $$ tail
return 1
fi
done
However, when we set the processes to print logging in DEBUG mode, they print so much logging that this script cannot keep up, and it takes about 15 minutes after the process is complete for the bash script to catch up. Is there a way of optimizing this, like changing 'while read line' to 'while read 100 lines', or something like that?
How about not forking up to two grep processes per log line?
tail -Fn +1 "$log_file" | grep -Ei "$search_text|^error\b" | while read line; do
So one long running grep process shall do preprocessing if you will.
Edit: As noted in the comments, it is safer to add --line-buffered to the grep invocation.
Some tips relevant for this script:
Checking that the service is doing its job is a much better check for daemon startup than looking at the log output
You can use grep ... <<<"$line" to execute fewer echos.
You can use tail -f | grep -q ... to avoid the while loop by stopping as soon as there's a matching line.
If you can avoid -i on grep it might be significantly faster to process the input.
Thou shalt not kill -9.
I'm trying to have a lightweight memory profiler for the matlab jobs that are run on my machine. There is either one or zero matlab job instance, but its process id changes frequently (since it is actually called by another script).
So here is the bash script that I put together to log memory usage:
#!/bin/bash
pid=`ps aux | grep '[M]ATLAB' | awk '{print $2}'`
if [[ -n $pid ]]
then
\grep VmSize /proc/$pid/status
else
echo "no pid"
fi
when I run this script in bash like this:
./script.sh
it works fine, giving me the following result:
VmSize: 1289004 kB
which is exactly what I want.
Now, I want to run this periodically. So I run it with watch, like this:
watch ./script.sh
But in this case I only receive:
no pid
Please note that I know the matlab job is still running, because I can see it with the same pid on top, and besides, I know each matlab job take several hours to finish.
I'm pretty sure that something is wrong with the quotes I have when setting pid. I just can't figure out how to fix it. Anyone knows what I'm doing wrong?
PS.
In the man page of watch, it says that commands are executed by sh -c. I did run my script like sh -c ./script and it works just fine, but watch doesn't.
Why don't you use a loop with sleep command instead?
For example:
#!/bin/bash
pid=`ps aux | grep '[M]ATLAB' | awk '{print $2}'`
while [ "1" ]
do
if [[ -n $pid ]]
then
\grep VmSize /proc/$pid/status
else
echo "no pid"
fi
sleep 10
done
Here the script sleeps(waits) for 10 seconds. You can set the interval you need changing the sleep command. For example to make the script sleep for an hour use sleep 1h.
To exit the script press Ctrl - C
This
pid=`ps aux | grep '[M]ATLAB' | awk '{print $2}'`
could be changed to:
pid=$(pidof MATLAB)
I have no idea why it's not working in watch but you could use a cron job and make the script log to a file like so:
#!/bin/bash
pid=$(pidof MATLAB) # Just to follow previously given advice :)
if [[ -n $pid ]]
then
echo "$(date): $(\grep VmSize /proc/$pid/status)" >> logfile
else
echo "$(date): no pid" >> logfile
fi
You'd of course have to create logfile with touch.
You might try just running ps command in watch. I have had issues in the past with watch chopping lines and such when they get too long.
It can be fixed by making the terminal you are running the command from wider or changing the column like this (may need to adjust the 160 to your liking):
export COLUMNS=160;