With SBATCH you can use the job-id in automatically generated output files using the following syntax with %j:
#!/bin/bash
# omitting some other sbatch commands here ...
#SBATCH -o slurm-%j.out-%N # name of the stdout, using the job number (%j) and the first node (%N)
#SBATCH -e slurm-%j.err-%N # name of the stderr, using job and first node values
I've been looking for a similar syntax for using the job-name instead of the job-id. Does anyone have a reference for what other slurm/sbatch values can be referenced in the %j style syntax?
In the newest versions of SLURM there is an option %x that represents job name.
See the "Changes in Slurm 17.02.1" section on the github:
https://github.com/SchedMD/slurm/blob/master/NEWS
However on many current clusters the slurm version is older than that and this option is not implemented. You can view the version of the slurm scheduler on your system:
sbatch --version
However there is a workaround.
You can create your own bash script, that can take a name as an argument, create a submission script that uses that name for the job name and output files and then submit it. For example,
You can create a script submit.sh:
#!/bin/bash
echo "#!/bin/bash" > jobscript.sh
echo "#SBATCH -o $1-%j.out-%N" >> jobscript.sh
echo "#SBATCH -e $1-%j.err-%N" >> jobscript.sh
echo "#SBATCH -J $1" >> jobscript.sh
#other echo commands with SBATCH options
echo "srun mycommand" >> jobscript.sh
#submit the job
sbatch jobscript.sh
And then execute it with an argument that correspond to the job name you want to give to your job:
bash ./submit.sh myJobName
Related
I am using a simple shell script to read and write lots of files (more than 300 files) in a HPC and I want to submit it using slurm.
The script looks like this:
#!/bin/bash
#SBATCH -n 1
#SBATCH --ntasks-per-node=40
#SBATCH --exclusive
for in_file in ${in_files}; do
# do something with ${in_file} and ${out_file}
echo ${in_file} ${out_file}
done
I may not submit all my tasks at one time because the number of files are larger than the nodes I can use. So is there a better way I can deal with the large number of files?
Are the files totally independent? It sounds you could make a bash script that schedules a job for each file and passes it to each job. One way would be to as a environment variable.
Something like
#!/bin/bash
for in_file in ${in_files}; do
export in_file=${in_file}
sbatch slurmjob.sh
done
Where your slurmjob.sh is something like
#!/bin/bash
#SBATCH -n 1
#SBATCH --ntasks-per-node=1
do something with ${in_file} and ${out_file}
echo ${in_file} ${out_file} > somefile.txt
``
I create this batch file myfirst_slurm_job.sh that contain the following code:
#!/bin/bash
#SBATCH --output="slurm1.txt"
cd $HOME/..
echo $PWD
echo $SLURMD_NODENAME
echo $SLURM_NTASKS
and then I run this command line:
sbatch myfirst_slurm_job.sh
note: it's my first post
You need to specify the --ntasks/-n flag;
#SBATCH -n 1
else SLURM won't bother to define this variable for you.
I want to pass a parameter to as bash script in a cluster in order to name the job. I tried this:
#!/bin/bash
#SBATCH -J "$1" #<--- to name the job with the first parameter
#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
echo "this is a test job named" $1
Gate main.mac
When I launch the job with
sbatch my_script.sh test_sript
I'm getting a file named $1-23472.out . It appears that "$1" didn't be interpreted. How can I have a file named "test_script-23472.out" ?
Also, is the line Gate main.mac mandatory? Can anyone explains me why we should put it ?
Many thanks
You probably can't do it exactly as you want to, but here's a solution that comes pretty close:
Batch script:
#!/bin/bash
#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
echo "this is a test job named" $SLURM_JOB_NAME
(rest of your script here)
Submit with:
$ sbatch -J jobname my_script.sh
Slurm will not interpret the Bash variable in the comments. Bash either since it is in a comment.
One solution is a construct like this for submission:
ARG="<something>" sbatch -J "$ARG" my_script.sh test_sript "$ARG"
As for the Gate main.mac line, it is used to start the Gate program with main.mac as argument.
This is how I've been formatting slurm scripts to parse bash variables as job names.
#!/bin/bash
MYVAR=$1
sbatch --export=ALL -J ${MYVAR} --wrap="run something"
I'm on CentOS 6.9 running slurm 17.11.7. I've modified my /gpfs0/export/slurm/conf/epilog script. I'm ultimately would like to print out job resource utilization information to the stdout file used be each users' job.
I've been testing it within the conditional at the end of the script for myself before I roll it out to other users. Below is my modified epilog script:
#!/bin/bash
# Clear out TMPDIR on the shared file system after job completes
exec >> /var/log/epilog.log
exec 2>> /var/log/epilog.log
if [ -z $SLURM_JOB_ID ]
then
echo -e " This script should be executed from slurm."
exit 1
fi
TMPDIR="/gpfs0/scratch/${SLURM_JOB_ID}"
rm -rf $TMPDIR
### My additions to the existing script ###
if [ "$USER" == "myuserid" ]
then
STDOUT=`scontrol show jobid ${SLURM_JOB_ID} | grep StdOut | awk 'BEGIN{FS="="}{print $2}'`
# Regular stdout/stderr is not respected, must use python.
python -c "import sys; stdout=sys.argv[1]; f=open(stdout, 'a'); f.write('sticks\n'); f.close();" ${STDOUT}
fi
exit 0
From the Prolog and Epilog section of the slurm.conf user manual it seems that stdout/stderr are not respected. Hence I modify the stdout file with python.
I've picked the compute node node21 to run this job, so I logged into node21 and tried several things to get it to notice my changes to the epilog script.
Reconfiguring slurmd:
sudo scontrol reconfigure
Restart slurm daemon:
sudo service slurm stop
sudo service slurm start
Neither of which seems to get the changes to the epilog script when I submit jobs. When put the same conditional in a batch script it runs flawlessly:
#!/bin/bash
#SBATCH --nodelist=node21
echo "Hello you!"
echo $HOSTNAME
if [ "$USER" == "myuserid" ]
then
STDOUT=`scontrol show jobid ${SLURM_JOB_ID} | grep StdOut | awk 'BEGIN{FS="="}{print $2}'`
python -c "import sys; stdout=sys.argv[1]; f=open(stdout, 'a'); f.write('sticks\n'); f.close();" ${STDOUT}
#echo "HELLO! ${USER}"
fi
QUESTION : Where am I going wrong?
EDIT : This is a MWE from within the context of trying to print resource utilization of jobs at the end of the output.
To get this, append the end of the epilog.log script with
# writing job statistics into job output
OUT=`scontrol show jobid ${SLURM_JOB_ID} | grep StdOut | awk 'BEGIN{FS="="}{print $2}'`
echo -e "sticks" >> ${OUT} 2>&1
There was no need to restart the slurm daemons. Additional commands can be added to it to get resource utilization, e.g.
sleep 5s ### Sleep to give chance for job to be written to slurm database for job statistics.
sacct --units M --format=jobid,user%5,state%7,CPUTime,ExitCode%4,MaxRSS,NodeList,Partition,ReqTRES%25,Submit,Start,End,Elapsed -j $SLURM_JOBID >> $OUT 2>&1
Basically, you can still append the output file using >>. Evidently, it did not occur to me that regular output redirection still works. It is still unclear why the python statement to this did not work.
According to this page, you can print to stdout from the Slurm prolog by prefacing your output with the 'print' command.
For example, instead of
echo "Starting prolog"
You need to do
echo "print Starting Prolog"
Unfortunately this only seems to work for the prolog, not the epilog.
I have an executable that takes multiple options and multiple file inputs in order to run. The executable can be called with a variable number of cores to run.
E.g. executable -a -b -c -file fileA --file fileB ... --file fileZ --cores X
I'm trying to create an sbatch file that will enable me to have multiple calls of this executable with different inputs. Each call should be allocated in a different node (in parallel with the rest), using X cores. The parallelization at core level is taken care of the executable, while at the node level by SLURM.
I tried with ntasks and multiple sruns but the first srun was called multiple times.
Another take was to rename the files and use a SLURM process or node number as filename before the extension but it's not really practical.
Any insight on this?
i do these kind of jobs always with the help of bash script that i run by a sbatch command. The easiest approach would be to have a loop in a sbatch script where you spawn the different job and job steps under your executable with srun specifying i.e. the corresponding node name in your partion with -w . You may also read up the documentation of slurm array jobs (if that befits you better). Alternatively you could also store all parameter combinations in a file and than loop over them with the script of have a look at "array job" manual page.
Maybe the following script (i just wrapped it up) helps you to get a feeling for what i have in mind (i hope its what you need). Its not tested so dont just copy and paste it!
#!/bin/bash
parameter=(10 5 2)
node_names=(node1 node2 node3)
# lets run one job per node each time taking one parameter
for parameter in ${parameter[*]}
# asign parameter to node
#script some if else condition here to specify parameters
# -w specifies the name of the node to use
# -N specifies the amount of nodes
JOBNAME="jmyjob$node-$parameter"
# asign the first job to the node
$node=${node_names[0]}
#delete first node from list
unset node_names[0];
#reinstantiate list
node_names=("${Unix[#]}")
srun -N1 -w$node -psomepartition -JJOBNAME executable.sh model_parameter &
done;
You will have the problem that you need to force your sbatch script to wait for the last job step. In this case the follwoing additional while loop might help you.
# Wait for the last job step to complete
while true;
do
# wait for last job to finish use the state of sacct for that
echo "waiting for last job to finish"
sleep 10
# sacct shows your jobs, -R only running steps
sacct -s R,gPD|grep "myjob*" #your job name indicator
# check the status code of grep (1 if nothing found)
if [ "$?" == "1" ];
then
echo "found no running jobs anymore"
sacct -s R |grep "myjob*"
echo "stopping loop"
break;
fi
done;
I managed to find one possible solution, so I'm posting it for reference:
I declared as many tasks as calls to the executable, as well as nodes and the desired number of cpus per call.
And then a separate srun for each call, declaring the number of nodes and tasks at each call. All the sruns are bound with ampersands (&):
srun -n 1 -N 1 --exclusive executable -a1 -b1 -c1 -file fileA1 --file fileB1 ... --file fileZ1 --cores X1 &
srun -n 1 -N 1 --exclusive executable -a2 -b2 -c2 -file fileA2 --file fileB2 ... --file fileZ2 --cores X2 &
....
srun -n 1 -N 1 --exclusive executable -aN -bN -cN -file fileAN --file fileBN ... --file fileZN --cores XN
--Edit: After some tests (as I mentioned in a comment below), if the process of the last srun ends before the rest, it seems to end the whole job, leaving the rest unfinished.
--edited based on the comment by Carles Fenoy
Write a bash script to populate multiple xyz.slurm files and submit each of them using sbatch. Following script does a a nested for loop to create 8 files. Then iterate over them to replace a string in those files, and then batch them. You might need to modify the script to suit your need.
#!/usr/bin/env bash
#Path Where you want to create slurm files
slurmpath=~/Desktop/slurms
rm -rf $slurmpath
mkdir -p $slurmpath/sbatchop
mkdir -p /exports/home/schatterjee/reports
echo "Folder /slurms and /reports created"
declare -a threads=("1" "2" "4" "8")
declare -a chunks=("1000" "32000")
declare -a modes=("server" "client")
## now loop through the above array
for i in "${threads[#]}"
{
for j in "${chunks[#]}"
{
#following are the content of each slurm file
cat <<EOF >$slurmpath/net-$i-$j.slurm
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --output=$slurmpath/sbatchop/net-$i-$j.out
#SBATCH --wait-all-nodes=1
echo \$SLURM_JOB_NODELIST
cd /exports/home/schatterjee/cs553-pa1
srun ./MyNETBench-TCP placeholder1 $i $j
EOF
#Now schedule them
for m in "${modes[#]}"
{
for value in {1..5}
do
#Following command replaces placeholder1 with the value of m
sed -i -e 's/placeholder1/'"$m"'/g' $slurmpath/net-$i-$j.slurm
sbatch $slurmpath/net-$i-$j.slurm
done
}
}
}
You can also try this python wrapper which can execute your command on the files you provide