I create this batch file myfirst_slurm_job.sh that contain the following code:
#!/bin/bash
#SBATCH --output="slurm1.txt"
cd $HOME/..
echo $PWD
echo $SLURMD_NODENAME
echo $SLURM_NTASKS
and then I run this command line:
sbatch myfirst_slurm_job.sh
note: it's my first post
You need to specify the --ntasks/-n flag;
#SBATCH -n 1
else SLURM won't bother to define this variable for you.
Related
I'm trying to run the following script with sbatch on our cluster.
#!/bin/bash
#SBATCH -o /SCRATCH-BIRD/users/lindenbaum-p/work/NEXTFLOW/work/chunkaa/work/a4/6d0605f453add1d97d609839cfd318/command.log
#SBATCH --no-requeue
#SBATCH --partition=Bird
set -e
echo "Hello" 1>&2
sbatch displays a job-id on stdout, there is nothing listed in squeue and it looks like nothing was written/executed.
If the line #SBATCH -o /SCRATCH-BIRD/users/... is removed , then the script works.
the directory exists
$ test -w /SCRATCH-BIRD/users/lindenbaum-p/work/NEXTFLOW/work/chunkaa/work/a4/6d0605f453add1d97d609839cfd318/ && echo OK
OK
could it be a problem with the filesystem ? how can I test this ?
OK, got it the partition is visible from the login node but not from the cluster nodes.
I want to pass a parameter to as bash script in a cluster in order to name the job. I tried this:
#!/bin/bash
#SBATCH -J "$1" #<--- to name the job with the first parameter
#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
echo "this is a test job named" $1
Gate main.mac
When I launch the job with
sbatch my_script.sh test_sript
I'm getting a file named $1-23472.out . It appears that "$1" didn't be interpreted. How can I have a file named "test_script-23472.out" ?
Also, is the line Gate main.mac mandatory? Can anyone explains me why we should put it ?
Many thanks
You probably can't do it exactly as you want to, but here's a solution that comes pretty close:
Batch script:
#!/bin/bash
#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
echo "this is a test job named" $SLURM_JOB_NAME
(rest of your script here)
Submit with:
$ sbatch -J jobname my_script.sh
Slurm will not interpret the Bash variable in the comments. Bash either since it is in a comment.
One solution is a construct like this for submission:
ARG="<something>" sbatch -J "$ARG" my_script.sh test_sript "$ARG"
As for the Gate main.mac line, it is used to start the Gate program with main.mac as argument.
This is how I've been formatting slurm scripts to parse bash variables as job names.
#!/bin/bash
MYVAR=$1
sbatch --export=ALL -J ${MYVAR} --wrap="run something"
I want to pass arguments in a job. I write a wrapper in the file job.sh but when i use it i get a error:
sbatch: error: This does not look like a batch script. The first
sbatch: error: line must start with #! followed by the path to an interpreter.
sbatch: error: For instance: #!/bin/sh
main.sh
DATE=`date '+%Y-%m-%d %H:%M:%S'`
name='test__'$DATE
./job.sh $name
job.sh
#!/bin/bash
sbatch << EOT
#!/bin/sh
#SBATCH --job-name=$1
#SBATCH --nodes=1 # number of nodes
#SBATCH --ntasks-per-node=1 # number of cores
#module load R
Rscript script.R $1
EOT
script.R
args <- commandArgs(trailingOnly=TRUE)
print((args[[1]]))
(the three files are in the same directory)
I wanted to indicate the name and other values of some variables of a job from another file but I get an error.
sbatch: error: Unable to open file 10:12:35
file.sh
#!/bin/bash
DATE=`date '+%Y-%m-%d %H:%M:%S'`
name='test__'$DATE
sbatch -J $name -o $name'.out' -e $name'.err' job.sh
job.sh
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --nodes=1 # number of nodes
#SBATCH --ntasks-per-node=2 # number of cores
#SBATCH --output=.out
#SBATCH --error=.err
#module load R
Rscript script.R
script.R
for(i in 1:1e6){print(i)}
You are wrongly quoting the variables and the space requested in the date is creating two arguments to sbatch, hence he is complaining about that wrong parameter.
If I were you, I would avoid the space (as a general rule, cause it is more error prone and always requires quoting):
file.sh:
#!/bin/bash
DATE=$(date '+%Y-%m-%dT%H:%M:%S')
name="test__$DATE"
sbatch -J "$name" -o "${name}.out" -e "${name}.err" job.sh
With SBATCH you can use the job-id in automatically generated output files using the following syntax with %j:
#!/bin/bash
# omitting some other sbatch commands here ...
#SBATCH -o slurm-%j.out-%N # name of the stdout, using the job number (%j) and the first node (%N)
#SBATCH -e slurm-%j.err-%N # name of the stderr, using job and first node values
I've been looking for a similar syntax for using the job-name instead of the job-id. Does anyone have a reference for what other slurm/sbatch values can be referenced in the %j style syntax?
In the newest versions of SLURM there is an option %x that represents job name.
See the "Changes in Slurm 17.02.1" section on the github:
https://github.com/SchedMD/slurm/blob/master/NEWS
However on many current clusters the slurm version is older than that and this option is not implemented. You can view the version of the slurm scheduler on your system:
sbatch --version
However there is a workaround.
You can create your own bash script, that can take a name as an argument, create a submission script that uses that name for the job name and output files and then submit it. For example,
You can create a script submit.sh:
#!/bin/bash
echo "#!/bin/bash" > jobscript.sh
echo "#SBATCH -o $1-%j.out-%N" >> jobscript.sh
echo "#SBATCH -e $1-%j.err-%N" >> jobscript.sh
echo "#SBATCH -J $1" >> jobscript.sh
#other echo commands with SBATCH options
echo "srun mycommand" >> jobscript.sh
#submit the job
sbatch jobscript.sh
And then execute it with an argument that correspond to the job name you want to give to your job:
bash ./submit.sh myJobName