I have a Snakefile that has a rule which sends 7 different shell commands.
I want to run each of these shell commands in sbatch and I want them to run in different slurm nodes.
Right now when I include sbatch inside the shell command in the Snakemake rule I do not get the desired output file because it takes awhile to run and when sbatch returns the command is still running. I think Snakemake thinks that I don't have the required output file because it thinks that the command "finished executing" before the submitted job completed.
What can I do to submit each rule in one slurm node using sbatch command in Snakemake file
I suspect that what you are doing is:
rule one:
input:
...
output:
...
shell:
"""
sbatch [sbatch-options] "some-command-or-script"
"""
What you want maybe is:
rule one:
input:
...
output:
...
shell:
"""
some-command-or-script
"""
To be executed as
snakemake --cluster "sbatch [sbatch-options]"
In this way every rule will send its jobs to the cluster and snakemake will handle them. If you want a rule to execute its jobs locally (not via sbatch) mark that rule with the directive localrule (check documentation for more detail)
Related
I have seen similar questions, but not exactly the same as mine: Use Bash variable within SLURM sbatch script, because I am not talking about slurm parameters.
I want to launch a slurm job for each of my sample files, so imagine I have 3 vcfs and I want to run a job for each of them:
I created a script to loop through a file in which I wrote sampleIds to run another script with each sample, which would perfectly work if I wanted to run it directly with bash:
while read line
do
sampleID="${line[0]}"
myscript.sh $sampleID
The problem is that I need to run the script with slurm, so is there any way to indicate slurm the bash variable that it should include?
I was trying this, but it is not working:
sbatch myscrip.sh --export=$sampleID
Okay, I've solved it:
sbatch --export=sampleID=$sampleID myscript.sh
If I submitted a job using slurm, how can a see the full command file that was submitted, usually a bash script that start with :
#SBATCH --job-name=
how can I use slurm command to see the content of this script file (I know I can do scontorol show jobId -xxx and see where the script is, but assuming it was deleted or changed, and I still want to see the original script that was submitted to slurm)
Thanks.
On older versions of Slurm (17.02 and before), you can do scontrol show -dd job <jobid> to have Slurm display the content of the submission script it stored, and in newer versions of Slurm (17.11 and after), you would use scontrol write batch_script <jobid> -.
I am trying to write a script so I can use the 'qsub' command to submit a job to the cluster.
Pretty much, once I get into the cluster, I go to the directory with my files and I do these steps:
export PATH=$PATH:$HOME/program/bin
Then,
program > run.log&
Is there any way to make this into a script so I am able to submit the job to the queue?
Thanks!
Putting the lines into a bash script and then running qsub myscript.sh should do it.
Is there a way to stop sbatch from copying the script to the compute node. For example when I run:
sbatch --mem=300 /shared_between_all_nodes/test.sh
test.sh is copied to /var/lib/slurm-llnl/slurmd/etc/ on the executing compute node. The trouble with this is there are other scripts in /shared_between_all_nodes/ that test.sh needs to use and I would like to avoid hard coding the path.
In sge I could use qsub -b y to stop it from copying the script to the compute node. Is there a similar option or config in slurm?
Using sbatch --wrap is a nice solution for this
sbatch --wrap /shared_between_all_nodes/test.sh
quotes are required if the script has parameters
sbatch --wrap "/shared_between_all_nodes/test.sh param1 param2"
from sbatch docs http://slurm.schedmd.com/sbatch.html
--wrap=
Sbatch will wrap the specified command string in a simple "sh" shell script, and submit that script to the slurm controller. When --wrap is used, a script name and arguments may not be specified on the command line; instead the sbatch-generated wrapper script is used.
The script might be copied there, but the working directory will be the directory in which the sbatch command is launched. So if the command is launched from /shared_between_all_nodes/ it should work.
To be able to lauch sbatch form anywhere, use this option
-D, --workdir=<directory>
Set the working directory of the batch script to directory before
it is executed.
like
sbatch --mem=300 -D /shared_between_all_nodes /shared_between_all_nodes/test.sh
I'm trying to obtain a value from another file and use this within a SLURM submission script. However, I get an error that the value is non-numerical, in other words, it is not being dereferenced.
Here is the script:
#!/bin/bash
# This reads out the number of procs based on the decomposeParDict
numProcs=`awk '/numberOfSubdomains/ {print $2}' ./meshModel/decomposeParDict`
echo "NumProcs = $numProcs"
#SBATCH --job-name=SnappyHexMesh
#SBATCH --output=./logs/SnappyHexMesh.log
#
#SBATCH --ntasks=`$numProcs`
#SBATCH --time=240:00
#SBATCH --mem-per-cpu=4000
#First run blockMesh
blockMesh
#Now decompose the mesh
decomposePar
#Now run snappy in parallel
mpirun -np $numProcs snappyHexMesh -parallel -overwrite
When I run this as a normal Bash shell script, it prints out the number of procs correctly and makes the correct mpirun call. Thus the awk command parses out the number of procs correctly and the variable is dereferenced as expected.
However, when I submit this to SLURM using:
sbatch myScript.sh
I get the error:
sbatch: error: Invalid numeric value "`$numProcs`" for number of tasks.
Can anyone help with this?
This won't work. What happens when you run
sbatch myscript.sh
is that slurm parses the script for those special #SBATCH lines, generates a job record, stores the batch script somewhere. The batch script is executed only later when the job runs.
So you need to structure you workflow in a slightly different way, and first calculate the number of procs you need before submitting the job. Note that you can use something like
sbatch -n $numProcs myscript.sh
, you don't need to autogenerate the script (also, mpirun should be able to get the number of procs in your allocation automatically, no need to use "-np").
Slurm stops processing #SBATCH directives on the first line of executable code in a script. For users whose #SBATCH directives are not dependent on the code they're trying to run above those directives, just put the #SBATCH lines at the top.
See the other answer for a workaround/solution if, as with OP, your sbatch options are dependent on the commands you've placed above them.
The batch script may contain options preceded with "#SBATCH" before
any executable commands in the script. sbatch will stop processing
further #SBATCH directives once the first non-comment non-whitespace
line has been reached in the script.
From the sbatch docs, my emphasis.