The question title pretty much says it all.
As an example, the --mem parameter for srun is optional (or at least this is the case for the SLURM instance I have access to). I would like to know the value for this option that would have the same effect as not specifying the option at all. (I realize that this particular default may depend on the values of other parameters passed to srun, such as the partition, etc.)
Ditto for all the other optional srun and sbatch parameters.
By default, --mem gets DefMemPerNode.
You can check that value using the command:
scontrol show config
If it is not defined, its default values will be 0 and it means that the job will be given all the memory of the node.
You can have the other default values from the FAQ.
Related
I launch a bunch of jobs with the following format:
sbatch -p partitionA,partitionB --nice=${NICE} script_to_run.sh
Is there a way to specify the nice value per partition, or is the way to do this is to set a default niceness for each partition and use that?
Let's say I want to submit a slurm job just assigning the total amount of tasks (--ntasks=someNumber), without specifying the number of nodes and the tasks per node. Is there a way to know within the launched slurm script how many cores are assigned by slurm for each of the reserved nodes? I need to know this info to properly create a machinefile for the program I'm launching, that must be structured like this:
node02:7
node06:14
node09:3
Once the job is launched, the only way I figured out to see what cores have been allocated on the nodes is using the command:
scontrol show jobid -dd
In its output the abovementioned info is stored (together with plenty of other details).
Is there a better way to get this info?
The way the srun documentation illustrates creating a machine file is by running srun hostname. To get the output you want you could run
srun hostname -s | sort | uniq -c | awk '{print $2":"$1}' > $MACHINEFILE
You should check the documentation of your program to see if it accepts a machine file with repetitions rather than a suffix count. If so you can simplify the command as
srun hostname -s > $MACHINEFILE
And of course the first step is actually to make sure you indeed need a machine file in the first place as many parallel programs/libraries have Slurm support and can gather the needed information from the environment variables setup by Slurm upon job start.
I would like to distribute my job evenly across multiple nodes and have specified the --distribution=cyclic in my sbatch file, but slurm ignores that and uses the block distribution instead.
Before, the tasks were distributing evenly across the nodes. From reading the docs, this is what I expect the default behavior to be, unless otherwise specified in slurm.conf.
Starting today, the tasks are clustering on the first node with only one task on each of the other nodes. I've obviously changed something in the config, but can't figure out where to fix it. I did make a change to the image for the compute nodes and rebooted them today.
When I try to stop the slurmctld on the head node, it is restarted immediately by my Bright Cluster Manager monitor. Not sure if this is preventing configuration updates or not.
I've investigated the slurm.conf file but it looks OK. I've tried both SelectTypeParameters=CR_Core and CR_CPU but get the same result.
To try and work around this I added the --distribution=cyclic to my sbatch file, but slurm is still allocation using the 'block' method. But adding this to the sbatch should not be necessary, at leas according to my understanding of the docs.
Here are the relevant lines from slurm.conf and my sbatch script:
# RESOURCES
SelectType=select/cons_res
SelectTypeParameters=CR_Core
# Node Description
NodeName=DEFAULT Sockets=2 CoresPerSocket=20 ThreadsPerCore=1
# Scheduler
SchedulerType=sched/backfill
#SBATCH --ntasks=12
#SBATCH --nodes=3
#SBATCH --distribution=cyclic:cyclic
I would expect the tasks to be distributed evenly between the nodes, with 4 tasks on each of the 3 nodes.
Here is how the tasks are actually getting distributed:
NODELIST STATE CPUS(A/I/O/T) MEMORY TMP_DISK REASON
compute001 mixed 10/30/0/40 192006 2038 none
compute002 mixed 1/39/0/40 192006 2038 none
compute003 mixed 1/39/0/40 192006 2038 none
compute004 idle 0/40/0/40 192006 2038 none
According to https://slurm.schedmd.com/sbatch.html, the distribution flag is only useful for srun:
Specify alternate distribution methods for remote processes. In sbatch, this only sets environment variables that will be used by subsequent srun requests.
(As to why it's like this… I have no idea. But it does appear it's by design.)
Depending on your configuration, you may be able to approximate what you want by setting SelectType=cons_res or cons_tres and SelectTypeParameters=CR_LLN. If either of these parameters changed recently, that might be the reason the behavior changed, as well.
I managed to manually distribute my processes across the nodes by modifying the sbatch file to limit the number of tasks assigned to each node:
#SBATCH --ntasks=12
#SBATCH --nodes=3
#SBATCH --tasks-per-node=4
This results in the expected distribution of the tasks across the nodes:
NODELIST STATE CPUS(A/I/O/T) MEMORY TMP_DISK REASON
compute001 mixed 4/36/0/40 192027 2038 none
compute002 mixed 4/36/0/40 192027 2038 none
compute003 mixed 4/36/0/40 192027 2038 none
compute004 idle 0/40/0/40 192027 2038 none
Is it possible to use the --array option as an argument? I mean, I have a R code where I use arrays. The number of arrays depends of the file on which I execute my R code. I would like to pass as argument the number of arrays into the sbatch my_code.R command line , in order to never modify my slurm code : for example, for a file with 550.000 columns, I will need 10 arrays, a file with 1.000.000 columns will needed 19 arrays etc. I must get something like this "sbatch --array 1-nb_of_arrays_needed my_code.R" . The goal is to make my code usable by everyone, without the user needs to go into the slurm code in order to change the line #SBATCH --array=x-y
My R code (I don't show it in full) :
data<-read.table(opt$file, h=T, row.names=1, sep="\t")+1
ncol=ncol(data)
nb_arrays=ncol/55000
nb_arrays=round(nb_arrays)
opt$number=nb_arrays
...
Bests
Your R script will start only when the job is scheduled. To be scheduled, it must be submitted, and to be submitted, it must know the argument to --array.
So you have two options:
Either split your R script into to, one part that will run before the job is submitted, and the other that will run when the job starts. The first part will compute the necessary number of jobs in the array (and possibly submit the job array automatically) and the other part will do the actual calculations.
If you prefer having only one R script, you can differentiate the behaviour based on the presence or absence of SLURM_JOB_ID variable in the environment. If it is not present, compute the number of jobs and submit, if it is present, do the actual calculations.
The other option is to set --array in the submission job to a large value, and when the first job in the array starts, it computes the number of jobs that are necessary, and cancels the superfluous jobs.
I have set of an array job as follows:
sbatch --array=1:100%5 ...
which will limit the number of simultaneously running tasks to 5. The job is now running, and I would like to change this number to 10 (i.e. I wish I'd run sbatch --array=1:100%10 ...).
The documentation on array jobs mentions that you can use scontrol to change options after the job has started. Unfortunately, it's not clear what this option's variable name is, and I don't think it is listed in the documentation of the sbatch command here.
Any pointers well received.
You can change the array throttling limit with the following command:
scontrol update ArrayTaskThrottle=<count> JobId=<jobID>