hello everyones so im trying to set up a new hpc cluster i made an account and added users and im using a partition but whenerver i run a job it gives me an error that request node configuration is not available i checked my slurm.conf but it seems good to me i need some help
the error Batch job submission failed: Requested node configuration is not available
#
# See the slurm.conf man page for more information.
#
SlurmUser=slurm
#SlurmdUser=root
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
#JobCredentialPrivateKey=
#JobCredentialPublicCertificate=
SlurmdSpoolDir=/cm/local/apps/slurm/var/spool
SwitchType=switch/none
MpiDefault=none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
#ProctrackType=proctrack/pgid
ProctrackType=proctrack/cgroup
#PluginDir=
#FirstJobId=
ReturnToService=2
#MaxJobCount=
#PlugStackConfig=
#PropagatePrioProcess=
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#SrunProlog=
#SrunEpilog=
#TaskProlog=
#TaskEpilog=
TaskPlugin=task/cgroup
#TrackWCKey=no
#TreeWidth=50
#TmpFs=
#UsePAM=
#
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
#
# SCHEDULING
#SchedulerAuth=
#SchedulerPort=
#SchedulerRootFilter=
#PriorityType=priority/multifactor
#PriorityDecayHalfLife=14-0
#PriorityUsageResetPeriod=14-0
#PriorityWeightFairshare=100000
#PriorityWeightAge=1000
#PriorityWeightPartition=10000
#PriorityWeightJobSize=1000
#PriorityMaxAge=1-0
#
# LOGGING
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurmctld
SlurmdDebug=3
SlurmdLogFile=/var/log/slurmd
#JobCompType=jobcomp/filetxt
#JobCompLoc=/cm/local/apps/slurm/var/spool/job_comp.log
#
# ACCOUNTING
JobAcctGatherType=jobacct_gather/linux
#JobAcctGatherType=jobacct_gather/cgroup
#JobAcctGatherFrequency=30
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageUser=slurm
# AccountingStorageLoc=slurm_acct_db
# AccountingStoragePass=SLURMDBD_USERPASS
# This section of this file was automatically generated by cmd. Do not edit manually!
# BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE
# Server nodes
SlurmctldHost=omics-master
AccountingStorageHost=master
# Nodes
NodeName=omics[01-05] Procs=48 Feature=local
# Partitions
PartitionName=defq Default=YES MinNodes=1 DefaultTime=UNLIMITED MaxTime=UNLIMITED AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 OverSubscribe=NO PreemptMode=OFF AllowAccounts=ALL AllowQos=ALL Nodes=omics[01-05]
ClusterName=omics
# Scheduler
SchedulerType=sched/backfill
# Statesave
StateSaveLocation=/cm/shared/apps/slurm/var/cm/statesave/omics
PrologFlags=Alloc
# Generic resources types
GresTypes=gpu
# Epilog/Prolog section
Prolog=/cm/local/apps/cmd/scripts/prolog
Epilog=/cm/local/apps/cmd/scripts/epilog
# Power saving section (disabled)
# END AUTOGENERATED SECTION -- DO NOT REMOVE
and this is my sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
defq* up infinite 5 idle omics[01-05]
and this is my test script
#!/bin/bash
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks-per-node=4
#SBATCH --ntasks-per-socket=2
#SBATCH --output=std.out
#SBATCH --error=std.err
#SBATCH --mem-per-cpu=1gb
echo "hello from:"
hostname; pwd; date;
sleep 10
echo "going to sleep during 10 seconds"
echo "wake up, exiting
"
and thanks in advance
In the node definition, you do not specify RealMemory so Slurm assumes the default of 1MB (!) per node. Hence the request of 1GB per CPU cannot be fulfilled.
You should run slurmd -C on the compute node, that will give you the line to insert in the slurm.conf file for Slurm to correctly know the hardware resources it can allocate.
$ slurmd -C | head -1
NodeName=node002 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=128547
I'm completly new to slurm and debian.
I have a cluster with 27 nodes with debian each has a processor with 24 cores. On my main node runing slurmctld i have a script which takes as much resources as it can. The question is, can I run it with all the cpu's in a cluster? if so then how. I know how to run a script that uses stress package and uses all cpus on 2 nodes :
#!/bin/bash
#
#SBATCH --job-name=test
#
#SBATCH --nodes=2
#SBATCH --ntasks=48
#SBATCH --time=00:00:20
srun sudo stress --cpu 48 --timeout 20
but i have to install the stress package on every I'm using.
I am running Slurm 19.05.2 with Cloud nodes only. I specified
SelectType = select/cons_tres
SelectTypeParameters = CR_CORE_MEMORY,CR_CORE_DEFAULT_DIST_BLOCK
To make sure that a node is fully utilised before allocating a second node.
It seem to work well with jobs that have many tasks. If I have 8 nodes which each have 16 cores and I submit a job which has 8 tasks and each task requires 2 cores this will be scheduled to one node.
For example the script:
#!/bin/bash
#
#SBATCH --job-name=batch
#SBATCH --output=o_batch.%A.%a.txt
#
#SBATCH --ntasks=8
#SBATCH --time=10:00
#SBATCH --cpus-per-task 2
#SBATCH --mem-per-cpu=100
srun hostname
will output
node-010000
node-010000
node-010000
node-010000
node-010000
node-010000
node-010000
node-010000
If I specify a job array with --array=1-8 (--ntasks=1) all jobs of the array will be scheduled on a different node (even though one node could fit all job requirements)
#!/bin/bash
#
#SBATCH --job-name=array
#SBATCH --output=array.%A.%a.txt
#
#SBATCH --ntasks=1
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#SBATCH --array=1-8
srun hostname
will output
node-010000
node-010001
node-010002
node-010003
node-010004
node-010005
node-010006
node-010007
Is there a way of configuring slurm to behave the same with arrays as with task?
I have a setup/installation of SLURM on my desktop computer to do some testing and understand how it works before deploying it to a cluster.
The desktop computer is running Ubuntu 18.10 (Cosmic), as the nodes in the cluster are all running on. The used version of SLURM is 17.11.9.
I have tested some of the features of SLURM, e.g. job-arrays and its deployment of tasks.
However I would like to communicate with the different tasks sent out to each node or CPU in the cluster, in order to collect its results (without disk I/O). For that reason, I have looked in how manage that with e.g. message queuing, and MPI, or OpenMPI. (Any other implementation strategy, as an advice or recommendation is much appreciated.)
I have tested MPI with a simple Python snippet, starting a communication between two processes. I am using MPI4PY to handle this communication.
This code snippet runs fine with mpiexec-command, but running it via SLURM and sbatch-command I cannot get it to work. SLURM is configured with OpenMPI and opmi_info states that SLURM is supported.
OpenMPI version 3.1.2-6 (from dpkg -l | grep mpi)
SLURM_VERSION 17.11.9
Ubuntu 18.10 (Cosmic)
MPI4PY version 3.0.1. (from pip list)
This is the Python3.6 code snippet:
$cat mpi_test.py
from mpi4py import MPI
if __name__=='__main__':
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
if rank==0:
data={'param1':1, 'param2':2, 'param3':3}
destinationNode = 1
print('Im', rank, 'sending to ', destinationNode)
comm.send(data, dest=destinationNode, tag=11)
elif rank!=0:
sourceNode = 0
dataRx=comm.recv(source=sourceNode, tag=11)
print('Im', rank, 'recieving from ', sourceNode)
for keys in dataRx.keys():
print('Data recieved: ',str(dataRx[keys]))
The python.mpi.sbatch used at the call with sbatch is:
$cat python.mpi.sbatch
#!/bin/bash -l
#SBATCH --job-name=mpiSimpleExample
#SBATCH --nodes=1
#SBATCH --error=slurm-err-%j.err
#SBATCH --export=all
#SBATCH --time=0-00:05:00
#SBATCH --partition=debug
srun -N 1 mpiexec -n 2 python3 mpi_test.py
#mpiexec -n 2 python3 mpi_test.py
exit 0
Running "sbatch python.mpi.sbatch" with this setup yields the following output:
$sbatch python.mpi.sbatch
$cat slurm-err-104.err
----------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 2
slots
that were requested by the application:
python3
Either request fewer slots for your application, or make more slots
available for use.
--------------------------------------------------------------------
Modifying python.mpi.sbatch to instead use:
"srun -n 1 mpiexec -n 1 python3 mpi_test.py" yields the error:
$cat slurm-err-105.error
Traceback (most recent call last):
File "mpi_test.py", line 18, in <module>
comm.send(data, dest=destinationNode, tag=11)
File "mpi4py/MPI/Comm.pyx", line 1156, in mpi4py.MPI.Comm.send
File "mpi4py/MPI/msgpickle.pxi", line 174, in mpi4py.MPI.PyMPI_send
mpi4py.MPI.Exception: MPI_ERR_RANK: invalid rank
---------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero
status, thus causing the job to be terminated. The first process to do
so was:
Process name: [[44366,1],0]
Exit code: 1
---------------------------------------------------------------------
Which is expected since it is only started with 1 node.
Running mpirun hostname, yields four instances of the machine, thus there should be four slots available for this machine.
I may run the Python3.6 with up to four (after modification of mpi_test.py) processess with the command "mpiexec -n 4 python3 mpi_test.py", with success.
Any help is much appreciated.
slurm.conf-file:
# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=desktop-comp
#ControlAddr=
#BackupController=
#BackupAddr=
#
AuthType=auth/munge
#CheckpointType=checkpoint/none
CryptoType=crypto/munge
#DisableRootJobs=NO
#EnforcePartLimits=NO
#Epilog=
#EpilogSlurmctld=
#FirstJobId=1
#MaxJobId=999999
#GresTypes=
#GroupUpdateForce=0
#GroupUpdateTime=600
#JobCheckpointDir=/var/slurm/checkpoint
#JobCredentialPrivateKey=
#JobCredentialPublicCertificate=
#JobFileAppend=0
#JobRequeue=1
#JobSubmitPlugins=1
#KillOnBadExit=0
#LaunchType=launch/slurm
#Licenses=foo*4,bar
#MailProg=/bin/mail
#MaxJobCount=5000
#MaxStepCount=40000
#MaxTasksPerNode=128
MpiDefault=openmpi
#MpiParams=ports=#-#
#PluginDir=
#PlugStackConfig=
#PrivateData=jobs
#ProctrackType=proctrack/cgroup
#Prolog=
#PrologFlags=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#RebootProgram=
ReturnToService=1
#SallocDefaultCommand=
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
#SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/lib/slurm-llnl/slurmd
SwitchType=switch/none
#TaskEpilog=
#TaskPlugin=task/affinity
#TaskPluginParam=Sched
#TaskProlog=
#TopologyPlugin=topology/tree
#TmpFS=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UsePAM=0
#
#
# TIMERS
#BatchStartTimeout=10
#CompleteWait=0
#EpilogMsgTime=2000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=120
SlurmdTimeout=300
#UnkillableStepTimeout=60
#VSizeFactor=0
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0
FastSchedule=1
#MaxMemPerCPU=0
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
#
#
# JOB PRIORITY
#PriorityFlags=
#PriorityType=priority/basic
#PriorityDecayHalfLife=
#PriorityCalcPeriod=
#PriorityFavorSmall=
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
#PriorityWeightFairshare=
#PriorityWeightJobSize=
#PriorityWeightPartition=
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING
#AccountingStorageEnforce=0
#AccountingStorageHost=
#AccountingStorageLoc=
#AccountingStoragePass=
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
#AccountingStorageUser=
AccountingStoreJobComment=YES
ClusterName=cluster
#DebugFlags=
#JobCompHost=
#JobCompLoc=
#JobCompPass=
#JobCompPort=
JobCompType=jobcomp/none
#JobCompUser=
#JobContainerType=job_container/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=3
#SlurmctldLogFile=
SlurmdDebug=3
#SlurmdLogFile=
#SlurmSchedLogFile=
#SlurmSchedLogLevel=
#
#
# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
#ResumeTimeout=
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
#
# COMPUTE NODES
NodeName=desktop-comp CPUs=1 State=UNKNOWN
PartitionName=debug Nodes=desktop-compDefault=YES MaxTime=INFINITE State=UP
In your update question you have in your slurm.conf the line
NodeName=desktop-comp CPUs=1 State=UNKNOWN
This tells slurm that you have only one CPU available on your node. You can try running slurmd -C to see what slurm discovers about your computer and copypaste the CPUs, CoresPerSocket etc. values to your slurm.conf.
On newly installed and configured compute nodes in our small cluster I am unable to submit slurm jobs using a batch script and the 'sbatch' command. After submitting, the requested node changes to the 'drained' status. However, I can run the same command interactively using 'srun'.
Works:
srun -p debug --ntasks=1 --nodes=1 --job-name=test --nodelist=node6 -l echo 'test'
Does not work:
sbatch test.slurm
with test.slurm:
#!/bin/sh
#SBATCH --job-name=test
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --nodelist=node6
#SBATCH --partition=debug
echo 'test'
It gives me:
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
debug up 1:00:00 1 drain node6
and I have to resume the node.
All nodes run Debian 9.8, use Infiniband and NIS.
I have made sure that all nodes have the same config, version of packages and daemons running. So, I don't see what I am missing.
Seems like the issue was connected to the present NIS. Just needed to add to the end of /etc/passwd this line:
+::::::
and restart slurmd on the node:
/etc/init.d/slurmd restart