I am trying to run ArangoDB in cluster-mode on a Cray-supercomputer.
It runs on a login node.
I followed these instructions:
https://docs.arangodb.com/3.3/Manual/Deployment/Local.html
To make proper use of the Cray-cluster I however need to submit it as a batch-job (Slurm / sbatch).
I am having issues getting it running because "arangod" goes silent, that is its command-line output does not end up in the slurm-log-file.
I have tried to change the log-settings using this link:
https://docs.arangodb.com/3.3/Manual/Administration/Configuration/Logging.html
If I put the logging to "info" then I get nothing. If I use "trace" like this:
build/bin/arangod --server.endpoint tcp://0.0.0.0:5003 --agency.my-address tcp://148.187.32.9:5001 --server.authentication false --agency.activate true --agency.size 3 --agency.supervision true --database.directory db_dir/agency_2 --log.level startup=trace --log.level agency=trace --log.level queries=trace --log.level replication=trace --log.level threads=trace
I get something, but it does not print any line I'm interested in, namely if it created the database-directory, if it ends up in gossip-mode and so on. I don't get a line of the expected output I would get in the console if I just ran it from the terminal.
As I said: on the login-node it all works. I suspect the problem might be in the interaction of Slurm and arangod.
Can you help me?
* EDIT *
I ran a small experiment. First I ran this (expecting an error message):
#!/bin/bash -l
#SBATCH --job-name=slurm_test
#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-core=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=debug
#SBATCH --constraint=mc
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun build/bin/arangod --server.endpoint tcp://0.0.0.0:5001
And got this (first line from arangodb, what we expect):
slurm-....out:
no database path has been supplied, giving up, please use the '--database.directory' option
srun: error: nid00008: task 0: Exited with exit code 1
srun: Terminating job step 8106415.0
Batch Job Summary Report for Job "slurm_test" (8106415) on daint
-----------------------------------------------------------------------------------------------------
Submit Eligible Start End Elapsed Timelimit
------------------- ------------------- ------------------- ------------------- ---------- ----------
2018-06-20T22:41:54 2018-06-20T22:41:54 Unknown Unknown 00:00:00 00:30:00
-----------------------------------------------------------------------------------------------------
Username Account Partition NNodes Energy
---------- ---------- ---------- ------ --------------
peterem g34 debug 1 joules
This job did not utilize any GPUs
----------------------------------------------------------
Scratch File System Files Quota
-------------------- ---------- ----------
/scratch/snx3000 85020 1000000
Then I ran this:
#!/bin/bash -l
#SBATCH --job-name=slurm_test
#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-core=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=debug
#SBATCH --constraint=mc
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun build/bin/arangod --server.endpoint tcp://0.0.0.0:5001 --agency.my-address tcp://127.0.0.1:5001 --server.authentication false --agency.activate true --agency.size 1 --agency.supervision true --database.directory agency1
This created the "agency1" directory but did not complete (ran for over 3min). So after a few minutes I "scancel" the job. This is the only output (slurm-....out:):
srun: got SIGCONT
slurmstepd: error: *** STEP 8106340.0 ON nid00008 CANCELLED AT 2018-06-20T22:38:03 ***
slurmstepd: error: *** JOB 8106340 ON nid00008 CANCELLED AT 2018-06-20T22:38:03 ***
srun: forcing job termination
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
Batch Job Summary Report for Job "slurm_test" (8106340) on daint
-----------------------------------------------------------------------------------------------------
Submit Eligible Start End Elapsed Timelimit
------------------- ------------------- ------------------- ------------------- ---------- ----------
2018-06-20T22:32:15 2018-06-20T22:32:15 Unknown Unknown 00:00:00 00:30:00
-----------------------------------------------------------------------------------------------------
Username Account Partition NNodes Energy
---------- ---------- ---------- ------ --------------
peterem g34 debug 1 joules
This job did not utilize any GPUs
----------------------------------------------------------
Scratch File System Files Quota
-------------------- ---------- ----------
/scratch/snx3000 85020 1000000
So: I know it is running in both cases (gives output or crates folder). But I have no idea why it gives no output in the second case.
I hope this clarifies my issue.
Thanks, Emanuel
Would you please print your entire slurm job command / job file? arangod logs to stdout. When stdout is redirected to an output file, as cluster batch systems do per default, you should monitor that file. As far as I remember slurm per default writes to slurm-$jobid.out
Related
I want to run about 400 jobs in GCP-slurm from about 2,000 arrays.
The slurm settings and slurm.config settings in my bash file are as follows.
run.sh
#SBATCH -o ./out/vs.%j.out
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH -W
slurm.config
MaxArraySize=50000
MaxJobCount=50000
#COMPUTE NODE
NodeName=DEFAULT CPUs=16 RealMemory=63216 State=UNKNOWN
NodeName=node-0-[0-599] State=CLOUD
Currently, 100 nodes are being used for work other than that task.
If you proceed with this task, only about 130-150 node tasks in total are executed and the rest are not executed.
Are there any additional parameters that need to be set?
-- additional error log
[2022-06-20T01:18:41.294] error: get_addr_info: getaddrinfo() failed: Name or service not known
[2022-06-20T01:18:41.294] error: slurm_set_addr: Unable to resolve “node-333"
[2022-06-20T01:18:41.294] error: fwd_tree_thread: can't find address for host node-333, check slurm.conf
I found a workaround for the additional error.
https://groups.google.com/g/slurm-users/c/y-QZKDbYfIk
I referred to the article, and you can edit slurcmtld.service / slurmd.service / slurmdbd.service.
network.target -> network-online.target
However, the maximum number of execution nodes is still maintained.
I am struggling with a basic python script that uses Pytorch to print the CUDA devices on Slurm.
This is the output of sinfo.
(ml) [s.1915438#sl2 pytorch_gpu_check]$ sinfo -o "%.10P %.5a %.10l %.6D %.6t %.20N %.10G"
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST GRES
compute* up 3-00:00:00 1 drain* scs0123 (null)
compute* up 3-00:00:00 1 down* scs0050 (null)
compute* up 3-00:00:00 120 alloc scs[0001-0009,0011-0 (null)
compute* up 3-00:00:00 1 down scs0010 (null)
developmen up 30:00 1 drain* scs0123 (null)
developmen up 30:00 1 down* scs0050 (null)
developmen up 30:00 120 alloc scs[0001-0009,0011-0 (null)
developmen up 30:00 1 down scs0010 (null)
gpu up 2-00:00:00 2 mix scs[2001-2002] gpu:v100:2
gpu up 2-00:00:00 2 idle scs[2003-2004] gpu:v100:2
accel_ai up 2-00:00:00 1 mix scs2041 gpu:a100:8
accel_ai up 2-00:00:00 4 idle scs[2042-2045] gpu:a100:8
accel_ai_d up 2:00:00 1 mix scs2041 gpu:a100:8
accel_ai_d up 2:00:00 4 idle scs[2042-2045] gpu:a100:8
accel_ai_m up 12:00:00 1 idle scs2046 gpu:1g.5gb
s_highmem_ up 3-00:00:00 1 mix scs0151 (null)
s_highmem_ up 3-00:00:00 1 idle scs0152 (null)
s_compute_ up 3-00:00:00 2 idle scs[3001,3003] (null)
s_compute_ up 1:00:00 2 idle scs[3001,3003] (null)
s_gpu_eng up 2-00:00:00 1 idle scs2021 gpu:v100:4
I've access to accel_ai partition.
This is the Python file I am trying to run.
(ml) [s.1915438#sl2 pytorch_gpu_check]$ cat gpu.py
import torch
print(torch.__version__)
print(f"Is available: {torch.cuda.is_available()}")
try:
print(f"Current Devices: {torch.cuda.current_device()}")
except :
print('Current Devices: Torch is not compiled for GPU or No GPU')
print(f"No. of GPUs: {torch.cuda.device_count()}")
And this is my bash file to submit the job.
(ml) [s.1915438#sl2 pytorch_gpu_check]$ cat check_gpu.sh
#!bin/bash
#SBATCH --nodes=1
#SBATCH --time=00:00:40
#SBATCH --ntasks=1
#SBATCH --job-name=gpu
#SBATCH --output=gpu.%j.out
#SBATCH --error=gpu.%j.err
#SBATCH --mem-per-cpu=10
#SBATCH --gres=gpu:1
#SBATCH --account=scs2045
#SBATCH --partition=accel_ai
module load CUDA/11.3
module load anaconda/3
source activate
conda activate ml
python gpu.py
This is what happends when I run the bash script to submit the job.
(ml) [s.1915438#sl2 pytorch_gpu_check]$ bash check_gpu.sh
1.11.0
Is available: False
Current Devices: Torch is not compiled for GPU or No GPU
No. of GPUs: 0
One thing I would like to make clear is that this Pytorch version comes with CUDA 11.3 from Pytorch's website.
Can anyone tell, what am I doing wrong?
Also, here even I exclude these lines the output is the same.
module load CUDA/11.3
module load anaconda/3
source activate
conda activate ml
As per your sinfo, you have separate partitions with gpu access. You need to run your program on one of those. The job submission script can be modified as follows. You also need to specify gpu type using --gres.
...
...
#SBATCH --partition=gpu
#SBATCH --gres=<Enter gpu type>
...
...
There is a couple of blunders in my approach. In the job file, the first line should be #!/bin/bash not #!bin/bash.
Also, Slurm has a special command SBATCH to submit your job file. So in order to run your job file, for example check_gpu.sh, we should use sbatch check_gpu.sh not bash check_gpu.sh.
The reason I was getting the following output is that bash thinks # is a comment.
(ml) [s.1915438#sl2 pytorch_gpu_check]$ bash check_gpu.sh
1.11.0
Is available: False
Current Devices: Torch is not compiled for GPU or No GPU
No. of GPUs: 0
Thus, only the following lines are executed from the job script.
module load CUDA/11.3
module load anaconda/3
source activate
conda activate ml
python gpu.py
After the correction, I ran the job script and it works as expected.
[s.1915438#sl1 pytorch_gpu_check]$ sbatch check_gpu.sh
Submitted batch job 7133028
[s.1915438#sl1 pytorch_gpu_check]$ cat gpu.7133029.out
1.11.0
Is available: True
Current Devices: 0
No. of GPUs: 1
GPU Name:NVIDIA A100-PCIE-40GB
[s.1915438#sl1 pytorch_gpu_check]$ cat gpu.7133029.err
I am trying to send an sbatch file to run a code in 200 cores in the system. My code is
#!/bin/sh
#SBATCH --job-name=sm #Job name
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=sankalpmathur#ufl.edu # Where to send mail
#SBATCH --mem-per-cpu=3gb # Per processor memory
#SBATCH --array=1-200
#SBATCH -t 199:00:00 # Walltime
#SBATCH -o output_%a.out # Name output file
#
pwd; hostname; date
module load stata
stata-mp -b do array_${SLURM_ARRAY_TASK_ID}.do
When I run the file I get this error
sbatch HS8_main.sbatch
sbatch: error: Unable to open file HS8_main.sbatch
I have run the same sbatch before and it ran fine. What could possibly be the reason for it to not run this time?
Thank you
That's the error one gets when the sbatch script isn't in the current directory, or the name is wrong. Are you sure HS8_main.sbatch is the name of your script, and it's in the same place you're running sbatch from?
just try "pwd" to check out the
On newly installed and configured compute nodes in our small cluster I am unable to submit slurm jobs using a batch script and the 'sbatch' command. After submitting, the requested node changes to the 'drained' status. However, I can run the same command interactively using 'srun'.
Works:
srun -p debug --ntasks=1 --nodes=1 --job-name=test --nodelist=node6 -l echo 'test'
Does not work:
sbatch test.slurm
with test.slurm:
#!/bin/sh
#SBATCH --job-name=test
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --nodelist=node6
#SBATCH --partition=debug
echo 'test'
It gives me:
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
debug up 1:00:00 1 drain node6
and I have to resume the node.
All nodes run Debian 9.8, use Infiniband and NIS.
I have made sure that all nodes have the same config, version of packages and daemons running. So, I don't see what I am missing.
Seems like the issue was connected to the present NIS. Just needed to add to the end of /etc/passwd this line:
+::::::
and restart slurmd on the node:
/etc/init.d/slurmd restart
I submit some cluster mode spark jobs which run just fine when I do it one by one with below sbatch specs.
#!/bin/bash -l
#SBATCH -J Spark
#SBATCH --time=0-05:00:00 # 5 hour
#SBATCH --partition=batch
#SBATCH --qos qos-batch
###SBATCH -N $NODES
###SBATCH --ntasks-per-node=$NTASKS
### -c, --cpus-per-task=<ncpus>
### (multithreading) Request that ncpus be allocated per process
#SBATCH -c 7
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --dependency=singleton
If I use a launcher to submit the same job with different node and task numbers, the system gets confused and tries to assign according to $SLURM_NTASK which gives 16. However I ask for example only 1 node,3tasks.
#!/bin/bash -l
for n in {1..4}
do
for t in {3..4}
do
echo "Running benchmark with ${n} nodes and ${t} tasks per node"
sbatch -N ${n} --ntasks-per-node=${t} spark-teragen.sh
sleep 5
sbatch -N ${n} --ntasks-per-node=${t} spark-terasort.sh
sleep 5
sbatch -N ${n} --ntasks-per-node=${t} spark-teravalidate.sh
sleep 5
done
done
How can I fix the error below by preventing slurm assign weird number of tasks per node which exceeds the limit.
Error:
srun: Warning: can't honor --ntasks-per-node set to 3 which doesn't match the
requested tasks 16 with the number of requested nodes 1. Ignoring --ntasks-per-node.
srun: error: Unable to create step for job 233838: More processors requested than
permitted