I am trying to get the location of the stdout and stderr file of an already completed job.
Indeed, while the job is running, I could do
scontrol show job $JobId
However, this does not work after a job is finished
I am able to get information about previous completed jobs with sacct,
However, there is no option to display the location of stderr and stdout with this command.
The only information, I found about this issue is this https://groups.google.com/g/slurm-users/c/e4cZMbtrMM0 . However, this suggests changing slurm.conf so that scontrol show job $JobId retains information. This is not possible in my case because I do not have access to slurm.conf
So I was wondering if there was a way with slurm to get the location of the stdout and stderr of a completed job?
Thanks for your help
---- edit ----
The jobs are submitted with a bash file
#SBATCH --output=...
#SBATCH --error=...
By running the command sbatch $submission_file
This means retrieving the command to submitted the file does not help. Indeed this will retrieve only sbatch $jobfile and not give any further information on the output and error directory.
Although Slurm does not seem to save that information in the accounting database explicitly, it does save the information of the working directory, which you can obtain with
sacct -j <JOBID> -o workdir%-100
Most of the time, chances are the output and error files will be relative to that directory.
Slurm also saves the submission command, which you can retrieve with
sacct -j <JOBID> -o SubmitLine%-100
which will reveal output and error files in the case they were provided in the command line.
Finally, note that Slurm will also save the full submission script if configured to do so (which could not be the case on your cluster). If so, you can retrieve it with
sacct -j <JOBID> --batch-script
I often have lots of SLURM jobs running from different directories. Therefore, it is useful to query the workdir of the jobs. I can do this for jobs in the queue (e.g. pending, running, etc.) something like this:
squeue -u $USER -o "%i %Z"
and I can do this for finished jobs (e.g. completed, timeout, cancelled, etc.) something like this:
sacct -u $USER -o JobID,WorkDir
The problem is, sometimes I have a directory with two (or more) SLURM batch scripts in it, e.g. submit.sh and restart.sh. Therefore, it is also useful to query the "command" of the jobs, i.e. the filename of the batch script. I can do this for jobs in the queue something like this:
squeue -u $USER -o "%i %o"
However, from checking the documentation of sacct and playing around with sacct, there appears to be no equivalent option for sacct so I cannot currently get the command for finished jobs. I also cannot use the squeue method for finished jobs - it just says slurm_load_jobs error: Invalid job id specified because finished jobs are not included in the squeue list. So, how can I find out the command of a finished SLURM job (using sacct or otherwise)?
Slurm does not indeed store the command in the accounting database. Two workarounds:
For a single user: use the JobName or Comment to store the script name upon submission. These are stored in the database, but this approach is error-prone;
Cluster-wise: enable job completion plugin to ElastiSearch as this stores not only the script name but the whole contents as well.
I am using a Torque+MAUI cluster.
The cluster's utilization now is ~10 node/40 nodes available, a lot of job being queued but cannot be started.
I submitted the following PBS script using qsub:
#!/bin/bash
#
#PBS -S /bin/bash
#PBS -o STDOUT
#PBS -e STDERR
#PBS -l walltime=500:00:00
#PBS -l nodes=1:ppn=32
#PBS -q zone0
cd /somedir/workdir/
java -Xmx1024m -Xms256m -jar client_1_05.jar
The job gets R(un) status immediately, but I had this abnormal information from qstat -n
8655.cluster.local user zone0 run.sh -- 1 32 -- 500:00:00 R 00:00:31
z0-1/0+z0-1/1+z0-1/2+z0-1/3+z0-1/4+z0-1/5+z0-1/6+z0-1/7+z0-1/8+z0-1/9
+z0-1/10+z0-1/11+z0-1/12+z0-1/13+z0-1/14+z0-1/15+z0-1/16+z0-1/17+z0-1/18
+z0-1/19+z0-1/20+z0-1/21+z0-1/22+z0-1/23+z0-1/24+z0-1/25+z0-1/26+z0-1/27
+z0-1/28+z0-1/29+z0-1/30+z0-1/31
The abnormal part is -- in run.sh -- 1 32, as the sessionId is missing, and evidently the script does not run at all, i.e. the java program does not ever had traces of being started.
After this kind of strange running for ~5 minutes, the job will be set back to Q(ueue) status and seemingly will not being run again (I had monitored this for ~1 week and it does not run even being queued to the top most job).
I tried submit the same job 14 times, and monitored its node in qstat -n, 7 copies ran successfully, having varied node numbers, but all jobs being allocated z0-1/* get stuck with this strange startup behavior.
Anyone know a solution to this issue?
For a temporary workaround, how can I specify NOT to use those strange nodes in PBS script?
It sounds like something is wrong with those nodes. One solution would be to offline the nodes that aren't working: pbsnodes -o <node name> and allow the cluster to continue to work. You may need to release the holds on any jobs. I believe you can run releasehold ALL to accomplish this in Maui.
Once you take care of that I'd investigate the logs on those nodes (start with the pbs_mom logs and the syslogs) and figure out what is wrong with them. Once you figure out and correct what is wrong with them, you can put the nodes back online: pbsnodes -c <node_name>. You may also want to look into setting up some node health scripts to proactively detect and handle these situations.
For users, contact your administrator and in the mean time, run the job using this workaround.
Use pbsnodes to check for free and healthy nodes
Modify PBS directive #PBS -l nodes=<freenode1>:ppn=<ppn1>+<freenode2>:ppn=<ppn2>+...
submit the job using qsub
I suppose it's a pretty trivial question but nevertheless, I'm looking for the (sacct I guess) command that will display the CPU time and memory used by a slurm job ID.
If your job is finished, then the sacct command is what you're looking for. Otherwise, look into sstat. For sacct the --format switch is the other key element. If you run this command:
sacct -e
you'll get a printout of the different fields that can be used for the --format switch. The details of each field are described in the Job Account Fields section of the man page. For CPU time and memory, CPUTime and MaxRSS are probably what you're looking for. cputimeraw can also be used if you want the number in seconds, as opposed to the usual Slurm time format.
sacct --format="CPUTime,MaxRSS"
The other answers all detail formats for output of sacct, which is great for looking at multiple jobs aggregated in a table.
However, sometimes you want to look at a specific job in more detail, so you can tell whether your job efficiently used the allocated resources. For that, seff is very useful. The syntax is simply seff <Jobid>. For example, here's a recent job of mine (that failed):
$ seff 15780625
Job ID: 15780625
Cluster: mycluster
User/Group: myuser/mygroup
State: OUT_OF_MEMORY (exit code 0)
Nodes: 1
Cores per node: 16
CPU Utilized: 12:06:01
CPU Efficiency: 85.35% of 14:10:40 core-walltime
Job Wall-clock time: 00:53:10
Memory Utilized: 1.41 GB
Memory Efficiency: 70.47% of 2.00 GB
Note that the key CPU metric, CPU Utilized, corresponds to the TotalCPU field from sacct, while Memory Utilized corresponds to MaxRSS.
sacct is indeed the command to use for finished jobs. For running jobs, you can look at the sstat command.
#aaron.kizmiller is right, sacct is the command to use.
One can fetch all of the following fields by passing them into saact --format="field,field"
Fields:
Account AdminComment AllocCPUS AllocGRES
AllocNodes AllocTRES AssocID AveCPU
AveCPUFreq AveDiskRead AveDiskWrite AvePages
AveRSS AveVMSize BlockID Cluster
Comment ConsumedEnergy ConsumedEnergyRaw CPUTime
CPUTimeRAW DerivedExitCode Elapsed ElapsedRaw
Eligible End ExitCode GID
Group JobID JobIDRaw JobName
Layout MaxDiskRead MaxDiskReadNode MaxDiskReadTask
MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask MaxPages
MaxPagesNode MaxPagesTask MaxRSS MaxRSSNode
MaxRSSTask MaxVMSize MaxVMSizeNode MaxVMSizeTask
McsLabel MinCPU MinCPUNode MinCPUTask
NCPUS NNodes NodeList NTasks
Priority Partition QOS QOSRAW
ReqCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov
ReqCPUS ReqGRES ReqMem ReqNodes
ReqTRES Reservation ReservationId Reserved
ResvCPU ResvCPURAW Start State
Submit Suspended SystemCPU Timelimit
TotalCPU UID User UserCPU
WCKey WCKeyID WorkDir
For example, to list all job ids, elapsed time, and max VM size, you can run:
sacct --format='JobID,Elapsed,MaxVMSize'
Although there already exist fantastic solutions, I share an another perspective.
This method can do the real time monitoring of a lot of nodes.
We can write a script monitor.sh to obtain the statistic(memory as an example), then logged it into file.
#! /bin/sh
if [ -f "./free.log_"`hostname` ];then
echo "file existed , now deleting it !"
rm ./free.log_`hostname`
fi
echo "start recording!"
while true
do
echo "******["`date +%Y-%m-%d_%H:%M:%S`"]******" >> free.log_`hostname`
free -s 1 -c 2 -h|sed -n 1,2p >> free.log_`hostname`
done
Then write your job script sbatch_input.sh, which can be called by sbatch.
#! /bin/sh
#SBATCH -N 2
#SBATCH -p cnall
srun hostname
srun ./monitor.sh
Call the script
sbatch ./sbatch_input.sh
We can see some log generated.
You can export SACCT_FORMAT and just type sacct every time.
$ export SACCT_FORMAT="JobID%20,JobName,User,Partition,NodeList,Elapsed,CPUTime,State,AllocTRES%32"
$ sacct
JobID JobName User Partition NodeList Elapsed CPUTime State AllocTRES
-------------------- ---------- --------- ---------- --------------- ---------- ---------- ---------- --------------------------------
249527_4 xgb_tune zhaoqi cn cn12 00:26:50 1-11:46:40 RUNNING billing=80,cpu=80,mem=100G,node+
249527_1 xgb_tune zhaoqi cn cn09 00:26:50 1-11:46:40 RUNNING billing=80,cpu=80,mem=100G,node+
249527_2 xgb_tune zhaoqi cn cn10 00:26:50 1-11:46:40 RUNNING billing=80,cpu=80,mem=100G,node+
249527_3 xgb_tune zhaoqi cn cn11 00:26:50 1-11:46:40 RUNNING billing=80,cpu=80,mem=100G,node+
ref: https://docs.ycrc.yale.edu/clusters-at-yale/job-scheduling/resource-usage/
sacct -a -j <job_id> --format=user%10,jobname%10,node%10,start%10,end%10,elapsed%10,MaxRS
use the command sacct to access finished slurm job history
for <job_id>, it's referring to the slurm job id
then, this --format= to mention the different details to display, with which format:
the user: the user run the job
the jobname: the job or process name
the node: this to indicate in which machine the job was done
the start and end are indicating successively the job start and end dates
for elapsed it's about the runtime of job or process,
and MaxRS for max cpus used to get the job done
for % it's used to determine how much characters to dedicated for printing a given info (e.g jobname%25: Jobname will be displayed in 25 characters)
I want to get the details of the last run cron job. If the job is interrupted due to some internal problems, I want to re-run the cron job.
Note: I don't have superuser privilege.
You can see the date, time, user and command of previously executed cron jobs using:
grep CRON /var/log/syslog
This will show all cron jobs. If you only wanted to see jobs run by a certain user, you would use something like this:
grep CRON.*\(root\) /var/log/syslog
Note that cron logs at the start of a job so you may want to have lengthy jobs keep their own completion logs; if the system went down halfway through a job, it would still be in the log!
Edit: If you don't have root access, you will have to keep your own job logs. This can be done simply by tacking the following onto the end of your job command:
&& date > /home/user/last_completed
The file /home/user/last_completed would always contain the last date and time the job completed. You would use >> instead of > if you wanted to append completion dates to the file.
You could also achieve the same by putting your command in a small bash or sh script and have cron execute that file.
#!/bin/bash
[command]
date > /home/user/last_completed
The crontab for this would be:
* * * * * bash /path/to/script.bash
/var/log/cron contains cron job logs. But you need a root privilege to see.
CentOs,
sudo grep CRON /var/log/cron