I'm looking for the SGE equivalent to PBS's "qstat -freenodes", basically I want to find all nodes that have nothing running. Suggestions?
If you have an exclusive resource associated with the hosts you can
use qhost:
qhost -l exclusive=TRUE
This should work even if you make the exclusive resource non-requestable.
It should be:
pbsnodes -l free
Related
When I submit by SBATCH job to our HPC, I believe slurm allocates nodes based on resources, and in my case, the Host is always spawned on Node 0 which is set as being the 1st in an alphabetical sort of the node/machine names. This is causing problems because (sometimes) this Host node may only have 1 core running, (and thus a small amountof memory) meaning it is unable to write large results/data files I need.
Is there any way to set the host node manually, given the resources slurm allocates in my nodefile?
I could fix this with -mincpus but I only need >1 cpu for this one purpose. Other solutions increasing --mem-per-cpu or just --mem also just add more resources to the job and delay it from starting.
You can use the --nodelist parameter to set specific nodes that should be used:
sbatch --nodelist=<NODE-NAME> script.sh
Or even --exclude the ones you do not want to use (e.g. node 0):
sbatch --exclude=node0 script.sh
The official documentation provides more information on both options.
I'm looking for a way to submit an OpenMP job to a Grid Engine scheduler, while specifying the number of cores it should run on. Something equivalent to LSF's -n option, or PBS's -l nodes=[count] option.
When I search on this, I'm see a bunch of answers specifying syntax like "-pe threaded [number of cores]". In those answers, there is no mention of having to create a parallel environment called "threaded". But when I try this syntax, it fails, saying that the requested parallel environment threaded does not exist. And when I type "qconf -spl", the only result I get is "make". So - should this "threaded" parallel environment exist by default, or is this something that has to be manually created on the cluster?
If it has to be manually created, is there any other syntax to submit jobs to multiple cores that does not rely on configurable naming on a cluster? This is for a third party program submitting to a cluster, so I don't want to have to rely not only on the client having created this pe, but naming it the same, etc... I was hoping the -l option might have something, but I haven't been able to find any permutation of that to achieve this.
If you get only "make" as possible parallel environment then this means that there are no parallel environments set on your cluster.
There are two solutions to your problem, depending on these 2 situations:
A) you have root/admin access to the cluster
B) you don't
In case B, well ask your administrator to create a parallel environment. In case A, you have to create a parallel environment. To create a new parallel environment you must type (requires root/admin privilege):
qconf -ap <pe_name>
And the default editor will start with a default pe_conf file that you must edit. If you need to setup only an openMP parallel environment you can use these options:
pe_name smp
slots 9999
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule $pe_slots
control_slaves FALSE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE
and for a MPI parallel environment:
pe_name mpi
slots 9999
user_lists NONE
xuser_lists NONE
start_proc_args /opt/sge/mpi/startmpi.sh $pe_hostfile
stop_proc_args /opt/sge/mpi/stopmpi.sh
allocation_rule $fill_up
control_slaves FALSE
job_is_first_task TRUE
urgency_slots min
accounting_summary TRUE
as you notice, in the latter case you will point SGE to the right initialization script and shutdown script for your MPI configuration. In the first case, you simply point to /bin/true.
The allocation_rule are different in this example. $fill_up means that SGE will fill any CPU it can find with parts of the MPI job, while for smp configuration you simply allocate the correct number of slots on the same machine, i.e. $pe_slots.
If you use MPI, your nodes should be connected using a high performance switch such as infiniband otherwise your jobs will spend much more time communicating than calculating.
EDIT:
oh, btw: the correct synthax to submit a job with a parallel environment is effectively:
qsub -pe <pe_name> <nb_slots>
FINAL EDIT:
the final answer to the question comes in the comments here below. In practice, SGE cannot handle multi-thread jobs if a parallel environment (PE) is not set on the cluster. If you do not have admin privileges on the cluster, you must either guess for the correct PE that has to be used using qconf -spl and inspect the different PEs with qconf -sp <pe_name>, or add an option in your software that allows the users to specify the PE that has to be used.
Otherwise, i.e. if no PE are available on the cluster, you cannot use a parallel version of your software.
See the comments for further information.
I'm having troubles with cpusets it will be great if you could help me
I've defined two cpuset groups: "cpuset_0" which has only one task, and "cpuset_1" which is for all the other tasks in my system.
"cpuset_0" has cpus="0", cpu_exclusive="1" and only the one task assign to it.
and "cpuset_1" has cpus="1-3", cpu_exclusive="0" and all the tasks I could move as root from the root cpuset.
Both cpusets has mems="0".
The problem is that for some reason I see tasks which assigned to "cpuset_1" which are running on the exclusive cpu "cpuset_0"
For example running ps H -eo tid,psr,cgroup,cmd
gives me:
2199 0 6:cpuset:/cpuset_1?5:freeze /usr/lib/chromium-browser/chromium-browser
among other processes which shouldn't be running on cpu 0.
BTW: I'm running kernel version 3.2.0
Were you able to actually make it work without using cpuset.mems ?. It is mandatory. What does your config looks like. or you have used the mount command.
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpuset.html
try to follow the below
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch-Using_Control_Groups.html
I want to use the /proc to find the resource usage of a particular process every second. The resources include cputime, disk usage and network usage. I looked at /proc/pid/stat , but I am not sure whether I am getting the required details. I want all 3 resource usage and I want to monitor them every second.
Some newer kernels have /proc/<pid_of_process>/io file. This is where IO stats are.
It is not documented in man proc, but you can try to figure out the numbers yourself.
Hope it helps.
Alex.
getrusage() accomplishes cpu, memory and disk etc.
man 2 getrusage
I don't know about network.
checkout glances.
It's got cpu disk and network all on one screen. It's not per process but it's better than looking at 3 separate tools.
Don't think there is a way to get the disk and network information on a per process basis.
The best you can have is the global disk and network, and the per process CPU time.
All documented in man proc
netstat -an
Shows all connections to the server including the source and destination ips and ports if you have proper permissions.
ac
Prints statistics about users' connect time
The best way to approach problems like this is to look up the source code of tools that perform similar monitoring and reporting.
Although there is no guarantee that they are using /proc directly, they will lead you to an efficient way to tackle the problem.
For your case, top(1), iotop(8) and nethogs(8) come to mind.
You can use SAR
-x report statistics for a given process.
See this for more details:
http://www.linuxcommand.org/man_pages/sar1.html
Example:
sar -u -r -b 1 -X PID | grep -v Average | grep -v Linux
You can use top
SYNOPSIS
top -hv|-bcHiOSs -d delay -n limit -u|U user -p PID -o champ -w [columns]
This is a screen capture of top in a terminal
I am a little concerned with the amount of resources that I can use in a shared machine. Is there any way to test if the administrator has a limit in the amount of resources that I can use? And if does, to make a more complete question, how can I set up such limit?
For process related limits, you can have a look in /etc/security/limits.conf (read the comments in the file, use google or use man limits.conf for more information). And as jpalecek points out, you may use ulimit -a to see (and possibly modify) all such limits currently in effect.
You can use the command quota to see if a disk quota is in effect.
You can try running
ulimit -a
to see what resource limits are in effect. Also, if you are allowed to change such limits, you can change them by the ulimit command, eg.
ulimit -c unlimited
lifts any limit for a size of a core file a process can make.
At the C level, the relevant functions (actually syscalls(2)...) could be setrlimit(2) and setpriority(2) and sched_setattr(2). You probably would want them to be called from your shell.
See also proc(5) -and try cat /proc/self/limits and sched(7).
You may want to use the renice(1) command.
If you run a long-lasting program (for several hours) not requiring user interaction, you could consider using some batch processing. Some Linux systems have a batch or at command.