What happens if I would try to run a multithreaded job in 1 SGE slot? Would it fail to start multiple threads? Or would it still start these multiple threads and potentially overload the SGE cluster node, because it is going to run more threads than there are slots?
I know I should use the -pe threaded nrThreads parameter. But I am running a program of which I am not sure how many threads it is using for every step.
It's been a while since I've used SGE, but at least back then, a job which launched more computational threads than allocated would not be prevented from launching those threads, usually then stealing CPU time from other jobs.
Perhaps current SGE versions are capable of using cpusets, which allow the administrator to limit the CPU's used by a job. At least the slurm scheduler can do this.
Related
I developed a Python script that enables multiple-threads. I sent my code to Slurm. I would like to verify if the multiple threads work well.
Therefore I want to monitor the real-time CPUs' usage. Is there any command can do this?
Most clusters allow logins to nodes if the user has a job running on it. So, just ssh to the node that is running your job, and run top(1)
If your code is multithreaded, the value in the %CPU column should be greater than 100%. Each thread, if it is fully busy, can consume up to 100% CPU (i.e. all the cycles on a single CPU core). So, N threads can consume up to N*100%.
Let's say, I've got a 4-core CPU, which basically means, my computer could run 4 processes at the same time; no more no less.
Now let's look at cluster module of nodejs; there are
1 master process
4 worker processes
Can I say, each worker process is to be assigned to each core of the CPU?
And if it is, then where is its master process located?
The master process is fired as a standard single-thread process (or service) so it "is assigned to" and lives on one of your CPU cores (or threads, depending on the case).
Each worker process is then spawned (forked) as described in the docs:
https://nodejs.org/api/cluster.html
Just so you know, Node now supports the use of worker_threads to execute JavaScript in parallel on multiple threads, so it's technically not true anymore that Node is limited by single-thread execution:
https://nodejs.org/api/worker_threads.html
I'm an intern that's been tasked with installing slurm across three compute units running ubuntu. How things work now is people ssh into one of the compute units and run a job on there, since all three units share memory through nfs mounting. Otherwise they are separate machines though.
My issue is that from what I've read in the documentation, it seems like when installing slurm I would specify each of these compute units as a completely separate node, and any jobs I'd like to run that use multiple cores would still be limited by how many cores are available on the individual node. My supervisor has told me however that the three units should be installed as a single node, and when a job comes in that needs more cores than available on a single compute unit, slurm should just use all the cores. The intention is that we won't be changing how we execute jobs (like a parallelized R script), just "wrapping" them in a sbatch script before sending them to slurm for scheduling and execution.
So is my supervisor correct in that slurm can be used to run our parallelized scripts unchanged with more cores than available on a single machine?
Running a script on more cores than available is nonsense. It does not provide any performance increase, rather the oposite, as more threads have to be managed but the computing power is the same.
But he is right in the sense that you can wrap your current script and send it to SLURM for execution using the whole node. But the three machines will be three nodes. They cannot work as a single node because, well, they are not a single node/machine. They do not share either memory nor busses, nor peripherals... they just share some disk thru the network.
You say that
any jobs I'd like to run that use multiple cores would still be limited by how many cores are available on the individual node
but that's the current situation with SSH. Nothing is lost by using SLURM to manage the resources. In fact, SLURM will take care of giving each job the proper resources and avoiding other users interfering with your computations.
Your best bet: create a cluster of three nodes as usual and let people send their jobs asking for as many resources they need without exceeding the available resources.
I'm using back-fill scheduler with Slurm to manage a small GPU cluster. The backfill scheduler makes a scheduling decision every bf_interval seconds (default value is 30 seconds). This means even when GPU resources are available sometimes I have to wait for a while until the they are allocated. I can obviously reduce bf_interval but given that we don't have a lot of job submissions it'd be good if I could force slurm to run the scheduling routine the moment a job is queued. Is this possible?
By default Slurm does it. From the documentation:
Slurm is designed to perform a quick and simple scheduling attempt at events such as job submission or completion and configuration changes.
Have you change the default configuration for this? And, are you sure that not scheduling on submission is your problem?
When utilizing multi cores via Node.js' cluster module is it guaranteed that each forked node worker is assigned to a different core?
If it's not guaranteed is there any way to control or manage it and eventually guarantee that all end up in different cores? Or the OS' scheduler distributes them evenly?
A while ago I did some tests with cluster module which you can check in this post that I wrote. Looking at the system monitor screenshots it is this pretty straightforward to understand what happens under the hood (with and without cluster module).
It is indeed up to the OS to distribute processes over the cores. You could obtain the pid of a child process and use an external utility to set the CPU affinity of that process. That would, of course, not be very portable.