I want to measure the memory usage of a job over time. I understand that I can use qstat -f <job id> to obtain the current memory usage. However, is there a way to track the memory usage over time? Perhaps I could write a bash script that would keep polling qstat and record the measurements over time.
The information that TORQUE reports is the virtual memory usage found in /proc/pid/stat. TORQUE does not offer a utility for saving this over time and profiling a job, but you could create one by regularly polling qstat (or writing an application that calls the API) and storing the results as you have suggested.
Related
We have a massively scaled nodejs project (~1m+ users) that is suddenly taking a massive beating on our CPU. (Epyc 24c 2ghz)
We've been trying to debug what's using all our CPU using a profiler, (and I can show you the output down below) and it's behaving really weirdly whatever it is.
We have a master process that spawns 48 clusters, after they're all loaded the cpu usage slowly grows to max. After killing a cluster, the LA doesn't dip at all. However after killing the master process, it all goes back to normal.
The master process obviously isn't maxing all threads, and killing a cluster should REALLY do the trick?
We even stopped the user input of the application and a cluster entirely and it didn't reduce cpu usage at all.
We've got plenty of log files we could send if you want them.
Based on the profile, it looks like the code is spending a lot of time getting the current time from the system. Do you maybe have Date.now() (or oldschool, extra-inefficient +new Date()) calls around a bunch of frequently used, relatively quick operations? Try removing those, you should see a speedup (or drop in CPU utilization, respectively).
As for stopping user input not reducing CPU load: are you maybe scheduling callbacks? Or promises, or other async requests? It's not difficult to write a program that only needs to be kicked off and then keeps the CPU busy forever on its own.
Beyond these rough guesses, there isn't enough information here to dig deeper. Is there anything other than time-related stuff on the profile, further down? In particular, any of your own code? What does the bottom-up profile say?
We're using SLURM to manage job scheduling on our computing cluster, and we experiencing a problem with memory management. Specifically, we can't find out how we can allocate memory for a specific task.
Consider the following setup:
Each node has 32GB memory
We have a SLURM job that sets --mem=24GB
Now, assume we want to run that SLURM job twice, concurrently. Then what I expect (or want) to happen is that when I queue it twice by calling sbatch runscript.sh twice, one of the two jobs will run on one node, and the other will run on another node. However, as it currently is, SLURM schedules both tasks on the same node.
One of the possible causes we've identified is that it appears to check only whether the 24GB of memory is available (i.e., not actively used by other node), instead of checking whether it is requested/allocated.
The question here is: is it possible to allocate/reserve memory per task in SLURM?
Thanks for your help!
In order to be able to manage memory slurm needs the parameter in SchedTypeParameters to include MEMORY. So just changing that parameter to CR_Core_Memory should be enough for Slurm to start to manage the memory.
If that is not set --mem will not reserve memory and only ensure that the node has enough memory configured.
More information here
#CarlesFenoy's answer is good, but to answer
The question here is: is it possible to allocate/reserve memory per
task in SLURM?
the parameter you are looking for is --mem-per-cpu, to use in combination with --cpus-per-task
How to execute a process for n cpu cycles on Linux? I have a batch processing system on a multi-core server and would like to ensure that each task gets exactle the same amount of cpu time. Once the cpu amount is consumed I would like to stop the process. So far I tried to do some thing with /proc/pid/stats utime and stime, but I did not succeed.
I believe it is impossible (to give the exact same number of cycles to several processes; a CPU cycle is often less than a nanosecond). You could execute a process for x CPU seconds. For that use setrlimit(2) with RLIMIT_CPU
Your batch processor could also manage time itself, see time(7). You could use timers (see timer_create(2) & timerfd_create(2)), have an event loop around poll(2), measure time with clock_getttime(2)
I'm not sure it is useful to write your own batch processing system. You could use the existing batch or slurm or gnqs (see also commercial products like pbsworks, lsf ..)
I am looking for a script that will spawn each command in a list until CPU/memory/network reaches a specified bound.
Some commercial scheduling tools will run as many jobs as it can until CPU utilization hits 90%. At that point, they wait until CPU utilization goes below a specified point and start another job. This maximizes utilization to finish the set faster.
An obvious example is in copying files. With 100+ files to copy, it is ludicrous to copy them one at a time. Such little CPU time is used, there should be many copies started. I/O bandwidth and network bandwidth become the constraint to manage.
I would like to not reinvent the wheel if there is already something available. Anyone know of something like this?
I was benchmarking the synchronous read performance on a linux box with SATA disk. I timed each read call with gettimeofday(2) and fired iostat -x to see disk statistics when the program was running. The disk IO time shown by iostat on the await column had an average of about 8msec, but the read time given by the program had an average of about 12msec. Where can these 4msec be spent?
Copying memory, doing context switches, and running other processes.
Other processes can get sheduled and run before your read starts or after it completes, which will extend the time that your process sees.