When I execute shell scripts bash.sh like this：
for((i = 0; i<=5 ;i++))
do
./test.sh &
done
and test.sh like this:
for((i = 0; i<=10000000 ;i++))
do
echo 'hh'
done
the cpu us usage should be very high.
But when I use top command to check the result, the cpu us value of the very first time of the top command result, before it refreshes, is very small.
So the cpu usage is not credible!
Are the other values credible the first time top command result refresh?
The load average reported on the top right is the average over the last 1, 5 and 15 minutes (from the man top page):
system load avg over the last 1, 5 and 15 minutes
So this value will be averaged over the last minute, and as a result your process with a high CPU usage will not have an immediate effect on it.
The %CPU column shows the usage since the last update:
The task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time
So on the first time, the %CPUs are off and will change when the screen updates.
Related
I'm trying to find out how I can get \ calculate the CPU utilization of a specific process over X amount of time ( I write my code in python over a Linux based system ).
What I want to get for example is the average CPU of a process in the last hour\day\10 minutes...
Is there a command or a calculation I can run?
*I can't run a command like "top" in the background for X time and calculate the CPU, I need it to be in one set of commands or calculation.
I tried top research on top command but I didn't found useful info for my case.
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu - give the average consumption on the process lifetime
Can there be a way to use uptime or proc[pid]\stat to calculate this?
Thanks,
What about using pidstat?
$ pidstat -p 12345 10
Will output the stats for pid 12345 every 10 seconds. This include CPU%
From there you can run it on background, and redirect output to a file:
$ pidstat -p 12345 10 > my_pid_stats.txt &
Here is a Link with several examples. There is a lot of flexibility in the output, so you can probably customize it to better suit your needs:
https://www.thegeekstuff.com/2014/11/pidstat-examples/
pidstat is part of the sysstat package on ubuntu, in case you decide to install it.
I am running code in a loop for multiple iterations on a dedicated CPU with RT priority and want to observe its behaviour over a long time. I found a very strange periodic behaviour of the code.
Briefly, this is what the code does:
Arraythread
{
while(1)
{
if(flag)
Multiply matrix
record time;
reset flag;
}
}
mainthread
{
for(30 mins)
{
set flag;
record time;
busy while(500 μs)
}
}
Here are the details about the machine I am using:
CPU: Intel(R) Xeon(R) Gold 6230 CPU # 2.10 GHz
L1 cache: 32K d and 32K i
L2 cache: 1024K
L3 cache: 28160K
Kernel: 3.10.0-693.2.2.rt56.623.el7.x86_64 #1 SMP PREEMPT RT
OS: CentOS
Current active profile: latency-performance
I modified the global limit of Linux real time scheduling (sched_rt_runtime_us) from 95% to 100%
Both the above mentioned threads are bound on a single NUMA node each with priority 99
More details about the code:
mainthread sets a flag every 500 μs. I used CLOCK_MONOTOMIC_RAW with clock_gettime function to read the time (let's say T0).
I put all the variables in a structure to reduce the cache misses.
Arraythread runs a busy while loop and waits for the flag to set.
Once the flag is set it multiplies two big arrays.
Once the multiplication is done it reset the flag and record the time (let's say T1).
I run this experiment for 30 mins (= 3600000 iterations)
I measure the time difference T1-T0 once the experiment is over.
Here is the clock:
The average time of the clock is ~500.5 microseconds. There are flactuations which are expected.
Here is the time taken by the array multiplication:
This is the full 30 minute view of the result.
There are four peaks in the results. The first peak is expected since for the very first time data comes from main memory and the CPU was on sleep.
Apart from the first peak, there are three more peaks and the time difference between peak_3 and peak_2 is 11.99364 mins where the time difference between peak_4 and peak_3 is 11.99358 mins. (I assumed the clock to be 500 μsec)
If I zoom it further:
This image shows what happened over 5 minutes.
If I zoom it further:
This image shows what happened over ~1.25 mins.
You notice that average time is around 113 μsec of the multiplication and there are peaks everywhere.
If I zoom it further:
This image shows what happened over 20 seconds.
If I zoom it further:
This image shows what happened over 3.5 seconds.
The time differences between the starting line of these peaks are: 910 ms, 910 ms, 902 ms (assuming two consecutive points are at 500 μs difference)
If I zoom it further:
This image shows what happened over 500 ms
~112.6 μs is the average time here and complete data is under 1 μs range.
Here are my questions:
Given that L3 cache is good enough to store the complete executable and there is no file read right and there is nothing else is running on the machine, no context switch is happening as well, why do some of the executions take almost double (or sometimes more than double) time? [see the peaks in first result image]
If we forget about those four peaks from the first image, how do I justify the periodic peaks in the results with almost constant time difference? What does the CPU do? These periodic peaks lasts few milliseconds.
I expect the results to be near constant like in the last image. Is there a way or OS/CPU settings I can apply to run the code like last image for infinite time?
Here is the complete code:
https://github.com/sghoslya/kite/blob/main/multiThreadProfCheckArray.c
I am using the iostat utility on my RedHat Linux server to monitor the performance of a disk. When I use "iostat -xd sdh 1", I get the perf result printed every one second. When I use "iostat -xd sdh 5", I get the perf result printed every five second. My feeling is the latter command is printing a snapshot of the perf every five second, rather than averaging over the past 5 seconds. Am I correct in my understanding?
If so, is there a way I can make iostat print the perf. number averaged over n seconds, or is there some other utility that will do that.
Currently, the perf number is fluctuating within a range, and I want to get a somewhat "stable" number. I am hoping that averaging over a period of time will give me such a number.
Thank you,
Ahmed.
In a GCC bug report (Bug 51617), the poster times his asynchronous (C++11) program's execution with an output looking like this:
/tmp/tst 81.54s user 0.23s system 628% cpu 13.001 total
What could the poster be using which gives that (or similar) output?
NB: An inspection of my man entry for time doesn't suggest anything useful to this end.
The time(1) man page on my (Ubuntu) Linux system says:
-f FORMAT, --format FORMAT
Use FORMAT as the format string that controls the output of
time. See the below more information.
:
FORMATTING THE OUTPUT
The format string FORMAT controls the contents of the time output. The
format string can be set using the `-f' or `--format', `-v' or `--ver‐
bose', or `-p' or `--portability' options. If they are not given, but
the TIME environment variable is set, its value is used as the format
string. Otherwise, a built-in default format is used. The default
format is:
%Uuser %Ssystem %Eelapsed %PCPU (%Xtext+%Ddata %Mmax)k
%Iinputs+%Ooutputs (%Fmajor+%Rminor)pagefaults %Wswaps
:
The resource specifiers, which are a superset of those recognized by
the tcsh(1) builtin `time' command, are:
% A literal `%'.
C Name and command line arguments of the command being
timed.
D Average size of the process's unshared data area, in
Kilobytes.
E Elapsed real (wall clock) time used by the process, in
[hours:]minutes:seconds.
F Number of major, or I/O-requiring, page faults that oc‐
curred while the process was running. These are faults
where the page has actually migrated out of primary memo‐
ry.
I Number of file system inputs by the process.
K Average total (data+stack+text) memory use of the
process, in Kilobytes.
M Maximum resident set size of the process during its life‐
time, in Kilobytes.
O Number of file system outputs by the process.
P Percentage of the CPU that this job got. This is just
user + system times divided by the total running time. It
also prints a percentage sign.
R Number of minor, or recoverable, page faults. These are
pages that are not valid (so they fault) but which have
not yet been claimed by other virtual pages. Thus the
data in the page is still valid but the system tables
must be updated.
S Total number of CPU-seconds used by the system on behalf
of the process (in kernel mode), in seconds.
U Total number of CPU-seconds that the process used direct‐
ly (in user mode), in seconds.
W Number of times the process was swapped out of main memo‐
ry.
X Average amount of shared text in the process, in Kilo‐
bytes.
Z System's page size, in bytes. This is a per-system con‐
stant, but varies between systems.
c Number of times the process was context-switched involun‐
tarily (because the time slice expired).
e Elapsed real (wall clock) time used by the process, in
seconds.
k Number of signals delivered to the process.
p Average unshared stack size of the process, in Kilobytes.
r Number of socket messages received by the process.
s Number of socket messages sent by the process.
t Average resident set size of the process, in Kilobytes.
w Number of times that the program was context-switched
voluntarily, for instance while waiting for an I/O opera‐
tion to complete.
x Exit status of the command.
So you can get CPU percentage as %P in the format.
Note that this is for the /usr/bin/time binary -- the shell time builtin is usually different (and less capable)
When issuing this command on Linux:
# cat /proc/loadavg
0.75 0.35 0.25 1/25 1747
The first three numbers are load averages. What are the last 2 numbers?
The last one keeps increasing by 2 every second, should I be worried?
/proc/loadavg
The first three fields in this file are load average figures giving
the number of jobs in the run queue (state R) or waiting for disk
I/O (state D) averaged over 1, 5, and 15 minutes. They are the
same as the load average numbers given by uptime(1) and other
programs.
The fourth field consists of two numbers separated by a
slash (/). The first of these is the number of currently executing
kernel scheduling entities (processes, threads); this will be less
than or equal to the number of CPUs. The value after the slash is the
number of kernel scheduling entities that currently exist on the
system.
The fifth field is the PID of the process that was most
recently created on the system.
I would like to comment the accepted answer.
The fourth field consists of two numbers separated by a slash (/). The
first of these is the number of currently executing kernel scheduling
entities (processes, threads); this will be less than or equal to the
number of CPUs.
I did a test program that reads integer N from input and then creates N threads and their run them forever. On RHEL 6.5 computer I have 8 processor and each processor has hyper threading. Anyway if I run my test and it creates 128 threads I see in the fourth field values that are greater than 128, for example 135. It is clearly greater than the number of CPU. This post supports my observation: http://juliano.info/en/Blog:Memory_Leak/Understanding_the_Linux_load_average
It is worth noting that the current explanation in proc(5) manual page
(as of man-pages version 3.21, March 2009) is wrong. It reports the
first number of the forth field as the number of currently executing
scheduling entities, and so predicts it can't be greater than the
number of CPUs. That doesn't match the real implementation, where this
value reports the current number of runnable threads.
The first three columns measure CPU and I/O utilization of the last one, five, and 15 minute periods. The fourth column shows the number of currently running processes and the total number of processes. The last column displays the last process ID used.
https://docs.fedoraproject.org/en-US/Fedora/17/html/System_Administrators_Guide/s2-proc-loadavg.html
The following page explains these in detail:
http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
Some interpretations:
If the averages are 0.0, then your system is idle.
If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
If they are higher than your CPU count, then you might have a performance problem (it depends).
You can consult the proc manual page for /proc/loadavg :
$ man proc | sed -n '/loadavg/,/^$/ p'
/proc/loadavg
The first three fields in this file are load average figures giving the number of jobs in the run queue
(state R) or waiting for disk I/O (state D) averaged over 1, 5, and 15 minutes. They are the same as
the load average numbers given by uptime(1) and other programs. The fourth field consists of two num‐
bers separated by a slash (/). The first of these is the number of currently runnable kernel schedul‐
ing entities (processes, threads). The value after the slash is the number of kernel scheduling enti‐
ties that currently exist on the system. The fifth field is the PID of the process that was most
recently created on the system.
For that, you need to install the man-pages package on CentOS7/RedHat7 or the manpages package on Ubuntu 20.04/22.04 LTS.