Could somebody please tell me what is "Memory Page Out Rate".
I have seen this in "HP Open View" server monitoring tool and tried googling it.
Would appreciate if some expert can clarify.
If page out rate is too high as 200+ per second, can it crash the server?
Thanks in advance
This link may help:
http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/index.jsp?topic=/com.ibm.itm.doc/main_unix65.htm
"Page Out Rate (KB per Second) The number of kilobytes that the virtual memory manager pages out per second averaged over the previous 30-second interval. Note: the value -1 indicates Not Available, -2 indicates Not Collected, 2147483648 indicates Value_Exceeds_Minimum, and -2147483647 indicates Value_Exceeds_Maximum."
A page-out rate of 200kb/s could be fine on some systems and service affecting on others. It all depends on how fast your disks/san can keep up.
To be honest you'd be better off asking this question in one of these more suitable sites, SO is for programming related questions, not sysadmin queries:
https://stackoverflow.com/questions/321618/where-can-i-ask-questions-that-arent-programming-questions
What you are referring to is the GBL_MEM_PAGEOUT variable of HP OVPM.
To my understanding it means that some process allocated memory and never de-allocated it, and that process is not currently active.
So basically it should be as low as possible.
However, according to the HP OVPM Metrics document, the definition is this:
GBL_MEM_PAGEOUT
PLATFORMS: HP-UX AIX NCR Sinix DEC SunOS Win3X/95
HP-UX AIX NCR Sinix DEC
The total number of page outs to the disk
during the interval. This includes pages paged out to paging space and
to the file system.
HP-UX
This metric is available on HP-UX 11.0 and beyond.
AIX
This is the same as the “page outs” value from the “vmstat -s”
command. Remember that “vmstat -s” reports cumulative counts.
This metric cannot be compared to the “po” value from the “vmstat”
command. The “po” value only reports the number of pages paged out to
paging space.
To determine the count (that is, GBL_MEM_PAGEOUT) for the current
interval, subtract the previous value from the current value. To
determine the rate (that is, GBL_MEM_PAGEOUT_RATE) for the current
interval, subtract the previous value from the current value and then
divide by the length of the interval. Keep in mind that whenever any
comparisons are made with other tools, both tools must be interval
synchronized with each other in order to be valid.
HP-UX SunOS
The total number of page outs to the disk during the
interval. This includes pages paged out to paging space and to the
file system.
Win3x/95
The total number of page outs to the disk during the interval
(Windows 95 only).
Related
I have a Red hat server where I can see the load average on the system is 23 24 23 (1min 5min 15min) using the top command. And i can see in /proc/cpuinfo there are 24 processors entry (0-23). But in each processor entry the cpu cores value is 6 and in each processor entry the physical id is either 1 or 0.
I want to know if my system is overloaded. Can anyone please tell me.
It seems you have a system with two processors, each with 6 cores. Each of the cores can likely run hyperthread => 2 x 6 x 2 = 24. In /proc/cpuinfo, top etc, you will see each hyperthread: that's the number of parallel processes or threads that your hardware can run.
The quick answer is that your system is probably not overloaded and that it processes a relatively stable amount of work over time (as the 1, 5 and 15 minute values are about the same). A rule of thumb is that the load average value should stay below the number of hyperthreads -- this is not, however, exactly true.
You'll find a more in-depth discussion here:
https://unix.stackexchange.com/questions/303699/how-is-the-load-average-interpreted-in-top-output-is-it-the-same-for-all-di
and here:
https://superuser.com/questions/23498/what-does-load-average-mean-on-unix-linux
and perhaps here:
https://linuxhint.com/load_average_linux/
However, please keep in mind that load average does not tell you everything about your system -- though it is usually a pretty good indicator in my experience. You'd have to check many other factors to determine overloadedness, like memory pressure, I/O wait times, I/O bandwidth utilization. It also depends on what kind of processing the system is performing.
I want to verify transparent huge page(THP) would cause large page fault latency, because Linux must zero pages before returning them to user. THP is 512x larger than 4KB pages, thus slower to clear. When memory is fragmented, the OS often compact memory to generate THP.
So I want to measure minor page fault latency(cost), but I still have no idea.
Check https://www.kernel.org/doc/Documentation/vm/transhuge.txt documentation and search LWN & RedHat docs for THP latency and THP faults.
https://www.kernel.org/doc/Documentation/vm/transhuge.txt says about zero THP:
By default kernel tries to use huge zero page on read page fault to
anonymous mapping. It's possible to disable huge zero page by writing 0
or enable it back by writing 1:
echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
You can vary the setting (introduced around 2012: https://lwn.net/Articles/517465/ Adding a huge zero page) and do measurements of page mapping and access latency. Just read some system time with rdtsc/rdtscp/CLOCK_MONOTONIC, do accesses to the page, reread time; record stats about the time difference, like min/max/avg; draw a histogram - count how many differences were in 0..100, 101..300, 301..600 ... ranges and how many were bigger than some huge value. Array to count histogram many be small enough.
You may try mmap() with MAP_POPULATE flag - (http://d3s.mff.cuni.cz/teaching/advanced_operating_systems/slides/10_huge_pages.pdf page 17)
RedHat blog has post about THP & page fault latency (with help of their stap SystemTap tracing): https://developers.redhat.com/blog/2014/03/10/examining-huge-pages-or-transparent-huge-pages-performance/
To prevent information leakage from the previous user of the page the kernel writes zeros in the entire page. For a 4096 byte page this is a relatively short operation and will only take a couple of microseconds. The x86 hugepages are 2MB in size, 512 times larger than the normal page. Thus, the operation may take hundreds of microseconds and impact the operation of latency sensitive code. Below is a simple SystemTap command line script to show which applications have huge pages zeroed out and how long those operations take. It will run until cntl-c is pressed.
stap -e 'global huge_clear probe kernel.function("clear_huge_page").return {
huge_clear [execname(), pid()] <<< (gettimeofday_us() - #entry(gettimeofday_us()))}'
Also, I'm not sure about this, but in theory, Linux Kernel may have some kernel thread to do prezeroing of huge pages before they will be required by any application.
I have a linux dedicated server machine(8cores 8gbRAM) where i run some crawler php scripts. The load on the system ends up being arround 200, which sounds a lot. Since i am not using the machine to host content, what could be the sideeffects of such high level of load for the purposes stated above.
Machines were made to work so there are no issues with high load average, per se. But, a high load average can be an indicator of a performance issue, often. Such investigation is usually application specific, but here is a very general guideline:
Since load average is a combined metric of (CPU, IO .. etc) you want to examine all separately. I would start with making sure the machine is not thrashing, by checking swap usage (vmstat comes in handy), and disk performance (using iostat). You may also check if your operations are CPU intensive.
You should read your load average value as a 3 component value (1 minute load, 5 minutes load and 15 minutes load respectively).
Take a look at the example taken from Wiki:
For example, one can interpret a load average of "1.73 0.60 7.98" on a single-CPU system as:
during the last minute, the system was overloaded by 73% on average (1.73 runnable processes, so that 0.73 processes had to wait for a turn for a single CPU system on average).
during the last 5 minutes, the CPU was idling 40% of the time on average.
during the last 15 minutes, the system was overloaded 698% on average (7.98 runnable processes, so that 6.98 processes had to wait for a turn for a single CPU system on average).
Full article
Please note that this value depends on the resources of your machine.
Cheers!
My research group is sharing time on a CentOS server, and we've been using renice +15 to try to lower the priority of long-running background tasks. When running top, these processes do show up as having a nice value of 15, but the "%ni" measure of nice cpu load is always very low (less than 1%) even when these processes are churning along on 30 cores (as reported in the %CPU column). This has made us think that we are not actually using renice correctly (though the nice processes do seem to yield to higher-priority tasks). How exactly is the nice cpu percentage calculated in top?
The numbers in top come from reading the file /proc/stat. The first line contains a summary for all the cpus combined. The first column is usr time, the second nice time. These times are in clock ticks, usually 100 per second, and are cumulative, so you have to look over and interval and subtract the start value from the end value. You can see the docs for more details, I like http://man7.org/linux/man-pages/man5/proc.5.html
The Linux kernel adds CPU time to the nice column if the nice value is greater than 0, otherwise it puts it in the usr column.
The nice value for an individual process can be found by looking at column 19 of /proc/[pid]/stat. That number should be 15 for you, and the number in column 18 should be 35 (The kernel's internal interpretation of a nice of 15.) However, if top is showing those as 15 in the NI column, it is getting that value from /proc/[pid]/stat.
Comparing the CPU time used in usr and sys in /proc/[pid]/stat and then usr, nice and sys in /proc/stat will give you a good idea of where the time is going. Maybe there are just a ton of CPUs on the system.
We have multiple servers in our lab and I tried to determine which one has more resources currently available. I tried to interpret the information htop displays but I'm not 100% understanding all those numbers.
I have taken a screen shot for each server after issuing htop:
Server #1:
Server #2:
Does server #1 have more memory available than server #2? Should I look at Avg or Mem? Or what other parameter should I look at?
Thanks!
htop author here.
Does server #1 have more memory available than server #2?
Yes.
From the htop faq:
The memory meter in htop says a low number, such as 9%, when top shows something like 90%! (Or: the MEM% number is low, but the bar looks almost full. What's going on?)
The number showed by the memory meter is the total memory used by processes. The additional available memory is used by the Linux kernel for buffering and disk cache, so in total almost the entire memory is in use by the kernel. I believe the number displayed by htop is a more meaningful metric of resources used: the number corresponds to the green bars; the blue and brown bars correspond to buffers and cache, respectively (as explained in the Help screen accessible through the F1 key). Numeric data about these is also available when configuring the memory meter to display as text (in the Setup screen, F2).
Hope that clears things up! Cheers!