I am running my code on AWS Lambda and I have some performance issues depending on the memory I allocate to the Lambda.
For the same call:
128 MB | 256 MB
+--------+-------+
Cold start | 3.5s | 2s |
-----------+--------+-------+
Nominal | 0.14s | 0.14s |
The maximum used memory is always 85-90 MB.
My guess is that the GC is triggered too often.
I have tried to look the RTS doc, it looks like -c <n> would allow to increase the threshold but it requires me to use -M <size> but I do not know what to put in there.
Is there a way to base a threshold on the physical memory or the total memory?
I guess one way would be via shell substitution. Then you can put whatever calculation you want in there. For example, on my machine, something like this computes half the available memory:
(echo scale=0; sed '/^MemTotal:/!d;s/[^0-9]//g;s.$./2.' /proc/meminfo) | bc
Explanation: extract the MemTotal: ... line from /proc/meminfo, delete everything that isn't the number, and append /2 to divide by 2. The initial scale=0 tells bc to round to the nearest whole number.
Rolling that into your command's argument list then looks like this:
myFancyHaskellProgram +RTS -M $( (echo scale=0; sed '/^MemTotal:/!d;s/[^0-9]//g;s.$./2.' /proc/meminfo) | bc )K
Related
I am working on a per process memory monitoring (Bash) script but it turns out to be more of a headache than I thought. Especially on forked processes such as PostgreSQL. There are a couple of reasons:
RSS is a potential value to be used as memory usage, however this also contains shared libraries etc which are used in other processes
PSS is another potential value which (should) show only the private memory of a process. Problem here is that PSS can only be retrieved from /proc//smaps which requires elevated capability privileges (or root)
USS (calculated as Private_Dirty + Private_Clean, source How does smem calculate RSS, USS and PSS?) could also be a potential candidate but here again we need access to /proc//smaps
For now I am trying to solve the forked process problem by looping through each PID's smaps (as suggested in https://www.depesz.com/2012/06/09/how-much-ram-is-postgresql-using/), for example:
for pid in $(pgrep -a -f "postgres" | awk '{print $1}' | tr "\n" " " ); do grep "^Pss:" /proc/$pid/smaps; done
Maybe some of the postgres processes should be excluded, I am not sure.
Using this method to calculate and sum the PSS and USS values, resulting in:
PSS: 4817 MB - USS: 4547 MB - RES: 6176 MB - VIRT: 26851 MB used
Obviously this only works with elevated privileges, which I would prefer to avoid. If these values actually represent the truth is not known because other tools/commands show yet again different values.
Unfortunately top and htop are unable to combine the postgres processes. atop is able to do this and seems to be (from a feeling) the most accurate with the following values:
NPROCS SYSCPU USRCPU VSIZE RSIZE PSIZE SWAPSZ RDDSK WRDSK RNET SNET MEM CMD 1/1
27 56m50s 16m40s 5.4G 1.1G 0K 2308K 0K 0K 0 0 11% postgres
Now to the question: What is the suggested and best way to retrieve the most accurate memory usage of an application with forked processes, such as PostgreSQL?
And in case atop already does an accurate calculation, how does atop get the to RSIZE value? Note that this value shown as root and non-root user, which would probably mean that /proc/<pid>/smaps is not used for the calculation.
Please comment if more information is needed.
EDIT: I actually found a bug in my pgrep pattern in my final script and it falsely parsed a lot more than just the postgres processes.
The new output now shows the same RES value as seen in atop RSIZE:
Script output:
PSS: 205 MB - USS: 60 MB - RES: 1162 MB - VIRT: 5506 MB
atop summarized postgres output:
NPROCS SYSCPU USRCPU VSIZE RSIZE PSIZE SWAPSZ RDDSK WRDSK RNET SNET MEM CMD
27 0.04s 0.10s 5.4G 1.1G 0K 2308K 0K 32K 0 0 11% postgres
But the question remains of course. Unless I am now using the most accurate way with the summarized RSS (RES) memory value. Let me know your thoughts, thanks :)
The SLURM documentation says
RealMemory
Size of real memory on the node in megabytes (e.g. "2048"). The default value is 1. Lowering RealMemory with the goal of setting aside some amount for the OS and not available for job allocations will not work as intended if Memory is not set as a consumable resource in SelectTypeParameters. So one of the *_Memory options need to be enabled for that goal to be accomplished. Also see MemSpecLimit.
In several places I have seen the recommendation to set this value equal to the value reported by slurmd -C, for example: How to set RealMemory in slurm?
However, I am confused about how this value is calculated and relates to other information, as for example MemTotal from /proc/meminfo.
RealMem is in MBs
slurmd -C
RealMemory=193086
MemTotal is in KBs
cat /proc/meminfo
MemTotal: 197721028 kB
Just divide MemTotal by 1024
197721028 / 1024 = 193086
I have been playing around with the bashrc and one of the thing I want to see at all time is my cpu usage in percentage. I decided to set this data in my PS1. The problem is that to have an accurate estimation of my CPU usage I need to do operations that require waiting for at least 0.5 seconds.
As a result of this, my new command line only displays at the end of the CPU calculation, 0.5 seconds later, which is really not acceptable. To deal with this I thought that I could maybe use a thread to do the CPU calculation and only display it at the end but I don't know how to do so.
One of the problem is that I display other information after CPU percentage so I don't know if it is even possible to delay the CPU display while still displaying the rest of the command line. I thought that maybe I could display a temporary string such as ??.?? and then replace it by the real value but I am not sure how to do so since if I type commands fast the position of the ??.?? can change (for example typing ls 5 times in a row very fast).
Maybe there is an even simpler solution to my problem such as calculating the CPU percentage in an other way ?
My CPU percentage calculating function:
function cpuf(){
NonIdle=0;Idle=0;Total=0;TotalD=0;Idled=0
NonIdle=$((`cat /proc/stat | awk '/^cpu / {print$2+$3+$4+$7+$8+$9}'` - $NonIdle))
Idle=$((`cat /proc/stat | awk '/^cpu / {print$5+$6}'` - $Idle))
sleep 0.5
NonIdle=$((`cat /proc/stat | awk '/^cpu / {print$2+$3+$4+$7+$8+$9}'` - $NonIdle))
Idle=$((`cat /proc/stat | awk '/^cpu / {print$5+$6}'` - $Idle))
Total=$((Idle+NonIdle))
CPU=$(((Total-Idle)/Total))
echo `echo "scale=2;($Total*100-$Idle*100)/$Total" | bc -l`
}
How I call it in the bashrc:
alias cpu="cpuf"
PS1+="(\[${MAGENTA}\]CPU $(cpu)%"
There is no need to reinvent the wheel here, linux already takes care of capturing system stats in /proc/loadavg. The first number is average load in the last minute across all cpus, so we just need to divide by the number of cpus, which we can determine by reading /proc/cpuinfo. Rolling this into .bashrc we get:
.bashrc
...
# My fancy prompt, adjust as you like...
FANCY_PROMPT="$GREEN\u$YELLOW#\h:$PURPLE\w$BLUE$ $RESET"
CPUS=$( grep -c bogomips /proc/cpuinfo )
_prompt_command() {
LOAD_AVG_1_MIN=$( cut -d ' ' -f 1 /proc/loadavg )
PERCENT=$( echo "scale=0; $LOAD_AVG_1_MIN * 100 / $CPUS" | bc -l )
PS1="CPU $PERCENT% $FANCY_PROMPT"
true
}
PROMPT_COMMAND="_prompt_command"
In Use:
SO linux /proc/loadavg
I wonder how do I get the percentage of my processor usage from 0% to 100%?
to know how many percent'm using my processor preferably in bash or other methods provided that percentage.
I have this script that I found on google however it is very much imprecisso
I tried to make more improvements could not, does anyone know any method to get the percentage of CPU utilization in% 0-100
my script
NUMCPUS=`grep ^proc /proc/cpuinfo | wc -l`; FIRST=`cat /proc/stat | awk '/^cpu / {print $5}'`; sleep 1; SECOND=`cat /proc/stat | awk '/^cpu / {print $5}'`; USED=`echo 2 k 100 $SECOND $FIRST - $NUMCPUS / - p | dc`; echo ${USED}% CPU Usage
Processor use or utilization is a measurement over time. One way to measure utilization in % is by computation over two successive reads of /proc/stat. A simple common bash script to compute the percentage is:
#!/bin/bash
# Read /proc/stat file (for first datapoint)
read cpu user nice system idle iowait irq softirq steal guest< /proc/stat
# compute active and total utilizations
cpu_active_prev=$((user+system+nice+softirq+steal))
cpu_total_prev=$((user+system+nice+softirq+steal+idle+iowait))
usleep 50000
# Read /proc/stat file (for second datapoint)
read cpu user nice system idle iowait irq softirq steal guest< /proc/stat
# compute active and total utilizations
cpu_active_cur=$((user+system+nice+softirq+steal))
cpu_total_cur=$((user+system+nice+softirq+steal+idle+iowait))
# compute CPU utilization (%)
cpu_util=$((100*( cpu_active_cur-cpu_active_prev ) / (cpu_total_cur-cpu_total_prev) ))
printf " Current CPU Utilization : %s\n" "$cpu_util"
exit 0
use/output:
$ bash procstat-cpu.sh
Current CPU Utilization : 10
output over 5 iterations:
$ ( declare -i cnt=0; while [ "$cnt" -lt 5 ]; do bash procstat-cpu.sh; ((cnt++)); done )
Current CPU Utilization : 20
Current CPU Utilization : 18
Current CPU Utilization : 18
Current CPU Utilization : 18
Current CPU Utilization : 18
top -bn1 | sed -n '/Cpu/p'
gives the following line
Cpu(s): 15.4%us, 5.3%sy, 0.0%ni, 78.6%id, 0.5%wa, 0.0%hi, 0.1%si, 0.0%st
You can pull any CPU field with the following will take the user CPU (us)
top -bn1 | sed -n '/Cpu/p' | awk '{print $2}' | sed 's/..,//'
Output:
15.4%
If you want another field like system CPU (sy) you can change the awk field from $2,
top -bn1 | sed -n '/Cpu/p' | awk '{print $3}' | sed 's/..,//'
Output:
5.3%
If you want other CPU:
us: user CPU used by user processes
sy: system CPU used by system/kernel processes
ni: nice CPU used by processes that were reniced
id: idle CPU not used
wa: io wait Essentially idle CPU waiting on IO devices
hi: hardware irq CPU used to service hardware IRQs
si: software irq CPU used to service soft IRQs
st: steal time CPU time which the hypervisor dedicated (or ‘stole’) for other guests in the system.
To get usage percent total since bringing the system up:
awk '/cpu /{print 100*($2+$4)/($2+$4+$5)}' /proc/stat
To get the usage percentage over the last second:
awk -v a="$(awk '/cpu /{print $2+$4,$2+$4+$5}' /proc/stat; sleep 1)" '/cpu /{split(a,b," "); print 100*($2+$4-b[1])/($2+$4+$5-b[2])}' /proc/stat
Explanation
From man 5 proc, the meaning of the first four numbers on the cpu line in /proc/stat is given by:
cpu 3357 0 4313 1362393
The amount of time, measured in units of USER_HZ (1/100ths of a second on most architectures, use sysconf(_SC_CLK_TCK)
to obtain the right value), that the system spent
in user mode, user mode with low priority (nice), system mode, and the idle task, respectively. The last value
should be USER_HZ times the second entry in the uptime
pseudo-file.
The get the CPU usage, we add the user and system times and divide by the total of user, system, and idle time.
Let's look again at the calculation for total CPU usage since system up:
awk '/cpu /{print 100*($2+$4)/($2+$4+$5)}' /proc/stat
By requiring that the line match cpu, we get system totals. The second column is user time, the fourth is system time, and the fifth is idle time. The ratio is multiplied by 100 to get a percentage.
Now, let's consider the recent CPU usage:
awk -v a="$(awk '/cpu /{print $2+$4,$2+$4+$5}' /proc/stat; sleep 1)" '/cpu /{split(a,b," "); print 100*($2+$4-b[1])/($2+$4+$5-b[2])}' /proc/stat
This reads /proc/cpu twice, a second apart. The first time, the CPU user + system, and user+system+idle times are saved in the variable a. sleep is called to delay for a second. Then, /proc/cpu is read a second time. Tne old user+system total is subtracted from the new total and divided by the change in the total of all times. The result is multiplied by 100 to convert it to percent and printed.
Using vmstat the command is short, moderately accurate and takes one second :
vmstat 1 2 | awk 'END { print 100 - $15 }'
Very simple script that considers only System, Idle and User.
The benefit over the other answers is that it requires no utilities, not even top, and also displays fractions, which the current top answer does not.
#!/bin/bash
read u1 s1 i1 <<< $(grep 'cpu ' /proc/stat | awk '{print $2" "$4" "$5}' )
sleep 1
read u2 s2 i2 <<< $(grep 'cpu ' /proc/stat | awk '{print $2" "$4" "$5}' )
u=$(echo "scale=4;$u2-$u1" | bc)
s=$(echo "scale=4;$s2-$s1" | bc)
i=$(echo "scale=4;$i2-$i1" | bc)
cpu=$(echo "scale=4;($u+$s)*100/($u+$s+$i)" | bc)
echo $cpu
Brief description - we pull data from /proc/stat from the line that starts with 'cpu'. We then get parse out the second token which is user time, the fourth token which is system time and fifth token which is idle time.
At this point, you may be tempted to just do the math, but all that will give you is the utilization since boot time. We need one more data point.
We sleep 1 second and we pull the data from /proc/stat again. Now we get the difference between the first pull and the second pull. This is the CPU utilization for that 1 second while we slept.
We get the difference for each of the variables, and then do the math on the difference. The strange 'scale=4' in front of each calculation is to force a float answer with 4 digit precision.
I'm running some JMeter tests against a Java process to determine how responsive a web application is under load (500+ users). JMeter will give the response time for each web request, and I've written a script to ping the Tomcat Manager every X seconds which will get me the current size of the JVM heap.
I'd like to collect stats on the server of the % of CPU being used by Tomcat. I tried to do it in a shell script using ps like this:
PS_RESULTS=`ps -o pcpu,pmem,nlwp -p $PID`
...running the command every X seconds and appending the results to a text file. (for anyone wondering, pmem = % mem usage and nlwp is number of threads)
However I've found that this gives a different definition of "% of CPU Utilization" than I'd like - according to the manpages for ps, pcpu is defined as:
cpu utilization of the process in "##.#" format. It is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage.
In other words, pcpu gives me the % CPU utilization for the process for the lifetime of the process.
Since I want to take a sample every X seconds, I'd like to be collecting the CPU utilization of the process at the current time only - similar to what top would give me
(CPU utilization of the process since the last update).
How can I collect this from within a shell script?
Use top -b (and other switches if you want different outputs). It will just dump to stdout instead of jumping into a curses window.
The most useful tool I've found for monitoring a server while performing a test such as JMeter on it is dstat. It not only gives you a range of stats from the server, it outputs to csv for easy import into a spreadsheet and lets you extend the tool with modules written in Python.
User load: top -b -n 2 |grep Cpu |tail -n 1 |awk '{print $2}' |sed 's/.[^.]*$//'
System load: top -b -n 2 |grep Cpu |tail -n 1 |awk '{print $3}' |sed 's/.[^.]*$//'
Idle load: top -b -n 1 |grep Cpu |tail -n 1 |awk '{print $5}' |sed 's/.[^.]*$//'
Every outcome is a round decimal.
Off the top of my head, I'd use the /proc filesystem view of the system state - Look at man 5 proc to see the format of the entry for /proc/PID/stat, which contains total CPU usage information, and use /proc/stat to get global system information. To obtain "current time" usage, you probably really mean "CPU used in the last N seconds"; take two samples a short distance apart to see the current rate of CPU consumption. You can then munge these values into something useful. Really though, this is probably more a Perl/Ruby/Python job than a pure shell script.
You might be able to get the rough data you're after with /proc/PID/status, which gives a Sleep average for the process. Pretty coarse data though.
also use 1 as iteration count, so you will get current snapshot without waiting to get another one in $delay time.
top -b -n 1
This will not give you a per-process metric, but the Stress Terminal UI is super useful to know how badly you're punishing your boxes. Add -c flag to make it dump the data to a CSV file.