I have a one a one node Kubernetes cluster and the memory usage reported by the metrics server does not seem to be the same as the memory usage shown with the free command
# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
<node_ip> 1631m 10% 13477Mi 43%
# free -m
total used free shared buff/cache available
Mem: 32010 10794 488 81 20727 19133
Swap: 16127 1735 14392
And the difference is significant ~ 3 GB.
I have also tested this on a 3 node cluster, and the issue is present there too:
# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
<node_ip1> 1254m 8% 26211Mi 84%
<node_ip2> 221m 1% 5021Mi 16%
<node_ip3> 363m 2% 8731Mi 28%
<node_ip4> 1860m 11% 20399Mi 66%
# free -m (this is on node 1)
total used free shared buff/cache available
Mem: 32010 5787 369 1676 25853 24128
Swap: 16127 0 16127
Why is there a difference?
The answer for your question can be found here. It is a duplicate so you can remove this post from StackOverflow.
The metrics exposed by the Metrics Server are collected by an instance of cAdvisor on each node. What you see in the output of kubectl top node is how cAdvisor determines the current resource usage.
So, apparently cAdvisor and free determine the resource usage in different ways. To find out why, you would need to dig into internals of how cAdvisor and free work.
Related
I want to test Pod eviction events that caused by memorypressure for taintbasedeviction on my pods, for to do that I created a memory load on my instance that have 2 vcpu and 8GB Ram.
For create a load I have run this command :
stress-ng --vm 2 --vm-bytes 10G --timeout 60s
Output of memory usage
$ free -h
total used free shared buff/cache available
Mem: 7.8Gi 2.7Gi 1.0Gi 3.9Gi 4.1Gi 984Mi
Swap: 0B 0B 0B
But in my nodes states there is no memorypressure I have updated kubelet eviction parameters at below :
evictionHard:
memory.available: "200Mi"
As summary, How Can I create memory pressure on my worker nodes for test the taint based eviction ?
Thanks
You could invoke the stress command multiple times. Check the script here.
The value for memory.available is derived from the cgroupfs instead of tools like free -m. This is important because free -m does not work in a container, and if users use the node allocatable feature, out of resource decisions are made local to the end user Pod part of the cgroup hierarchy as well as the root node. This script reproduces the same set of steps that the kubelet performs to calculate memory.available. The kubelet excludes inactive_file (i.e. # of bytes of file-backed memory on inactive LRU list) from its calculation as it assumes that memory is reclaimable under pressure.
I recently saw my AWS EC2 instance's states at SSH helper program (Not a putty program).
I saw below.
[centos#ip-172-31-xx-xx ~]$ free -h
total used free shared buffers cached
Mem: 1.8G 1.0G 869M 144K 137M 267M
-/+ buffers/cache: 600M 1.2G
Swap: 0B 0B 0B
I understand buffers and cached usage are reserved usage, so it is empty usage. But, I didn't understand this.
-/+ buffers/cache: 600M 1.2G
What does it mean?
As an alternative look at the contents of: /proc/meminfo
For example:
grep MemAvailable /proc/meminfo
and:
cat /proc/meminfo
Notice that MemAvailable is only available in modern Linux kernels (not RHEL/CentOS 6 unless you run it with a newer kernel, Like Oracle Unbreakable Linux does)
For fun and education look also at: https://www.linuxatemyram.com/
For a more convenient info on your systems resource usage you may be interested in something like atop: https://haydenjames.io/use-atop-linux-server-performance-analysis/ or one of the other top tools like these: https://haydenjames.io/alternatives-top-htop/
I'm just no big fan of free so I avoid it like the plague ;-)
According to the post Meaning of the buffers/cache line in the output of free.
It seems to be the used memory minus the free memory in cache and buffers and the free memory plus the free memory in cache and buffers.
You can calculate the value if you form the sum of buffers and cached (400M) and substract the value from used (1000M - 400M = 600M) and add it on free (869M + 400~ 1,2G).
I am trying to configure monitoring on a variety of Kubernetes (GKE) nodes, specifically to identify [near] out-of-memory conditions. The documentation for node/memory/allocatable_utilization states:
This value cannot exceed 1 as usage cannot exceed allocatable memory bytes.
However, it reports a non-evictable value > 1 (1.015), which contradicts that constraint. Also, it's not clear to me how this corresponds with the actual condition on the node, as shown by free -m:
$ free -m
total used free shared buff/cache available
Mem: 15038 10041 184 67 4812 4606
Swap: 0 0 0
This node is designed to run memory-intensive workloads (Java) and as such this is in line with what I'd expect per our heap size planning.
Why would Stackdriver report this value with those conditions on the node?
We have two machines with identical configuration and use (we have two balanced Siebel application servers in them).
Normally, we have a very similar RAM usage in them (around 7 Gb).
Recently, we've have a sudden increase of RAM in only one of them and now we have close to 14 Gb utilization of RAM in that machine.
So, for very similar boxes, we have one of them using 7Gb of RAM while the other one is consuming 14 Gb.
Now, using ps aux command to determine which process it's using all this additional memory, we see memory consumption is very similar in both machines. Somehow, we don't see any process that's using those 7 Gb of additional RAM.
Let's see:
Machine 1:
total used free shared buffers cached
Mem: 15943 15739 204 0 221 1267
-/+ buffers/cache: 14249 1693
Swap: 8191 0 8191
So, we have 14249 Mb usage of RAM.
Machine 2:
total used free shared buffers cached
Mem: 15943 15636 306 0 962 6409
-/+ buffers/cache: 8264 7678
Swap: 8191 0 8191
So, we have 8264 Mb usage of RAM.
I guess, the sum of Resident Set Size memory of ps should be equal or bigger to this value. According to this answer is how much memory is allocated to the process and is in RAM (including memory from shared libraries). We don't have any memory in SWAP.
However:
Machine 1:
ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024}'
8357.08
8357.08 < 14249 -> NOK!
Machine 2:
ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024}'
8468.63
8468.63 > 8264 -> OK
What do I get wrong? How can I find where this "missing" memory is?
Thank you in advance
If them two are virtual machines, maybe the "missing" memory is occupied by Balloon driver, especially they are hosted by VMware ESXi.
Recently I encounter the similar scenario. Sum of all process RSS is 14GB, command free shows 26GB used, so there are 12GB memory missing.
After search on internet, I follow this article and execute command vmware-toolbox-cmd stat balloon on my VM, console shows 12xxxMB (used by balloon), BINGO!
I was running an application which was to load about 60 mil items in memcache. I had two servers added in a bucket. After about 65% of the data was loaded, I saw 1.3 mil items evicted in both servers. And these were statistics at that point.
On server 1
STAT bytes_written 619117542
STAT limit_maxbytes 3145728000
On server 2
STAT bytes_written 619118863
STAT limit_maxbytes 3145728000
Here's the output of free -m at that point of time.
On server 1
total used free shared buffers cached
Mem: 7987 5965 2021 0 310 441
-/+ buffers/cache: 5213 2774
Swap: 4095 0 4095
On sever 2
total used free shared buffers cached
Mem: 11980 11873 106 0 207 5860
-/+ buffers/cache: 5805 6174
Swap: 5119 0 5119
As we can see, on both servers, limit_maxbytes was not reached. Only about 600MB was used at both the places. However on server 2, free memory dipped to as low as 100 mb. Now I know that cached is 5.8 GB and that linux could free that memory for running processes. But it looks like that didn't happen and seeing memory reaching critical level, memcached started evicting items.
Or is there any other reason? When exactly does linux free up cache memory? Is 100 mb of free ram is still not critical enough for linux to free up cache? Please help me understanding why such an even occured.
The 'slabs' refer to how Memcached allocates memory. Rather than a complex exact-match,it puts your data into a close-enough (slightly larger) piece of memory within the server. This means that it will frequently 'waste' memory that isn't storing your data.
You can tweak how big each potential slot is though when you start the memcached server with the factor (-f) and the initial chunk-size (-s) options. How you set those, depends on the mix of sizes you are storing in cache.