Understand output of vmstat memory utilization - linux

Question

I have a solaris box and im trying to know whether its running out of memory or if its stable.
below is the output of vmstat.
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr vc vc vc vc in sy cs us sy id
1 0 0 11426696 4603520 613 1477 449 6 6 0 0 78 22 28 29 8970 37714 22961 43 6 51
4 0 0 4975280 0 1747 3487 805 0 0 0 0 233 41 33 44 9558 53713 15845 74 8 18
4 0 0 4936944 0 933 1837 0 0 0 0 0 56 28 12 39 9317 46898 14648 82 7 11
5 0 0 4943080 0 1056 2806 805 0 0 0 0 103 21 18 18 9286 46900 14866 78 8 14
5 0 0 4942264 0 1088 2173 804 6 6 0 0 109 8 40 31 9927 56484 16495 84 8 8
3 0 0 4942520 0 308 1018 1756 3 3 0 0 166 87 29 44 10638 64146 21413 83 9 8
0 0 0 4942512 0 156 326 1740 0 0 0 0 370 12 33 52 11554 40375 21897 75 9 16
2 0 0 4947384 0 294 560 845 0 0 0 0 121 18 23 20 9445 52382 17016 77 6 17
I can see the free column shows 0 however the sr column also shows 0
And output from top command doesn't show how much free memory available. Swap shows 0.0%
load averages: 11.4, 9.12, 9.24;
9021 processes: 9018 sleeping, 1 running, 2 on cpu
CPU states: 0.0% idle, 71.4% user, 28.6% kernel, 0.0% iowait, 0.0% swap
Memory: 24G phys mem, 16G total swap, 13G free swap
Am i running out of RAM?
Please suggest how to interpret this data. Do i need to increase my physical memory?
Appreciate some insights.

Answer 1

From the Solaris 11.4 vmstat man page, there's one important thing to note:
Without options, vmstat displays a one-line summary of the virtual memory activity since the system was booted.
That also applies to the first line of output from Solaris vmstat: it's a summary of all activity since the system was booted.
A good description of the output fields is found in the EXAMPLES section of the Solaris man vmstat page:
Examples
Example 1 Using vmstat
The following command displays a summary of what the system is doing
every five seconds.
example% vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi p fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 11456 4120 1 41 19 1 3 0 2 0 4 0 0 48 112 130 4 14 82
0 0 1 10132 4280 0 4 44 0 0 0 0 0 23 0 0 211 230 144 3 35 62
0 0 1 10132 4616 0 0 20 0 0 0 0 0 19 0 0 150 172 146 3 33 64
0 0 1 10132 5292 0 0 9 0 0 0 0 0 21 0 0 165 105 130 1 21 78
1 1 1 10132 5496 0 0 5 0 0 0 0 0 23 0 0 183 92 134 1 20 79
1 0 1 10132 5564 0 0 25 0 0 0 0 0 18 0 0 131 231 116 4 34 62
1 0 1 10124 5412 0 0 37 0 0 0 0 0 22 0 0 166 179 118 1 33 67
1 0 1 10124 5236 0 0 24 0 0 0 0 0 14 0 0 109 243 113 4 56 39
example%
The fields of vmstat's display are
kthr
Report the number of kernel threads in each of the three following
states:
r
the number of kernel threads in run queue
b
the number of blocked kernel threads that are waiting for
resources I/O, paging, and so forth
w
the number of swapped out lightweight processes (LWPs) that
are waiting for processing resources to finish.
memory
Report on usage of virtual and real memory.
swap
available swap space (Kbytes)
free
size of the free list (Kbytes)
page
Report information about page faults and paging activity. The
information on each of the following activities is given in units per
second.
re
page reclaims — but see the –S option for how this field is modified.
mf
minor faults — but see the –S option for how this field is modified.
pi
kilobytes paged in
po
kilobytes paged out
fr
kilobytes freed
de
anticipated short-term memory shortfall (Kbytes)
sr
pages scanned by clock algorithm
When executed in a zone and if the pools facility is active, all of
the above (except for ‘de’) only report activity on the processors in
the processor set of the zone's pool.
disk
Report the number of disk operations per second. There are slots for
up to four disks, labeled with a single letter and number. The letter
indicates the type of disk (s = SCSI, i = IPI, and so forth); the
number is the logical unit number.
faults
Report the trap/interrupt rates (per second).
in
interrupts
sy
system calls
cs
CPU context switches
When executed in a zone and if the pools facility is active, all of
the above only report activity on the processors in the processor set
of the zone's pool.
cpu
Give a breakdown of percentage usage of CPU time. On MP systems, this
is an average across all processors.
us
user time
sy
system time
id
idle time
When executed in a zone and if the pools facility is active, all of
the above only report activity on the processors in the processor set
of the zone's pool.

Answer 2

This can help you https://www.howtogeek.com/424334/how-to-use-the-vmstat-command-on-linux/. There is explanation of those shorts.
Memory
swpd: the amount of virtual memory used. In other words, how much memory has been swapped out.,
free: the amount of idle (currently unused) memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
Swap
si: Amount of virtual memory swapped in from swap space.
so: Amount of virtual memory swapped out to swap space.
IO
bi: Blocks received from a block device. The number of data blocks used to swap virtual memory back into RAM.
bo: Blocks sent to a block device. The number of data blocks used to swap virtual memory out of RAM and into swap space.
System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second. A context switch is when the kernel swaps from system mode processing into user mode processing.

Answer 3

"0" is not a valid free memory value.
By design, Solaris always makes sure a minimal amount of free memory is available. The fact the sr column is also equals to zero suggests there is no memory shortage. In any case, you wouldn't have been able to run vmstat or top in the first place with such an extreme RAM shortage.
You should investigate further to understand why the free memory is reported a zero. mdb's ::memstat command would be a good start:
# echo "::memstat" | mdb -k