I downloaded and built dpdk-stable-16.11.4 version (using the x86_64-native-linuxapp-gcc target). I'm running Ubuntu 16.04.3 LTS. After setting up hugepages according to http://dpdk.org/doc/quick-start or http://dpdk.org/doc/guides-16.04/linux_gsg/sys_reqs.html
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
I can see the hugepages fine.
cat /proc/meminfo | grep Huge
HugeAnonHugePages: 284672 kB
ShmemHugePages: 0 kB
HugePages_Total: 64
HugePages_Free: 64
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
But when I run the helloWorld example, it complained about no free hugepages, see below.
./build/helloworld -l 0-3 -n 2
EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
hello from core 1
hello from core 2
hello from core 3
hello from core 0
Also, the memory info showed free hugepages were all gone.
cat /proc/meminfo | grep Huge
AnonHugePages: 380928 kB
ShmemHugePages: 0 kB
HugePages_Total: 64
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Any idea why this is happening?
1. 1GB Hugepages
The example complains about 1048576kB hugepages, i.e. 1GB. This is correct, you don't have 1GB pages configured as we can see in /proc/meminfo
But you do not need 1GB pages to start a DPDK application, so it just informs and continues.
2. Free Hugepages
The free hugepages are all gone because if we do not specify the amount of memory to use for the DPDK application with -mor --socket-mem option, it will reserve all the available hugepages.
More on that in DPDK Getting Started Guide here.
In case the hugepages remain allocated even when the application has ended (or crashed), you can free the hugepages manually by removing files in /mnt/huge
Related
We develop a Linux application that uses DPDK and which must also be heavily optimised for speed.
We must specify huge pages for use by DPDK and also for general dynamic memory allocation. For the latter we use the libhugetlbfs library.
sudo mount -t hugetlbfs none /mnt/hugetlbfs
We specify the huge pages in the bootcmd line as follows:
hugepagesz=1G hugepages=20 hugepages=0 default_hugepagesz=1G
We are using Centos 7 and full boot cmdline is:
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet hugepagesz=1G hugepages=20 default_hugepagesz=1G irqaffinity=0,1 isolcpus=4-23 nosoftlockup mce=ignore_ce idle=poll
These values are fairly arbitrary. With these values, I see:
$ free -m
total used free shared buff/cache available
Mem: 47797 45041 2468 9 287 2393
Swap: 23999 0 23999
So 2.468GB RAM is free out of 48GB. So a very large amount of memory is allocated to huge pages and I want to reduce this.
My question is what would be sensible values for them?
I am confused by the interpretation of the parameters. I see:
$ cat /proc/meminfo
<snip>
HugePages_Total: 43
HugePages_Free: 43
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
and also:
$ cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
43
Why are 43 pages reported when my parameters only specify 20 pages of 1G?
I would like some guidelines on:
huge page size/quantity that I might need for DPDK?
huge page size/quantity that I might need for malloc?
I know these are highly application dependant but some guidelines would be helpful. Also how could I detect if the huge pages were insufficent for the application?
Additional info:
$ cat /proc/mounts | grep huge
cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
Update 4 March:
My boot cmdline is now:
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet hugepagesz=1G hugepages=20 default_hugepagesz=1G irqaffinity=0,1 isolcpus=4-23 nosoftlockup mce=ignore_ce idle=poll transparent_hugepage=never
and transparent hugepages are disabled (I activated a custom tune profile):
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
but I still see 43 hugepages:
$ cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
43
whereas I have only specified 20 in cmdline. Why is this?
I used gdb to produce core dump file with gcore and used the dump in symbolic execution. However I found it did not include read-only segments in memory, for example, the loaded library libc.so.6. Unfortunately, I need it for symbolic execution.
Is there a way to make core dump also include those read-only segments?
EDIT:
I changed value in /proc//coredump_filter to 0x3c to allow:
" Dump file-backed shared mappings" according to core(5). However, gcore now gives dozens of warnings:
warning: Memory read failed for corefile section, 1048576 bytes at xxxxxx
,where xxxxxx is roughly the memory region where shared libraries (including libc, etc.) lies in the memory. However I want the core file to include shared libraries.
What is wrong?
ps: I can read from memory regions of shared libraries inside gdb normally with a plain gdb command x /16i.
EDIT:
I also checked /proc/<pid>/smaps, inside which, for example, memory region of libc is like:
7f65930f8000-7f65932df000 r-xp 00000000 08:01 789499 /lib/x86_64-linux-gnu/libc-2.27.so
Size: 1948 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 836 kB
Pss: 82 kB
Shared_Clean: 836 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 836 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
VmFlags: rd ex mr mw me sd
I presume the default value 0x33 in coredump_filter should be sufficient to dump this region.
Also as VM flags do not include dd (do not include area into core dump), it was not prevented from being written to core file, but why gcore did not dump it???
Trying to execute this sample:
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/vm/hugepage-shm.c
but get:
shmget: Cannot allocate memory
shmget(0x2, 268435456, IPC_CREAT|SHM_HUGETLB|0600) = -1 ENOMEM (Cannot allocate memory)
It is a machine with 8GB RAM and Ubuntu 16.04, Kernel 4.4.0 ia64
free -m
total used free shared buff/cache available
Mem: 7170 1452 4604 77 1112 5361
Swap: 7623 0 7623
cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
grep -i huge /proc/meminfo
AnonHugePages: 413696 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
cat /proc/sys/vm/hugetlb_shm_group
0
cat /proc/sys/kernel/shmmax
18446744073692774399
cat /proc/sys/kernel/shmall
18446744073692774399
Why it does not work?
From the meminfo, you have no available hugepage can be used,
You can sysctl vm.hugepages=20 to enable 40M for this.
For x86_64 arch, 2Mi size is enabled by default, if you want to use 1Gi size, you have to add some parameters as kernel boot arguments.
Refer to:
https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
I'm running Ubuntu Trusty 14.04 on a new machine with 8GB of RAM, and it seems to be locking up periodically and nothing is in syslog file. I've installed Nagios and have been watching the graphs, and it looks like memory is going high from 7% to 72% in just a span of 10 mins. Only node process are running on server. In top I found all process are running very normal memory consumption. Even after stopping node process. Memory remains with same utilization.
free agrees, claiming I'm using more than 5.7G of memory:
free -h
total used free shared buffers cached
Mem: 7.8G 6.5G 1.3G 2.2M 233M 612M
-/+ buffers/cache: 5.7G 2.1G
Swap: 2.0G 0B 2.0G
This other formula for totaling the memory roughly agrees:
# ps -e -orss=,args= | sort -b -k1,1n | awk '{total = total + $1}END{print total}'
503612
If the processes only total 500 MiB, where's the rest of the memory going?
I've got solution on this... so just wanna to update the same...
echo 2 > /proc/sys/vm/drop_caches
This resolved my issue. So I have added the same in my cron for every 5 mins on each of ubuntu server
On one of my server, I have some memory/disk KV service,
Memory KV behave like memcached, ask for a big trunk of memory(10GB) when initialized,
Disk Kv behave like leveldbd, its random read and sequential write, and it frequently reads a lot files.
Memory are all alloced using libc malloc.
My KV server process do not consume a lot of memory as below (since lack of memory, I have killed memory KV, leaving only disk KV, but free memory still goes down):
:~$top
PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 0 5030m 3.9g 2772 S 8 6.1 10430:52 tair_server
20 0 4833m 3.9g 4560 S 8 6.1 10171:07 tair_server
20 0 4844m 3.9g 3844 S 38 6.1 10073:32 tair_server
20 0 4765m 3.8g 4144 S 8 6.0 10552:39 tair_server
20 0 2941m 2.4g 9.8m S 0 3.8 256:45.70 tair_server
20 0 2953m 2.4g 12m S 1 3.7 276:54.64 tair_server
But, my memory are gone.
$free -m
total used free shared buffers cached
Mem: 64552 57778 6774 0 16 326
-/+ buffers/cache: 57435 7117
Swap: 0 0 0
I can see slab consume lots of my memory, and it's unreclaimable.
$cat /proc/meminfo
MemTotal: 66101892 kB
MemFree: 6816228 kB
Buffers: 17024 kB
Cached: 456640 kB
SwapCached: 0 kB
Active: 19697712 kB
Inactive: 3197312 kB
Active(anon): 19546504 kB
Inactive(anon): 2875632 kB
Active(file): 151208 kB
Inactive(file): 321680 kB
Unevictable: 48 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 6612 kB
Writeback: 72 kB
AnonPages: 22421152 kB
Mapped: 54408 kB
Shmem: 332 kB
Slab: 28870400 kB
SReclaimable: 213344 kB
SUnreclaim: 28657056 kB
KernelStack: 30000 kB
PageTables: 62776 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 33050944 kB
Committed_AS: 37517224 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 388624 kB
VmallocChunk: 34324313700 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 5696 kB
DirectMap2M: 2082816 kB
DirectMap1G: 65011712 kB
Here is the slab info.
$slabtop -s c
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
69842766 69838389 38% 0.19K 1663025 42 13304200K kmalloc-192
69314912 69314796 38% 0.12K 2166091 32 8664364K kmalloc-128
70866624 70866323 39% 0.06K 1107291 64 4429164K kmalloc-64
69299968 69299592 38% 0.03K 541406 128 2165624K kmalloc-32
128388 72434 56% 0.99K 4230 32 135360K ext4_inode_cache
208782 94112 45% 0.19K 4971 42 39768K dentry
I don't understand what consumes lots memory, why it does so, and how to solve this.
Can this be an interval kernel error?
OR it's a glibc problem, it do not return memory back to system, due to frequently disk read?
It looks like your distro version is a bit old, but that's ok. Don't listen to people who tell you that you have to upgrade before viewing your unmae -a output. However it would be nice if you supply it...
In the newer versions of server and desktop distros, the free command output and the /proc/meminfo contain one more line for the purpose of eliminating exactly that kinds of confusions that you have. The line name is "MemAvailable" in /proc/meminfo and "available" in free output.
The column "free" in free -m does not show free memory in the way that humans understand that (and so the "MemFree" line in /proc/meminfo). It does not exclude kernel's page cache and other caches which are not "used" in the way that humans understand that.
That's the first thing. If you think that I'm wrong and you understand the free output correctly try: echo 3 > /proc/sys/vm/drop_caches and see what happens then with memory usage. Please provide the output of free after executing that command as root.
If it's still that bad please read that: https://www.linuxquestions.org/questions/linux-server-73/very-high-slab-usage-hard-to-understand-901323/. It says that your kernel may have to be upgraded.
With the excerpts of top , free , slabtop provide
Looks like ur kernel is eating up the memory
Slab: 28870400 kB
A very naice way of finding this is .
do a Top , and do a sum of RES memory (resident memory on RAM ), top given only user view of memory .
do a free -m and see how much memory is free and used (free given kernel + user ) . The diff between the total memory - top RES , and what is clamined as free in FREE command shouldnt be not much ~1 GB
"Its time to bump ur OS version"