I used gdb to produce core dump file with gcore and used the dump in symbolic execution. However I found it did not include read-only segments in memory, for example, the loaded library libc.so.6. Unfortunately, I need it for symbolic execution.
Is there a way to make core dump also include those read-only segments?
EDIT:
I changed value in /proc//coredump_filter to 0x3c to allow:
" Dump file-backed shared mappings" according to core(5). However, gcore now gives dozens of warnings:
warning: Memory read failed for corefile section, 1048576 bytes at xxxxxx
,where xxxxxx is roughly the memory region where shared libraries (including libc, etc.) lies in the memory. However I want the core file to include shared libraries.
What is wrong?
ps: I can read from memory regions of shared libraries inside gdb normally with a plain gdb command x /16i.
EDIT:
I also checked /proc/<pid>/smaps, inside which, for example, memory region of libc is like:
7f65930f8000-7f65932df000 r-xp 00000000 08:01 789499 /lib/x86_64-linux-gnu/libc-2.27.so
Size: 1948 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 836 kB
Pss: 82 kB
Shared_Clean: 836 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 836 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
VmFlags: rd ex mr mw me sd
I presume the default value 0x33 in coredump_filter should be sufficient to dump this region.
Also as VM flags do not include dd (do not include area into core dump), it was not prevented from being written to core file, but why gcore did not dump it???
Related
We are running a large MATLAB program. After about 24 hours of calculation it stops with an "Out of Memory" error.
Not long before this error occured the process status was quite healty, the process using about 3 Gb on a 12 Gb RAM machine with 32 Gb swap. We have no reason to believe that the process should behave much different, it was in iteration 3 of similar process where the first three iterations lead to a VmPeak of 2.7 Gb. The process was run without GUI using the -nojvm option, using MATLAB r2013a. The error occured in two different computers. Depending on exact parameterization the error occurs at different places in the program.
My question is: is there in MATLAB/Linux/64 bit another resource than plain RAM which can link to an Out of Memory message? Can it be memory fragmentation related, and how can we measure the current level of memory fragmentation?
Matlab error:
CATCH in vsProcessVideosetByFrame.Error using cat
Out of memory. Type HELP MEMORY for your options.
Error in ttClassDetections/horzcat (line 170)
output.data.(fn{j}) = cat(2,temp{:});
Process status not long before it crashed with an Out of Memory message:
schuttek#pc-07843:~$ cat /proc/14123/status
Name: MATLAB
State: S (sleeping)
Tgid: 14123
Ngid: 0
Pid: 14123
PPid: 1
TracerPid: 0
Uid: 8793 8793 8793 8793
Gid: 100 100 100 100
FDSize: 256
Groups: 4 24 27 30 46 100 108 124 501
NStgid: 14123
NSpid: 14123
NSpgid: 14123
NSsid: 11423
VmPeak: 2744640 kB
VmSize: 2613568 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 1045376 kB
VmRSS: 1025692 kB
VmData: 2095720 kB
VmStk: 132 kB
VmExe: 12 kB
VmLib: 179980 kB
VmPTE: 2920 kB
VmPMD: 24 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
Threads: 18
SigQ: 1/64005
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000008080006
SigIgn: 0000000000000001
SigCgt: 00000001880804ee
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
Seccomp: 0
Cpus_allowed: ffffffff
Cpus_allowed_list: 0-31
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 37
nonvoluntary_ctxt_switches: 5
Since this is probably not the OOM killer (as dmesg | grep kill is empty), the error should be internal to MATLAB.
One possible cause is indeed the memory fragmentation; at least some MATLAB routines seem to request explicitly continuous areas of memory that you may run out of. That may be alleviated with pack - a MATLAB function that reconciles the memory.
Here's the documentation page. Make sure to heed the warning about variables larger than 2GB!
https://www.mathworks.com/help/matlab/ref/pack.html
So far running on a Windows 10 64 bit system seems to solve our problem.
I downloaded and built dpdk-stable-16.11.4 version (using the x86_64-native-linuxapp-gcc target). I'm running Ubuntu 16.04.3 LTS. After setting up hugepages according to http://dpdk.org/doc/quick-start or http://dpdk.org/doc/guides-16.04/linux_gsg/sys_reqs.html
mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
I can see the hugepages fine.
cat /proc/meminfo | grep Huge
HugeAnonHugePages: 284672 kB
ShmemHugePages: 0 kB
HugePages_Total: 64
HugePages_Free: 64
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
But when I run the helloWorld example, it complained about no free hugepages, see below.
./build/helloworld -l 0-3 -n 2
EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
hello from core 1
hello from core 2
hello from core 3
hello from core 0
Also, the memory info showed free hugepages were all gone.
cat /proc/meminfo | grep Huge
AnonHugePages: 380928 kB
ShmemHugePages: 0 kB
HugePages_Total: 64
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Any idea why this is happening?
1. 1GB Hugepages
The example complains about 1048576kB hugepages, i.e. 1GB. This is correct, you don't have 1GB pages configured as we can see in /proc/meminfo
But you do not need 1GB pages to start a DPDK application, so it just informs and continues.
2. Free Hugepages
The free hugepages are all gone because if we do not specify the amount of memory to use for the DPDK application with -mor --socket-mem option, it will reserve all the available hugepages.
More on that in DPDK Getting Started Guide here.
In case the hugepages remain allocated even when the application has ended (or crashed), you can free the hugepages manually by removing files in /mnt/huge
Trying to execute this sample:
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/vm/hugepage-shm.c
but get:
shmget: Cannot allocate memory
shmget(0x2, 268435456, IPC_CREAT|SHM_HUGETLB|0600) = -1 ENOMEM (Cannot allocate memory)
It is a machine with 8GB RAM and Ubuntu 16.04, Kernel 4.4.0 ia64
free -m
total used free shared buff/cache available
Mem: 7170 1452 4604 77 1112 5361
Swap: 7623 0 7623
cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
grep -i huge /proc/meminfo
AnonHugePages: 413696 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
cat /proc/sys/vm/hugetlb_shm_group
0
cat /proc/sys/kernel/shmmax
18446744073692774399
cat /proc/sys/kernel/shmall
18446744073692774399
Why it does not work?
From the meminfo, you have no available hugepage can be used,
You can sysctl vm.hugepages=20 to enable 40M for this.
For x86_64 arch, 2Mi size is enabled by default, if you want to use 1Gi size, you have to add some parameters as kernel boot arguments.
Refer to:
https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
I have a router with embedded-linux, kernel version 2.4.22. I know its very old version but I have to deal with it. The following is free command output on the box,
[8300002902-3] Debug> free
total used free shared buffers
Mem: 128104 123928 4176 0 2164
Swap: 0 0 0
Total: 128104 123928 4176
Now my problem is that whenever it execute some commands and redirects its output to some file using the > operator the buffer size is increased by 4KB and even if I remove the same file using rm -rf the buffer again increase by 4KB. due to this when my buffer size reaches at near 5.3MB of size the OOM problem occurs as you can see that I only have 128MB RAM available. I think the kernel is not reclaiming the buffer space. Please tell me what can be done to get rid of this problem. I have researched the setting drop_caches value in the /proc/sys/vm but this command is not available at my kernel version.
Here is the proc/meminfo dump may be it can help.
[8300002902-3] Debug> cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 131178496 126545920 4632576 0 2211840 54927360
Swap: 0 0 0
MemTotal: 128104 kB
MemFree: 4524 kB
MemShared: 0 kB
Buffers: 2160 kB
Cached: 53640 kB
SwapCached: 0 kB
Active: 66628 kB
Inactive: 36692 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 128104 kB
LowFree: 4524 kB
SwapTotal: 0 kB
SwapFree: 0 kB
On one of my server, I have some memory/disk KV service,
Memory KV behave like memcached, ask for a big trunk of memory(10GB) when initialized,
Disk Kv behave like leveldbd, its random read and sequential write, and it frequently reads a lot files.
Memory are all alloced using libc malloc.
My KV server process do not consume a lot of memory as below (since lack of memory, I have killed memory KV, leaving only disk KV, but free memory still goes down):
:~$top
PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 0 5030m 3.9g 2772 S 8 6.1 10430:52 tair_server
20 0 4833m 3.9g 4560 S 8 6.1 10171:07 tair_server
20 0 4844m 3.9g 3844 S 38 6.1 10073:32 tair_server
20 0 4765m 3.8g 4144 S 8 6.0 10552:39 tair_server
20 0 2941m 2.4g 9.8m S 0 3.8 256:45.70 tair_server
20 0 2953m 2.4g 12m S 1 3.7 276:54.64 tair_server
But, my memory are gone.
$free -m
total used free shared buffers cached
Mem: 64552 57778 6774 0 16 326
-/+ buffers/cache: 57435 7117
Swap: 0 0 0
I can see slab consume lots of my memory, and it's unreclaimable.
$cat /proc/meminfo
MemTotal: 66101892 kB
MemFree: 6816228 kB
Buffers: 17024 kB
Cached: 456640 kB
SwapCached: 0 kB
Active: 19697712 kB
Inactive: 3197312 kB
Active(anon): 19546504 kB
Inactive(anon): 2875632 kB
Active(file): 151208 kB
Inactive(file): 321680 kB
Unevictable: 48 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 6612 kB
Writeback: 72 kB
AnonPages: 22421152 kB
Mapped: 54408 kB
Shmem: 332 kB
Slab: 28870400 kB
SReclaimable: 213344 kB
SUnreclaim: 28657056 kB
KernelStack: 30000 kB
PageTables: 62776 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 33050944 kB
Committed_AS: 37517224 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 388624 kB
VmallocChunk: 34324313700 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 5696 kB
DirectMap2M: 2082816 kB
DirectMap1G: 65011712 kB
Here is the slab info.
$slabtop -s c
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
69842766 69838389 38% 0.19K 1663025 42 13304200K kmalloc-192
69314912 69314796 38% 0.12K 2166091 32 8664364K kmalloc-128
70866624 70866323 39% 0.06K 1107291 64 4429164K kmalloc-64
69299968 69299592 38% 0.03K 541406 128 2165624K kmalloc-32
128388 72434 56% 0.99K 4230 32 135360K ext4_inode_cache
208782 94112 45% 0.19K 4971 42 39768K dentry
I don't understand what consumes lots memory, why it does so, and how to solve this.
Can this be an interval kernel error?
OR it's a glibc problem, it do not return memory back to system, due to frequently disk read?
It looks like your distro version is a bit old, but that's ok. Don't listen to people who tell you that you have to upgrade before viewing your unmae -a output. However it would be nice if you supply it...
In the newer versions of server and desktop distros, the free command output and the /proc/meminfo contain one more line for the purpose of eliminating exactly that kinds of confusions that you have. The line name is "MemAvailable" in /proc/meminfo and "available" in free output.
The column "free" in free -m does not show free memory in the way that humans understand that (and so the "MemFree" line in /proc/meminfo). It does not exclude kernel's page cache and other caches which are not "used" in the way that humans understand that.
That's the first thing. If you think that I'm wrong and you understand the free output correctly try: echo 3 > /proc/sys/vm/drop_caches and see what happens then with memory usage. Please provide the output of free after executing that command as root.
If it's still that bad please read that: https://www.linuxquestions.org/questions/linux-server-73/very-high-slab-usage-hard-to-understand-901323/. It says that your kernel may have to be upgraded.
With the excerpts of top , free , slabtop provide
Looks like ur kernel is eating up the memory
Slab: 28870400 kB
A very naice way of finding this is .
do a Top , and do a sum of RES memory (resident memory on RAM ), top given only user view of memory .
do a free -m and see how much memory is free and used (free given kernel + user ) . The diff between the total memory - top RES , and what is clamined as free in FREE command shouldnt be not much ~1 GB
"Its time to bump ur OS version"