On Android shell:
/data/local/valgrind/enter code herebin/valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --log-file=/sdcard/valgrind.log /data/local/Test
========================================================================
valgrind.log--HEAP SUMMARY:
==21314== HEAP SUMMARY:
==21314== in use at exit: 2,098,270 bytes in 6 blocks
==21314== total heap usage: 6 allocs, 0 frees, 2,098,270 bytes allocated
==21314==
==21314== 4 bytes in 1 blocks are definitely lost in loss record 1 of 6
==21314== at 0x482BAEC: malloc (vg_replace_malloc.c:291)
==21314== by 0x864B: ??? (in /data/local/Test)
==21314==
==21314== 10 bytes in 1 blocks are definitely lost in loss record 2 of 6
==21314== at 0x482BAEC: malloc (vg_replace_malloc.c:291)
==21314== by 0x863B: ??? (in /data/local/Test)
==21314==
==21314== 80 bytes in 1 blocks are definitely lost in loss record 3 of 6
==21314== at 0x482C2E4: operator new[](unsigned int) (vg_replace_malloc.c:378)
==21314== by 0x85DF: ??? (in /data/local/Test)
==21314==
==21314== 1,024 bytes in 1 blocks are still reachable in loss record 4 of 6
==21314== at 0x482BAEC: malloc (vg_replace_malloc.c:291)
==21314== by 0x4852DB3: __smakebuf (in /system/lib/libc.so)
==21314==
==21314== 1,048,576 bytes in 1 blocks are possibly lost in loss record 5 of 6
==21314== at 0x482BAEC: malloc (vg_replace_malloc.c:291)
==21314== by 0x86C3: ??? (in /data/local/Test)
==21314==
==21314== 1,048,576 bytes in 1 blocks are definitely lost in loss record 6 of 6
==21314== at 0x482BAEC: malloc (vg_replace_malloc.c:291)
==21314== by 0x869F: ??? (in /data/local/Test)
==21314==
==21314== LEAK SUMMARY:
==21314== definitely lost: 1,048,670 bytes in 4 blocks
==21314== indirectly lost: 0 bytes in 0 blocks
==21314== possibly lost: 1,048,576 bytes in 1 blocks
==21314== still reachable: 1,024 bytes in 1 blocks
==21314== suppressed: 0 bytes in 0 blocks
==21314==
==21314== For counts of detected and suppressed errors, rerun with: -v
==21314== ERROR SUMMARY: 137 errors from 18 contexts (suppressed: 0 from 0)
the Android.mk:
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Test
LOCAL_SRC_FILES := helloworld.cpp
APP_CPPFLAGS += -O0
include $(BUILD_EXECUTABLE)**
On Linux I run ndk-build NDK_DEBUG=1 so this Test is debugable, why doesn't the valgrind log show the line numbers?
Related
Here's the code I'm wondering about:
final class Foo {
var subscriptions = Set<AnyCancellable>()
init () {
Timer
.publish(every: 2, on: .main, in: .default)
.autoconnect()
.zip(Timer.publish(every: 3, on: .main, in: .default).autoconnect())
.sink {
print($0)
}
.store(in: &subscriptions)
}
}
This is the output it produces:
(2020-12-08 15:45:41 +0000, 2020-12-08 15:45:42 +0000)
(2020-12-08 15:45:43 +0000, 2020-12-08 15:45:45 +0000)
(2020-12-08 15:45:45 +0000, 2020-12-08 15:45:48 +0000)
(2020-12-08 15:45:47 +0000, 2020-12-08 15:45:51 +0000)
Would this code eventually crash from memory shortage? It seems like the zip operator is storing every value that it receives but can't yet publish.
zip does not limit its upstream buffer size. You can prove it like this:
import Combine
let ticket = (0 ... .max).publisher
.zip(Empty<Int, Never>(completeImmediately: false))
.sink { print($0) }
The (0 ... .max) publisher will try to publish 263 values synchronously (that, is, before returning control to the Zip subscriber). Run this and watch the memory gauge in Xcode's Debug navigator. It will climb steadily. You probably want to kill it after a few seconds, because it will eventually use up an awful lot of memory and make your Mac unpleasant to use before finally crashing.
If you run it in Instruments for a few seconds, you'll see that all of the allocations happen in this call stack, indicating that Zip internally uses a plain old Array to buffer the incoming values.
66.07 MB 99.8% 174 main
64.00 MB 96.7% 45 Publisher<>.sink(receiveValue:)
64.00 MB 96.7% 42 Publisher.subscribe<A>(_:)
64.00 MB 96.7% 41 Publishers.Zip.receive<A>(subscriber:)
64.00 MB 96.7% 12 Publisher.subscribe<A>(_:)
64.00 MB 96.7% 2 Empty.receive<A>(subscriber:)
64.00 MB 96.7% 2 AbstractZip.Side.receive(subscription:)
64.00 MB 96.7% 2 AbstractZip.receive(subscription:index:)
64.00 MB 96.7% 2 AbstractZip.resolvePendingDemandAndUnlock()
64.00 MB 96.7% 2 protocol witness for Subscription.request(_:) in conformance Publishers.Sequence<A, B>.Inner<A1, B1, C1>
64.00 MB 96.7% 2 Publishers.Sequence.Inner.request(_:)
64.00 MB 96.7% 1 AbstractZip.Side.receive(_:)
64.00 MB 96.7% 1 AbstractZip.receive(_:index:)
64.00 MB 96.7% 1 specialized Array._copyToNewBuffer(oldCount:)
64.00 MB 96.7% 1 specialized _ArrayBufferProtocol._forceCreateUniqueMutableBufferImpl(countForBuffer:minNewCapacity:requiredCapacity:)
64.00 MB 96.7% 1 swift_allocObject
64.00 MB 96.7% 1 swift_slowAlloc
64.00 MB 96.7% 1 malloc
64.00 MB 96.7% 1 malloc_zone_malloc
When I type vmstat -m in command line, it shows:
Cache Num Total Size Pages
fuse_request 0 0 424 9
fuse_inode 0 0 768 5
pid_2 0 0 128 30
nfs_direct_cache 0 0 200 19
nfs_commit_data 0 0 704 11
nfs_write_data 36 36 960 4
nfs_read_data 0 0 896 4
nfs_inode_cache 8224 8265 1048 3
nfs_page 0 0 128 30
fscache_cookie_jar 2 48 80 48
rpc_buffers 8 8 2048 2
rpc_tasks 8 15 256 15
rpc_inode_cache 17 24 832 4
bridge_fdb_cache 14 59 64 59
nf_conntrack_expect 0 0 240 16
For the nfs_write_data line(line 7), why the "pages" is less than "total"?
For some of them, the "total" is always equal to "pages".
Taken from vmstat man page
...
The -m switch displays slabinfo.
...
Field Description For Slab Mode
cache: Cache name
num: Number of currently active objects
total: Total number of available objects
size: Size of each object
pages: Number of pages with at least one active object
totpages: Total number of allocated pages
pslab: Number of pages per slab
Thus, total is the number of slabinfo objects (objects used by the OS as inodes, buffers and so on) and a page can contain more than one object
When I run the !finalizequeue command on a dump file, it shows me different heap and the the total objects to be finalized under this. What are the different heaps like Heap 0, Heap 1 and so on in the result below? My understanding is that there would be one heap per process, is it correct?
0:000> !finalizequeue
SyncBlocks to be cleaned up: 0
Free-Threaded Interfaces to be released: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
----------------------------------
------------------------------
Heap 0
generation 0 has 19 finalizable objects (41c7ed80->41c7edcc)
generation 1 has 19 finalizable objects (41c7ed34->41c7ed80)
generation 2 has 2283 finalizable objects (41c7c988->41c7ed34)
Ready for finalization 0 objects (41c7edcc->41c7edcc)
------------------------------
Heap 1
generation 0 has 101 finalizable objects (41ccc27c->41ccc410)
generation 1 has 25 finalizable objects (41ccc218->41ccc27c)
generation 2 has 2636 finalizable objects (41cc98e8->41ccc218)
Ready for finalization 0 objects (41ccc410->41ccc410)
------------------------------
Heap 2
generation 0 has 6 finalizable objects (41d4195c->41d41974)
generation 1 has 11 finalizable objects (41d41930->41d4195c)
generation 2 has 2328 finalizable objects (41d3f4d0->41d41930)
Ready for finalization 0 objects (41d41974->41d41974)
------------------------------
Heap 3
generation 0 has 21 finalizable objects (41c96188->41c961dc)
generation 1 has 16 finalizable objects (41c96148->41c96188)
generation 2 has 2584 finalizable objects (41c938e8->41c96148)
Ready for finalization 0 objects (41c961dc->41c961dc)
Those are the GC heaps. The GC in this process is running in server mode, which has one GC heap per processor. The output is showing you the locations of the finalizable objects by generation and by heap.
I have a computer with 128 GB of RAM, running Linux (3.19.5-200.fc21.x86_64). However, I cannot allocate more than ~30 GB of RAM in a single process. Beyond this, malloc fails:
#include <stdlib.h>
#include <iostream>
int main()
{
size_t gb_in_bytes = size_t(1)<<size_t(30); // 1 GB in bytes (2^30).
// try to allocate 1 block of 'i' GB.
for (size_t i = 25; i < 35; ++ i) {
size_t n = i * gb_in_bytes;
void *p = ::malloc(n);
std::cout << "allocation of 1 x " << (n/double(gb_in_bytes)) << " GB of data. Ok? " << ((p==0)? "nope" : "yes") << std::endl;
::free(p);
}
}
This produces the following output:
/tmp> c++ mem_alloc.cpp && a.out
allocation of 1 x 25 GB of data. Ok? yes
allocation of 1 x 26 GB of data. Ok? yes
allocation of 1 x 27 GB of data. Ok? yes
allocation of 1 x 28 GB of data. Ok? yes
allocation of 1 x 29 GB of data. Ok? yes
allocation of 1 x 30 GB of data. Ok? yes
allocation of 1 x 31 GB of data. Ok? nope
allocation of 1 x 32 GB of data. Ok? nope
allocation of 1 x 33 GB of data. Ok? nope
allocation of 1 x 34 GB of data. Ok? nope
I searched for quite some time, and found that this is related to the maximum virtual memory size:
~> ulimit -all
[...]
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
virtual memory (kbytes, -v) 32505856
[...]
I can increase this limit to ~64 GB via ulimit -v 64000000, but not further. Beyond this, I get operation not permitted errors:
~> ulimit -v 64000000
~> ulimit -v 65000000
bash: ulimit: virtual memory: cannot modify limit: Operation not permitted
~> ulimit -v unlimited
bash: ulimit: virtual memory: cannot modify limit: Operation not permitted
Some more searching revealed that in principle it should be possible to set these limits via the "as" (address space) entry in /etc/security/limits.conf. However, by doing this, I could only reduce the maximum amount of virtual memory, not increase it.
Is there any way to either lift this limit of virtual memory per process completely, or to increase it beyond 64 GB? I would like to use all of the physical memory in a single application.
EDIT:
Following Ingo Leonhardt, I tried ulimits -v unlimited after logging in as root, not as standard user. Doing this solves the problem for root (the program can then allocate all the physical memory while logged in as root). But this works only for root, not for other users. However, at the least this means that in principle the kernel can handle this just fine, and that there is only a configuration problem.
Regarding limits.conf: I tried explicitly adding
hard as unlimited
soft as unlimited
to /etc/security/limits.conf, and rebooting. This had no effect. After login as standard user, ulimit -v still returns about 32 GB, and ulimit -v 65000000 still says permission denied (while ulimit -v 64000000 works). The rest of limits.conf is commented out, and in /etc/security/limits.d there is only one other, unrelated entry (limiting nproc to 4096 for non-root users). That is, the virtual memory limit must be coming from elsewhere than limits.conf. Any ideas what else could lead to ulimits -v not being "unlimited"?
EDIT/RESOLUTION:
It was caused by my own stupidity. I had a (long forgotten) program in my user setup which used setrlimit to restrict the amount of memory per process to prevent Linux from swapping to death. It was unintentionally copied from a 32 GB machine to the 128 GB machine. Thanks to Paul and Andrew Janke and everyone else for helping to track it down. Sorry everyone :/.
If anyone else encounters this: Search for ulimit/setrlimit in the bash and profile settings, and programs potentially calling those (both your own and the system-wide /etc settings) and make sure that /security/limits.conf does not include this limit... (or at least try creating a new user, to see if this happens in your user or the system setup)
This is a ulimit and system setup problem, not a c++ problem.
I can run your appropriately modified code on an Amazon EC2 instance type r3.4xlarge with no problem. These cost less than $0.20/hour on the spot market, and so I suggest you rent one, and perhaps take a look around in /etc and compare to your own setup... or maybe you need to recompile a Linux kernel to use that much memory... but it is not a C++ or gcc problem.
Ubuntu on the EC2 machine was already set up for unlimited process memory.
$ sudo su
# ulimit -u
--> unlimited
This one has 125GB of ram
# free
total used free shared buffers cached
Mem: 125903992 1371828 124532164 344 22156 502248
-/+ buffers/cache: 847424 125056568
Swap: 0 0 0
I modified the limits on your program to go up to 149GB.
Here's the output. Looks good up to 118GB.
root#ip-10-203-193-204:/home/ubuntu# ./memtest
allocation of 1 x 25 GB of data. Ok? yes
allocation of 1 x 26 GB of data. Ok? yes
allocation of 1 x 27 GB of data. Ok? yes
allocation of 1 x 28 GB of data. Ok? yes
allocation of 1 x 29 GB of data. Ok? yes
allocation of 1 x 30 GB of data. Ok? yes
allocation of 1 x 31 GB of data. Ok? yes
allocation of 1 x 32 GB of data. Ok? yes
allocation of 1 x 33 GB of data. Ok? yes
allocation of 1 x 34 GB of data. Ok? yes
allocation of 1 x 35 GB of data. Ok? yes
allocation of 1 x 36 GB of data. Ok? yes
allocation of 1 x 37 GB of data. Ok? yes
allocation of 1 x 38 GB of data. Ok? yes
allocation of 1 x 39 GB of data. Ok? yes
allocation of 1 x 40 GB of data. Ok? yes
allocation of 1 x 41 GB of data. Ok? yes
allocation of 1 x 42 GB of data. Ok? yes
allocation of 1 x 43 GB of data. Ok? yes
allocation of 1 x 44 GB of data. Ok? yes
allocation of 1 x 45 GB of data. Ok? yes
allocation of 1 x 46 GB of data. Ok? yes
allocation of 1 x 47 GB of data. Ok? yes
allocation of 1 x 48 GB of data. Ok? yes
allocation of 1 x 49 GB of data. Ok? yes
allocation of 1 x 50 GB of data. Ok? yes
allocation of 1 x 51 GB of data. Ok? yes
allocation of 1 x 52 GB of data. Ok? yes
allocation of 1 x 53 GB of data. Ok? yes
allocation of 1 x 54 GB of data. Ok? yes
allocation of 1 x 55 GB of data. Ok? yes
allocation of 1 x 56 GB of data. Ok? yes
allocation of 1 x 57 GB of data. Ok? yes
allocation of 1 x 58 GB of data. Ok? yes
allocation of 1 x 59 GB of data. Ok? yes
allocation of 1 x 60 GB of data. Ok? yes
allocation of 1 x 61 GB of data. Ok? yes
allocation of 1 x 62 GB of data. Ok? yes
allocation of 1 x 63 GB of data. Ok? yes
allocation of 1 x 64 GB of data. Ok? yes
allocation of 1 x 65 GB of data. Ok? yes
allocation of 1 x 66 GB of data. Ok? yes
allocation of 1 x 67 GB of data. Ok? yes
allocation of 1 x 68 GB of data. Ok? yes
allocation of 1 x 69 GB of data. Ok? yes
allocation of 1 x 70 GB of data. Ok? yes
allocation of 1 x 71 GB of data. Ok? yes
allocation of 1 x 72 GB of data. Ok? yes
allocation of 1 x 73 GB of data. Ok? yes
allocation of 1 x 74 GB of data. Ok? yes
allocation of 1 x 75 GB of data. Ok? yes
allocation of 1 x 76 GB of data. Ok? yes
allocation of 1 x 77 GB of data. Ok? yes
allocation of 1 x 78 GB of data. Ok? yes
allocation of 1 x 79 GB of data. Ok? yes
allocation of 1 x 80 GB of data. Ok? yes
allocation of 1 x 81 GB of data. Ok? yes
allocation of 1 x 82 GB of data. Ok? yes
allocation of 1 x 83 GB of data. Ok? yes
allocation of 1 x 84 GB of data. Ok? yes
allocation of 1 x 85 GB of data. Ok? yes
allocation of 1 x 86 GB of data. Ok? yes
allocation of 1 x 87 GB of data. Ok? yes
allocation of 1 x 88 GB of data. Ok? yes
allocation of 1 x 89 GB of data. Ok? yes
allocation of 1 x 90 GB of data. Ok? yes
allocation of 1 x 91 GB of data. Ok? yes
allocation of 1 x 92 GB of data. Ok? yes
allocation of 1 x 93 GB of data. Ok? yes
allocation of 1 x 94 GB of data. Ok? yes
allocation of 1 x 95 GB of data. Ok? yes
allocation of 1 x 96 GB of data. Ok? yes
allocation of 1 x 97 GB of data. Ok? yes
allocation of 1 x 98 GB of data. Ok? yes
allocation of 1 x 99 GB of data. Ok? yes
allocation of 1 x 100 GB of data. Ok? yes
allocation of 1 x 101 GB of data. Ok? yes
allocation of 1 x 102 GB of data. Ok? yes
allocation of 1 x 103 GB of data. Ok? yes
allocation of 1 x 104 GB of data. Ok? yes
allocation of 1 x 105 GB of data. Ok? yes
allocation of 1 x 106 GB of data. Ok? yes
allocation of 1 x 107 GB of data. Ok? yes
allocation of 1 x 108 GB of data. Ok? yes
allocation of 1 x 109 GB of data. Ok? yes
allocation of 1 x 110 GB of data. Ok? yes
allocation of 1 x 111 GB of data. Ok? yes
allocation of 1 x 112 GB of data. Ok? yes
allocation of 1 x 113 GB of data. Ok? yes
allocation of 1 x 114 GB of data. Ok? yes
allocation of 1 x 115 GB of data. Ok? yes
allocation of 1 x 116 GB of data. Ok? yes
allocation of 1 x 117 GB of data. Ok? yes
allocation of 1 x 118 GB of data. Ok? yes
allocation of 1 x 119 GB of data. Ok? nope
allocation of 1 x 120 GB of data. Ok? nope
allocation of 1 x 121 GB of data. Ok? nope
allocation of 1 x 122 GB of data. Ok? nope
allocation of 1 x 123 GB of data. Ok? nope
allocation of 1 x 124 GB of data. Ok? nope
allocation of 1 x 125 GB of data. Ok? nope
allocation of 1 x 126 GB of data. Ok? nope
allocation of 1 x 127 GB of data. Ok? nope
allocation of 1 x 128 GB of data. Ok? nope
allocation of 1 x 129 GB of data. Ok? nope
allocation of 1 x 130 GB of data. Ok? nope
allocation of 1 x 131 GB of data. Ok? nope
allocation of 1 x 132 GB of data. Ok? nope
allocation of 1 x 133 GB of data. Ok? nope
allocation of 1 x 134 GB of data. Ok? nope
allocation of 1 x 135 GB of data. Ok? nope
allocation of 1 x 136 GB of data. Ok? nope
allocation of 1 x 137 GB of data. Ok? nope
allocation of 1 x 138 GB of data. Ok? nope
allocation of 1 x 139 GB of data. Ok? nope
allocation of 1 x 140 GB of data. Ok? nope
allocation of 1 x 141 GB of data. Ok? nope
allocation of 1 x 142 GB of data. Ok? nope
allocation of 1 x 143 GB of data. Ok? nope
allocation of 1 x 144 GB of data. Ok? nope
allocation of 1 x 145 GB of data. Ok? nope
allocation of 1 x 146 GB of data. Ok? nope
allocation of 1 x 147 GB of data. Ok? nope
allocation of 1 x 148 GB of data. Ok? nope
allocation of 1 x 149 GB of data. Ok? nope
Now, about that US$0.17 I spent on this...
I'm analysing the X-Loader settings for the POP mDDR on the Beagleboard xM.
The amount of mDDR POP memory in the BB xM is 512MB (according to the Manual).
More precisely the Micron variant: 256MB on CS0 + 256MB on CS1 = 512MB total.
The bus width is 32 bits, this can be verified in the SDRC_MCFG_p register settings in the X-Loader.
The type of memory used is the MT46H128M32L2KQ-5 as mentioned in this group:
https://groups.google.com/forum/#!topic/beagleboard/vgrq2bOxXrE
Reading the data sheet of that memory, the 32 bit configuration with the maximum capacity is 16Meg x 32 x 4 = 64Meg x 32.
So 64MB are not 256MB, 128 MB are feasible but only with 16 bit bus width, and even then, we are still not at 256MB.
The guy in the group mentioned above says that the memory is a 4Gb, but the data sheet says that it is a 2Gb.
My question:
How can 512MB be achieved by using 2 memory chips of the above type and 32 bit bus width?
Thanks in advance for your help.
Martin
According to datasheet MT46H128M32L2KQ-5 has following configuration:
MT46H128M32L2 – 16 Meg x 32 x 4 Banks x 2
16 Meg x 32 x 4 Banks x 2 = 4096 Meg (bits, not bytes)
4096 Meg (bits) / 8 = 512 MB (Megabytes)
More from datasheet:
The 2Gb Mobile low-power DDR SDRAM is a high-speed CMOS, dynamic
random-access memory containing 2,147,483,648 bits.
Each of the x32’s 536,870,912-bit banks is organized as 16,384 rows by 1024
columns by 32 bits. (p. 8)
So, if you multiply the number of rows by the number of columns by the number of bits (it's specified in the datasheet), you will get the size of a bank in bits. Bank size is = 16384 x 1024 x 32 = 16 Megs x 32 = 536870912 (bits).
Next, you need to multiply the bank size (in bits) by the number of banks in chip: chip size = 536870912 x 4 = 2147483648 (bits).
In order to get result in bytes, you have to dived it by 8.
chip size (bytes) = 2147483648 (bits) / 8 = 268435456
In order to get result in megabytes, you have to dived it by 1024 x 1024
chip size = 268435456 / 1024 / 1024 = 256 MB (Megabytes)
This is dual LPDDR chip internally organized as 2 x 256 MB chips (it has two chip selects: CS0#, CS1#) (it's specified in the datasheet). The single chip contains two memory chips inside, 256MB each. For BB this single chip must be configured like 2 memories 256MB each in order to get 512MB. So, you have to setup CS0 as 256MB and CS1 as 256MB.