Valgrind says 0 bytes lost but also says one less Frees than Mallocs
Because I have used malloc only once, I'm only posting those segments and not all the 3 files.
When loading a dictionary.txt file into a hash table:
bool load(const char *dictionary)
{
(dictionary.c:54) FILE *dict_file = fopen(dictionary, "r");
if (dict_file == NULL)
return false;
int key;
node *n = NULL;
int mallocs = 0;
while (1)
{
n = malloc(sizeof(node));
printf("malloced: %i\n", ++mallocs);
if (fscanf(dict_file, "%s", n->word) == -1)
{
printf("malloc freed\n");
free(n);
break;
}
key = hash(n->word);
n->next = table[key];
table[key] = n;
words++;
}
return true;
}
And the Unloading part:
bool unload(void)
{
int deleted = 0;
node *n;
for (int i = 0; i < N; i++)
{
n = table[i];
while(n != NULL)
{
n = n->next;
free(table[i]);
table[i] = n;
deleted++;
}
}
printf("DELETED: %i", deleted);
return true;
}
Check50 says there are memory leaks. But can't understand where.
Command: ./speller dictionaries/small texts/cat.txt
==4215==
malloced: 1
malloced: 2
malloced: 3
malloced: 4
malloc freed
DELETED: 3
WORDS MISSPELLED: 2
WORDS IN DICTIONARY: 3
WORDS IN TEXT: 6
TIME IN load: 0.03
TIME IN check: 0.00
TIME IN size: 0.00
TIME IN unload: 0.00
TIME IN TOTAL: 0.03
==4215==
==4215== HEAP SUMMARY:
==4215== in use at exit: 552 bytes in 1 blocks
==4215== total heap usage: 9 allocs, 8 frees, 10,544 bytes allocated
==4215==
==4215== 552 bytes in 1 blocks are still reachable in loss record 1 of 1
==4215== at 0x4C31B0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4215== by 0x525AF29: __fopen_internal (iofopen.c:65)
==4215== by 0x525AF29: fopen##GLIBC_2.2.5 (iofopen.c:89)
==4215== by 0x40114E: load (dictionary.c:54)
==4215== by 0x40095E: main (speller.c:40)
==4215==
==4215== LEAK SUMMARY:
==4215== definitely lost: 0 bytes in 0 blocks
.
.
.
==4215== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
speller.c has distribution code. I hope the rest of the question is clear and understandable.
The pointer to the opened file (dict_file) needs to be closed. See man fclose.
Related
I am learning pthreads programming.
I understood that there are two states of thread:
1. Joinable
2. Detachable
In case of Joinable, we need to call pthread_join to free the resources(stack), whereas in case of detached there is no need to call pthread_join and the resources will be freed on thread exit.
I wrote a sample program to observe the behavior
#include <stdio.h>
#include <pthread.h>
#include <stdlib.h>
void *threadFn(void *arg)
{
pthread_detach(pthread_self());
sleep(1);
printf("Thread Fn\n");
pthread_exit(NULL);
}
int main(int argc, char *argv[])
{
pthread_t tid;
int ret = pthread_create(&tid, NULL, threadFn, NULL);
if (ret != 0) {
perror("Thread Creation Error\n");
exit(1);
}
printf("After thread created in Main\n");
pthread_exit(NULL);
}
When i try to check any mem leaks with valgrind it gave me leaks of 272 bytes. Can you show me why is the leak happening here.
$valgrind --leak-check=full ./app
==38649==
==38649== HEAP SUMMARY:
==38649== in use at exit: 272 bytes in 1 blocks
==38649== total heap usage: 7 allocs, 6 frees, 2,990 bytes allocated
==38649==
==38649== 272 bytes in 1 blocks are possibly lost in loss record 1 of 1
==38649== at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==38649== by 0x40134A6: allocate_dtv (dl-tls.c:286)
==38649== by 0x40134A6: _dl_allocate_tls (dl-tls.c:530)
==38649== by 0x4E44227: allocate_stack (allocatestack.c:627)
==38649== by 0x4E44227: pthread_create##GLIBC_2.2.5 (pthread_create.c:644)
==38649== by 0x108902: main (2.c:18)
==38649==
==38649== LEAK SUMMARY:
==38649== definitely lost: 0 bytes in 0 blocks
==38649== indirectly lost: 0 bytes in 0 blocks
==38649== possibly lost: 272 bytes in 1 blocks
==38649== still reachable: 0 bytes in 0 blocks
==38649== suppressed: 0 bytes in 0 blocks
==38649==
==38649== For counts of detected and suppressed errors, rerun with: -v
==38649== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Your expectation is correct that there shouldn't be any leaks in main thread once you call pthread_exit.
However, what you observe is a quirk of the implementation you're using (which is likely to be glibc) - pthreads library (glibc implementation) re-uses the initially allocated stack for threads - like a cache so that previously allocated stacks can be re-used whenever possible.
Valgrind simply reports what it "sees" (something was allocated but not de-allocated). But it's not a real leak, so you don't need to worry about this.
If you "reverse" the logic (main thread exits as the last thread) then you wouldn't see leaks because the initially allocated stack space is properly free'd by the main thread. But this leak isn't a real leak in any case and you can safely ignore this.
You can also setup a suppression file so that Valgrind doesn't complain about this (which is to inform Valgrind that "I know this isn't not real leak, so don't report this"), such as:
{
Pthread_Stack_Leaks_Ignore
Memcheck:Leak
fun:calloc
fun:allocate_dtv
fun:_dl_allocate_tls
fun:allocate_stack
fun:pthread_create*
}
Using mbind, one can set the memory policy for a given mapped memory segment.
Q: How can I tell mbind to interleave a segment on all nodes?
If done after allocation but before usage, MPOL_INTERLEAVE on all nodes will do what we expect -- memory will be allocated uniformly on all nodes.
However, if the segment has already been written to and is allocated in e.g. node zero, there is no way to tell the kernel to uniformly interleave it on all NUMA nodes.
The operation simply becomes a no-op, as the kernel interprets it as "please place this segment on this set of nodes". Since we're passing the set of all NUMA nodes, there is no memory allocated outside that requires being moved.
Minimal, Complete, and Verifiable example
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sched.h>
#include <sys/syscall.h>
#include <numaif.h>
#include <numa.h>
#define N ((1<<29) / sizeof(int))
#define PAGE_SIZE sysconf(_SC_PAGESIZE)
#define PAGE_MASK (~(PAGE_SIZE - 1))
void print_command(char *cmd) {
FILE *fp;
char buf[1024];
if ((fp = popen(cmd, "r")) == NULL) {
perror("popen");
exit(-1);
}
while(fgets(buf, sizeof(buf), fp) != NULL) {
printf("%s", buf);
}
if(pclose(fp)) {
perror("pclose");
exit(-1);
}
}
void print_node_allocations() {
char buf[1024];
snprintf(buf, sizeof(buf), "numastat -c %d", getpid());
printf("\x1B[32m");
print_command(buf);
printf("\x1B[0m");
}
int main(int argc, char **argv) {
int *a = numa_alloc_local(N * sizeof(int));
size_t len = (N * sizeof(int)) & PAGE_MASK;
unsigned long mymask = *numa_get_mems_allowed()->maskp;
unsigned long maxnode = numa_get_mems_allowed()->size;
// pin thread to core zero
cpu_set_t mask;
CPU_ZERO(&mask);
CPU_SET(0, &mask);
if (sched_setaffinity(syscall(SYS_gettid), sizeof(mask), &mask) < 0) {
perror("sched_setaffinity");
exit(-1);
}
// initialize array
printf("\n\n(1) array allocated on local node\n");
a[0] = 997;
for(size_t i=1; i < N; i++) {
a[i] = a[i-1] * a[i-1] % 1000000000;
}
print_node_allocations();
// attempt to get it to be uniformly interleaved on all nodes
printf("\n\n(2) array interleaved on all nodes\n");
if (mbind(a, len, MPOL_INTERLEAVE, &mymask, maxnode, MPOL_MF_MOVE_ALL | MPOL_MF_STRICT) == -1) {
perror("mbind failed");
exit(-1);
}
print_node_allocations();
// what if we interleave on all but the local node?
printf("\n\n(3) array interleaved on all nodes (except local node)\n");
mymask -= 0x01;
if (mbind(a, len, MPOL_INTERLEAVE, &mymask, maxnode, MPOL_MF_MOVE_ALL | MPOL_MF_STRICT) == -1) {
perror("mbind failed");
exit(-1);
}
print_node_allocations();
return 0;
}
Compiling and running with gcc -o interleave_all interleave_all.c -lnuma && sudo ./interleave_all yields:
(1) array allocated on local node
Per-node process memory usage (in MBs) for PID 20636 (interleave_all)
Node 0 Node 1 Node 2 Node 3 Total
------ ------ ------ ------ -----
Huge 0 0 0 0 0
Heap 0 0 0 0 0
Stack 0 0 0 0 0
Private 514 0 0 0 514
------- ------ ------ ------ ------ -----
Total 514 0 0 0 514
(2) array interleaved on all nodes
Per-node process memory usage (in MBs) for PID 20636 (interleave_all)
Node 0 Node 1 Node 2 Node 3 Total
------ ------ ------ ------ -----
Huge 0 0 0 0 0
Heap 0 0 0 0 0
Stack 0 0 0 0 0
Private 514 0 0 0 514
------- ------ ------ ------ ------ -----
Total 514 0 0 0 514
(3) array interleaved on all nodes (except local node)
Per-node process memory usage (in MBs) for PID 20636 (interleave_all)
Node 0 Node 1 Node 2 Node 3 Total
------ ------ ------ ------ -----
Huge 0 0 0 0 0
Heap 0 0 0 0 0
Stack 0 0 0 0 0
Private 2 171 171 171 514
------- ------ ------ ------ ------ -----
Total 2 171 171 171 514
I am using Pango library alongside Cairo, without GTK, in a test-drive application which I'm currently compiling on MacOSX. I have a memory leakage problem, that I have traced to this function:
void draw_with_cairo (void)
{
PangoLayout *layout;
PangoFontDescription *desc;
int i;
cairo_save (cr);
cairo_scale (cr, 1, -1);
cairo_translate (cr, 0, -HEIGHT);
cairo_translate (cr, 400, 300);
layout = pango_cairo_create_layout (cr);
pango_layout_set_text (layout, "Test", -1);
desc = pango_font_description_from_string ("BMitra 32");
pango_layout_set_font_description (layout, desc);
pango_font_description_free (desc);
for (i = 0; i < 12; i++)
{
int width, height;
double angle = iter + (360.0 * i) / 12;
double red;
cairo_save (cr);
red = (1 + cos ((angle - 60) * G_PI / 180.)) / 2;
cairo_set_source_rgb (cr, red, 0, 1.0 - red);
cairo_rotate (cr, angle * G_PI / 180.);
pango_cairo_update_layout (cr, layout);
pango_layout_get_size (layout, &width, &height);
cairo_move_to (cr, - ((double)width / PANGO_SCALE) / 2, - 250);
pango_cairo_show_layout (cr, layout);
cairo_restore (cr);
}
cairo_restore (cr);
g_object_unref (layout);
}
This routine is being called a lot, maybe a hundred times in a second. And the memory leak is huge, around 30MB in 3secs, and has a constant rate. When I compare this code, it seems quite fine to me. I have searched for this, have found many references to memory leaks while using pango in Gtk applications, and they all look for a patch in pango or gtk. I am really puzzled and can't believe there would be such a bug in a heavily used library like pango and think this is a problem with my own code. Any suggestions is appreciated.
This is the vmmap result for Uli's code:
Executing vmmap -resident 25897 | grep TOTAL at beginning of main()
TOTAL 321.3M 126.2M 485
TOTAL 18.0M 200K 1323 173K 0% 2
Executing vmmap -resident 25897 | grep TOTAL after cairo init
TOTAL 331.3M 126.4M 489
TOTAL 27.0M 224K 1327 1155K 4% 6
Executing vmmap -resident 25897 | grep TOTAL after one iteration
TOTAL 383.2M 143.9M 517
TOTAL 37.2M 3368K 18634 3423K 8% 5
Executing vmmap -resident 25897 | grep TOTAL after loop
TOTAL 481.6M 244.1M 514
TOTAL 137.2M 103.7M 151961 66.4M 48% 6
Executing vmmap -resident 25897 | grep TOTAL at end
TOTAL 481.6M 244.1M 520
TOTAL 136.3M 103.1M 151956 65.4M 48% 11
And this is the unfiltered output of the last stage:
Executing vmmap -resident 25751 at end
Process: main [25751]
Path: /PATH/OMITTED/main
Load Address: 0x109b9c000
Identifier: main
Version: ???
Code Type: X86-64
Parent Process: bash [837]
Date/Time: 2016-01-30 23:28:35.866 +0330
Launch Time: 2016-01-30 23:27:35.148 +0330
OS Version: Mac OS X 10.11.2 (15C50)
Report Version: 7
Analysis Tool: /Applications/Xcode.app/Contents/Developer/usr/bin/vmmap
Analysis Tool Version: Xcode 7.0.1 (7A1001)
----
Virtual Memory Map of process 25751 (main)
Output report format: 2.4 -- 64-bit process
VM page size: 4096 bytes
==== Non-writable regions for process 25751
==== Legend
SM=sharing mode:
COW=copy_on_write PRV=private NUL=empty ALI=aliased
SHM=shared ZER=zero_filled S/A=shared_alias
==== Summary for process 25751
ReadOnly portion of Libraries: Total=219.6M resident=112.2M(51%) swapped_out_or_unallocated=107.5M(49%)
Writable regions: Total=155.7M written=5448K(3%) resident=104.1M(67%) swapped_out=0K(0%) unallocated=51.6M(33%)
VIRTUAL RESIDENT REGION
REGION TYPE SIZE SIZE COUNT (non-coalesced)
=========== ======= ======== =======
Activity Tracing 2048K 12K 2
Dispatch continuations 8192K 32K 2
Kernel Alloc Once 8K 8K 3
MALLOC guard page 32K 0K 7
MALLOC metadata 364K 84K 11
MALLOC_LARGE 260K 260K 2 see MALLOC ZONE table below
MALLOC_LARGE (empty) 980K 668K 2 see MALLOC ZONE table below
MALLOC_LARGE metadata 4K 4K 2 see MALLOC ZONE table below
MALLOC_SMALL 32.0M 880K 3 see MALLOC ZONE table below
MALLOC_TINY 104.0M 102.1M 7 see MALLOC ZONE table below
STACK GUARD 56.0M 0K 3
Stack 8264K 60K 3
VM_ALLOCATE 16K 8K 2
__DATA 16.7M 13.6M 217
__IMAGE 528K 104K 2
__LINKEDIT 92.4M 22.5M 34
__TEXT 127.2M 89.6M 220
__UNICODE 552K 476K 2
mapped file 32.2M 13.7M 4
shared memory 328K 172K 10
=========== ======= ======== =======
TOTAL 481.6M 244.3M 518
VIRTUAL RESIDENT ALLOCATION BYTES REGION
MALLOC ZONE SIZE SIZE COUNT ALLOCATED % FULL COUNT
=========== ======= ========= ========= ========= ====== ======
DefaultMallocZone_0x109bd0000 136.3M 103.2M 151952 65.4M 48% 10
GFXMallocZone_0x109bd3000 0K 0K 0 0K 0
=========== ======= ========= ========= ========= ====== ======
TOTAL 136.3M 103.2M 151952 65.4M 48% 10
I have omitted the non-writable regions part because it was overflowing stackoverflow limits!
I don't see any memory leaks. The following program prints its memory usage before and after running your above function 100.000 times. Both numbers are the same for me.
#include <cairo.h>
#include <math.h>
#include <pango/pangocairo.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define HEIGHT 500
#define WIDTH 500
void draw_with_cairo (cairo_t *cr)
{
PangoLayout *layout;
PangoFontDescription *desc;
int i;
cairo_save (cr);
cairo_scale (cr, 1, -1);
cairo_translate (cr, 0, -HEIGHT);
cairo_translate (cr, 400, 300);
layout = pango_cairo_create_layout (cr);
pango_layout_set_text (layout, "Test", -1);
desc = pango_font_description_from_string ("BMitra 32");
pango_layout_set_font_description (layout, desc);
pango_font_description_free (desc);
for (i = 0; i < 12; i++)
{
int width, height;
double angle = i + (360.0 * i) / 12;
double red;
cairo_save (cr);
red = (1 + cos ((angle - 60) * G_PI / 180.)) / 2;
cairo_set_source_rgb (cr, red, 0, 1.0 - red);
cairo_rotate (cr, angle * G_PI / 180.);
pango_cairo_update_layout (cr, layout);
pango_layout_get_size (layout, &width, &height);
cairo_move_to (cr, - ((double)width / PANGO_SCALE) / 2, - 250);
pango_cairo_show_layout (cr, layout);
cairo_restore (cr);
}
cairo_restore (cr);
g_object_unref (layout);
}
static void print_memory_usage(const char *comment)
{
char buffer[1024];
sprintf(buffer, "grep -E VmPeak\\|VmSize /proc/%d/status", getpid());
printf("Executing %s %s\n", buffer, comment);
system(buffer);
}
int main()
{
cairo_surface_t *s;
cairo_t *cr;
int i;
print_memory_usage("at beginning of main()");
s = cairo_image_surface_create(CAIRO_FORMAT_ARGB32, WIDTH, HEIGHT);
cr = cairo_create(s);
print_memory_usage("after cairo init");
draw_with_cairo(cr);
print_memory_usage("after one iteration");
for (i = 0; i < 100 * 1000; i++)
draw_with_cairo(cr);
print_memory_usage("after loop");
cairo_surface_destroy(s);
cairo_destroy(cr);
print_memory_usage("at end");
return 0;
}
Output for me (with no traces of any memory leaks):
Executing grep -E VmPeak\|VmSize /proc/31881/status at beginning of main()
VmPeak: 76660 kB
VmSize: 76660 kB
Executing grep -E VmPeak\|VmSize /proc/31881/status after cairo init
VmPeak: 77640 kB
VmSize: 77640 kB
Executing grep -E VmPeak\|VmSize /proc/31881/status after one iteration
VmPeak: 79520 kB
VmSize: 79520 kB
Executing grep -E VmPeak\|VmSize /proc/31881/status after loop
VmPeak: 79520 kB
VmSize: 79520 kB
Executing grep -E VmPeak\|VmSize /proc/31881/status at end
VmPeak: 79520 kB
VmSize: 78540 kB
P.S.: I tested this on an up-to-date debian testing amd64.
I am doing a small Project. I am checking about memory leaks using the tool Valgrind. When I use this tool, I got the bellow information.
> 584 bytes in 74 blocks are definitely lost in loss record 103 of 104
> ==4628== at 0x402BE68: malloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
> ==4628== by 0x41CF8D0: strdup (strdup.c:43)
> ==4628== by 0x8060B95: main (in mycall)
>
> LEAK SUMMARY:
> ==4628== definitely lost: 584 bytes in 74 blocks
> ==4628== indirectly lost: 0 bytes in 0 blocks
> ==4628== possibly lost: 0 bytes in 0 blocks
> ==4628== still reachable: 21,414 bytes in 383 blocks
> ==4628== suppressed: 0 bytes in 0 blocks
> ==4628==
> ==4628== For counts of detected and suppressed errors, rerun with: -v
> ==4628== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
These are the codings I have used the function strdup. I have used in LEX code.
{string} {
yylval.string = strdup(yytext + 1);
yylval.string[yyleng - 2] = 0;
return PPSTRING;
}
{numvar} { yylval.string = strdup(yytext);return(PPNUMVAR); }
{sysnumvar} { yylval.string = (char *) strdup(yytext);return(PPSYSNUMVAR); }
I don't know in which point the memory has been leaked.
strdup function allocate necessary memory to store the sourcing string implicitly, you need to free the returned string (i.e., yylval.string in your code) manually.
From the official documentation (source):
process.memoryUsage()
Returns an object describing the memory usage of the Node process
measured in bytes.
var util = require('util');
console.log(util.inspect(process.memoryUsage()));
This will generate:
{ rss: 4935680, heapTotal: 1826816, heapUsed: 650472 }
heapTotal and heapUsed refer to V8's memory usage.
Exactly what do rss, heapTotal, and heapUsed stand for?
It might seem like a trivial question, but I've been looking and I could not find a clear answer so far.
In order to answer this question, one has to understand V8’s Memory Scheme first.
A running program is always represented through some space allocated in memory. This space is called Resident Set. V8 uses a scheme similar to the Java Virtual Machine and divides the memory into segments:
Code: the actual code being executed
Stack: contains all value types (primitives like integer or Boolean) with pointers referencing objects on the heap and pointers defining the control flow of the program
Heap: a memory segment dedicated to storing reference types like objects, strings and closures.
Now it is easy to answer the question:
rss: Resident Set Size
heapTotal: Total Size of the Heap
heapUsed: Heap actually Used
Ref: http://apmblog.dynatrace.com/2015/11/04/understanding-garbage-collection-and-hunting-memory-leaks-in-node-js/
RSS is the resident set size, the portion of the process's memory held in RAM (as opposed to the swap space or the part held in the filesystem).
The heap is the portion of memory from which newly allocated objects will come from (think of malloc in C, or new in JavaScript).
You can read more about the heap at Wikipedia.
The Node.js documentation describes it as follows:
heapTotal and heapUsed refer to V8's memory usage. external refers to
the memory usage of C++ objects bound to JavaScript objects managed by
V8. rss, Resident Set Size, is the amount of space occupied in the
main memory device (that is a subset of the total allocated memory)
for the process, which includes the heap, code segment and stack.
All mentioned values are expressed in bytes. So, if you just want to print them, you probably want to rescale them to MB:
const used = process.memoryUsage();
for (let key in used) {
console.log(`Memory: ${key} ${Math.round(used[key] / 1024 / 1024 * 100) / 100} MB`);
}
That will give you an output like:
Memory: rss 522.06 MB
Memory: heapTotal 447.3 MB
Memory: heapUsed 291.71 MB
Memory: external 0.13 MB
Let's do this with an Example
The following example will show you how the increase in memory usage will actually increase the rss and heapTotal
const numeral = require('numeral');
let m = new Map();
for (let i = 0; i < 100000; i++) {
m.set(i, i);
if (i % 10000 === 0) {
const { rss, heapTotal } = process.memoryUsage();
console.log( 'rss', numeral(rss).format('0.0 ib'), heapTotal, numeral(heapTotal).format('0.0 ib') )
}
}
Running The above will give you something like this:
rss 22.3 MiB 4734976 4.5 MiB
rss 24.2 MiB 6483968 6.2 MiB
rss 27.6 MiB 9580544 9.1 MiB
rss 27.6 MiB 9580544 9.1 MiB
rss 29.3 MiB 11419648 10.9 MiB
rss 29.3 MiB 11419648 10.9 MiB
rss 29.3 MiB 11419648 10.9 MiB
rss 32.8 MiB 15093760 14.4 MiB
rss 32.9 MiB 15093760 14.4 MiB
rss 32.9 MiB 15093760 14.4 MiB
This clearly shows you how using variable and continuously incrementing the space required by it increases the heapTotal and correspondingly the Resident Set Size(rss)
RSS
RSS is a reasonable measure for the "total memory usage of the Node.js interpreter process". You simply be able to run your program if that goes above the available RAM. Note however that it excludes some types of memory, so the actual memory consumption on a server that just runs a single process could be higher (VSZ is the worst case).
The concept of RSS is defined in the Linux kernel itself as mentioned at: What is RSS and VSZ in Linux memory management and measures the total memory usage of the process. This value can therefore be measured by external programs such as ps without knowledge of Node.js internals, e.g. as shown at: Retrieve CPU usage and memory usage of a single process on Linux?
heapTotal and heapUsed
These are concepts internal to the Node.js implementation. It would be good to look at the v8 source code to understand them more precisely, notably I wonder if they just obtain those values from glibc with functions such as those mentioned at: API call to get current heap size of process? of if it has its own heap management done on top of it.
For the concept of heap in general see also: What and where are the stack and heap? and What is the function of the push / pop instructions used on registers in x86 assembly? The heap is overwhelmingly likely to take the majority of memory in a JavaScript program, I don't think you will ever bother to try and look for that memory elsewhere (besides perhaps typed arrays perhaps, which show separately under process.memoryUsage()).
Runnable test
The following code example can be used to do simple tests which I have tried to analyze at: https://cirosantilli.com/javascript-memory-usage-benchmark But unlike languages without garbage collection like C++, it is very difficult to predict why memory usage is so overblown sometimes, especially when we have smaller numbers of objects. I'm not sure other garbage collected languages do any better though.
You have to run the program with:
node --expose-gc main.js
main.js
#!/usr/bin/env node
// CLI arguments.
let arr = false
let array_buffer = false
let dealloc = false
let klass = false
let obj = false
let n = 1000000
let objn = 0
for (let i = 2; i < process.argv.length; i++) {
switch (process.argv[i]) {
case 'arr':
arr = true
break
case 'array-buffer':
array_buffer = true
break
case 'class':
klass = true
break
case 'dealloc':
dealloc = true
break
case 'obj':
obj = true
break
case 'n':
i++
n = parseInt(process.argv[i], 10)
break
case 'objn':
i++
objn = parseInt(process.argv[i], 10)
break
default:
console.error(`unknown option: ${process.argv[i]}`);
break
}
}
class MyClass {
constructor(a, b) {
this.a = a
this.b = b
}
}
let a
if (array_buffer) {
a = new Int32Array(new ArrayBuffer(n * 4))
for (let i = 0; i < n; i++) {
a[i] = i
}
} else if (obj) {
a = []
for (let i = 0; i < n; i++) {
a.push({ a: i, b: -i })
}
} else if (objn) {
a = []
for (let i = 0; i < n; i++) {
const obj = {}
for (let j = 0; j < objn; j++) {
obj[String.fromCharCode(65 + j)] = i
}
a.push(obj)
}
} else if (klass) {
a = []
for (let i = 0; i < n; i++) {
a.push({ a: i, b: -i })
}
} else if (klass) {
a = []
for (let i = 0; i < n; i++) {
a.push(new MyClass(i, -i))
}
} else if (arr) {
a = []
for (let i = 0; i < n; i++) {
a.push([i, -i])
}
} else {
a = []
for (let i = 0; i < n; i++) {
a.push(i)
}
}
if (dealloc) {
a = undefined
}
let j
while (true) {
if (!dealloc) {
j = 0
// The collector somehow removes a if we don't reference it here.
for (let i = 0; i < n; i++) {
if (obj || klass) {
j += a[i].a + a[i].b
} else if (objn) {
const obj = a[i]
for (let k = 0; k < objn; k++) {
j += obj[String.fromCharCode(65 + k)]
}
} else if (arr) {
j += a[i][0] + a[i][1]
} else {
j += a[i]
}
}
console.error(j)
}
global.gc()
console.error(process.memoryUsage())
}
Some things we learn on Node 16 Ubuntu 21.10:
with node --expose-gc bench_mem.js n 1 we see that the minimum RSS is 30 MiB and the minimum heapUsed 3.7 MB. RSS for a C hello world on the same system is 770 kB for comparison