realloc function gives SIGABRT due to limited heap size - linux

I am trying to reproduc a problem .
My c code giving SIGABRT , i traced it back to this line number :3174
https://elixir.bootlin.com/glibc/glibc-2.27/source/malloc/malloc.c
/* Little security check which won't hurt performance: the allocator
never wrapps around at the end of the address space. Therefore
we can exclude some size values which might appear here by
accident or by "design" from some intruder. We need to bypass
this check for dumped fake mmap chunks from the old main arena
because the new malloc may provide additional alignment. */
if ((__builtin_expect ((uintptr_t) oldp > (uintptr_t) -oldsize, 0)
|| __builtin_expect (misaligned_chunk (oldp), 0))
&& !DUMPED_MAIN_ARENA_CHUNK (oldp))
malloc_printerr ("realloc(): invalid pointer");
My understanding is that when i call calloc function memory get allocated when I call realloc function and try to increase memory area ,heap is not available for some reason giving SIGABRT
My another question is, How can I limit the heap area to some bytes say, 10 bytes to replicate the problem. In stackoverflow RSLIMIT and srlimit is mentioned but no sample code is mentioned. Can you provide sample code where heap size is 10 Bytes ?

How can I limit the heap area to some bytes say, 10 bytes
Can you provide sample code where heap size is 10 Bytes ?
From How to limit heap size for a c code in linux , you could do:
You could use (inside your program) setrlimit(2), probably with RLIMIT_AS (as cited by Ouah's answer).
#include <sys/resource.h>
int main() {
setrlimit(RLIMIT_AS, &(struct rlimit){10,10});
}
Better yet, make your shell do it. With bash it is the ulimit builtin.
$ ulimit -v 10
$ ./your_program.out
to replicate the problem
Most probably, limiting heap size will result in a different problem related to heap size limit. Most probably it is unrelated, and will not help you to debug the problem. Instead, I would suggest to research address sanitizer and valgrind.

Related

How much locked memory does io_uring_setup need?

When using io_uring_queue_init it calls io_uring_setup. There's an ENOMEM returned when there is insufficient amount of locked memory available for the process.
A strace will look something like:
[pid 37480] io_uring_setup(2048, {flags=0, sq_thread_cpu=0, sq_thread_idle=0}) = -1 ENOMEM (Cannot allocate memory)
What is the formula for how much locked memory is required per entry (first argument)? and if possible, based on the sq_entries/cq_entries in the params structure? Kernel code for the particularly keen. Please don't expand the kernel page size from the formula, as I do want this to be an architecture dependent answer (if it is).
I don't want a dodgy just set ulimit -l to unlimited as an answer. There's this outstanding feature request feature request that would help when implemented.
Thanks to Jens Axboe the following to liburing library calls where added (>=liburing-2.1) returning the size in bytes, 0 if not required, or -errno for errors.
ssize_t io_uring_mlock_size(unsigned entries, unsigned flags);
ssize_t io_uring_mlock_size_params(unsigned entries, struct io_uring_params *p);

What is the unit for memory usage in VTK?

I wonder what is the unit for reporting a memory usage returned by vtkPVMemoryUseInformation.GetProcMemoryUse (reference)? Is it a bit, byte, kilobyte? Where can I find this in the documentation?
Update 1
I'm calling the mentioned function from a Python-script with servermanager.vtkPVMemoryUseInformation().GetProcMemoryUse(<index>). We don't have size_t in Python, right? The main question is how can I convert the value into a human-readable value like MB or GB returned by a function call?
This method internally uses vtksys::SystemInformation, which returns system RAM used in units of KiB.
https://github.com/Kitware/VTK/blob/master/Utilities/KWSys/vtksys/SystemInformation.hxx.in
The doc should be improved here.

segmentation fault of C program with security/limits.conf stacksize 8192kb

in linux, at least in SUSE/SLES version 11, if you type 'limit' it will respond with
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 1 kbytes
memoryuse 449929820 kbytes
vmemoryuse 423463360 kbytes
descriptors 1024
memorylocked 64 kbytes
maxproc 4135289
maxlocks unlimited
maxsignal 4135289
maxmessage 819200
maxnice 0
maxrtprio 0
maxrttime unlimited
those are my default settings in /etc/security/limits.conf
I have a C program that is around 5500 lines of code, minimal comments.
I declare some big arrays, the program deals with mesh structures so there is a structured "nodes" array having x,y,z variables of type double, along with some other integer and double variables if needed. And there's a structured "elements" array having n1, n2, n3 variables of type integer. In many of the functions I'll declare something like
struct NodeTYPE nodes[200000];
struct ElemTYPE elements[300000];
When running the program, it spits out a menu to choose what to do, and you enter a number. Enter a 1 then calls some function, enter a 2 calls some different function, and so on. Most of them work but one did not, and debugging the program as soon as the function is called it segmentation faults.
If i modify /etc/security/limits.conf and do
* hard stacksize unlimited
* soft stacksize unlimited
then the program works without changing anything in the code and without recompiling. The program is compiled via
gcc myprogram.c -O2 -o myprogram.x -lm
Can someone explain in some detail why this happens,
and what the best method for fixing this type of problem?
Is it poor programming on my part causing the segmentation fault?
I want to say in the past... when i figured out making stacksize unlimited... if i put my large arrays globally in the program outside of main() then the program would not seg fault... it was only when those big array declarations were within functions that causes this problem.
Is having a limit on stacksize of 8MB ridiculously small (my system has over 128GB of RAM) ?

malloc/realloc/free capacity optimization

When you have a dynamically allocated buffer that varies its size at runtime in unpredictable ways (for example a vector or a string) one way to optimize its allocation is to only resize its backing store on powers of 2 (or some other set of boundaries/thresholds), and leave the extra space unused. This helps to amortize the cost of searching for new free memory and copying the data across, at the expense of a little extra memory use. For example the interface specification (reserve vs resize vs trim) of many C++ stl containers have such a scheme in mind.
My question is does the default implementation of the malloc/realloc/free memory manager on Linux 3.0 x86_64, GLIBC 2.13, GCC 4.6 (Ubuntu 11.10) have such an optimization?
void* p = malloc(N);
... // time passes, stuff happens
void* q = realloc(p,M);
Put another way, for what values of N and M (or in what other circumstances) will p == q?
From the realloc implementation in glibc trunk at http://sources.redhat.com/git/gitweb.cgi?p=glibc.git;a=blob;f=malloc/malloc.c;h=12d2211b0d6603ac27840d6f629071d1c78586fe;hb=HEAD
First, if the memory has been obtained via mmap() instead of sbrk(), which glibc malloc does for large requests, >= 128 kB by default IIRC:
if (chunk_is_mmapped(oldp))
{
void* newmem;
#if HAVE_MREMAP
newp = mremap_chunk(oldp, nb);
if(newp) return chunk2mem(newp);
#endif
/* Note the extra SIZE_SZ overhead. */
if(oldsize - SIZE_SZ >= nb) return oldmem; /* do nothing */
/* Must alloc, copy, free. */
newmem = public_mALLOc(bytes);
if (newmem == 0) return 0; /* propagate failure */
MALLOC_COPY(newmem, oldmem, oldsize - 2*SIZE_SZ);
munmap_chunk(oldp);
return newmem;
}
(Linux has mremap(), so in practice this is what is done).
For smaller requests, a few lines below we have
newp = _int_realloc(ar_ptr, oldp, oldsize, nb);
where _int_realloc is a bit big to copy-paste here, but you'll find it starting at line 4221 in the link above. AFAICS, it does NOT do the constant factor optimization increase that e.g. the C++ std::vector does, but rather allocates exactly the amount requested by the user (rounded up to the next chunk boundaries + alignment stuff and so on).
I suppose the idea is that if the user wants this factor of 2 size increase (or any other constant factor increase in order to guarantee logarithmic efficiency when resizing multiple times), then the user can implement it himself on top of the facility provided by the C library.
Perhaps you can use malloc_usable_size (google for it) to find the answer experimentally. This function, however, seems undocumented, so you will need to check out if it is still available at your platform.
See also How to find how much space is allocated by a call to malloc()?

issue in putting custom fields in struct malloc_chunk

i am trying to put a head-tag and a foot tag inside struct malloc_chunk, like this:
struct malloc_shunk {
INTERNAL_SIZE prev_size;
INTERNAL_SIZE size;
}
Here is what i did:
1.
struct malloc_shunk {
INTERNAL_SIZE foot_tag;
INTERNAL_SIZE prev_size;
INTERNAL_SIZE size;
}
before putting the head_tag inside malloc_chunk, i just added foot_tag only and compiled glibc. I made a small test program that mallocs 60 bytes from the system and then frees it. Thou i can see that the malloc returned properly, free complained, saying "invalid pointer". The pointer malloc returned was 0x9313010. That makes the pointer to the start of malloc_chunk to be 0x9313004.
So when the free is passed 0x9313010, it converts it to 0x9313004 through (mem2chunk) and the check it for alignment. Since my wordsize is 4, alignment check with 0x9313010 is where i am getting problems. Can you please tell me if the Mem pointer (returned by malloc) needs to be absolutely double-word aligned. (as here it may not satisfy that criterion, as difference betwwen the pointer returned by malloc and start of chunk will be 12 bytes here and not 8).
To ever come 1. issue , i just added 1 head-tag to the structure so that it becomes
struct malloc_shunk {
INTERNAL_SIZE foot_tag;
INTERNAL_SIZE prev_size;
INTERNAL_SIZE size;
INTERNAL_SIZE head_tag;
}
Now the difference betwwen the pointer returned by malloc and start of chunk will be 16 which will always be double word aligned. But Here i am facing a bigger problem as the time when first malloc is called the arena is setup and bins are initallised. The size of the victim here is not coming out to be zero as it should be in normal cases. The problem is that the victim->size is actually comes out to be the place where 'top' is stored rather than 'last_remainder'. i'd like to ask for you opinion if there is any other way/workaround/solution, so that i can over come this initialization of arena issue i am currently facing.
Thanks and Regards,
Kapil
From what ever i have learnt, try not to modify the structure malloc_chunk. Even if you have to then make sure that the top pointer is initially 0 otherwise heap will never be formed. Its the sYSMALLOC() function in _int_malloc() that first MMAPs a bigger chunk from kernel. It will be invoked only if the top is 0 initially.

Resources