A question about malloc implementation in glibc - linux

Question

I was reading source code of glibc.
In function void *__libc_malloc(size_t bytes):
void *__libc_malloc(size_t bytes) {
mstate ar_ptr;
void *victim;
_Static_assert(PTRDIFF_MAX <= SIZE_MAX / 2, "PTRDIFF_MAX is not more than half of SIZE_MAX");
if (!__malloc_initialized) ptmalloc_init();
...
}
It shows that if the first thread was created, it calls ptmalloc_init(), and links thread_arena with main_arena, and sets __malloc_initialized to true.
On the other hand, the second thread was blocked by the following code in ptmalloc_init():
static void ptmalloc_init(void) {
if (__malloc_initialized) return;
__malloc_initialized = true;
thread_arena = &main_arena;
malloc_init_state(&main_arena);
...
Thus the thread_arena of the second thread is NULL, and it has to mmap() additional arena.
My question is:
It seems possible to cause race condition because there's no any lock with __malloc_initialized, and thread_arenas of the first thread and second thread may both link with main_arena, why not use lock to protect __malloc_initialized?

Answer 1

It seems possible to cause race condition because there's no any lock with __malloc_initialized
It is impossible1 for a program to create a second running thread without having called an allocation routine (and therefore ptmalloc_init) while it was still single-threaded.
Because of that, ptmalloc_init can assume that it runs while there is only a single thread.
1Why is it impossible? Because creating a thread itself calls calloc.
For example, in this program:
#include <pthread.h>
void *fn(void *p) { return p; }
int main()
{
pthread_t tid;
pthread_create(&tid, NULL, fn, NULL);
pthread_join(tid, NULL);
return 0;
}
ptmalloc_init is called here (only a single thread exists at that point):
Breakpoint 2, ptmalloc_init () at /usr/src/debug/glibc-2.34-42.fc35.x86_64/malloc/arena.c:283
283 if (__malloc_initialized)
(gdb) bt
#0 ptmalloc_init () at /usr/src/debug/glibc-2.34-42.fc35.x86_64/malloc/arena.c:283
#1 __libc_calloc (n=17, elem_size=16) at malloc.c:3526
#2 0x00007ffff7fdd6c3 in calloc (b=16, a=17) at ../include/rtld-malloc.h:44
#3 allocate_dtv (result=result#entry=0x7ffff7dae640) at ../elf/dl-tls.c:375
#4 0x00007ffff7fde0e2 in __GI__dl_allocate_tls (mem=mem#entry=0x7ffff7dae640) at ../elf/dl-tls.c:634
#5 0x00007ffff7e514e5 in allocate_stack (stacksize=<synthetic pointer>, stack=<synthetic pointer>,
pdp=<synthetic pointer>, attr=0x7fffffffde30)
at /usr/src/debug/glibc-2.34-42.fc35.x86_64/nptl/allocatestack.c:429
#6 __pthread_create_2_1 (newthread=0x7fffffffdf58, attr=0x0, start_routine=0x401136 <fn>, arg=0x0)
at pthread_create.c:648
#7 0x0000000000401167 in main () at p.c:7

Answer 2

GLIBC's dynamic memory allocator is designed to deliver performances in both mono-threaded and multi-threaded programs. Several mutexes are used instead of having a centralized unique one which would at the end serialize every concurrent accesses to the dynamic memory allocator. The concept of arenas protected by one mutex has been introduced to have a kind of reserved memory area for each thread. Hence, the threads can access the memory allocator data structures in parallel as long as they use different arenas.
The main goal is to avoid as much as possible the contention on the mutexes.
The initialization step is critical because the main arena must be set up once. The __malloc_initialized global variable is a flag to prevent multiple initializations. Of course, in a multi-threaded environment, the latter should be protected by a mutex because checking the value of a variable is not multi-thread safe. But doing this would break the main design principle consisting to avoid a centralized mutex which would somehow serialize the execution of the concurrent threads during the process life time.
So, the unprotected __malloc_initialized is a trade-off that works as long as the first access to the memory allocator is done in mono-threaded mode.
Under Linux, a process starts mono-threaded (the main thread). With dynamically and statically linked programs, the GLIBC library has an initialization entry point (CSU = C Start Up) called __libc_start_main()_ defined in csu/libc-start.c in the library's source tree. It performs many initializations before calling the main() function. This is where a first call to the dynamic allocator occurs to initialize the main arena.
Let's look at the following program which does not explicitly call any service from the dynamic memory allocator and does not create any thread:
#include <unistd.h>
int main(void)
{
pause();
return 0;
}
Let's compile it and run it with gdb and a breakpoint on malloc():
$ gcc -g mm.c -o mm
$ gdb ./mm
[...]
(gdb) br malloc
Function "malloc" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (malloc) pending.
(gdb) run
Starting program: /.../mm
Breakpoint 1, malloc (n=1441) at dl-minimal.c:49
49 dl-minimal.c: No such file or directory.
(gdb) where
#0 malloc (n=1441) at dl-minimal.c:49
#1 0x00007ffff7fec5e5 in calloc (nmemb=<optimized out>, size=size#entry=1) at dl-minimal.c:103
#2 0x00007ffff7fdc284 in _dl_new_object (realname=realname#entry=0x7ffff7ff4342 "", libname=libname#entry=0x7ffff7ff4342 "", type=type#entry=0, loader=loader#entry=0x0,
mode=mode#entry=536870912, nsid=nsid#entry=0) at dl-object.c:89
#3 0x00007ffff7fd1d2f in dl_main (phdr=0x555555554040, phnum=<optimized out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:1330
#4 0x00007ffff7febc4b in _dl_sysdep_start (start_argptr=start_argptr#entry=0x7fffffffdf70, dl_main=dl_main#entry=0x7ffff7fd15e0 <dl_main>) at ../elf/dl-sysdep.c:252
#5 0x00007ffff7fd104c in _dl_start_final (arg=0x7fffffffdf70) at rtld.c:449
#6 _dl_start (arg=0x7fffffffdf70) at rtld.c:539
#7 0x00007ffff7fd0108 in _start () from /lib64/ld-linux-x86-64.so.2
#8 0x0000000000000001 in ?? ()
#9 0x00007fffffffe2e2 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)
The above display shows that even if malloc() is not called explicitly in the main program, the GLIBC's internals call at least once the memory allocator triggering the initialization of the main arena.
We may consequently wonder why we need to check the __malloc_initialized variable during the process life time after the internal initialization step. The GLIBC initialization sets up various internal modules (main stack, pthreads...) and some of them may call the dynamic memory allocator. Hence __malloc_initialized is here to allow calling the allocator at any time during the initialization step. And, if the allocator is not needed because of some specific esoteric configuration, then it will not be initialized at all.