malloc() allocates a memory chunk which is virtually contiguous inside the process memory space. malloc() takes a size as a parameter in bytes and returns pointer to that allocated memory space but what if the requirement is to allocate memory which is 4k aligned?
That would almost certainly be achieved using something like posix_memalign.
Since 4Kbytes is often the size of a page (see sysconf(3) with _SC_PAGESIZE or the old getpagesize(2) syscall) you could use mmap(2) syscall (which is used by malloc and posix_memalign) to get 4Kaligned memory.
you can not allocate physically contiguous memory in user space. Because in User space kernel always allocates memory from highmem zone. But if you are writing a kernel module or a system space code then you can use _get_page() or _get_pages().
Related
In the linux man page,
By default, Linux follows an optimistic memory allocation
strategy. This means that when malloc() returns non-NULL there
is no guarantee that the memory really is available.
How is Linux able to lazily allocate memory?
My guess is that sbrk is called, Linux remembers the process ID and stores some kind of mapping to determine which virtual memory address it has allocated a physical address for. Where can I read to get more information on this?
From https://man7.org/linux/man-pages/man3/malloc.3.html :
Normally, malloc() allocates memory from the heap, and adjusts
the size of the heap as required, using sbrk(2). When allocating
blocks of memory larger than MMAP_THRESHOLD bytes, the glibc
malloc() implementation allocates the memory as a private
anonymous mapping using mmap(2).
See Why does calling mmap() with large size not fail? .
Many articles say that memory allocated by malloc is in the heap. And we also know that, normally, when the size requested by malloc is more than 128KB, memory is allocated in the memory mapping segment using the mmap syscall. When less than 128KB, memory is allocated in the heap by using the brk syscall.
So what is the difference between the heap and the memory mapping segment? Is "memory allocated by malloc is in the heap" wrong?
I thought a little about virtual memory management, and came to the result that there can be two types of memory fragmentation. The first happens on the physical memory side where pages can not be freed because there are some bytes of it used. Mostly the last bytes will be freed sooner or later and then the physical memory page will become free again and is unmapped.
But what happens to the pointer (virtual address) returned by malloc. Let's assume a 32-bit system. The program "randomly" allocates and frees memory but there is never used more than some MByte. Let's assume further that the program will never free the memory in the order it is allocated. So the "top of heap" pointer can never be decreased as the free will never occur at the end of the heap. I assume that malloc has to map the memory always to the end of the heap memory space. This means the pointer value will increase with every call.
Earlier or later the returned pointer will reach the highest possible address (e.g. 0xffffffff) and it becomes impossible to further add memory while the system has enough free pages available as most pages have been freed. It is just a matter of the highest possible pointer value.
To solve this an algorithm would be needed that maintains unmapped address spaces and let them grow as more memory is beeing freed at the beginning or the end of the space. Is there an algorithm like this implemented by malloc?
I assume that malloc has to map the memory always to the end of the heap memory space.
This assumption is actually incorrect. Some implementations may keep multiple pools that different sizes of blocks are allocated from. (For instance, one common approach is a slab allocator, which keeps a separate pool for each size of block that the allocator will return.)
In any case, yes — all meaningful implementations of malloc() will track memory that has been freed and will reuse it when possible.
I had a short look at the slab allocator. This seems to be more related to memory page management used inside kernel. My question is related to the user space and the fact that whenever memory is allocated it needs to get an address in address space of the calling process's heap. What happens to this address space when it is limited as given in a 32-bit system.
It is clear that the system does not loose the memory at all. What I mean is that there is no address space left to get an address where the memory can be mapped while all memory at lower addresses has been freed and unmapped already.
I want to allocate a large portion of memory using malloc() for an indefinite amount of time. I may touch the memory for a long time let say 1 minute. How do i prevent the kernel from taking that memory away from the process?
I can not re allocate that memory because it is being used by another device that is outside of the kernels control.
In the Linux, you can allocate memory in user space, such as with malloc or a mmap, pass it down to the kernel, and then in the kernel, obtain references to the memory with get_user_pages. This will prevent the pages from going away, and also allow them to be accessed from any address space as struct page * references (and requiring kmap and kunmap if CONFIG_HIGHMEM is in effect). These pages will not be contiguous physical memory, however, and they may not be in a suitable range for DMA.
Memory to be accessed by devices is usually allocated in the kernel (e.g. using kmalloc with GFP_DMA. For allocations larger than a page, kmalloc finds consecutive physical pages, too. Once obtained, kmalloc-ed memory can be mapped into user space with remap_pfn_range.
What is a memory heap ?
Presumably you mean heap from a memory allocation point of view, not from a data structure point of view (the term has multiple meanings).
A very simple explanation is that the heap is the portion of memory where dynamically allocated memory resides (i.e. memory allocated via malloc). Memory allocated from the heap will remain allocated until one of the following occurs:
The memory is free'd
The program terminates
If all references to allocated memory are lost (e.g. you don't store a pointer to it anymore), you have what is called a memory leak. This is where the memory has still been allocated, but you have no easy way of accessing it anymore. Leaked memory cannot be reclaimed for future memory allocations, but when the program ends the memory will be free'd up by the operating system.
Contrast this with stack memory which is where local variables (those defined within a method) live. Memory allocated on the stack generally only lives until the function returns (there are some exceptions to this, e.g. static local variables).
You can find more information about the heap in this article.
A memory heap is a location in memory where memory may be allocated at random access. Unlike the stack where memory is allocated and released in a very defined order, individual data elements allocated on the heap are typically released in ways which is asynchronous from one another. Any such data element is freed when the program explicitly releases the corresponding pointer, and this may result in a fragmented heap. In opposition only data at the top (or the bottom, depending on the way the stack works) may be released, resulting in data element being freed in the reverse order they were allocated.
Heap is just an area where memory is allocated or deallocated without any order. This happens when one creates an object using the new operator or something similar. This is opposed to stack where memory is deallocated on the first in last out basis.
It's a chunk of memory allocated from the operating system by the memory manager in use by a process. Calls to malloc() et alia then take memory from this heap instead of having to deal with the operating system directly.
You probably mean heap memory, not memory heap.
Heap memory is essentially a large pool of memory (typically per process) from which the running program can request chunks. This is typically called dynamic allocation.
It is different from the Stack, where "automatic variables" are allocated. So, for example, when you define in a C function a pointer variable, enough space to hold a memory address is allocated on the stack. However, you will often need to dynamically allocate space (With malloc) on the heap and then provide the address where this memory chunk starts to the pointer.
A memory heap is a common structure for holding dynamically allocated memory.
See Dynamic_memory_allocation on wikipedia.
There are other structures, like pools, stacks and piles.
Memory organization is divided into two parts: heap memory and stack memory.
Heap memory is the main working memory, lowest address is the starting address.
In stack memory, the flow of data is driven by bottom to up approach. Then the memory Arch is named as stack.
every running process has its own private fake virtual memory provided by the OS.
the OS can map this to physical memory at any point as long as it is available otherwise it will map to disk and swap as needed.
this virtual memory is logically divided into segments for organizing different kinds of data.
the code segment holds the executable instructions.
the data segment holds static data such as global or static variables.
the stack holds local data that is automatically managed by called and returning functions.
all of these segments are fixed size even the stack its just the portion used can grow or shrink and is reclaimed as functions returned.
the only segment that is not preallocated at app startup and fixed size is the heap.
the app can request from the OS at runtime new memory to be allocated and the OS will reserve a part of your apps virtual space and commit that to physical memory as needed.
the OS will return a pointer to that newly allocated heap memory and that pointer holds the base or starting address of the new block. that pointer sits on the stack and when that stack space is reclaimed your pointer will be no longer in scope and therefore you have no means of access to that block of memory. and if you dont tell the OS you are done with it so it can reclaim it that is just zombie memory sitting there with no means of access and if your app keeps requesting memory while never giving it back it will crash when the system runs out of memory. so it is important to free or at least pass the pointer to another pointer external to the scope it was defined in so you can maintain an interface to that memory allocated in heap space. i would suggest looking into virtual memory further and understanding segments.