I'm writing my own memory allocator. If my notion is right, there is kernel's addres space from zero to 1-GB, and user-space from 1 to 4, that contains different sections like: code, stack, data, heap and others.
Is it right, that heap section's size and position can't be changed during the program execution?
How can I get the size and the position?
All I need to do, after getting heap area, is allocate the memory at my discretion, isn't I?
Why do you worry about this?
If you're writing an allocator a-la libc, use sbrk and/or mmap to reserve memory from the kernel. You do not need to worry about where the heap is placed with neither of those system calls.
If you want to instrument libc's malloc/calloc/realloc, things are even simpler - just wrap them in your allocator functions.
Yes, the allocator effectively manages the heap by requesting memory from the kernel. Typically, as is the case with brk, its position lies after the end of the data segment, and it grows at increasing addresses (or allocates in multiples of page size with mmap, etc.)
The allocator manages the size; the position of the heap is not relevant as far as the allocator is concerned, but it's at the position of knowing it.
The allocator effectively requests memory from the kernel. Once it has that memory, it can distribute it to programs however it deems fit.
It is an allocator which defines a heap. If you have a custom allocator, and it determines that all memory clients have returned all memory, it is perfectly valid for it to delete its heap or create a new one to supply memory requests from.
Since the allocator itself defines the heap it should know its size and position. If what you are talking about is usurping the OS allocator's responsibility with your own allocator, you should only do this by using the OS allocator to obtain a block of memory then use that as the heap for your own allocator.
Again, once your allocator owns the memory block it may be used at your discretion. You cannot simply take memory which is managed by another allocator and in its free pool and use it without serious potential consequences.
Related
HVM is a functional runtime which represents pointers as 32-bit values. Its allocator reserves a huge (4 GB) buffer preemptively, which it uses to create internal objects. This is not ideal. Instead, I'd like to use the system allocator, but that's not possible, since it returns 64-bit pointers, which may be larger than the space available to store them. Is there any cross-platform way in Rust to allocate a buffer, such that the pointer to the buffer is guaranteed to fit in an u32? In other words, I'm looking for something akin to:
let ptr = Box::new_with_small_ptr(size);
assert!(ptr as u64 + size < u32::MAX);
There isn't, because it's an extremely niche need and requires a lot of care.
It's not as simple as "just returning a low pointer" - you need to actually allocate that space from the OS. Your entry point into that would be mmap. Be prepared to do some low-level work with MAP_FIXED and reading /proc/self/maps, and also implementing an allocator on top of the memory region you get from mmap.
If your concern is just excess memory usage, note that Linux overcommits memory by default - allocating 4GB of memory won't reserve physical memory unless you actually try to use it all.
If a process initially has a number of pages allocated to it in the heap, but a lot of the data in the pages has been deallocated, is there some sort of optimization that the OS does to consolidate the data into one page so that the other pages can be freed?
In general, nothing happens, the heap will continue to have "holes" in it.
Since the (virtual) memory addresses known by a process must remain valid, the operating system cannot perform "heap compaction" on its own. However, some runtimes like .Net do it.
If you are using C or C++, all you can hope for by default is that malloc() will be able to reuse previously deallocated chunks. But if your usage pattern is "allocate a lot of small objects then deallocate half of them at random," the memory utilization will probably not decrease much from the peak.
If a process initially has a number of pages allocated to it in the heap
A process will not initially have pages allocates in a heap.
is there some sort of optimization that the OS does to consolidate the data into one page so that the other pages can be freed
The operating system has no knowledge of user heaps. It allocates pages to the process. What that process does with those pages is up to it (i.e., use them for a heap, stack, code, etc.).
A process's heap manager can consolidate freed chunks of memory. When this occurs, it is normally done to fight heap fragmentation. However, I have never seen a heap manager on a paging system that unmaps pages once they are mapped by the operating system.
The heap of a process never has holes on it. The heap is part of the data segment allocated to a process, that grows dynamically upwards to the top of the stack segment, basically with the use of the sbrk(2) system call (that fixes a new size to the data segment) so the heap is a continuous segment (at least in terms of virtual address space) of allocated pages. malloc(3) never returns the heap space (or part of it) to the system. See malloc(3) for info about this. While there are memory allocators that allow a process to have several heaps (by means of allocating new memory segments, by use of the mmap(2) system call) the segments allocated by a memory allocator are commonly never returned back to the system.
What happens is that the memory allocator reuses the heap space allocated with sbrk(2) and mmap(2) and manages memory for being reused, but it is never returned back to the system.
But don't fear, as this is handled in a good and profitable way by the system, anyway.
That should not affect the overall system management, except from the fact that it consumes virtual address space, and probably page contents will end in the swap device if you don't use them until the process references them again and makes the system to reload them from the swap device(s). If your process doesn't reuse the holes it creates in the heap, the most probable destination is for the system to move them to the swap device and continue reusing it for other processes.
At this moment, I don't know if the system optimices swap allocation by not swapping out zeroed pages, as it does, for example, with text segments of executables (they never go to a swap device, because their contents are already swapped off in the executable file ---this was the reason you couldn't erase in ancient unices a program executable, or the reason there's not need anymore to use the sticky bit in frequently used programs---) but I think it doesn't (and the reason is that it's most improbable the unused pages will be zeroed by the application)
Be warned only in the case you have a 15Gb single process' heap use in your system and 90% of heap use is not in use most of the time. But think better in optimising the allocation resources because a process that consumes 15Gb of heap while most of the time 90%+ is unused, seems to be a poor design. If you have no other chance, simply provide enough swap space to your system to afford that.
I thought a little about virtual memory management, and came to the result that there can be two types of memory fragmentation. The first happens on the physical memory side where pages can not be freed because there are some bytes of it used. Mostly the last bytes will be freed sooner or later and then the physical memory page will become free again and is unmapped.
But what happens to the pointer (virtual address) returned by malloc. Let's assume a 32-bit system. The program "randomly" allocates and frees memory but there is never used more than some MByte. Let's assume further that the program will never free the memory in the order it is allocated. So the "top of heap" pointer can never be decreased as the free will never occur at the end of the heap. I assume that malloc has to map the memory always to the end of the heap memory space. This means the pointer value will increase with every call.
Earlier or later the returned pointer will reach the highest possible address (e.g. 0xffffffff) and it becomes impossible to further add memory while the system has enough free pages available as most pages have been freed. It is just a matter of the highest possible pointer value.
To solve this an algorithm would be needed that maintains unmapped address spaces and let them grow as more memory is beeing freed at the beginning or the end of the space. Is there an algorithm like this implemented by malloc?
I assume that malloc has to map the memory always to the end of the heap memory space.
This assumption is actually incorrect. Some implementations may keep multiple pools that different sizes of blocks are allocated from. (For instance, one common approach is a slab allocator, which keeps a separate pool for each size of block that the allocator will return.)
In any case, yes — all meaningful implementations of malloc() will track memory that has been freed and will reuse it when possible.
I had a short look at the slab allocator. This seems to be more related to memory page management used inside kernel. My question is related to the user space and the fact that whenever memory is allocated it needs to get an address in address space of the calling process's heap. What happens to this address space when it is limited as given in a 32-bit system.
It is clear that the system does not loose the memory at all. What I mean is that there is no address space left to get an address where the memory can be mapped while all memory at lower addresses has been freed and unmapped already.
By reading "understanding linux network internals" and "understanding linux kernel" the two books as well as other references, I am quite confused and need some clarifications about the "memory cache" and "memory pool" techniques.
1) Are they the same or different techniques?
2) If not the same, what makes the difference, or the distinct goals?
3) Also, how does the Slab Allocator come in?
Regarding the slab allocator:
So imagine memory is flat that is you have a block of 4 gigs contiguous memory. Then one of your programs reqeuests a 256 bytes of memory so what the memory allocator has to do is choose a suitable block of 256 bytes from this 4 gigs. So now you your memory looks something like
<============256bytes=======================>
(each = is a contiguous block of memory). Some time passes and a lot of programs operating with the memory require more 256 blocks or more or less so in the end your memory might look like:
<==256==256=256=86=68=121===>
so it gets fragmented and then there is no trace of your beautiful 4gig block of memory - this is fragmentation. Now, what the slab allocator would do is keep track of allocated objects and once they are not used anymore it will say that the memory is free when in fact it will be retained in some sort of List (You might wanna read about FreeLists).
So now imagine that the first program relinquish the 256 bytes allocated and then a new would like to have 256 bytes so instead of allocating a new chunk of the main memory it might re-use the lastly freed 256 bytes without having to go through the burden of searching the physical memory for appropriate contiguous block of space. This is how you essentially implement the memory cache. This is done so that memory fragmentation is reduced overall because you might end up in situation where memory is so fragmented that it is unusable and the memory-manager has to do some magic to get you block of appropriate size. Where as using a slab allocator pro-actively combats (but doesn't eliminate) the problem.
Linux memory allocator A.K.A slab allocator maintains the frequently used list/pool of memory objects of similar or approximate size. slab is giving extra flexibility to programmer to create their own pool of frequently used memory objects of same size and label it as programmer want,allocate, deallocate and finally destroy it.This cache is known to your driver and private to it.But there is a problem, during memory pressure there are high chances of allocation failures which could be not acceptable in some drivers, then what to do better always reserve some memory handy so that we never feel the memory crunch, since kmem cache is more generic pool mechanism we need some one who can always maintain minimum required memory and that's our buddy memory pool .
Lookaside Caches - The cache manager in the Linux kernel is sometimes called the slab allocator. You might end up allocating many objects of the same size over and over so by using this mechanism you just can allocate many objects in the same size and then use them later, without the need to allocate many objects over and over.
Memory Pool is just a form of lookaside cache that tries to always keep a list of memory around for use in emergencies, so when the memory pool is created, the allocation functions (slab allocators) create a pool of preallocated objects so you can acquire them when you need.
What is a memory heap ?
Presumably you mean heap from a memory allocation point of view, not from a data structure point of view (the term has multiple meanings).
A very simple explanation is that the heap is the portion of memory where dynamically allocated memory resides (i.e. memory allocated via malloc). Memory allocated from the heap will remain allocated until one of the following occurs:
The memory is free'd
The program terminates
If all references to allocated memory are lost (e.g. you don't store a pointer to it anymore), you have what is called a memory leak. This is where the memory has still been allocated, but you have no easy way of accessing it anymore. Leaked memory cannot be reclaimed for future memory allocations, but when the program ends the memory will be free'd up by the operating system.
Contrast this with stack memory which is where local variables (those defined within a method) live. Memory allocated on the stack generally only lives until the function returns (there are some exceptions to this, e.g. static local variables).
You can find more information about the heap in this article.
A memory heap is a location in memory where memory may be allocated at random access. Unlike the stack where memory is allocated and released in a very defined order, individual data elements allocated on the heap are typically released in ways which is asynchronous from one another. Any such data element is freed when the program explicitly releases the corresponding pointer, and this may result in a fragmented heap. In opposition only data at the top (or the bottom, depending on the way the stack works) may be released, resulting in data element being freed in the reverse order they were allocated.
Heap is just an area where memory is allocated or deallocated without any order. This happens when one creates an object using the new operator or something similar. This is opposed to stack where memory is deallocated on the first in last out basis.
It's a chunk of memory allocated from the operating system by the memory manager in use by a process. Calls to malloc() et alia then take memory from this heap instead of having to deal with the operating system directly.
You probably mean heap memory, not memory heap.
Heap memory is essentially a large pool of memory (typically per process) from which the running program can request chunks. This is typically called dynamic allocation.
It is different from the Stack, where "automatic variables" are allocated. So, for example, when you define in a C function a pointer variable, enough space to hold a memory address is allocated on the stack. However, you will often need to dynamically allocate space (With malloc) on the heap and then provide the address where this memory chunk starts to the pointer.
A memory heap is a common structure for holding dynamically allocated memory.
See Dynamic_memory_allocation on wikipedia.
There are other structures, like pools, stacks and piles.
Memory organization is divided into two parts: heap memory and stack memory.
Heap memory is the main working memory, lowest address is the starting address.
In stack memory, the flow of data is driven by bottom to up approach. Then the memory Arch is named as stack.
every running process has its own private fake virtual memory provided by the OS.
the OS can map this to physical memory at any point as long as it is available otherwise it will map to disk and swap as needed.
this virtual memory is logically divided into segments for organizing different kinds of data.
the code segment holds the executable instructions.
the data segment holds static data such as global or static variables.
the stack holds local data that is automatically managed by called and returning functions.
all of these segments are fixed size even the stack its just the portion used can grow or shrink and is reclaimed as functions returned.
the only segment that is not preallocated at app startup and fixed size is the heap.
the app can request from the OS at runtime new memory to be allocated and the OS will reserve a part of your apps virtual space and commit that to physical memory as needed.
the OS will return a pointer to that newly allocated heap memory and that pointer holds the base or starting address of the new block. that pointer sits on the stack and when that stack space is reclaimed your pointer will be no longer in scope and therefore you have no means of access to that block of memory. and if you dont tell the OS you are done with it so it can reclaim it that is just zombie memory sitting there with no means of access and if your app keeps requesting memory while never giving it back it will crash when the system runs out of memory. so it is important to free or at least pass the pointer to another pointer external to the scope it was defined in so you can maintain an interface to that memory allocated in heap space. i would suggest looking into virtual memory further and understanding segments.