In an approach to detect the heap corruption I am trying to implement a hash table to keep some information about the malloced memory. This is being done inside glibc itself. When we malloc(), we put the information like address and size in the hash table and when we free(), we deallocate the corresponding hash table entry, again in glibc's free() itself.
To allocate memory for the hash table i have mmap'd some memory (refrained from using malloc for this, as the chances of process induced heap corruption can corrupt my hash table also).
The issue is that there is no limit to the number of mallocs that a process can ask for, this requires my hash table to be extensible. Since my hash table works on array indices, memory used for hash table need to be contiguous so that using an index we can easily reach the bucket or record. Now, when hash table uses all the memory, i need to do a 'mmap' again in such a way that this memory starts where the previous ended. man page of mmap says that we can provide an address to mmap, that will act as a hint to kernel to map virtual memory at that address. To the hash table, it will look like a contguouis chunk of memory. I would like to ask for you suggestions how reliable is this approach and what are the potential pitfalls using this.
If this is Linux, you can use mremap.
If you write your hash table to be based on offsets and not absolute pointers, you can pass the MREMAP_MAYMOVE flag and never have to worry about allocation failures. (Well, until you exhaust your virtual memory, anyway.)
how reliable is this approach
MAP_FIXED is very reliable: IF the memory you are asking for is available, the kernel will give it to you.
what are the potential pitfalls
The obvious one: something else may have gotten mmaped into the region you want to extend into, and you lose.
If you are doing this for 64-bit process, you can mmap e.g. 1TB of memory as your initial hash table allocation. So long as you don't actually access it, this mmap is effectively free (of cost), assuming you are doing MA_ANON mapping.
BTW, I hope you realize that you are re-inventing a bicycle here, as many existing solutions (such as tcmalloc and jemalloc) will already provide debugging facilities that are likely to be better that whatever you concoct yourself.
Related
I've read that sbrk is a deprecated call and one should prefer mmap with MAP_ANONYMOUS flag. I need one continous (logical) memory block that can grow. However, mmap treats first parameter as a hint, so it can make gaps, which is unacceptable in my case. I tried to use MAP_FIXED flag (which as documentation states is not recommended) and I can get continuos memory, but after mapping several pages I get strange behaviour of my program: system functions like printf and clock_gettime begin to fail. I guess the first mmap which I call without MAP_FIXED returns page that has some mapped pages after it, which contain system data. So what is the right way to use mmap instead of sbrk?
With Linux you can use mmap with MAP_NORESERVE (and possibly PROT_NONE) to claim a large chunk of address space without actually allocating any memory. You map the largest area you could possibly want (and can get), and then remap bits of it with MAP_FIXED to actually allocate memory as needed.
I've read that sbrk is a deprecated call
Don't believe everything you read, especially if the source is not authoritative.
I need one continous (logical) memory block that can grow.
In that case, mmap is not for you, unless you are willing to declare the maximum size to which that block can grow.
I tried to use MAP_FIXED flag (which as documentation states is not recommended) and I can get continuos memory, but after mapping several pages I get strange behaviour of my program
With MMAP_FIXED you have to be very careful: the system will happily map over whatever (if anything) was there before, including libc data and code.
We need to store a large 1GB of contiguous bytes in memory for long periods of time (weeks to months), and are trying to choose a Vector/Array library. I had two concerns that I can't find the answer to.
Vector.Unboxed seems to store the underlying bytes on the heap, which can be moved around at will by the GC.... Periodically moving 1GB of data would be something I would like to avoid.
Vector.Storable solves this problem by storing the underlying bytes in the c heap. But everything I've read seems to indicate that this is really only to be used for communicating with other languages (primarily c). Is there some reason that I should avoid using Vector.Storable for internal Haskell usage.
I'm open to a third option if it makes sense!
My first thought was the mmap package, which allows you to "memory-map" a file into memory, using the virtual memory system to manage paging. I don't know if this is appropriate for your use case (in particular, I don't know if you're loading or computing this 1GB of data), but it may be worth looking at.
In particular, I think this prevents the GC moving the data around (since it's not on the Haskell heap, it's managed by the OS virtual memory subsystem). On the other hand, this interface handles only raw bytes; you couldn't have, say, an array of Customer objects or something.
I'm writing a memory allocation routine, and it's currently running smoothly. I get my memory from the OS with mmap() in 4096-byte pages. When I start my memory allocator I allocate 1gig of virtual address space with mmap(), and then as allocations are made I divide it up into hunks according to the specifics of my allocation algorithm.
I feel safe allocating as much as a 1gig of memory on a whim because I know mmap() doesn't actually put pages into physical memory until I actually write to them.
Now, the program using my allocator might have a spurt where it needs a lot of memory, and in this case the OS would have to eventually put a whole 1gig worth of pages into physical RAM. The trouble is that the program might then go into a dormant period where it frees most of that 1gig and then uses only minimal amounts of memory. Yet, all I really do inside of my allocator's MyFree() function is to flip a few bits of bookkeeping data which mark the previously used gig as free, but I know this doesn't cause the OS remove those pages from physical memory.
I can't use something like munmap() to fix this problem, because the nature of the allocation algorithm is such that it requires a continuous region of memory without any holes in it. Basically I need a way to tell the OS "Listen, you can take these pages out of physical memory and clear them to 0, but please remap them on the fly when I need them again, as if they were freshly mmap()'d"
What would be the best way to go about this?
Actually, after writing this all up I just realized that I can probably do an munmap() followed immediately by a fresh mmap(). Would that be the correct way to go about? I get the sense that there's probably some more efficient way to do this.
You are looking for madvise(addr, length, MADV_DONTNEED). From the manpage:
MADV_DONTNEED: Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.
Note especially the language about how subsequent accesses will succeed but revert to zero-fill-on-demand (for mappings without an underlying file).
Your thinking-out-loud alternative of an munmap followed immediately by another mmap will also work but risks kernel-side inefficiencies because it is no longer tracking the allocation a single contiguous region; if there are many such unmap-and-remap events the kernelside data structures might wind up being quite bloated.
By the way, with this kind of allocator it's very important that you use MAP_NORESERVE for the initial allocation, and then touch each page as you allocate it, and trap any resulting SIGSEGV and fail the allocation. (And you'll need to document that your allocator installs a handler for SIGSEGV.) If you don't do this your application will not work on systems that have disabled memory overcommit. See the mmap manpage for more detail.
Is there any way to ensure key material gets securely erased from the memory after the program exits? Being able to erase it manually and keep the program running would be even better. As Haskell uses automated garbage collection (which may not happen at all if there is loads of free memory?), I assume that the second task is impossible. Could something that serves the purpose be implemented using FFI?
GHC can return memory to the OS when it is no longer needed, so merely blanking the memory on exit won't achieve your goal. Garbage collection is a complicated business, but there is in general no way to ensure that old copies of your secure data are not returned to the OS memory pool.
However the OS will blank your memory before allocating it to another process. If you don't trust the OS to keep your memory secure then you have a much bigger problem.
I'm not sure what you mean by "unreliable"; Haskell GC is reliable, but the program has comparatively little visibility of what is happening.
However if you are concerned merely with a cryptographic key rather than a big, complicated data structure then life gets a bit better. You can use a Foreign Pointer to point to a memory location for your key, and then make blanking that bit of memory into a part of your finaliser. You can even write a bit of code that allocates a block of memory, mlocks it, and then hands off foreign pointers to key-sized chunks of that memory on request, with finalisers that wipe the key. That would probably do what you want.
The point of a ForeignPtr is that it is guaranteed not to be moved or re-interpreted by the GC.
On a 32 bit Linux system, a process can access up to 4 GB of virtual address space; however, processes seem to be conservative in varying degrees in reserving any of that. So a program that uses malloc will occasionally grow its data segment by a syscall sbrk/brk. Even those pages aren't claimed in physical memory yet. What I don't fully understand is why we need to sbrk in the first place, why not just give me 4 GB address space avoiding any sbrk call, as until we touch/claim those blocks, it is essentially a free operation right?
What happens if you memory-map a file (a very common thing to do under Linux)? It has to go somewhere in the address space, so there must be some means of defining "used" and "not used" parts.
Shared memory (which is really just mapping a file without an actual file) is the same. It has to go somewhere, and the OS must be sure it can place it without overwriting something.
Also, it is preferrable to maintain locality of reference for obvious (and less obvious) efficiency reasons. If you were allowed to just write to and read from any location in your address space, you can bet that some people would do just that.
There's a couple of reasons that come to mind:
You'd no longer get segfaults when accessing unmapped memory
The Translation lookaside buffer (TLB) would be larger, possibly requiring more time to set it up
You'd have to unmap some of that memory anyway if you load in a new shared library or mmap() something