malloc returns a null pointer whenever I'm trying to allocate more than 4gb. sizeof(void*) is 8 and the machine has 256gb of physical memory of which most is unused so finding a contiguous block should be no problem. What could be the possible reasons for this?
Probably due to a limitation in the malloc function. Maybe you'll find your answer in This previous post.
If you want an example of an obvious limitation, it will not give you more than the size of your RAM in terms of space.
Related
I have a doubt related to buddy system allocator in Linux kernel.
Consider this example:
Assume there is 128KB RAM on which buddy system is allocating.
If i allocate 32KB using kmalloc, my understanding is,
128KB is split into two 64KB in step 1 and the first 64KB is split into two 32KB.
Then the first 32KB is allocated to me.
Clearly 96KB of contiguous memory is still available.
Now if i allocate 90KB at this stage, what happens?
will kmalloc fail or buddy (or even slab) allocator handle this and successfully allocate requested memory?
After first 32KB, you could not allocate anything bigger than 64KB. kmalloc() returns nullptr in such a case.
This is true for a buddy allocator. A slab allocator works differently, but this is off topic.
I am trying to allocate a 5-page-800x600 frame buffer(roughly 5mb). But during DRAM memory map initialization, dma_alloc_coherent() only returns a zero pointer or does not allocate the buffer.
It used to work with just allocating a 4-page frame buffer(4mb). I have already tried setting CONSISTENT_DMA_SIZE to 8mb, 10mb, and 12mb. But this doesn't seem to have any effect.
Is there any other setting I'm over looking?
thanks alot,
nazekimi
P.S.
working on a Linux 2.6.10 Mobilinux kernel
kernel does power-of-2 allocation. so 5MB means 8MB allocation. so probably you need to increase CONSISTENT_DMA_SIZE even more.
Thx,
Jeffrey
I am trying to write a custom memory manager and right now I am allocating a global array of bytes like:
char g_retail_memory[512*MB];
When i then look in the resource monitor of Windows 7 it reports that around 512 MBs of virtual memory has been allocated. can someone please explain why I am not getting physical memory? If i use malloc() instead i get physical memory. Am I doing something wrong? In that case, is there a correct way of obtaining physical memory under visual c++?
See http://en.wikipedia.org/wiki/Virtual_memory
All physical memory you allocate will also be virtual memory, because VM is the way the system presents memory to applications in modern operating systems. The reason you may not see some allocations appear as physical could be that the memory has never been used, so the OS didn't actually need to find physical memory for it, but will when/if the memory is first used.
Note that virtual memory's backing physical memory can later "go away" if the OS needs it for another application. This is sometimes referred to as "swapping" or "paging."
The Memory that i'm trying to allocate is not huge or any thing. i just cant allocate
1.5 to 1.7 gigabyte of contagious memory. From what i know, windows gives you 2 gigabytes of virtual space to use in your application. so, a call like malloc(1500*1024*1024) is not totally crazy. i tried malloc ,new[], VirtualAlloc all didn't work.
is there something i'm missing here?
someone told me it has something to do with physical memory, i totally dismissed that because why was virtual space ,address translation tables and TLBs invented if i'm allocating physical memory.
if i'm allocating a 1.5 gig on a machine with 256 megabytes of ram and i try to access shouldn't the system be thrashing but working?
Different versions of Windows have different memory restrictions. If you're using a 32-bit version, you may need to use the 4GB tuning techniques to allocate more than 2GB.
If you are running a 32 bit version of windows, you have a max of 2Gb of virtual space. Your compiled programs and the C/C++ runtime libraries each use up some part of it, along with preallocated code and data segments. If you run a 32 bit windows, you have less memory space than you think. I'll agree that 1.5 Gb doesn't sound unreasonable, but then you would think that MS products weren't unreasonable, too, right?
Try samller pieces as a sanity check, (e.g., 1Gb); I suspect that will succeed. And try big allocations on a 64 bit system, where there isn't any practical upper limit.
Are you using ODBC? The ODBC dlls in 32-bit windows seem to insert themselves at an awkward place in the virtual address space, causing large allocations like yours to fail. A workaround is to configure your app to delay load the ODBC dlls, and then make sure you allocate your big chunk before you call anything that uses ODBC.
In linux kernel, mem_map is the array which holds all "struct page" descriptors. Those pages includes the 128MiB memory in lowmem for dynamically mapping highmem.
Since the lowmem size is 1GiB, so the mem_map array has only 1GiB/4KiB＝256KiB entries. If each entry size is 32 byte, then the mem_map memory size = 8MiB. But if we could use mem_map to map all 4GiB physical memory(if we have so much physical memory available on x86-32), then the mem_map array would occupy 32MiB, that is not a lot of kernel memory(or am i wrong?).
So my question is: why do we need to use that 128MiB in low for indirect highmem mapping in the first place? Or put another way, why not to map all those max 4GiB physical memory(if available) in the kernel space directly?
Note: if my understanding of the kernel source above is wrong, please correct. Thanks!
Look Here: http://www.xml.com/ldd/chapter/book/ch13.html
Kernel low memory is the 'real' memory map, addressed with 32-bit pointers on x86.
Kernel high memory is the 'virtual' memory map, addressed with virtual structures on x86.
You don't want to map it all into the kernel address space, because you can't always address all of it, and you need most of your memory for virtual memory segments (virtual, page-mapped process space.)
At least, that's how I read it. Wow, that's a complicated question you asked.
To throw more confusion, chapter 13 talks about some PCI devices not being able to address the 32-bit space, which was the genesis of my previous comment:
On x86, some kernel memory usage is limited to the first Gigabyte of memory bacause of DMA addressing concerns. I'm not 100% familiar with the topic, but there's a comapatibility mode for DMA on the PCI bus. That may be what you are looking at.
3.6 GB is not the ceiling when using physical address extension, which is commonly needed on most modern x86 boards, especially with memory hotplug.
Or put another way, why not to map all those max 4GiB physical
memory(if available) in the kernel space directly?
One reason is userspace: every usespace process have its own virtual address space. Suppose you have 4Gb of RAM on x86. So if we suggest that kernel owns 1Gb of memory (~800 directly mapped + ~200 vmalloc) all other ~3Gb should be dynamically distributed between processes spinning in user space. So how can you map your 4Gbs directly when you have a several address spaces?
why do we need zone_highmem on x86?
The reason is the same. Kernel reserves only ~800Mb for low mem. All other memory will be allocated and connected with particular virtual address only on demand. For example if you will execute a binary a new virtual address space will be created and some pages will be allocated for storing your binary code and data (heap ,stack ...). So the key attribute of high mem is to serve dynamic memory allocation requests, you never know in advance what will be triggered by userspace...