I have started to learn about Virtual Address Space (VAS) and I have few questions:
How much of VAS is created for each process depending on the architecture (32-bit and 64-bit)?
Is VAS for each process created on hard disk? If so, what happens if there is not enough space?
What is the difference between VAS and Virtual Memory (VM)?
Virtual address versus physical address
During the execution of your program, the variables (integers, arrays, strings, etc.) are stored somewhere in the main memory of your computer (RAM). Some programming languages (like C or C++) allow you to obtain the memory address at which a given variable is stored (with the & operator), and to manipulate that address (add to it, subtract from it, print it, etc.).
Here is a C program that prints the memory address of a variable:
#include <stdio.h>
int main(void) {
int variable = 1234;
void *address = &variable;
printf("Memory address of variable: %p\n", address);
return 0;
}
Output:
Memory address of variable: 0x7ffc9e9662a4
Now, if you compile and execute this program on a typical desktop computer, with a typical operating system (like GNU/Linux or Windows), the memory address that is printed by this program is not the hardware address at which the data 1234 is actually located in the memory chip. This may be surprising, but there is a level of indirection between the addresses used by your program and the hardware addresses.
Virtual address space on 64-bit computers
On a 64-bit computer, a memory address manipulated by your program is an integer between 0 and 18446744073709551615 inclusive. Such an address is called a virtual memory address. The range of those addresses is called the virtual address space of the process. You can ask the operating system to map a range of virtual memory addresses to the physical memory of your computer, so that when you try to read or write bytes at thoses addresses, your program doesn't crash for accessing unmapped virtual memory addresses.
Typically, on x86-64 computers, only 248 virtual memory addresses can be successfully mapped to physical memory, because 256 TiB of usable virtual address space is considered sufficient. In the future, processor manufacturers may raise or remove this limit if there is a need for it.
Virtual address space on 32-bit computers
On 32-bit computers, there are 232 virtual memory addresses. On those computers, a memory address manipulated by your program is an integer between 0 and 4294967295 inclusive.
On x86 32-bit computers, there is usually no restriction on the range of virtual memory addresses that can be mapped to physical memory addresses.
Mapping a range of virtual memory addresses
On GNU/Linux, you can request a mapping by calling the function mmap(). On Windows, you can request a mapping by calling the function VirtualAlloc(). Those functions take the size of the mapping as argument, and return the first virtual address that is now backed by actual physical memory. Those functions can fail to create a new mapping if the physical memory is already completely used by other processes. And again, if you try to access (read or write) the content of a virtual memory address that is outside an area mapped by mmap() or VirtualAlloc(), the operating system will terminate your program (by sending a segmentation fault signal).
On GNU/Linux, a process can examine the mappings created in its virtual address space just by reading the file /proc/self/maps. You can learn a lot by reading the output of the command cat /proc/self/maps.
Hard disk drive
On a typical computer, the main memory is a semiconductor memory, and the hard disk drive is only a secondary storage device.
On a typical operating system, a range of virtual memory addresses can only be mapped to the main memory (which is usually a semiconductor memory device). Such a range cannot be directly mapped to a secondary storage device (usually a hard disk drive) without using the main memory as intermediary.
On an n-bit machine, the VAS is 2n bytes large. So, on a 32-bit machine, the VAS 232 = 4 GiB large.
Virtual memory is not created on disk. In fact, the existence of a disk is not needed for implementing virtual memory. Most implementations of virtual memory are paged. So, when a 4 GiB VAS is created, only the pages that are needed are mapped into that VAS. For example, suppose a process only uses 16 pages of memory on a 32-bit system with 4k-sized pages. Despite having a 4 GiB VAS, only 16 * 4k = 216 bytes of memory are mapped into the VAS. The rest of the memory is unmapped. If the CPU tries to access this unmapped memory, a segmentation fault will occur. If a process wants to map memory at this address, then (in a POSIX-complaint OS) it can request the mapping from the OS using mmap(2). This will make a lot more sense once you learn about page tables.
Virtual memory is a concept. A virtual address space is an entity that stems from the concept of virtual memory. These terms go hand in hand, but refer to different things.
I will list a couple of caveats.
Caveat 1.1
I am not not aware of any 64-bit processor that truly supports a 64-bit VAS. The addresses themselves are 64 bits wide, but a certain number of upper bits are ignored. AMD's first implementation of x86_64 only supported 48-bit addresses. The upper 16 bits of an address were effectively ignored. In such a system, the addresses are 64 bits wide, but the real size of the VAS is limited to 248 bytes. Subsequent architectures supported 56-bit addresses.
Caveat 1.2
If a processor supports PAE, then the VAS on an n-bit machine may be larger than 2n bytes. This is how 32-bit processors can support VASs larger than 4 GiB.
Caveat 2.1
Not really a caveat, but this is related to your question. You asked what happens when there isn't enough space on disk to create a VAS. As I mentioned in the main answer, the VAS is not created on disk. However, any computer only has a finite amount of physical memory. What happens when a process requests a page be mapped, but there is no physical memory available? There are there several ways to handle this:
Swapping is done by temporarily moving a page that is mapped in virtual memory to disk. The entire contents of the page are copied to disk. Then, the process that requested the page has the physical page mapped into their memory. Eventually, the old page may be requested. If this occurs, then the OS copies the page from disk and remaps it into the corresponding VAS. This is what Linux and most modern operating systems do.
The process is simply told there is no memory available, for example, through an error number like ENOMEM.
The process is blocked until memory is available. I haven't seen this in practice.
Swapping implies the use of a disk, but virtual memory does not imply the use of swapping, hence a disk is not necessary for virtual memory.
Virtual Address Space - wikipedia
When a new application on a 32-bit OS is executed, the process has a 4 GiB VAS: each one of the memory addresses (from 0 to 232 − 1) in that space can have a single byte as a value. Initially, none of them have values.
For n-bit OS, these n-address lines allow address space upto 2n addresses, i.e., 0 to 2n - 1. This would mean 16 EiB for 64-bit OS. (Though in actual implementations, less space is used as this much space is unnecessary.)
CPU Cache - wikipedia
Most general purpose CPUs implement some form of virtual memory. To summarize, either each program running on the machine sees its own simplified address space, which contains code and data for that program only, or all programs run in a common virtual address space. A program executes by calculating, comparing, reading and writing to addresses of its virtual address space, rather than addresses of physical address space, making programs simpler and thus easier to write.
For example, in C++, program memory is divided in stack, heap, data, code. I'm not sure if analogy is correct (may be), but it somewhat presents an insight if you're aware.
Virtual memory - wikipedia
In computing, virtual memory is a memory management technique that provides an "idealized abstraction of the storage resources that are actually available on a given machine"3 which "creates the illusion to users of a very large (main) memory".[4]
The computer's operating system, using a combination of hardware and software, maps memory addresses used by a program, called virtual addresses, into physical addresses in computer memory. Main storage, as seen by a process or task, appears as a contiguous address space or collection of contiguous segments. The operating system manages virtual address spaces and the assignment of real memory to virtual memory.
Address translation hardware in the CPU, often referred to as a memory management unit (MMU), automatically translates virtual addresses to physical addresses. Software within the operating system may extend these capabilities to provide a virtual address space that can exceed the capacity of real memory and thus reference more memory than is physically present in the computer.
If you know about computer architecture (which I'm sure you do from the question), it'd be clarified by now.
Still, for anyone in general, I'm giving a bit of explanation.
Assume addresses as pointers in C++. If you don't know C++, closest analogy would be array/list indices in any language. Now the addresses point to the memory locations, just like pointers point to the variable. The actual data is stored in the variable. To get the variable data using pointer/index, you provide address location from where the data is to be extracted. Now in physical memory, there won't be a thing like a variable. There is memory and it's location address through which it is accessed.
The real memory is physical memory, which is the hard disks. It is accessed with physical addresses, which would be unique for each byte.
Accessing physical memory directly with physical addresses would be cumbersome. Thus the addresses are simplified by the OS to virtual addresses. These addresses may or may not be unique (these aren't physical addresses, remember). Thus, multiple virtual addresses may point to same location.
The virtual memory is not actually existent, rather it's just a concept of physical memory simplified using virtual addresses to give the user an illusionous space say where next memory location is stored at next address (virtual address to be precise).
Since multiple virtual addresses can be mapped, by using MMU, to same physical address, and thus to point to same phyical memory location, the virtual memory size can be made to exceed the physical memory size (virtually). But effectively, the memory size would still be same as physical.
Thus, to access a memory data, Virtual Addresses are specified by user/program to OS, which are converted to Physical Addresses by memory management unit (mmu) and then applied to the address lines of the computer architecture (electronics spotted!!), which yields the data at the corresponding physical location. And this concept is called Virtual Memory.
-Himanshu
I'm currently programming a Linux Kernel driver, which needs to tell a FPGA a base address in RAM to write to.
The memory is allocated in the kernel driver with dma_alloc_coherent.
This will generate a 32 bit physical address and a kernel virtual address, the physical address is being passed to the FPGA.
The FPGA is a Cyclone V with embedded ARM Cortex-A9, on which an embedded Linux with the driver is running.
The problem now is, that the FPGA fabric only generates a 27 bit wide bus to address the sdram, while the physical address, that is being generated by the dma call has 32 bits, e.g. th physical address has been 0x2f220000, which exceeds 27 bit span.
I want to know, if it is okay to mask the most significant 5 bits and tell the FPGA the address 0x7220000 and still have the correct behaviour (In doc it is stated, that the physical address shall be casted to buswidth, that would mean masking, because I can't use 27 bit in processor).
Also is it okay to access the DMA memory with a simple memcpy command, that copies from the kernel virtual address to a buffer?
Thanks in advance.
The answer truly depends on the physical memory layout of your device. If the address bus of the FPGA complements the missing bits so that the actual address resolves to the correct memory, then masking is, probably, okay. If not, then it is possible that the memory that the Linux kernel returned to you is simply in accessible to the FPGA. If that's the case, you will have to find a way to ask Linux to only give you buffers from memory that is accessible.
The linear address beyond 896MB correspond to High memory region ZONE_HIGHMEM.
So the page allocator functions will not work on this region, since they give the linear address of directly mapped page frames in ZONE_NORMAL and ZONE_DMA.
I am confused about these lines specified in Undertanding linux Kernel:
What do they mean when they say "In 64 bit hardware platforms ZONE_HIGHMEM is always empty."
What does this highlighted statement mean: "The allocation of high-memory page frames is done only through alloc_pages() function. These functions do not return linear address since they do not exist. Instead the functions return linear address of the page descriptor of the first allocated page frame. These linear addresses always exist, because all page descriptors are allocated in low memory once and forever during kernel initialization."
What are these Page descriptors and does the 896MB already have all page descriptors of entire RAM.
The x86-32 kernel needs high memory to access more than 1G of physical memory, as it is impossible to permanently map more than 2^{32} addresses within a 32-bit address space and the kernel/user split is 1G/3G.
The x86-64 kernel has no such limitation, as the amount of physically-addressable memory (currently 256T) fits within its 64-bit address space and thus may always be permanently mapped.
High memory is a hack. Ideally you don't need it. Indeed, the point of x86-64 is to be able to directly address all the memory you could possibly want. Taken
from https://www.quora.com/Linux-Kernel/What-is-the-difference-between-high-memory-and-normal-memory
I think page descriptor means struct page. And considering the sizeof struct page. Yes all of them can be stored in ZONE_NORMAL
I am allocating a block of memory with kmalloc in a device driver in Linux. The address that I get is 0xffff880000180000. I am using an IvyBridge processor with a 46-bit physical address space. That means that the CPU doesn't have more than 46 addressing pins, so it can't access any address above 0x00003fffffffffff. The address I'm getting is obviously greater than that, as it has bit 47 set.
Assuming that kmalloc returns a physical pointer (that is, a pointer where the virtual, linear and physical addresses are the same), how can I be getting such an address, if it's not accessible by the CPU?
This question is related to "Why am I getting a high address when I use kmalloc with GFP_DMA in Linux?". However, it is not identical. This question is about the physical addresses kmalloc returns in general, whereas the linked question is about use of GFP_DMA and its relationship to the returned address.
It's clearly not a physical address but a linear or virtual address, and is in 'canonical form' where the value of bit 47 is 'sign extended' up to bit 63.
Processes on Linux 3.0 on x86_64 architecture have a 64-bit virtual address space.
It is clear that 0 is guaranteed to be an invalid memory address [see definition below] in this address space, as this is used to indicate a NULL pointer.
What other 64-bit numbers (if any) are guaranteed never to be valid memory addresses, and why?
For example, can 1 ever be a valid address? What about 2^64-1?
Definition: What do you mean "guaranteed to be an invalid memory address" ?
void deref_and_assign(uint64_t i)
{
char* p = (char*) i;
*p = 42;
}
For the purposes of this question a guaranteed invalid memory reference means that the function deref_and_assign will always raise a SIGSEGV.
On x86/64 if page translation enabled and the memory at virtual address 0 isn't accessible (because of the way physical memory is mapped into the virtual address space), 1 ... 4095 won't be accessible either because all these 4096 addresses correspond to a single page of memory and it can only be available or unavailable as a whole. It is a good idea to never map memory at virtual address 0. Not mapping it will help to catch many NULL pointer dereferences. The CPU here will generate a page fault (aka #PF) on unmapped locations or locations requiring higher privilege than the currently executing code.
In 64-bit mode the CPU may implement fewer (48+) than 64 virtual address bits and 64-bit addresses must contain either all zeroes or all ones in the bits that aren't implemented (the value, 0 or 1, must be the same as the value of the most significant implemented address bit, all of which can be interpreted as address sign-extension). Such addresses are called canonical. If you try to read or write memory using a non-canonical address, you'll get a general protection fault (AKA #GP).
So, depending on the OS (effectively, on its memory layout) and actual CPU you may come up with ranges of "invalid" memory addresses. If you try to read/write the kernel's memory from a user mode application, you'll get #PF. If you try to read/write unmapped memory (e.g. at address 0 through 4095), you'll get #PF. If you try to read/write at a non-canonical address, you'll get a #GP.
Is that the kind of thing you're looking for?
You checked that a Linux process cannot mmap with MAP_FIXED a segment starting at (void*)0.
So for practical purposes you can safely assume that the very first page 0 - 0xfff is never mmap-ed (the 4Kb size of pages is processor and system dependent, but it is very often 4kb). Then you can assume that dereferencing pointers (from inside a Linux application) inside this first page gives SIGSEGV
Likewise for the last page ending at 0xffffffffffffffff (ie 2^64-1)