Hello everyone,
I am a newbie to Linux-Kernel and I'm presently referring to the book Understanding Linux Kernel.I read about the memory management where everything is given well about the paging and segmentation but my question is not answered yet.
If two different processes have same Linear Addresses then can they be different locations in Physical Addresses .Because since each CPU has only one Global Page Directory which is mapped again to the physical addresses by observing the 32 bits Linear Address.But how does two processes can have memory upto 4GB .Please explain.
Yes, two different processes can both be using the same linear pointer, but it can dereference to two different locations in physical memory. That is because each process has its own page tables, and when switching from one process to another, the CPU's page table register is also switched to point to the page tables of the new process.
Have you cloned your own local copy of the Linux source code yet? If not, go and do it right now. You'll need to refer to it as you read your book.
Cloned now? Good. Go to the cloned working directory and open up arch/x86/include/asm/mm_context.h. Go down to line 51, you'll find static inline void switch_mm. This is the function which switches the CPU from the virtual memory space of one process to another. (I'm assuming you are most interested in x86.) Look down now to line 64: load_cr3(next->pgd). That's where the magic happens: the page tables are switched and now the CPU will interpret all pointers using the new process' page tables.
Related
This user asked why two Linux processes of the same file could not share the same text segment, to which the responder replied that modern Operating Systems sandbox programs by default and shared libraries are the only entities not duplicated by the OS (not including storage memory).
While I was researching earlier, however, I found(1) other(2) questions(3) that talked about how the operating system is trying to balance optimization with process-memory protection, and so through a combination of virtual memory addressing and page table lookups, as well as optimizations like copy-on-write, two programs could be sharing the same .TEXT segment, although they wouldn't be aware of it. Am I correct in saying this?
Excerpt: So the real thing is that text section could be shared by mapping different virtual pages to the same physical pages (called frames).
The reason I ask is because we're working on a molecular dynamics simulator that runs in a shared memory-model and we're trying to determine whether there is just one processor tasked with the burden of holding the program code, or if every processor requires their own local copy. (There are 6,000 processors.)
The thing that confuses me as well is that the OpenMP Memory Model is "shared", but each thread has "private access to each variable" (slide 5/19), which sounds a lot like the shared library [so/dll] model, and I'm wondering if that's how shared resources operate in general.
Additional Source - This question was asking about shared memory, but in terms of whether it was a specific location that all processes have access to. In his diagram, I want to know whether those three text segments can be simplified to one text segment and n-pointers, essentially.
It looks like as misunderstanding here, partially because of different usage of "shared" term. I.e., sharing of text segment doesn't have much in common with OpenMP memory model.
Speaking on techical side, it's easy to just look at state of text page sharing under Linux. There are useful /proc//smaps files, where for each mapping Linux kernel reports size in Shared_Clean state (see man 5 proc for more details).
For example, when I run single less /proc/self/smaps
for 1st mapping I see
Shared_Clean: 0 kB
Private_Clean: 108 kB
But after running 2nd less /proc/self/smaps
in another terminal at same machine I got
Shared_Clean: 108 kB
Private_Clean: 0 kB
So, we see the code for /usr/bin/less is actually shared between different processes.
To me, the answer about "sandboxes" from 1st reference is not valid. We can see that sharing is here. You can try yor particular workload and see what occurs for real codes.
I am trying to understand the nature of the operations carried out regarding the deallocation of physical memory when a process terminates.
Assumed that page table for the process is a multi-level tree structure thats implemented on Linux.
My current understanding is that the OS would need to deallocate each physical page frame that is mapped to whatever subset of the virtual addresses for which the Page Table entry (PTE) exists. This could happen by a traversal of the multi-level tree PT structure & for the PTEs that have their valid bit set, the physical frame descriptor corresponding to the PTE is added to the free list (which is used in the Buddy allocation process).
My question is: Is the traversal of the Page Table actually done for this? An alternative, faster way would be to maintain a linked list of the page frame descriptors allotted to a process, for each process & then traverse that linearly during process termination. Is this more generic & faster method instead followed?
I'm not sure that page gets physically deallocated at process ending.
My understanding is that MMU is managed by the kernel.
But each process has its own virtual address space, which the kernel changes:
for explicit syscalls changing it, ie. mmap(2)
at program start thru execve(2) (which can be thought of several virtual mmap-s as described by the segments of the ELF program executable file)
at process termination, as if each segment of the address space was virtually munmap-ed
And when a process terminates, it is its virtual address space (but not any physical RAM pages) which gets destroyed or deallocated!
So the page table (whatever definition you give to it) is probably managed inside the kernel by a few primitives like adding a segment to virtual address space and removing a segment from it. The virtual space is lazily managed, since the kernel uses copy on write techniques to make fork fast.
Don't forget that some pages (e.g. the code segment of shared libraries) are shared between processes and that every task of a multi-threaded process are sharing the same virtual address space.
BTW, the Linux kernel is free software, so you should study its source code (from http://kernel.org/). Look also on http://kernelnewbies.org ; memory management happens inside the mm/ subtree of the kernel source.
There are lots of resources. Look into linux-kernel-slides, slides#245 for a start, and there are many books and resources about the Linux kernel... Look for vm_area_struct, pgetable, etc...
Im currently trying to understand systems programming for Linux and have a hard time understanding how virtual to physical memory mappings work.
What I understand so far is that two processes P1 and P2 can make references to the same virtual adress for example 0xf11001. Now this memory adress is split up into two parts. 0xf11 is the page number and 0x001 is the offset within that page (assuming 4096 page size is used). To find the physical adress the MMU has hardware registeres that maps the pagenumber to a physical adress lets say 0xfff. The last stage is to combine 0xfff with 0x001 to find the physical 0xfff001 adress.
However this understanding makes no sens, the same virtual adresses would still point to the same physical location??? What step is I missing inorder for my understanding to be correct???
You're missing one (crucial) step here. In general, MMU doesn't have hardware registers with mappings, but instead one register (page table base pointer) which points to the physical memory address of the page table (with mappings) for the currently running process (which are unique to every process). On context switch, kernel with change this register's value, so for each running process different mapping will be performed.
Here's nice presentation on this topic: http://www.eecs.harvard.edu/~mdw/course/cs161/notes/vm.pdf
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
This is a noob question about computer science: How is ram allocated?
Fo example, I use Windows. Can I know which adresses are used by a program? How does Windows allocate memory? Contiguous or non contiguous?
Is it the same thing on Linux OSes ?
And, can I have access to the whole ram with a program? (I don't believe in that, but...)
Do you know any good lectures/documentation on this?
First, when you think you are allocating RAM, you really are not. This is confusing, I know, but it's really not complicated once you understand how it works. Keep reading.
RAM is allocated by the operating systems in units called "pages". Usually, this means contiguous regions of 4kiB, but other sizes are possible (to complicate things further, there exists support for "large pages" (usually on the order of 1-4MiB) on modern processors, and the operating system may have an allocation granularity different from the page size, for example Windows has a page size of 4kiB with a granularity of 64kiB).
Let's ignore those additional details and just think of "pages" that have one particular size (4KiB).
If you allocate and use areas that are greater than the system's page size, you will usually not have contiguous memory, but you will nevertheless see it as contiguous, since your program can only "think" in virtual addresses. In reality you may be using two (or more) pages that are not contiguous at all, but they appear to be. These virtual addresses are transparently translated to the actual addresses by the MMU.
Further, not all memory that you believe to have allocated necessarily exists in RAM at all times, and the same virtual address may correspond to entirely different pieces of RAM at different times (for example when a page is swapped out and is later swapped in again -- your program will see it at the same address, but in reality it is most likely in a different piece of RAM).
Virtual memory is a very powerful instrument. While one address in your program can only refer to [at most] one physical address (in a particular page) in RAM, one physical page of RAM can be mapped to several different addresses in your program, and even in several independent programs.
It is for example possible to create "circular" memory regions, and code from shared libraries is often loaded into one memory location, but used by many programs at the same time (and it will have different addresses in those different programs). Or, you can share memory between programs with that technique so when one program writes to some address, the value in the other program's memory location changes (because it is the exact same memory!).
On a high level, you ask your standard library for memory (e.g. malloc), and the standard library manages a pool of regions that it has reserved in a more or less unspecified way (there are many different allocator implementations, they all have in common that you can ask them for memory, and they give back an address -- this is where you think that you are allocating RAM when you are not).
When the allocator needs more memory, it asks the operating system to reserve another block. Under Linux, this might be sbrk and mmap, under Windows, this would for example be VirtualAlloc.
Generally, there are 3 things you can do with memory, and it generally works the same under Linux and Windows (and every other modern OS), although the API functions used are different, and there are a few more minor differences.
You can reserve it, this does more or less nothing, apart from logically dividing up your address space (only your process cares about that).
Next, you can commit it, this again doesn't do much, but it somewhat influences other processes. The system has a total limit of how much memory it can commit for all processes (physical RAM plus page file size), and it keeps track of that. Which means that memory you commit counts against the same limit that another process could commit. Otherwise, again, not much happens.
Last, you can access memory. This, finally, has a noticeable effect. Upon first accessing a page, a fault occurs (because the page does not exist at all!), and the operating system either fetches some data from a file (if the page belongs to a mapping) or it clears some page (possibly after first saving it to disk). The OS then adjusts the structures in the virtual memory system so you see this physical page of RAM at the address you accessed.
From your point of view, none of that is visible. It just works as if by magic.
It is possible to inspect processes for what areas in their address space are used, and it is possible (but kind of meaningless) to translate this to physical addresses. Note that the same program run at different times might store e.g. one particular variable at a different address. Under Windows, you can for example use the VMMap tool to inspect process memory allocation.
You can only use all RAM if you write your own operating system, since there is always a little memory which the OS reserves that user processes cannot use.
Otherwise you can in principle use [almost] all memory. However, whether or not you can directly use that much depends on whether your process is 32 or 64 bits. Computers nowadays typically have more RAM than you can address with 32 bits, so either you need to use address windowing extensions or your process must be 64 bits. Also, even given an amount of RAM that is in principle addressable using 32 bits, some address space factors (e.g. fragentation, kernel reserve) may prevent you from directly using all memory.
I'm studying process execution on Linux 2.6.32 on a 64-bit box. While studying the outputs of /proc/$PID/maps, I observed one thing:
$ cat /proc/2203/maps | head -1
00400000-004d9000 r-xp 00000000 08:02 1050631 /bin/bash
$ cat /proc/27032/maps | head -1
00400000-00404000 r-xp 00000000 08:02 771580 /sbin/getty
It seems that the maps file for all the programs shows that the executable code for each program is loaded in a block of memory beginning at 0x00400000.
I understand that these are virtual addresses. However, I don't get how these addresses can be the same for multiple concurrently running processes. What is the reason behind using a common start address for loading all processes, and how does the OS distinguish between the virtual load point of one process from another?
Edit:
From my understanding of address space virtualization using paging, I thought part of the virtual address was used to look up the physical address of a memory block (a frame) by using it to index one or more page tables. Consider this case. The address looks 32-bit (this is another thing that baffles me -- why are the program addresses 32-bit, but the addresses of the loaded libraries are 64-bit?). Breaking the address into ten, ten, and twelve bits corresponding to the page directory entry, the page table entry, and the page offset respectively, shouldn't 0x00400000 always mean "page directory entry 1, page table entry 0, offset 0", no matter what program performs the address translation?
One way I can see how this can be done is if the OS modified the page directory entry #1 to point to the page table corresponding to the program each time a task switch is performed. If that's the case, it sounds like a lot of added complexity -- given that program code is position-independent, won't it be easier to just load the program at an arbitrary virtual address and just go from there?
The answer is that each process has its own page tables. They are switched when processes are switched.
More information at http://www.informit.com/articles/article.aspx?p=101760&seqNum=3.
The kernel switches the page tables when a context switch happens. On operating systems where the kernel is mapped into every process, the kernel pages can remain. On the other hand, operating systems (32bit) which provide 4GiB to user-processes have to do a context switch when going into the kernel (a syscall) as well.
While virtual addressing doesn't require different processes to have different page tables, (the dependency goes the other way), I can't think of any current operating systems that don't give each process its own page tables.
Q: I understand that these are virtual addresses.
A: Good...
Q: However, I don't get how these addresses can be the same for
multiple concurrently running processes.
A: I thought you just said you understood "virtual addresses" ;)?
Q: What is the reason behind using a common start address for loading
all processes?
A: Remember, it's a virtual address - not a physical address. Why not have some standard start address?
And remember - you don't want to make the start address "0" - there are a lot of specific virtual addresses (especially those under 640K) that a process might wish to map as though it were a physical address.
Here's a good article that touches on a few of these issues. Including "e_entry":
How main() is executed on Linux