While attempting to test Is it allowed to access memory that spans the zero boundary in x86? in user-space on Linux, I wrote a 32-bit test program that tries to map the low and high pages of 32-bit virtual address space.
After echo 0 | sudo tee /proc/sys/vm/mmap_min_addr, I can map the zero page, but I don't know why I can't map -4096, i.e. (void*)0xfffff000, the highest page. Why does mmap2((void*)-4096) return -ENOMEM?
strace ./a.out
execve("./a.out", ["./a.out"], 0x7ffe08827c10 /* 65 vars */) = 0
strace: [ Process PID=1407 runs in 32 bit mode. ]
....
mmap2(0xfffff000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0
Also, what check is rejecting it in linux/mm/mmap.c, and why is it designed that way? Is this part of making sure that creating a pointer to one-past-an-object doesn't wrap around and break pointer comparisons, because ISO C and C++ allow creating a pointer to one-past-the-end, but otherwise not outside of objects.
I'm running under a 64-bit kernel (4.12.8-2-ARCH on Arch Linux), so 32-bit user space has the entire 4GiB available. (Unlike 64-bit code on a 64-bit kernel, or with a 32-bit kernel where the 2:2 or 3:1 user/kernel split would make the high page a kernel address.)
I haven't tried from a minimal static executable (no CRT startup or libc, just asm) because I don't think that would make a difference. None of the CRT startup system calls look suspicious.
While stopped at a breakpoint, I checked /proc/PID/maps. The top page isn't already in use. The stack includes the 2nd highest page, but the top page is unmapped.
00000000-00001000 rw-p 00000000 00:00 0 ### the mmap(0) result
08048000-08049000 r-xp 00000000 00:15 3120510 /home/peter/src/SO/a.out
08049000-0804a000 r--p 00000000 00:15 3120510 /home/peter/src/SO/a.out
0804a000-0804b000 rw-p 00001000 00:15 3120510 /home/peter/src/SO/a.out
f7d81000-f7f3a000 r-xp 00000000 00:15 1511498 /usr/lib32/libc-2.25.so
f7f3a000-f7f3c000 r--p 001b8000 00:15 1511498 /usr/lib32/libc-2.25.so
f7f3c000-f7f3d000 rw-p 001ba000 00:15 1511498 /usr/lib32/libc-2.25.so
f7f3d000-f7f40000 rw-p 00000000 00:00 0
f7f7c000-f7f7e000 rw-p 00000000 00:00 0
f7f7e000-f7f81000 r--p 00000000 00:00 0 [vvar]
f7f81000-f7f83000 r-xp 00000000 00:00 0 [vdso]
f7f83000-f7fa6000 r-xp 00000000 00:15 1511499 /usr/lib32/ld-2.25.so
f7fa6000-f7fa7000 r--p 00022000 00:15 1511499 /usr/lib32/ld-2.25.so
f7fa7000-f7fa8000 rw-p 00023000 00:15 1511499 /usr/lib32/ld-2.25.so
fffdd000-ffffe000 rw-p 00000000 00:00 0 [stack]
Are there VMA regions that don't show up in maps that still convince the kernel to reject the address? I looked at the occurrences of ENOMEM in linux/mm/mmapc., but it's a lot of code to read so maybe I missed something. Something that reserves some range of high addresses, or because it's next to the stack?
Making the system calls in the other order doesn't help (but PAGE_ALIGN and similar macros are written carefully to avoid wrapping around before masking, so that wasn't likely anyway.)
Full source, compiled with gcc -O3 -fno-pie -no-pie -m32 address-wrap.c:
#include <sys/mman.h>
//void *mmap(void *addr, size_t len, int prot, int flags,
// int fildes, off_t off);
int main(void) {
volatile unsigned *high =
mmap((void*)-4096L, 4096, PROT_READ | PROT_WRITE,
MAP_FIXED|MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0);
volatile unsigned *zeropage =
mmap((void*)0, 4096, PROT_READ | PROT_WRITE,
MAP_FIXED|MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0);
return (high == MAP_FAILED) ? 2 : *high;
}
(I left out the part that tried to deref (int*)-2 because it just segfaults when mmap fails.)
The mmap function eventually calls either do_mmap or do_brk_flags which do the actual work of satisfying the memory allocation request. These functions in turn call get_unmapped_area. It is in that function that the checks are made to ensure that memory cannot be allocated beyond the user address space limit, which is defined by TASK_SIZE. I quote from the code:
* There are a few constraints that determine this:
*
* On Intel CPUs, if a SYSCALL instruction is at the highest canonical
* address, then that syscall will enter the kernel with a
* non-canonical return address, and SYSRET will explode dangerously.
* We avoid this particular problem by preventing anything executable
* from being mapped at the maximum canonical address.
*
* On AMD CPUs in the Ryzen family, there's a nasty bug in which the
* CPUs malfunction if they execute code from the highest canonical page.
* They'll speculate right off the end of the canonical space, and
* bad things happen. This is worked around in the same way as the
* Intel problem.
#define TASK_SIZE_MAX ((1UL << __VIRTUAL_MASK_SHIFT) - PAGE_SIZE)
#define IA32_PAGE_OFFSET ((current->personality & ADDR_LIMIT_3GB) ? \
0xc0000000 : 0xFFFFe000)
#define TASK_SIZE (test_thread_flag(TIF_ADDR32) ? \
IA32_PAGE_OFFSET : TASK_SIZE_MAX)
On processors with 48-bit virtual address spaces, __VIRTUAL_MASK_SHIFT is 47.
Note that TASK_SIZE is specified depending on whether the current process is 32-bit on 32-bit, 32-bit on 64-bit, 64-bit on 64-bit. For 32-bit processes, two pages are reserved; one for the vsyscall page and the other used as a guard page. Essentially, the vsyscall page cannot be unmapped and so the highest address of the user address space is effectively 0xFFFFe000. For 64-bit processes, one guard page is reserved. These pages are only reserved on 64-bit Intel and AMD processors because only on these processors the SYSCALL mechanism is used.
Here is the check that is performed in get_unmapped_area:
if (addr > TASK_SIZE - len)
return -ENOMEM;
If I look at the memory mapping for a 64-bit process on Linux (x86_64) I see that the a.out is mapped in fairly low memory:
$ cat /proc/1160/maps
00400000-004dd000 r-xp 00000000 103:03 536876177 /usr/bin/bash
006dc000-006dd000 r--p 000dc000 103:03 536876177 /usr/bin/bash
006dd000-006e6000 rw-p 000dd000 103:03 536876177 /usr/bin/bash
006e6000-006ec000 rw-p 00000000 00:00 0
00e07000-00e6a000 rw-p 00000000 00:00 0 [heap]
7fbeac11c000-7fbeac128000 r-xp 00000000 103:03 1074688839 /usr/lib64/libnss_files-2.17.so
7fbeac128000-7fbeac327000 ---p 0000c000 103:03 1074688839 /usr/lib64/libnss_files-2.17.so
I'd like to map a 2G memory region in the very lowest portion of memory, but have to put this in the region after these a.out mappings, crossing into the second 2G region.
Is the a.out being mapped here part of the x86_64 ABI, or can this load address be moved to a different region, using one of:
runtime loader flags
linker flags when the executable is created
?
Yes. Building a Linux x86-64 application as a position-independent executable will cause both it and its heap to be mapped into high memory, right along with libc and other libraries. This should leave the space under 2GB free for your use. (However, note that the kernel will probably protect the first 64KB or so of memory from being mapped to protect it from certain exploits; look up vm.mmap_min_addr for information.)
To build your application as a position-independent executable, pass -pie -fPIE to the compiler.
Why does this command:
[jalal#galapagos-20] (42)$ dmesg | egrep '(s|h)d[a-z]'
Shows the following? Can you please show step by step?
ACPI: SSDT 00000000d8ffbbd8 03528 (v01 SaSsdt SaSsdt 00003000 INTL 20091112)
NMI watchdog enabled, takes one hw-pmu counter.
sd 0:0:0:0: [sda] 500118192 512-byte logical blocks: (256 GB/238 GiB)
sd 0:0:0:0: [sda] 4096-byte physical blocks
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2 sda3 sda4 sda5 sda6 sda7
sd 0:0:0:0: [sda] Attached SCSI disk
EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts:
dracut: Mounted root filesystem /dev/sda3
snd_hda_intel 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
snd_hda_intel 0000:00:03.0: irq 33 for MSI/MSI-X
snd_hda_intel 0000:00:03.0: setting latency timer to 64
snd_hda_intel 0000:00:1b.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
snd_hda_intel 0000:00:1b.0: irq 34 for MSI/MSI-X
snd_hda_intel 0000:00:1b.0: setting latency timer to 64
sound hdaudioC1D2: autoconfig for ALC3220: line_outs=1 (0x1b/0x0/0x0/0x0/0x0) type:line
sound hdaudioC1D2: speaker_outs=1 (0x14/0x0/0x0/0x0/0x0)
sound hdaudioC1D2: hp_outs=1 (0x15/0x0/0x0/0x0/0x0)
sound hdaudioC1D2: mono: mono_out=0x0
sound hdaudioC1D2: inputs:
sound hdaudioC1D2: Front Mic=0x1a
sound hdaudioC1D2: Rear Mic=0x18
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts:
EXT4-fs (sda7): mounted filesystem with ordered data mode. Opts:
EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts:
EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts:
EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts:
Adding 67108860k swap on /dev/sda2. Priority:-1 extents:1 across:67108860k SS
snd_hda_intel 0000:00:1b.0: IRQ timing workaround is activated for card #1. Suggest a bigger bdl_pos_adj.
The command dmesg dumps the kernel log buffer to stdout. This output consists of a bunch of log messages, each on a separate line.
That output is piped to the command egrep with the option (s|h)d[a-z]. egrep checks its stdin (the output of dmesg) one line at a time for a match against the regular expression (s|h)d[a-z]. This regular expression will match a series of characters where:
The first character is s or h
The second character is d
The third character is in the ASCII range (inclusive) between a and z (the lowercase letters a to z).
egrep prints to stdout any lines which match the supplied regular expression.
The egrep command is using the regular expression (s|h)d[a-z] for filtering lines containing the sequences hda to hdz and sda to sdz on the output of dmesg - traditional names for disk devices in Linux.
This regular expression could be switched to [sh]d[a-z], which has the same effect but you may find easier to understand.
There are several false positives if you are looking for disks, like the lines containing "hdaudio".
I am debugging one issue where a same program behaves differently on different Linux boxes (all 2.6 kernels). Basically on some linux box, mmap() of 16MB always succeeds, but on other linux box the same mmap() would fail with "ENOMEM".
I checked /proc//maps and saw the virtual memory map on different linux boxes are quite different. One thing is: the address range for heap:
linux box1: the mmap() would return address of 31162000-31164000
linux box2: the mmap() would return address of a9a67000-a9a69000
My question is: for a particular process, how is linux virtual memory arranged? What decides the actual address ranges? Why the mmap() would fail even I can still see some large unused virtual address ranges?
UPDATE: here is an example of address space where mmap() of 16MB would fail:
10024000-10f6b000 rwxp 10024000 00:00 0 [heap]
30000000-3000c000 rw-p 30000000 00:00 0
3000c000-3000d000 r--s 00000000 00:0c 5461 /dev/shm/mymap1
3000d000-3000e000 rw-s 00001000 00:0c 5461 /dev/shm/mymap2
3000e000-30014000 r--s 00000000 00:0c 5463 /dev/shm/mymap3
30014000-300e0000 r--s 00000000 00:0c 5465 /dev/shm/mymap4
300e0000-310e0000 r--s 00000000 00:0b 2554 /dev/mymap5
310e0000-31162000 rw-p 310e0000 00:00 0
31162000-31164000 rw-s 00000000 00:0c 3554306 /dev/shm/mymap6
31164000-312e4000 rw-s 00000000 00:0c 3554307 /dev/shm/mymap7
312e4000-32019000 rw-s 00000000 00:0c 3554309 /dev/shm/mymap8
7f837000-7f84c000 rw-p 7ffeb000 00:00 0 [stack]
in the above example: there are still big space between the last mymap8 and [stack]. But further mmap() of 16MB would fail. My question is: how does Linux decide mmap() base and the allowed range?
Thanks.
I want to use fw_printenv for get U-boot's env.
$cat /proc/mtd
mtd0: 00060000 00004000 "bootloader"
mtd1: 00200000 00004000 "kernel"
mtd2: 03c00000 00004000 "root"
and
$vi /etc/fw_env.config
# Configuration file for fw_(printenv/saveenv) utility.
# Up to two entries are valid, in this case the redundand
# environment sector is assumed present.
# MTD device name Device offset Env. size Flash sector size
/dev/mtd0 0x0000 0x60000 0x4000
then
$ fw_printenv
Warning: Bad CRC, using default environment
bootcmd=bootp; setenv bootargs root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}::off; bootm
bootdelay=5
baudrate=115200
would you tell me what's wrong?
thanks a lot. ^^
I got it.
# MTD device name Device offset Env. size Flash sector size
/dev/mtdblock0 0x40000 0x20000 0x4000