How to create and write to memory mapped files? - rust

Question

Editor's note: This code example is from a version of Rust prior to 1.0 and the code it uses does not exist in Rust 1.0. Some answers have been updated to answer the core question for newer versions of Rust.
I'm trying to create a memory mapped file using std::os::MemoryMap. The current approach looks as follows:
use std::os;
use std::ptr;
use std::old_io as io;
use std::os::unix::prelude::AsRawFd;
use std::os::MapOption;
let path = Path::new("test.mmap");
let f = match io::File::open_mode(&path, io::Open, io::ReadWrite) {
Ok(f) => f,
Err(err) => panic!("Could not open file: {}", err),
};
let mmap_opts = &[
MapOption::MapReadable,
MapOption::MapWritable,
MapOption::MapFd(f.as_raw_fd())
];
let mmap = match os::MemoryMap::new(1024*1024, mmap_opts) {
Ok(mmap) => {
println!("Successfully created the mmap: {}", mmap.len());
mmap
}
Err(err) => panic!("Could not read the mmap: {}", err),
};
unsafe {
let data = mmap.data();
if data.is_null() {
panic!("Could not access data from memory mapped file")
}
let src = "Hello!";
ptr::copy_memory(data, src.as_ptr(), src.as_bytes().len());
}
This program fails with
Process didn't exit successfully: `target/mmap` (status=4)
when calling ptr::copy_memory or any other operations on data.
What is the reason I cannot write (or read) the data from the MemoryMap?
What is the correct way to use MemoryMap in Rust?

Answer 1

The real answer is to use a crate that provides this functionality, ideally in a cross-platform manner.
use memmap; // 0.7.0
use std::{
fs::OpenOptions,
io::{Seek, SeekFrom, Write},
};
const SIZE: u64 = 1024 * 1024;
fn main() {
let src = "Hello!";
let mut f = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open("test.mmap")
.expect("Unable to open file");
// Allocate space in the file first
f.seek(SeekFrom::Start(SIZE)).unwrap();
f.write_all(&[0]).unwrap();
f.seek(SeekFrom::Start(0)).unwrap();
let mut data = unsafe {
memmap::MmapOptions::new()
.map_mut(&f)
.expect("Could not access data from memory mapped file")
};
data[..src.len()].copy_from_slice(src.as_bytes());
}
Note that this is still possible for this code to lead to undefined behavior. Since the slice is backed by a file, the contents of the file (and thus the slice) may change from outside of the Rust program, breaking the invariants that the unsafe block is supposed to hold. The programmer needs to ensure that the file doesn't change during the lifetime of the map. Unfortunately, the crate itself does not provide much assistance to prevent this from happening or even any documentation warning the user.
If you wish to use lower-level system calls, you are missing two main parts:
mmap doesn't allocate any space on its own, so you need to set some space in the file. Without this, I get Illegal instruction: 4 when running on macOS.
MemoryMap (was) private by default so you need to mark the mapping as public so that changes are written back to the file (I'm assuming you want the writes to be saved). Without this, the code runs, but the file is never changed.
Here's a version that works for me:
use libc; // 0.2.67
use std::{
fs::OpenOptions,
io::{Seek, SeekFrom, Write},
os::unix::prelude::AsRawFd,
ptr,
};
fn main() {
let src = "Hello!";
let size = 1024 * 1024;
let mut f = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open("test.mmap")
.expect("Unable to open file");
// Allocate space in the file first
f.seek(SeekFrom::Start(size as u64)).unwrap();
f.write_all(&[0]).unwrap();
f.seek(SeekFrom::Start(0)).unwrap();
// This refers to the `File` but doesn't use lifetimes to indicate
// that. This is very dangerous, and you need to be careful.
unsafe {
let data = libc::mmap(
/* addr: */ ptr::null_mut(),
/* len: */ size,
/* prot: */ libc::PROT_READ | libc::PROT_WRITE,
// Then make the mapping *public* so it is written back to the file
/* flags: */ libc::MAP_SHARED,
/* fd: */ f.as_raw_fd(),
/* offset: */ 0,
);
if data == libc::MAP_FAILED {
panic!("Could not access data from memory mapped file")
}
ptr::copy_nonoverlapping(src.as_ptr(), data as *mut u8, src.len());
}
}

Answer 2

Up to date version:
use std::ptr;
use std::fs;
use std::io::{Write, SeekFrom, Seek};
use std::os::unix::prelude::AsRawFd;
use mmap::{MemoryMap, MapOption};
// from crates.io
extern crate mmap;
extern crate libc;
fn main() {
let size: usize = 1024*1024;
let mut f = fs::OpenOptions::new().read(true)
.write(true)
.create(true)
.open("test.mmap")
.unwrap();
// Allocate space in the file first
f.seek(SeekFrom::Start(size as u64)).unwrap();
f.write_all(&[0]).unwrap();
f.seek(SeekFrom::Start(0)).unwrap();
let mmap_opts = &[
// Then make the mapping *public* so it is written back to the file
MapOption::MapNonStandardFlags(libc::consts::os::posix88::MAP_SHARED),
MapOption::MapReadable,
MapOption::MapWritable,
MapOption::MapFd(f.as_raw_fd()),
];
let mmap = MemoryMap::new(size, mmap_opts).unwrap();
let data = mmap.data();
if data.is_null() {
panic!("Could not access data from memory mapped file")
}
let src = "Hello!";
let src_data = src.as_bytes();
unsafe {
ptr::copy(src_data.as_ptr(), data, src_data.len());
}
}

Answer 3

2022-version:
use memmap2::Mmap;
use std::fs::{self};
use std::io::{Seek, SeekFrom, Write};
use std::ops::DerefMut;
pub fn memmap2() {
// How to write to a file using mmap
// First open the file with writing option
let mut file = fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.open("mmap_write_example2.txt")
.unwrap();
// Allocate space in the file for the data to be written,
// UTF8-encode string to get byte slice.
let data_to_write: &[u8] =
"Once upon a midnight dreary as I pondered weak and weary; äåößf\n".as_bytes();
let size: usize = data_to_write.len();
file.seek(SeekFrom::Start(size as u64 - 1)).unwrap();
file.write_all(&[0]).unwrap();
file.seek(SeekFrom::Start(0)).unwrap();
// Then write to the file
let mmap = unsafe { Mmap::map(&file).unwrap() };
let mut mut_mmap = mmap.make_mut().unwrap();
mut_mmap.deref_mut().write_all(data_to_write).unwrap();
}