I'm currently learning Rust's lifetime and ownership. I encountered codes like this(minimized):
fn main() {
let mut text = Text::new();
println!("text: {:?}", text);
let mut byte = [1_u8];
let txt = text.get_text(byte);
println!("byte: {:?}", byte);
println!("txt: {:?}", txt);
byte[0] = 7_u8;
println!("byte: {:?}", byte);
println!("txt: {:?}", txt);
}
#[derive(Debug)]
struct Text<'a> {
temp: [u8; 1],
data: &'a [u8],
}
impl<'a> Text<'a> {
fn new() -> Self {
Text {
temp: [0_u8],
data: &[0],
}
}
fn get_lifetime(&self) -> &'a Self {
unsafe {
// &*(self as *const Self)
std::mem::transmute(self)
}
}
fn get_text(&mut self, byte: [u8; 1]) -> Text<'a> {
self.temp = byte;
let res = Text {
temp: [0_u8],
data: &self.get_lifetime().temp,
};
res
}
}
the output is
text: Text { temp: [0], data: [0] }
byte: [1]
txt: Text { temp: [0], data: [1] }
byte: [7]
txt: Text { temp: [0], data: [1] }
It seems that the slice data inside variable text point to some place independent of byte. I think maybe the byte is copied into get_text. But is slice data always refer to the copied one? How does Rust determine when to drop the copied byte?
Further more, I'm confused by the use of get_lifetime. Can anyone further explain the usage, and whether it is safe to use?
It seems that the slice data inside variable text point to some place independent of byte.
The value of data is initially set to &[0]. A slice literal is 'static and the reference is valid for the entire duration of the program.
After you call get_text, the value of byte is copied into temp. Then data is made into a reference to that field.
This is not safe. In the code you've actually shown, there is no problem. The problem is how this struct might be used later.
Consider this example:
fn main() {
let mut text = Text::new();
let mut byte = [1_u8];
let txt = text.get_text(byte);
println!("txt: {:?}", txt); // txt: Text { temp: [0], data: [1] }
use_text(text);
}
// Don't inline, so it's guaranteed that text is physically moved in memory
#[inline(never)]
fn use_text<'a>(text: Text<'a>) {
println!("txt: {:?}", text); // Undefined Behaviour
}
In this example text is moved into the use_text function's call frame. That means it is stored in a new location in memory. But data will still point to to the old place in memory where the temp field used to be stored.
This illustrates one of the big dangers of using unsafe. The consequences may be observed far away from where you used that keyword. This example was sound until text was moved, but after that point, it is Undefined Behaviour.
Related
This question already has answers here:
Rust: Is there an opposite for split_at_mut (i.e. join_mut)?
(2 answers)
Closed 8 months ago.
When writing a parser I ran into the problem that there are two string slices that come from the same origin string and are next to each other in memory. Of course it would be possible to simply copy the strings and merge them back into one, but that would require unnecessary computational resources. Is there a clean way to solve this in rust without unsafe code? For better illustration, here is an example of how I would like to solve it:
fn main() {
//This is the owned string.
//(Of course, this is also just a slice of a static string, but that makes no difference here).
let origin: &str = "Hello World";
//Substrings which borrow data from the original and should be adjacent in memory
let a: &str = &origin[0..5];
let b: &str = &origin[5..11];
//If the representation of a and b on the stack is:
// a: { ptr: PointerA, len: LenA }
// b: { ptr: PointerB, len: LenB }
//Then PointerA + LenA should be PointerB
//From this I conclude that there must be a way to combine these strings into c,
//which in turn would have this representation on the stack:
// c: { ptr: PointerA, len: LenA + LenB }
//The merge method doesn't actually exist, it's just a example of how I would imagen the api to look like.
let c = a.merge(b).unwrap();
assert!(c == origin)
}
Contrast this with the more inefficient current solution:
fn main() {
let origin: &str = "Hello World";
let a: &str = &origin[0..5];
let b: &str = &origin[5..11];
//Here both strings are simply copied to another location in the heap
//and need unnecessarily more memory, because the stored data exactly matches the data in origin
let c = a.to_owned() + b;
assert!(c == origin)
}
EDIT: This is a example of how i would implemented this with unsafe code, but i really don't know if it is actually safe
fn main() {
let origin: &str = "Hello World";
let a: &str = &origin[0..5];
let b: &str = &origin[5..11];
let c = merge(a, b).unwrap();
assert!(c == origin)
}
fn merge<'a>(one: &'a str, two: &'a str) -> Option<&'a str> {
unsafe {
let one: [usize; 2] = std::intrinsics::transmute(one);
let two: [usize; 2] = std::intrinsics::transmute(two);
if let Some(len) = one[1].checked_add(two[1]) {
if one[0] + one[1] == two[0] {
Some(std::intrinsics::transmute([one[0], len]))
} else {
None
}
} else {
None
}
}
}
You can chain the character within the splitted string:
fn main() {
//This is the owned string.
//(Of course, this is also just a slice of a static string, but that makes no difference here).
let origin: &str = "Hello World";
//Substrings which borrow data from the original and should be adjacent in memory
let a: &str = &origin[0..5];
let b: &str = &origin[5..11];
let c = a.chars().chain(b.chars());
assert_eq!(c.collect::<String>().as_str(), origin);
}
Playground
But notice that for most operation requiring &str, you would have to create a new string anyway.
So it bring us to unsafe life, and as pointed by #chayimfriedman it is UB:
use std::{slice, str};
fn main() {
//This is the owned string.
//(Of course, this is also just a slice of a static string, but that makes no difference here).
let origin: &str = "Hello World";
//Substrings which borrow data from the original and should be adjacent in memory
let a: &str = &origin[0..5];
let b: &str = &origin[5..11];
let c: &str = merge(a, b).unwrap();
assert_eq!(c, origin);
}
fn merge<'a>(a: &'a str, b: &'a str) -> Result<&'a str, String> {
let a_len = a.len();
let a_ptr = a.as_ptr();
let b_ptr = b.as_ptr();
let b_len = b.len();
if a_ptr as usize + a_len != b_ptr as usize {
return Err("Strings are not alighned".to_string());
}
Ok(unsafe {
str::from_utf8_unchecked(slice::from_raw_parts(a_ptr as *const u8, a_len + b_len))
})
}
Playground
Note that eventually you could avoid the pointers casting and instead use ptr::addr, which as for rust 1.62 it is still on nightly.
Assuming you have a binary file example.bin and you want to read that file in units of f64, i.e. the first 8 bytes give a float, the next 8 bytes give a number, etc. (assuming you know endianess) How can this be done in Rust?
I know that one can use std::fs::read("example.bin") to get a Vec<u8> of the data, but then you have to do quite a bit of "gymnastics" to convert always 8 of the bytes to a f64, i.e.
fn eight_bytes_to_array(barry: &[u8]) -> &[u8; 8] {
barry.try_into().expect("slice with incorrect length")
}
let mut file_content = std::fs::read("example.bin").expect("Could not read file!");
let nr = eight_bytes_to_array(&file_content[0..8]);
let nr = f64::from_be_bytes(*nr_dp_per_spectrum);
I saw this post, but its from 2015 and a lot of changes have happend in Rust since then, so I was wondering if there is a better/faster way these days?
Example without proper error handling and checking for cases when file contains not divisible amount of bytes.
use std::fs::File;
use std::io::{BufReader, Read};
fn main() {
// Using BufReader because files in std is unbuffered by default
// And reading by 8 bytes is really bad idea.
let mut input = BufReader::new(
File::open("floats.bin")
.expect("Failed to open file")
);
let mut floats = Vec::new();
loop {
use std::io::ErrorKind;
// You may use 8 instead of `size_of` but size_of is less error-prone.
let mut buffer = [0u8; std::mem::size_of::<f64>()];
// Using read_exact because `read` may return less
// than 8 bytes even if there are bytes in the file.
// This, however, prevents us from handling cases
// when file size cannot be divided by 8.
let res = input.read_exact(&mut buffer);
match res {
// We detect if we read until the end.
// If there were some excess bytes after last read, they are lost.
Err(error) if error.kind() == ErrorKind::UnexpectedEof => break,
// Add more cases of errors you want to handle.
_ => {}
}
// You should do better error-handling probably.
// This simply panics.
res.expect("Unexpected error during read");
// Use `from_be_bytes` if numbers in file is big-endian
let f = f64::from_le_bytes(buffer);
floats.push(f);
}
}
I would create a generic iterator that returns f64 for flexibility and reusability.
struct F64Reader<R: io::BufRead> {
inner: R,
}
impl<R: io::BufRead> F64Reader<R> {
pub fn new(inner: R) -> Self {
Self{
inner
}
}
}
impl<R: io::BufRead> Iterator for F64Reader<R> {
type Item = f64;
fn next(&mut self) -> Option<Self::Item> {
let mut buff: [u8; 8] = [0;8];
self.inner.read_exact(&mut buff).ok()?;
Some(f64::from_be_bytes(buff))
}
}
This means if the file is large, you can loop through the values without storing it all in memory
let input = fs::File::open("example.bin")?;
for f in F64Reader::new(io::BufReader::new(input)) {
println!("{}", f)
}
Or if you want all the values you can collect them
let input = fs::File::open("example.bin")?;
let values : Vec<f64> = F64Reader::new(io::BufReader::new(input)).collect();
I need rust code to read lines of a file, and break them into an array of slices. The working code is
use std::io::{self, BufRead};
fn main() {
let stdin = io::stdin();
let mut f = stdin.lock();
let mut line : Vec<u8> = Vec::new();
loop {
line.clear();
let sz = f.read_until(b'\n', &mut line).unwrap();
if sz == 0 {break};
let body : Vec<&[u8]> = line.split(|ch| *ch == b'\t').collect();
DoStuff(body);
}
}
However, that code is slower than I'd like. The code I want to write is
use std::io::{self, BufRead};
fn main() {
let stdin = io::stdin();
let mut f = stdin.lock();
let mut line : Vec<u8> = Vec::new();
let mut body: Vec<&[u8]> = Vec::new();
loop {
line.clear();
let sz = f.read_until(b'\n', &mut line).unwrap();
if sz == 0 {break};
body.extend(&mut line.split(|ch| *ch == b'\t'));
DoStuff(body);
body.clear();
}
}
but that runs afoul of the borrow checker.
In general, I'd like a class containing a Vec<u8> and an associated Vec<&[u8]>, which is the basis of a lot of C++ code I'm trying to replace.
Is there any way I can accomplish this?
I realize that I could replace the slices with pairs of integers, but that seems clumsy.
No, I can't just use the items from the iterator as they come through -- I need random access to the individual column values. In the simplified case where I do use the iterator directly, I get a 3X speedup, which is why I suspect a significant speedup by replacing collect with extend.
Other comments on this code is also welcome.
Just for sake of completeness, and since you are coming from C++, a more Rusty way of writing the code would be
use std::io::{self, BufRead};
fn do_stuff(body: &[&str]) {}
fn main() {
for line in io::stdin().lock().lines() {
let line = line.unwrap();
let body = line.split('\t').collect::<Vec<_>>();
do_stuff(&body);
}
}
This uses .lines() from BufRead to get an iterator over \n-delimited lines from the input. It assumes that your input is actually valid UTF8, which in your code was not a requirement. If it is not UTF8, use .split(b'\n'), .split(b'\t') and &[&u8] instead.
Notice that this does allocate and subsequently free a new Vec via .collect() every time the loop executes. We are somewhat relying on the allocator's free-list to make this cheap. But it is correct in all cases.
The reason your second example does not compile (after fixing the DoStuff(&body) is this:
12 | line.clear();
| ^^^^^^^^^^^^ mutable borrow occurs here
...
15 | body.extend(&mut line.split(|ch| *ch == b'\t'));
| ---- ---- immutable borrow occurs here
| |
| immutable borrow later used here
The problem here is the loop: Line 12 line.clear() will execute after line 15 body.extend() from the second iteration onwards. But the compiler has figured out that body borrows from line (it contains references to the fields inside line). The call to line.clear() mutably borrows line - all of line - and as far as the compiler is concerned is free to do anything it wants with the data it holds. This is an error because line.clear() could possibly mutate data that body has borrowed immutably. The compiler does not reason about the fact that .clear() obviously does not mutate the borrowed data, quite the opposite in fact, but the compiler's reasoning stops at the function signature.
I seems like the answer is
No, it's not possible to reuse the vector of slices.
The way to go is to make something like a slice, but with integer offsets rather than pointers. Code is attached, comments welcome.
Performance is currently 15% better than the C++, but the C++ is part of a larger system, and is probably doing some additional stuff.
/// pointers into a vector, simulating a slice without the ownership issues
#[derive(Debug, Clone)]
pub struct FakeSlice {
begin: u32,
end: u32,
}
/// A line of a text file, broken into columns.
/// Access to the `lines` and `parts` is allowed, but should seldom be necessary
/// `line` does not include the trailing newline
/// An empty line contains one empty column
///```
/// use std::io::BufRead;
/// let mut data = b"one\ttwo\tthree\n";
/// let mut dp = &data[..];
/// let mut line = cdx::TextLine::new();
/// let eof = line.read(&mut dp).unwrap();
/// assert_eq!(eof, false);
/// assert_eq!(line.strlen(), 13);
/// line.split(b'\t');
/// assert_eq!(line.len(), 3);
/// assert_eq!(line.get(1), b"two");
///```
#[derive(Debug, Clone)]
pub struct TextLine {
pub line: Vec<u8>,
pub parts: Vec<FakeSlice>,
}
impl TextLine {
/// make a new TextLine
pub fn new() -> TextLine {
TextLine {
line: Vec::new(),
parts: Vec::new(),
}
}
fn clear(&mut self) {
self.parts.clear();
self.line.clear();
}
/// How many column in the line
pub fn len(&self) -> usize {
self.parts.len()
}
/// How many bytes in the line
pub fn strlen(&self) -> usize {
self.line.len()
}
/// should always be false, but required by clippy
pub fn is_empty(&self) -> bool {
self.parts.is_empty()
}
/// Get one column. Return an empty column if index is too big.
pub fn get(&self, index: usize) -> &[u8] {
if index >= self.parts.len() {
&self.line[0..0]
} else {
&self.line[self.parts[index].begin as usize..self.parts[index].end as usize]
}
}
/// Read a new line from a file, should generally be followed by `split`
pub fn read<T: std::io::BufRead>(&mut self, f: &mut T) -> std::io::Result<bool> {
self.clear();
let sz = f.read_until(b'\n', &mut self.line)?;
if sz == 0 {
Ok(true)
} else {
if self.line.last() == Some(&b'\n') {
self.line.pop();
}
Ok(false)
}
}
/// split the line into columns
/// hypothetically you could split on one delimiter, do some work, then split on a different delimiter.
pub fn split(&mut self, delim: u8) {
self.parts.clear();
let mut begin: u32 = 0;
let mut end: u32 = 0;
#[allow(clippy::explicit_counter_loop)] // I need the counter to be u32
for ch in self.line.iter() {
if *ch == delim {
self.parts.push(FakeSlice { begin, end });
begin = end + 1;
}
end += 1;
}
self.parts.push(FakeSlice { begin, end });
}
}
So let's say I have a String, "Foo Bar" and I want to create a substring of "Bar" without allocating new memory.
So I moved the raw pointer of the original string to the start of the substring (in this case offsetting it by 4) and use the String::from_raw_parts() function to create the String.
So far I have the following code, which as far as I understand should do this just fine. I just don't understand why this does not work.
use std::mem;
fn main() {
let s = String::from("Foo Bar");
let ptr = s.as_ptr();
mem::forget(s);
unsafe {
// no error when using ptr.add(0)
let txt = String::from_raw_parts(ptr.add(4) as *mut _, 3, 3);
println!("{:?}", txt); // This even prints "Bar" but crashes afterwards
println!("prints because 'txt' is still in scope");
}
println!("won't print because 'txt' was dropped",)
}
I get the following error on Windows:
error: process didn't exit successfully: `target\debug\main.exe` (exit code: 0xc0000374, STATUS_HEAP_CORRUPTION)
And these on Linux (cargo run; cargo run --release):
munmap_chunk(): invalid pointer
free(): invalid pointer
I think it has something to do with the destructor of String, because as long as txt is in scope the program runs just fine.
Another thing to notice is that when I use ptr.add(0) instead of ptr.add(4) it runs without an error.
Creating a slice didn't give me any problems on the other Hand. Dropping that worked just fine.
let t = slice::from_raw_parts(ptr.add(4), 3);
In the end I want to split an owned String in place into multiple owned Strings without allocating new memory.
Any help is appreciated.
The reason for the errors is the way that the allocator works. It is Undefined Behaviour to ask the allocator to free a pointer that it didn't give you in the first place. In this case, the allocator allocated 7 bytes for s and returned a pointer to the first one. However, when txt is dropped, it tells the allocator to deallocate a pointer to byte 4, which it has never seen before. This is why there is no issue when you add(0) instead of add(4).
Using unsafe correctly is hard, and you should avoid it where possible.
Part of the purpose of the &str type is to allow portions of an owned string to be shared, so I would strongly encourage you to use those if you can.
If the reason you can't just use &str on its own is because you aren't able to track the lifetimes back to the original String, then there are still some solutions, with different trade-offs:
Leak the memory, so it's effectively static:
let mut s = String::from("Foo Bar");
let s = Box::leak(s.into_boxed_str());
let txt: &'static str = &s[4..];
let s: &'static str = &s[..4];
Obviously, you can only do this a few times in your application, or else you are going to use up too much memory that you can't get back.
Use reference-counting to make sure that the original String stays around long enough for all of the slices to remain valid. Here is a sketch solution:
use std::{fmt, ops::Deref, rc::Rc};
struct RcStr {
rc: Rc<String>,
start: usize,
len: usize,
}
impl RcStr {
fn from_rc_string(rc: Rc<String>, start: usize, len: usize) -> Self {
RcStr { rc, start, len }
}
fn as_str(&self) -> &str {
&self.rc[self.start..self.start + self.len]
}
}
impl Deref for RcStr {
type Target = str;
fn deref(&self) -> &str {
self.as_str()
}
}
impl fmt::Display for RcStr {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
fmt::Display::fmt(self.as_str(), f)
}
}
impl fmt::Debug for RcStr {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
fmt::Debug::fmt(self.as_str(), f)
}
}
fn main() {
let s = Rc::new(String::from("Foo Bar"));
let txt = RcStr::from_rc_string(Rc::clone(&s), 4, 3);
let s = RcStr::from_rc_string(Rc::clone(&s), 0, 4);
println!("{:?}", txt); // "Bar"
println!("{:?}", s); // "Foo "
}
I can't figure it out why my local var line does not live long enough. You can see bellow my code. It work on the Rust's playground.
I may have an idea of the issue: I use a structure (load is a function of this structure). As I want to store the result of the line in a member of my struct, it could be the issue. But I don't see what should I do to resolve this problem.
pub struct Config<'a> {
file: &'a str,
params: HashMap<&'a str, &'a str>
}
impl<'a> Config<'a> {
pub fn new(file: &str) -> Config {
Config { file: file, params: HashMap::new() }
}
pub fn load(&mut self) -> () {
let f = match fs::File::open(self.file) {
Ok(e) => e,
Err(e) => {
println!("Failed to load {}, {}", self.file, e);
return;
}
};
let mut reader = io::BufReader::new(f);
let mut buffer = String::new();
loop {
let result = reader.read_line(&mut buffer);
if result.is_ok() && result.ok().unwrap() > 0 {
let line: Vec<String> = buffer.split("=").map(String::from).collect();
let key = line[0].trim();
let value = line[1].trim();
self.params.insert(key, value);
}
buffer.clear();
}
}
...
}
And I get this error:
src/conf.rs:33:27: 33:31 error: `line` does not live long enough
src/conf.rs:33 let key = line[0].trim();
^~~~
src/conf.rs:16:34: 41:6 note: reference must be valid for the lifetime 'a as defined on the block at 16:33...
src/conf.rs:16 pub fn load(&mut self) -> () {
src/conf.rs:17 let f = match fs::File::open(self.file) {
src/conf.rs:18 Ok(e) => e,
src/conf.rs:19 Err(e) => {
src/conf.rs:20 println!("Failed to load {}, {}", self.file, e);
src/conf.rs:21 return;
...
src/conf.rs:31:87: 37:14 note: ...but borrowed value is only valid for the block suffix following statement 0 at 31:86
src/conf.rs:31 let line: Vec<String> = buffer.split("=").map(String::from).collect();
src/conf.rs:32
src/conf.rs:33 let key = line[0].trim();
src/conf.rs:34 let value = line[1].trim();
src/conf.rs:35
src/conf.rs:36 self.params.insert(key, value);
...
There are three steps in realizing why this does not work.
let line: Vec<String> = buffer.split("=").map(String::from).collect();
let key = line[0].trim();
let value = line[1].trim();
self.params.insert(key, value);
line is a Vec of Strings, meaning the vector owns the strings its containing. An effect of this is that when the vector is freed from memory, the elements, the strings, are also freed.
If we look at string::trim here, we see that it takes and returns a &str. In other words, the function does not allocate anything, or transfer ownership - the string it returns is simply a slice of the original string. So if we were to free the original string, the trimmed string would not have valid data.
The signature of HashMap::insert is fn insert(&mut self, k: K, v: V) -> Option<V>. The function moves both the key and the value, because these needs to be valid for as long as they may be in the hashmap. We would like to give the hashmap the two strings. However, both key and value are just references to strings which is owned by the vector - we are just borrowing them - so we can't give them away.
The solution is simple: copy the strings after they have been split.
let line: Vec<String> = buffer.split("=").map(String::from).collect();
let key = line[0].trim().to_string();
let value = line[1].trim().to_string();
self.params.insert(key, value);
This will allocate two new strings, and copy the trimmed slices into the new strings.
We could have moved the string out of the vector(ie. with Vec::remove), if we didn't trim the strings afterwards; I was unable to find a easy way of trimming a string without allocating a new one.
In addition, as malbarbo mentions, we can avoid the extra allocation that is done with map(String::from), and the creation of the vector with collect(), by simply omitting them.
In this case you have to use String instead of &str. See this to understand the difference.
You can also eliminate the creation of the intermediate vector and use the iterator return by split direct
pub struct Config<'a> {
file: &'a str,
params: HashMap<String, String>
}
...
let mut line = buffer.split("=");
let key = line.next().unwrap().trim().to_string();
let value = line.next().unwrap().trim().to_string();