I'm still learning Rust and have the following code.
use std::collections::BTreeMap;
#[derive(Debug)]
struct MyStruct {
a: String,
b: String,
}
fn main() {
let mut hash = BTreeMap::new();
let data = vec![
MyStruct {
a: "entry1".to_string(),
b: "entry1 body".to_string(),
},
MyStruct {
a: "entry2".to_string(),
b: "entry2 body".to_string(),
}
];
let re = regex::Regex::new(r#".(\d)"#).unwrap();
for item in &data {
for m in re.captures_iter(&item.b) {
hash.insert(&m[1].parse::<i32>().unwrap(), &item.a);
}
}
println!("{:#?}", hash);
}
It generates an error:
error[E0716]: temporary value dropped while borrowed
--> src\main.rs:26:26
|
26 | hash.insert(&m[1].parse::<i32>().unwrap(), &item.a);
| ---- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - temporary value is freed at the end of this statement
| | |
| | creates a temporary which is freed while still in use
| borrow later used here
|
= note: consider using a `let` binding to create a longer lived value
What's the correct way to fix it? I tried to put &m[1].parse::<i32>().unwrap() in a variable but of no avail.
The BTreeMap structure should either be the owner of the data and key inserted, or the data and key should have a 'static lifetime (same as HashMap and other collections). In this case, the key being used is i32, which has the Copy trait defined for it, so simply removing the & reference should pass the i32 value in as the key. For the data, you will either want to clone the string rather than & borrow, but you could also rewrite the loop to consume the data vector and pass the item.b string value in without the need to clone.
Related
I have a struct which maps ids to indices and vice versa.
struct IdMapping<'a> {
external_2_internal: HashMap<&'a str, usize>,
internal_2_external: HashMap<usize, String>,
}
impl<'a> IdMapping<'a> {
fn new() -> IdMapping<'a> {
IdMapping {
external_2_internal: HashMap::new(),
internal_2_external: HashMap::new(),
}
}
fn insert(&'a mut self, internal: usize, external: String) {
self.internal_2_external.insert(internal, external);
let mapped_external = self.internal_2_external.get(&internal).unwrap();
self.external_2_internal.insert(mapped_external, internal);
}
}
If I am using this structure the following way
fn map_ids<'a>(ids: Vec<String>) -> IdMapping<'a> {
let mut mapping = IdMapping::new();
for (i, id) in ids.iter().enumerate() {
mapping.insert(i, id.clone());
}
mapping
}
I receive the following compiler error:
error[E0499]: cannot borrow `mapping` as mutable more than once at a time
--> src/lib.rs:28:9
|
24 | fn map_ids<'a>(ids: Vec<String>) -> IdMapping<'a> {
| -- lifetime `'a` defined here
...
28 | mapping.insert(i, id.clone());
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `mapping` was mutably borrowed here in the previous iteration of the loop
...
31 | mapping
| ------- returning this value requires that `mapping` is borrowed for `'a`
Playground link
Why can't I mutably borrow the mapping for its insert method each iteration of the loop? How should I implement this use case?
The problem is due to the self reference in your struct.
Let's first look at whether this is theoretically sound (assuming we're writing unsafe code):
HashMap uses a flat array (quadratic probing), so objects in the HashMap aren't address stable under insertion of a new element. This means that an insertion into internal_2_external may move the existing Strings around in memory.
String stores its content in a separate heap allocation, the heap allocation remains at the same location. So even if the String is moved around, a &str referencing it will remain pointing to valid memory.
So this would actually work if implemented in unsafe code.
While it's logically sound to do those operations, the type system is unable to recognise that you can move a String while keeping borrowed ranges of it valid. This means that if you borrow a &str from a String, the type system will prevent you from doing any mutation operation on your String, including moving it. As such, you can't do any mutation operation on the hash map either, giving rise to your error.
I can see two safe ways to work around this:
Make external_2_internal keep its own copy of the strings, i.e. useHashMap<String, usize>.
Keep the strings in a separate Vec:
struct IdMapping {
strings: Vec<String>,
external_2_internal: HashMap<usize /* index */, usize>,
internal_2_external: HashMap<usize, usize /* index */>,
}
Alternatively you could work with unsafe code, if you don't want to change the layout of your struct.
Working code first:
use std::cell::{Ref, RefCell};
use std::rc::Rc;
struct ValueHolder {
value: i32
}
fn give_value(wrapped: &Rc<RefCell<ValueHolder>>) -> Ref<i32> {
Ref::map(
(**wrapped).borrow(),
|borrowed| { &(*borrowed).value },
)
}
fn main() {
println!("Our value: {}", *give_value(
&Rc::new(RefCell::new(ValueHolder { value: 1337 }))
));
}
The relevant part is the give_value function.
I want to return a Ref to something inside a Rc<RefCell<>>, in this case a value inside of a struct.
That works just fine, but since I just started to learn Rust, I wonder, how I could achieve the same thing without using Ref::map.
The "naive" approach:
fn give_value(wrapped: &Rc<RefCell<ValueHolder>>) -> &i32 {
&(*(**wrapped).borrow()).value
}
fails for obvious reasons:
error[E0515]: cannot return value referencing temporary value
--> src/bin/rust_example.rs:9:5
|
9 | &(*(**wrapped).borrow()).value
| ^^^--------------------^^^^^^^
| | |
| | temporary value created here
| returns a value referencing data owned by the current function
So my question is: How can I recreate what the Ref::map function does by myself?
You can't. RefCell requires that Ref be used whenever the value is used, so it knows when it should allow a borrow. When the Ref gets dropped, it signals to the RefCell that it can decrease the borrow count, and if the borrow count is 0, then a mutable borrow can happen. Being able to obtain a reference to the internal data that doesn't borrow the Ref (note that a reference you can return from a function cannot borrow the Ref otherwise you'd be referencing a temporary) would be problematic, because then the Ref could get dropped and the RefCell thinks that it can lend out a mutable borrow, but there's still an immutable borrow to the data. The source code for Ref::map is:
pub fn map<U: ?Sized, F>(orig: Ref<'b, T>, f: F) -> Ref<'b, U>
where
F: FnOnce(&T) -> &U,
{
Ref { value: f(orig.value), borrow: orig.borrow }
}
which uses private fields that you can't access. So, no, you cannot recreate Ref::map yourself.
I am trying to store into a HashMap the result of a parsing operation on a text file (parsed with nom). The result is comprised of a Vec buffer and some slices over that buffer. The goal is to store those together in a tuple or struct as a value in the hash map (with String key). But I can't work around the lifetime issues.
Context
The parsing itself takes an &[u8] and returns some data structure containing slices over that same input, e.g.:
struct Cmd<'a> {
pub name: &'a str
}
fn parse<'a>(input: &'a [u8]) -> Vec<Cmd<'a>> {
[...]
}
Now, because the parsing operates on slices without storage, I need to first store the input text in a Vec so that the output slices remain valid, so something like:
struct Entry<'a> {
pub input_data: Vec<u8>,
pub parsed_result: Vec<Cmd<'a>>
}
Then I would ideally store this Entry into a HashMap. This is were troubles arise. I tried two different approaches:
Attempt A: Store then parse
Create the HashMap entry first with the input, parse referencing the HashMap entry directly, and then update it.
pub fn store_and_parse(filename: &str, map: &mut HashMap<String, Entry>) {
let buffer: Vec<u8> = load_from_file(filename);
let mut entry = Entry{ input_data: buffer, parsed_result: vec![] };
let cmds = parse(&entry.input_data[..]);
entry.parsed_result = cmds;
map.insert(filename.to_string(), entry);
}
This doesn't work because the borrow checker complains that &entry.input_data[..] borrows with the same lifetime as entry, and therefore cannot be moved into map as there's an active borrow.
error[E0597]: `entry.input_data` does not live long enough
--> src\main.rs:26:23
|
23 | pub fn store_and_parse(filename: &str, map: &mut HashMap<String, Entry>) {
| --- has type `&mut std::collections::HashMap<std::string::String, Entry<'1>>`
...
26 | let cmds = parse(&entry.input_data[..]);
| ^^^^^^^^^^^^^^^^ borrowed value does not live long enough
27 | entry.parsed_result = cmds;
28 | map.insert(filename.to_string(), entry);
| --------------------------------------- argument requires that `entry.input_data` is borrowed for `'1`
29 | }
| - `entry.input_data` dropped here while still borrowed
error[E0505]: cannot move out of `entry` because it is borrowed
--> src\main.rs:28:38
|
26 | let cmds = parse(&entry.input_data[..]);
| ---------------- borrow of `entry.input_data` occurs here
27 | entry.parsed_result = cmds;
28 | map.insert(filename.to_string(), entry);
| ------ ^^^^^ move out of `entry` occurs here
| |
| borrow later used by call
Attempt B: Parse then store
Parse first, then try to store both the Vec buffer and the data slices into it all together into the HashMap.
pub fn parse_and_store(filename: &str, map: &mut HashMap<String, Entry>) {
let buffer: Vec<u8> = load_from_file(filename);
let cmds = parse(&buffer[..]);
let entry = Entry{ input_data: buffer, parsed_result: cmds };
map.insert(filename.to_string(), entry);
}
This doesn't work because the borrow checker complains that cmds has same lifetime as &buffer[..] but buffer will be dropped by the end of the function. It ignores the fact that cmds and buffer have the same lifetime, and are both (I wish) moved into entry, which is itself moved into map, so there should be no lifetime issue here.
error[E0597]: `buffer` does not live long enough
--> src\main.rs:33:21
|
31 | pub fn parse_and_store(filename: &str, map: &mut HashMap<String, Entry>) {
| --- has type `&mut std::collections::HashMap<std::string::String, Entry<'1>>`
32 | let buffer: Vec<u8> = load_from_file(filename);
33 | let cmds = parse(&buffer[..]);
| ^^^^^^ borrowed value does not live long enough
34 | let entry = Entry{ input_data: buffer, parsed_result: cmds };
35 | map.insert(filename.to_string(), entry);
| --------------------------------------- argument requires that `buffer` is borrowed for `'1`
36 | }
| - `buffer` dropped here while still borrowed
error[E0505]: cannot move out of `buffer` because it is borrowed
--> src\main.rs:34:34
|
31 | pub fn parse_and_store(filename: &str, map: &mut HashMap<String, Entry>) {
| --- has type `&mut std::collections::HashMap<std::string::String, Entry<'1>>`
32 | let buffer: Vec<u8> = load_from_file(filename);
33 | let cmds = parse(&buffer[..]);
| ------ borrow of `buffer` occurs here
34 | let entry = Entry{ input_data: buffer, parsed_result: cmds };
| ^^^^^^ move out of `buffer` occurs here
35 | map.insert(filename.to_string(), entry);
| --------------------------------------- argument requires that `buffer` is borrowed for `'1`
Minimal (non-)working example
use std::collections::HashMap;
#[derive(Debug, PartialEq)]
struct Cmd<'a> {
name: &'a str
}
fn parse<'a>(input: &'a [u8]) -> Vec<Cmd<'a>> {
Vec::new()
}
fn load_from_file(filename: &str) -> Vec<u8> {
Vec::new()
}
#[derive(Debug, PartialEq)]
struct Entry<'a> {
pub input_data: Vec<u8>,
pub parsed_result: Vec<Cmd<'a>>
}
// pub fn store_and_parse(filename: &str, map: &mut HashMap<String, Entry>) {
// let buffer: Vec<u8> = load_from_file(filename);
// let mut entry = Entry{ input_data: buffer, parsed_result: vec![] };
// let cmds = parse(&entry.input_data[..]);
// entry.parsed_result = cmds;
// map.insert(filename.to_string(), entry);
// }
pub fn parse_and_store(filename: &str, map: &mut HashMap<String, Entry>) {
let buffer: Vec<u8> = load_from_file(filename);
let cmds = parse(&buffer[..]);
let entry = Entry{ input_data: buffer, parsed_result: cmds };
map.insert(filename.to_string(), entry);
}
fn main() {
println!("Hello, world!");
}
Edit: Attempt with 2 maps
As Kevin pointed, and this is what threw me off the first time (above attempts), the borrow checker doesn't understand that moving a Vec doesn't invalidate the slices because the heap buffer of the Vec is not touched. Fair enough.
Side note: I am ignoring the parts of Kevin's answer related to using indexes (the Rust documentation explicitly states slices are a better replacement for indices, so I feel this is working against the language) and the use of external crates (which also are explicitly working against the language). I am trying to learn and understand how to do this "the Rust way", not at all costs.
So my immediate reaction to that was to change the data structure: first insert the storage Vec into a first HashMap, and once it's there call the parse() function to create the slices directly pointing into the HashMap value. Then store those into a second HashMap, which would naturally dissociate the two. However that also doesn't work as soon as I put all of that in a loop, which is the broader goal of this code:
fn two_maps<'a>(
filename: &str,
input_map: &'a mut HashMap<String, Vec<u8>>,
cmds_map: &mut HashMap<String, Vec<Cmd<'a>>>,
queue: &mut Vec<String>) {
{
let buffer: Vec<u8> = load_from_file(filename);
input_map.insert(filename.to_string(), buffer);
}
{
let buffer = input_map.get(filename).unwrap();
let cmds = parse(&buffer[..]);
for cmd in &cmds {
// [...] Find further dependencies to load and parse
queue.push("...".to_string());
}
cmds_map.insert(filename.to_string(), cmds);
}
}
fn main() {
let mut input_map = HashMap::new();
let mut cmds_map = HashMap::new();
let mut queue = Vec::new();
queue.push("file1.txt".to_string());
while let Some(path) = queue.pop() {
println!("Loading file: {}", path);
two_maps(&path[..], &mut input_map, &mut cmds_map, &mut queue);
}
}
The problem here is that once the input buffer is in the first map input_map, referencing it binds the lifetime of each new parsed result to the entry of that HashMap, and therefore the &'a mut reference (the 'a lifetime added). Without this, the compiler complains that data flows from input_map into cmds_map with unrelated lifetimes, which is fair enough. But with this, the &'a mut reference to input_map becomes locked on the first loop iteration and never released, and the borrow checker chokes on the second iteration, quite rightfully so.
So I am stuck again. Is what I am trying to do completely unreasonable and impossible in Rust? How can I approach the problem (algorithms, data structures) to make things work lifetime-wise? I really don't see what's the "Rust way" here to store a collection of buffers and slices over those buffers. Is the only solution (that I want to avoid) to first load all files, and then parse them? This is very impractical in my case because most files contain references to other files, and I want to load the minimum chain of dependencies (likely < 10 files), not the entire collection (which is something like 3000+ files), and I can only access dependencies by parsing each file.
It seems the core of the issue is that storing the input buffers into any kind of data structure requires a mutable reference to said data structure for the duration of the insert operation, which is incompatible with having long-lived immutable references to each single buffer (for the slices) because those references need to have the same lifetime as per the HashMap definition. Is there any other data structure (maybe immutable ones) that lifts this? Or am I completely on the wrong track?
Now, because the parsing operates on slices without storage, I need to first store the input text in a Vec so that the output slices remain valid, so something like:
struct Entry<'a> {
pub input_data: Vec<u8>,
pub parsed_result: Vec<Cmd<'a>>
}
What you are attempting here is a “self-referential structure”, where parsed_result refers to input_data. There is an incidental and a fundamental reason why this cannot work as written.
The incidental reason is that this struct declaration contains the lifetime parameter 'a, but actually the lifetime you're attempting to give parsed_result is the lifetime of the struct itself, and there is no Rust syntax to specify that lifetime.
The fundamental reason is that Rust allows structs (and other values) to be moved to other locations in memory, and references are just statically checked pointers. So, when you write
map.insert(filename.to_string(), entry);
you're causing the value of entry to be moved from the stack frame to the HashMap's storage. That move invalidates any references into entry, whether or not entry contains those references itself. That's what the error "cannot move out of entry because it is borrowed" means; the borrow checker is not allowing the move to happen.
In your Attempt B,
let buffer: Vec<u8> = load_from_file(filename);
let cmds = parse(&buffer[..]);
let entry = Entry{ input_data: buffer, parsed_result: cmds };
the problem is that you're moving buffer (into the Entry) while cmds borrows it. Again, that means the references (just fancy pointers!) into buffer would become invalid, so it's not allowed.
(Now, since Vec stores its actual data in a heap-allocated vector that will stay put while the Vec is moved, this might actually be safe, but the Rust borrow checker doesn't care about that.)
Solutions
The simplest solution (from a language perspective) is to have each Cmd store indices into input_data instead of references. Indices don't become invalid when the object is moved since they're relative. The disadvantage of this is of course that you have to slice the input data every time — code has to carry around the Entry as well as the Cmd.
However, there are tools available to make self-referential structures, without even needing to write any unsafe code. The crates ouroboros and rental both allow you to define self-referential structs, at the price of having to use special functions to access the struct fields.
For example, your code might look something like this using ouroboros (I haven't tested this):
use ouroboros::self_referencing;
#[self_referencing]
struct Entry {
input_data: Vec<u8>,
#[borrows(input_data)]
parsed_result: Vec<Cmd<'this>> // 'this is a special lifetime name provided by ouroboros
}
fn parse_and_store(filename: &str, map: &mut HashMap<String, Entry>) {
let entry = EntryBuilder { // EntryBuilder is defined by ouroboros to help construct Entry
input_data: load_from_file(filename),
// Note that instead of giving a value for parsed_result, we give
// a function to compute it.
parsed_result_builder: |input_data: &[u8]| parse(input_data),
}.build();
map.insert(filename.to_string(), entry);
}
fn do_something_with_entry(entry: &Entry) {
entry.with_parsed_result(|cmds| {
// cmds is a reference to `self.parsed_result` which only lives as
// long as this lambda and therefore can't be invalidated by a move.
});
}
ouroboros (and rental) provide a fairly odd interface for accessing fields. If, like me, you don't want to expose that interface to your users (or the rest of your code), you can write a wrapper struct around the self-referential struct whose impl contains methods designed for how you want the structure to be used, so all of the odd field access methods can remain private.
I thought I got the idea of move semantics until this code.
fn main() {
let v = Data {
body: vec![10, 40, 30],
};
p(&v);
}
fn p(d: &Data) {
for i in d.body {
// &d.body, Why d.body move?
println!("{}", i);
}
}
struct Data {
body: Vec<i32>,
}
error[E0507]: cannot move out of borrowed content
--> src/main.rs:9:14
|
9 | for i in d.body {
| ^^^^^^ cannot move out of borrowed content
error[E0507]: cannot move out of `d.body` which is behind a `&` reference
--> src/main.rs:9:14
|
8 | fn p(d: &Data) {
| ----- help: consider changing this to be a mutable reference: `&mut Data`
9 | for i in d.body {
| ^^^^^^
| |
| cannot move out of `d.body` which is behind a `&` reference
| `d` is a `&` reference, so the data it refers to cannot be moved
I passed a reference, and I accessed a field via auto-deref feature, so why is it a move?
What you are doing is field accessing on pointer.
Check Field Access Expression :
if the type of the expression to the left of the dot is a pointer, it
is automatically dereferenced as many times as necessary to make the
field access possible
Sample for how Rust evaluates Field Access Expression on Borrowed Content :
let d = Data { /*input*/}
let body = (&d).body // -> (*&d).body -> d.body
let ref_body = &(&d).body // -> &(*&).body -> &d.body -> &(d.body)
Note : d is still borrowed content, auto deref is just needed to access the fields.
Why move ?
Consider this code:
struct Data {
body: Vec<i32>,
id: i32,
}
fn p(mut d: &Data) {
let id = d.id;
}
This code will work as expected and there will be no moves in here so you will able to reuse d.id. In this situation:
Rust will try to copy the value of d.id. Since d.id is i32 and implements the Copy trait, it will copy the value to id.
Consider this code:
fn p(mut d: &Data) {
let id = d.id; // works
let body = d.body; // fails
}
This code will not work because:
Rust will try to copy d.body but Vec<i32> has no implementation of the Copy trait.
Rust will try to move body from d, and you will get the "cannot move out of borrowed content" error.
How does this effect the loop?
From the reference
A for expression is a syntactic construct for looping over elements provided by an implementation of std::iter::IntoIterator
A for loop is equivalent to the following block expression.
'label: for PATTERN in iter_expr {
/* loop body */
}
is equivalent to
{
let result = match IntoIterator::into_iter(iter_expr) {
mut iter => 'label: loop {
let mut next;
match Iterator::next(&mut iter) {
Option::Some(val) => next = val,
Option::None => break,
};
let PAT = next;
let () = { /* loop body */ };
},
};
result
}
This means your vector must have an implementation of IntoIterator because IntoIterator::into_iter(self) expects self as an argument. Luckily, both impl IntoIterator for Vec<T>, another is impl<'a, T> IntoIterator for &'a Vec<T> exist.
Why does this happen?
Simply:
When you use &d.body, your loop uses the &Vec implementation of IntoIterator.
This implementation returns an iterator which points at your vector's slice. This means you will get the reference of elements from your vector.
When you use d.body, your loop uses the Vec implementation of IntoIterator.
This implementation returns an iterator which is a consuming iterator. This means your loop will have the ownership of actual elements, not their references. For the consuming part this implementation needs the actual vector not the reference, so the move occurs.
You are accessing the field body in d. body itself is a Vec<i32> which is not a reference. If you would use d directly, no & would be necessary, but since you are accessing a field in d, you must specify that you want to have the reference to the field.
Basically d owns body. If you borrow d you cannot steal body, it belongs to d but you can borrow it.
This loop will be desugared into something similar to the following:
let mut iter = IntoIterator::into_iter(v);
loop {
match iter.next() {
Some(x) => {
// loop body
},
None => break,
}
}
As you can see, it's using into_iter, which moves the vector d.body.
I'm learning Rust and I'm trying to cargo-cult this code into compiling:
use std::vec::Vec;
use std::collections::BTreeMap;
struct Occ {
docnum: u64,
weight: f32,
}
struct PostWriter<'a> {
bytes: Vec<u8>,
occurrences: BTreeMap<&'a [u8], Vec<Occ>>,
}
impl<'a> PostWriter<'a> {
fn new() -> PostWriter<'a> {
PostWriter {
bytes: Vec::new(),
occurrences: BTreeMap::new(),
}
}
fn add_occurrence(&'a mut self, term: &[u8], occ: Occ) {
let occurrences = &mut self.occurrences;
match occurrences.get_mut(term) {
Some(x) => x.push(occ),
None => {
// Add the term bytes to the big vector of all terms
let termstart = self.bytes.len();
self.bytes.extend(term);
// Create a new occurrences vector
let occs = vec![occ];
// Take the appended term as a slice to use as a key
// ERROR: cannot borrow `*occurrences` as mutable more than once at a time
occurrences.insert(&self.bytes[termstart..], occs);
}
}
}
}
fn main() {}
I get an error:
error[E0499]: cannot borrow `*occurrences` as mutable more than once at a time
--> src/main.rs:34:17
|
24 | match occurrences.get_mut(term) {
| ----------- first mutable borrow occurs here
...
34 | occurrences.insert(&self.bytes[termstart..], occs);
| ^^^^^^^^^^^ second mutable borrow occurs here
35 | }
36 | }
| - first borrow ends here
I don't understand... I'm just calling a method on a mutable reference, why would that line involve borrowing?
I'm just calling a method on a mutable reference, why would that line involve borrowing?
When you call a method on an object that's going to mutate the object, you can't have any other references to that object outstanding. If you did, your mutation could invalidate those references and leave your program in an inconsistent state. For example, say that you had gotten a value out of your hashmap and then added a new value. Adding the new value hits a magic limit and forces memory to be reallocated, your value now points off to nowhere! When you use that value... bang goes the program!
In this case, it looks like you want to do the relatively common "append or insert if missing" operation. You will want to use entry for that:
use std::collections::BTreeMap;
fn main() {
let mut map = BTreeMap::new();
{
let nicknames = map.entry("joe").or_insert(Vec::new());
nicknames.push("shmoe");
// Using scoping to indicate that we are done with borrowing `nicknames`
// If we didn't, then we couldn't borrow map as
// immutable because we could still change it via `nicknames`
}
println!("{:?}", map)
}
Because you're calling a method that borrows as mutable
I had a similar question yesterday about Hash, until I noticed something in the docs. The docs for BTreeMap show a method signature for insert starting with fn insert(&mut self..
So when you call .insert, you're implicitly asking that function to borrow the BTreeMap as mutable.