This question already has an answer here:
Returning a reference from a HashMap or Vec causes a borrow to last beyond the scope it's in?
(1 answer)
Closed 3 years ago.
I'm trying to write a function that finds returns a mutable reference to an existing element in a Vec, or inserts it if it doesn't exist and returns a mutable reference to the new element.
I've tried a couple of times, but the borrow checker isn't convinced. I've simplified the code I was trying to write to the example below, which gives the same errors.
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(u) = vec.iter_mut().find(|u| **u == val) {
u
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
Playground link: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=cb12c38bcf3682b15a247d14aab48b6b
Rust gives me the following compiler error (full message through the playground link):
error[E0499]: cannot borrow `*vec` as mutable more than once at a time
This seems to be something that should be possible to implement in rust, however it's not clear to me how I reimplement this to avoid borrow checker errors.
The reason this doesn't work as written is because of a limitation in the current borrow checker. This is very similar to NLL case #3, in which the compiler borrows somewhat overzealously for an entire match statement when the borrow is only used in one of the branches. With the experimental "Polonius" borrow checker (available on the nightly compiler with the -Z polonius flag), your code is accepted as-is.
Working in the stable compiler, it's probably a good idea to redesign your data structures as Sébastien Renauld's answer also suggests, but if you need to make this work with a Vec, you can work around it by briefly using an index to end the borrow:
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(i) = vec.iter().position(|each| *each == val) {
&mut vec[i]
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
This works because the result of calling position is not a reference, so the borrow of vec is not held during the if let.
This is similar to the following questions, which manage to find the same limitation using early return from a loop:
Double mutable borrow error in a loop happens even with NLL on
"Variable does not live long enough" when returning a Result containing a reference but it does live long enough
Vec is an unordered, not very structured type. It has no way to look up the exact position of an item within it; the closest the default functions get to is contains(), which only tells you if the item is contained.
Furthermore, due to the fact that a Vec is not a Set, the behavior of "find the item or append and return " is undefined - "find the item", if there is a duplicate, needs to be defined further.
To solve this problem without changing to the correct type (HashSet is the type you really want for this. Note the existence of get_or_insert(), which is literally what you are after. It pays to use the proper structure for the job, rather than to try to make everything fit a Vec), we're going to have to build it ourselves. Keeping to your signature, it looks like this (Playground):
trait VecSeekOrAppend<T:PartialEq>:Sized {
fn get_or_insert(&mut self, item: T) -> &mut T;
}
impl<T> VecSeekOrAppend<T> for Vec<T>
where T: PartialEq + Clone {
fn get_or_insert(&mut self, item: T) -> &mut T {
if !self.contains(&item) {
self.push(item.clone());
}
for i in self.iter_mut() {
if i == &mut item {
return i;
}
}
unreachable!();
}
}
The reason your initial version does not work is due to the returned lifetime requirement; all methods returning a reference from a Vec require a lifetime validity for the duration of use. By returning such a &mut reference, if you attempt to do it in one go, the mutation of the Vec<_> will happen while there is already a mutable borrow of it.
Splitting the cycle in two, and performing the insertion (without keeping a reference) to then find the eference, allows us to sidestep this problem. Another way to perform this is to store items by a serializable or hashable identifier (the exact way HashMap and HashSet work) in order to innately provide this layer of indirection.
There is a rust feature in the works to ease some of this pain (non-lexical lifetimes), but, as you can see from the github issue, it's not in the near future.
Related
In attempting to refactor a Rust application that was working fine, I tried to separate out the contents of a loop into a new function. However, in this newly refactored out function, I needed to pass an argument that had to be mutable, and passed by reference. Suddenly the code that absolutely worked inline, broke just because of the mutable reference passing.
My question is: can someone please explain why this does not work with such a "simple" change? (i.e. refactoring out a new function of otherwise unchanged code)
I have a minimal demo of the problem, along with a couple of working comparisons below. Here is the error from that code:
error[E0499]: cannot borrow `str_to_int` as mutable more than once at a time
--> src/main.rs:30:22
|
30 | get_key(key, &mut str_to_int);
| ^^^^^^^^^^^^^^^ `str_to_int` was mutably borrowed here in the previous iteration of the loop
The sample code:
use std::collections::BTreeMap;
fn get_int (
key: u32,
values: &mut BTreeMap<u32, u32>,
) -> &u32 {
values.entry(key).or_insert_with(|| { 1 })
}
fn get_key<'a> (
key: &'a str,
values: &'a mut BTreeMap<&'a str, u32>,
) -> &'a u32 {
values.entry(key).or_insert_with(|| { 1 })
}
fn main() {
let mut int_to_int = BTreeMap::new();
for key in vec![1,2] {
get_int(key, &mut int_to_int);
}
let mut str_to_int_inline = BTreeMap::new();
for key in vec!["a","b"] {
str_to_int_inline.entry(key).or_insert_with(|| { 1 });
}
let mut str_to_int = BTreeMap::new();
for key in vec!["a","b"] {
get_key(key, &mut str_to_int);
}
}
Note that the first loop (int_to_int) is identical to the third loop (str_to_int) except for the data type of the key -- in that the key was not a reference, so no lifetime was required to be specified. And the second loop (str_to_int_inline) is identical to the third loop (str_to_int) except the behavior is inline instead of in a separate function.
There are many related questions and blogs on this topic, but they all seem more specifically focused on particular versions of this question, and I want to know the more generic explanation (to my current understanding). If the answer is already just to understand one of these links better, I will gladly mark this question as a duplicate.
Related questions:
How to fix ".. was mutably borrowed here in the previous iteration of the loop" in Rust?
https://users.rust-lang.org/t/mutable-borrow-starts-here-in-previous-iteration-of-loop/26145
https://github.com/rust-lang/rust/issues/47680#issuecomment-363131420
Why does linking lifetimes matter only with mutable references?
Something that I read also led me to https://github.com/rust-lang/polonius which also seemed like maybe it could make this work, in the future -- any thoughts?
Your problem is very simple: you are specifying the lifetimes for get_key() incorrectly.
The correct (and working) version is:
fn get_key<'a, 'b>(key: &'a str, values: &'b mut BTreeMap<&'a str, u32>) -> &'b u32 {
values.entry(key).or_insert_with(|| 1)
}
Maybe you can already guess what's going on.
Since you used 'a for both the HashMap itself and its keys, it means you were required to borrow the HashMap for as long as the keys' lifetime - 'static. This means two things:
You need a &'static mut HashMap, which you don't have.
You're borrowing the HashMap for 'static in the first iteration of the loop, then borrow it again in the next, while it is still borrowed (because it is borrowed for 'static). This error hides the first error, somewhat erroneously, and is the only error the compiler emits.
In general, using a lifetime twice with a mutable reference is almost always wrong (shared references are more tolerant, as they're covariant over their type).
This question already has an answer here:
Returning a reference from a HashMap or Vec causes a borrow to last beyond the scope it's in?
(1 answer)
Closed 3 years ago.
I'm trying to write a function that finds returns a mutable reference to an existing element in a Vec, or inserts it if it doesn't exist and returns a mutable reference to the new element.
I've tried a couple of times, but the borrow checker isn't convinced. I've simplified the code I was trying to write to the example below, which gives the same errors.
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(u) = vec.iter_mut().find(|u| **u == val) {
u
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
Playground link: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=cb12c38bcf3682b15a247d14aab48b6b
Rust gives me the following compiler error (full message through the playground link):
error[E0499]: cannot borrow `*vec` as mutable more than once at a time
This seems to be something that should be possible to implement in rust, however it's not clear to me how I reimplement this to avoid borrow checker errors.
The reason this doesn't work as written is because of a limitation in the current borrow checker. This is very similar to NLL case #3, in which the compiler borrows somewhat overzealously for an entire match statement when the borrow is only used in one of the branches. With the experimental "Polonius" borrow checker (available on the nightly compiler with the -Z polonius flag), your code is accepted as-is.
Working in the stable compiler, it's probably a good idea to redesign your data structures as Sébastien Renauld's answer also suggests, but if you need to make this work with a Vec, you can work around it by briefly using an index to end the borrow:
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(i) = vec.iter().position(|each| *each == val) {
&mut vec[i]
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
This works because the result of calling position is not a reference, so the borrow of vec is not held during the if let.
This is similar to the following questions, which manage to find the same limitation using early return from a loop:
Double mutable borrow error in a loop happens even with NLL on
"Variable does not live long enough" when returning a Result containing a reference but it does live long enough
Vec is an unordered, not very structured type. It has no way to look up the exact position of an item within it; the closest the default functions get to is contains(), which only tells you if the item is contained.
Furthermore, due to the fact that a Vec is not a Set, the behavior of "find the item or append and return " is undefined - "find the item", if there is a duplicate, needs to be defined further.
To solve this problem without changing to the correct type (HashSet is the type you really want for this. Note the existence of get_or_insert(), which is literally what you are after. It pays to use the proper structure for the job, rather than to try to make everything fit a Vec), we're going to have to build it ourselves. Keeping to your signature, it looks like this (Playground):
trait VecSeekOrAppend<T:PartialEq>:Sized {
fn get_or_insert(&mut self, item: T) -> &mut T;
}
impl<T> VecSeekOrAppend<T> for Vec<T>
where T: PartialEq + Clone {
fn get_or_insert(&mut self, item: T) -> &mut T {
if !self.contains(&item) {
self.push(item.clone());
}
for i in self.iter_mut() {
if i == &mut item {
return i;
}
}
unreachable!();
}
}
The reason your initial version does not work is due to the returned lifetime requirement; all methods returning a reference from a Vec require a lifetime validity for the duration of use. By returning such a &mut reference, if you attempt to do it in one go, the mutation of the Vec<_> will happen while there is already a mutable borrow of it.
Splitting the cycle in two, and performing the insertion (without keeping a reference) to then find the eference, allows us to sidestep this problem. Another way to perform this is to store items by a serializable or hashable identifier (the exact way HashMap and HashSet work) in order to innately provide this layer of indirection.
There is a rust feature in the works to ease some of this pain (non-lexical lifetimes), but, as you can see from the github issue, it's not in the near future.
To add the elements of two Vecs I wrote a function like
fn add_components(dest: &mut Vec<i32>, first: &Vec<i32>, second: &Vec<i32>){
for i in 0..first.len() {
dest[i] = first[i] + second[i];
}
}
And this works fine when dest is another Vec.
let mut new_components = Vec::with_capacity(components.len());
Vector::add_components(&mut new_comps, &components, &other_components);
But it blows up when I am trying to add in-place:
Vector::add_components(&mut components, &components, &other_components);
because now I borrow components as mutable and immutable at the same time. But this obviously is what I am trying to achieve.
Are there any conventional and general (meaning not only concerning Vecs) solutions to this problem which don't involve unsafe code and pointer magic?
Another example of this problem:
Suppose I want to overload AddAssign for a numeric type like
impl AddAssign<Output=&NumericType> for NumericType {
fn add_assign(&mut self, other: &NumericType) {
unimplemented!() // concrete implementation is not important
}
}
Notice that I want to take a reference as second argument to avoid copying. This works fine when adding two different objects, but adding an object to itself creates the exact same scenario:
let mut num = NumericType{};
num += &num
I am borrowing num mutably and immutably at the same time. So obviously this should work and is safe, but it also is against Rust's borrowing rules.
What are the best practices (apart from copying of course) to deal with this issue, which arises in many forms?
There is no generic solution to this. Rust can't generically abstract over mutability in borrow checking.
You will need to have two versions of the function for in-place and destination versions.
Rust has strict aliasing rules, so dest[i] = first[i] + second[i] actually compiles to different code depending on whether the compiler has a guarantee that dest and first are different. Don't try to fudge it with unsafe, because it will be Undefined Behavior and will get miscompiled.
I'm implementing Conway's game of life to teach myself Rust. The idea is to implement a single-threaded version first, optimize it as much as possible, then do the same for a multi-threaded version.
I wanted to implement an alternative data layout which I thought might be more cache-friendly. The idea is to store the status of two cells for each point on a board next to each other in memory in a vector, one cell for reading the current generation's status from and one for writing the next generation's status to, alternating the access pattern for each
generation's computation (which can be determined at compile time).
The basic data structures are as follows:
#[repr(u8)]
pub enum CellStatus {
DEAD,
ALIVE,
}
/** 2 bytes */
pub struct CellRW(CellStatus, CellStatus);
pub struct TupleBoard {
width: usize,
height: usize,
cells: Vec<CellRW>,
}
/** used to keep track of current pos with iterator e.g. */
pub struct BoardPos {
x_pos: usize,
y_pos: usize,
offset: usize,
}
pub struct BoardEvo {
board: TupleBoard,
}
The function that is causing me troubles:
impl BoardEvo {
fn evolve_step<T: RWSelector>(&mut self) {
for (pos, cell) in self.board.iter_mut() {
//pos: BoardPos, cell: &mut CellRW
let read: &CellStatus = T::read(cell); //chooses the right tuple half for the current evolution step
let write: &mut CellStatus = T::write(cell);
let alive_count = pos.neighbours::<T>(&self.board).iter() //<- can't borrow self.board again!
.filter(|&&status| status == CellStatus::ALIVE)
.count();
*write = CellStatus::evolve(*read, alive_count);
}
}
}
impl BoardPos {
/* ... */
pub fn neighbours<T: RWSelector>(&self, board: &BoardTuple) -> [CellStatus; 8] {
/* ... */
}
}
The trait RWSelector has static functions for reading from and writing to a cell tuple (CellRW). It is implemented for two zero-sized types L and R and is mainly a way to avoid having to write different methods for the different access patterns.
The iter_mut() method returns a BoardIter struct which is a wrapper around a mutable slice iterator for the cells vector and thus has &mut CellRW as Item type. It is also aware of the current BoardPos (x and y coordinates, offset).
I thought I'd iterate over all cell tuples, keep track of the coordinates, count the number of alive neighbours (I need to know coordinates/offsets for this) for each (read) cell, compute the cell status for the next generation and write to the respective another half of the tuple.
Of course, in the end, the compiler showed me the fatal flaw in my design, as I borrow self.board mutably in the iter_mut() method and then try to borrow it again immutably to get all the neighbours of the read cell.
I have not been able to come up with a good solution for this problem so far. I did manage to get it working by making all
references immutable and then using an UnsafeCell to turn the immutable reference to the write cell into a mutable one.
I then write to the nominally immutable reference to the writing part of the tuple through the UnsafeCell.
However, that doesn't strike me as a sound design and I suspect I might run into issues with this when attempting to parallelize things.
Is there a way to implement the data layout I proposed in safe/idiomatic Rust or is this actually a case where you actually have to use tricks to circumvent Rust's aliasing/borrow restrictions?
Also, as a broader question, is there a recognizable pattern for problems which require you to circumvent Rust's borrow restrictions?
When is it necessary to circumvent Rust's borrow checker?
It is needed when:
the borrow checker is not advanced enough to see that your usage is safe
you do not wish to (or cannot) write the code in a different pattern
As a concrete case, the compiler cannot tell that this is safe:
let mut array = [1, 2];
let a = &mut array[0];
let b = &mut array[1];
The compiler doesn't know what the implementation of IndexMut for a slice does at this point of compilation (this is a deliberate design choice). For all it knows, arrays always return the exact same reference, regardless of the index argument. We can tell that this code is safe, but the compiler disallows it.
You can rewrite this in a way that is obviously safe to the compiler:
let mut array = [1, 2];
let (a, b) = array.split_at_mut(1);
let a = &mut a[0];
let b = &mut b[0];
How is this done? split_at_mut performs a runtime check to ensure that it actually is safe:
fn split_at_mut(&mut self, mid: usize) -> (&mut [T], &mut [T]) {
let len = self.len();
let ptr = self.as_mut_ptr();
unsafe {
assert!(mid <= len);
(from_raw_parts_mut(ptr, mid),
from_raw_parts_mut(ptr.offset(mid as isize), len - mid))
}
}
For an example where the borrow checker is not yet as advanced as it can be, see What are non-lexical lifetimes?.
I borrow self.board mutably in the iter_mut() method and then try to borrow it again immutably to get all the neighbours of the read cell.
If you know that the references don't overlap, then you can choose to use unsafe code to express it. However, this means you are also choosing to take on the responsibility of upholding all of Rust's invariants and avoiding undefined behavior.
The good news is that this heavy burden is what every C and C++ programmer has to (or at least should) have on their shoulders for every single line of code they write. At least in Rust, you can let the compiler deal with 99% of the cases.
In many cases, there's tools like Cell and RefCell to allow for interior mutation. In other cases, you can rewrite your algorithm to take advantage of a value being a Copy type. In other cases you can use an index into a slice for a shorter period. In other cases you can have a multi-phase algorithm.
If you do need to resort to unsafe code, then try your best to hide it in a small area and expose safe interfaces.
Above all, many common problems have been asked about (many times) before:
How to iterate over mutable elements inside another mutable iteration over the same elements?
Mutating an item inside of nested loops
How can a nested loop with mutations on a HashMap be achieved in Rust?
What's the Rust way to modify a structure within nested loops?
Nesting an iterator's loops
While trying to answer on Rust array of functions (it's been answered with a great answer), I've coined the following code:
fn main() {
let mut a : Vec<proc() -> uint>;
for i in range(0u, 11) {
a[i] = proc(){i};
}
println!("{} {} {}", a[1](), a[5](), a[9]());
}
Please ignore the fact that proc is being deprecated, I just figured out that this is what should be used instead of closures (I didn't know about move and unboxed closures at that time).
However, I was not able to call elements of a vector due the following:
<anon>:6:26: 6:30 error: cannot move out of dereference (dereference is implicit, due to indexing)
<anon>:6 println!("{} {} {}", a[1](), a[5](), a[9]());
What this error means? Shouldn't it just return uint?
proc() were closures that could only be used once, and thus calling them consumed them.
In your situation, this would imply moving the closure out of the Vec<> in order to consume it, which is not possible as indexing is the dereferencing of a & pointer, which only allow immutable access.