In attempting to refactor a Rust application that was working fine, I tried to separate out the contents of a loop into a new function. However, in this newly refactored out function, I needed to pass an argument that had to be mutable, and passed by reference. Suddenly the code that absolutely worked inline, broke just because of the mutable reference passing.
My question is: can someone please explain why this does not work with such a "simple" change? (i.e. refactoring out a new function of otherwise unchanged code)
I have a minimal demo of the problem, along with a couple of working comparisons below. Here is the error from that code:
error[E0499]: cannot borrow `str_to_int` as mutable more than once at a time
--> src/main.rs:30:22
|
30 | get_key(key, &mut str_to_int);
| ^^^^^^^^^^^^^^^ `str_to_int` was mutably borrowed here in the previous iteration of the loop
The sample code:
use std::collections::BTreeMap;
fn get_int (
key: u32,
values: &mut BTreeMap<u32, u32>,
) -> &u32 {
values.entry(key).or_insert_with(|| { 1 })
}
fn get_key<'a> (
key: &'a str,
values: &'a mut BTreeMap<&'a str, u32>,
) -> &'a u32 {
values.entry(key).or_insert_with(|| { 1 })
}
fn main() {
let mut int_to_int = BTreeMap::new();
for key in vec![1,2] {
get_int(key, &mut int_to_int);
}
let mut str_to_int_inline = BTreeMap::new();
for key in vec!["a","b"] {
str_to_int_inline.entry(key).or_insert_with(|| { 1 });
}
let mut str_to_int = BTreeMap::new();
for key in vec!["a","b"] {
get_key(key, &mut str_to_int);
}
}
Note that the first loop (int_to_int) is identical to the third loop (str_to_int) except for the data type of the key -- in that the key was not a reference, so no lifetime was required to be specified. And the second loop (str_to_int_inline) is identical to the third loop (str_to_int) except the behavior is inline instead of in a separate function.
There are many related questions and blogs on this topic, but they all seem more specifically focused on particular versions of this question, and I want to know the more generic explanation (to my current understanding). If the answer is already just to understand one of these links better, I will gladly mark this question as a duplicate.
Related questions:
How to fix ".. was mutably borrowed here in the previous iteration of the loop" in Rust?
https://users.rust-lang.org/t/mutable-borrow-starts-here-in-previous-iteration-of-loop/26145
https://github.com/rust-lang/rust/issues/47680#issuecomment-363131420
Why does linking lifetimes matter only with mutable references?
Something that I read also led me to https://github.com/rust-lang/polonius which also seemed like maybe it could make this work, in the future -- any thoughts?
Your problem is very simple: you are specifying the lifetimes for get_key() incorrectly.
The correct (and working) version is:
fn get_key<'a, 'b>(key: &'a str, values: &'b mut BTreeMap<&'a str, u32>) -> &'b u32 {
values.entry(key).or_insert_with(|| 1)
}
Maybe you can already guess what's going on.
Since you used 'a for both the HashMap itself and its keys, it means you were required to borrow the HashMap for as long as the keys' lifetime - 'static. This means two things:
You need a &'static mut HashMap, which you don't have.
You're borrowing the HashMap for 'static in the first iteration of the loop, then borrow it again in the next, while it is still borrowed (because it is borrowed for 'static). This error hides the first error, somewhat erroneously, and is the only error the compiler emits.
In general, using a lifetime twice with a mutable reference is almost always wrong (shared references are more tolerant, as they're covariant over their type).
Related
In this leetcode invert binary tree problem, I'm trying to borrow a node wrapped in an Rc mutably. Here is the code.
use std::rc::Rc;
use std::cell::RefCell;
impl Solution {
pub fn invert_tree(root: Option<Rc<RefCell<TreeNode>>>) -> Option<Rc<RefCell<TreeNode>>> {
let mut stack: Vec<Option<Rc<RefCell<TreeNode>>>> = vec![root.clone()];
while stack.len() > 0 {
if let Some(node) = stack.pop().unwrap() {
let n: &mut TreeNode = &mut node.borrow_mut();
std::mem::swap(&mut n.left, &mut n.right);
stack.extend(vec![n.left.clone(), n.right.clone()]);
}
}
root
}
}
If I change the line let n: &mut TreeNode to just let n = &mut node.borrow_mut(), I get a compiler error on the next line, "cannot borrow *n as mutable more than once at a time"
It seems like the compiler infers n to be of type &mut RefMut<TreeNode>, but it all works out when I explicitly say it is &mut TreeNode. Any reason why?
A combination of borrow splitting and deref-coercion causes the seemingly identical code to behave differently.
The compiler infers n to be of type RefMut<TreeNode>, because that's what borrow_mut actually returns:
pub fn borrow_mut(&self) -> RefMut<'_, T>
RefMut is a funny little type that's designed to look like a &mut, but it's actually a separate thing. It implements Deref and DerefMut, so it will happily pretend to be a &mut TreeNode when needed. But Rust is still inserting calls to .deref() in there for you.
Now, why does one work and not the other? Without the type annotation, after deref insertion, you get
let n = &mut node.borrow_mut();
std::mem::swap(&mut n.deref_mut().left, &mut n.deref_mut().right);
So we're trying to call deref_mut (which takes a &mut self) twice in the same line on the same variable. That's not allowed by Rust's borrow rules, so it fails.
(Note that the &mut on the first line simply borrows an owned value for no reason. Temporary lifetime extension lets us get away with this, even though you don't need the &mut at all in this case)
Now, on the other hand, if you do put in the type annotation, then Rust sees that borrow_mut returns a RefMut<'_, TreeNode> but you asked for a &mut TreeNode, so it inserts the deref_mut on the first line. You get
let n: &mut TreeNode = &mut node.borrow_mut().deref_mut();
std::mem::swap(&mut n.left, &mut n.right);
Now the only deref_mut call is on the first line. Then, on the second line, we access n.left and n.right, both mutably, simultaneously. It looks like we're accessing n mutably twice at once, but Rust is actually smart enough to see that we're accessing two disjoint parts of n simultaneously, so it allows it. This is called borrow splitting. Rust will split borrows on different instance fields, but it's not smart enough to see the split across a deref_mut call (function calls could, in principle, do anything, so Rust's borrow checker refuses to try to do advanced reasoning about their return value).
This question already has an answer here:
Returning a reference from a HashMap or Vec causes a borrow to last beyond the scope it's in?
(1 answer)
Closed 3 years ago.
I'm trying to write a function that finds returns a mutable reference to an existing element in a Vec, or inserts it if it doesn't exist and returns a mutable reference to the new element.
I've tried a couple of times, but the borrow checker isn't convinced. I've simplified the code I was trying to write to the example below, which gives the same errors.
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(u) = vec.iter_mut().find(|u| **u == val) {
u
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
Playground link: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=cb12c38bcf3682b15a247d14aab48b6b
Rust gives me the following compiler error (full message through the playground link):
error[E0499]: cannot borrow `*vec` as mutable more than once at a time
This seems to be something that should be possible to implement in rust, however it's not clear to me how I reimplement this to avoid borrow checker errors.
The reason this doesn't work as written is because of a limitation in the current borrow checker. This is very similar to NLL case #3, in which the compiler borrows somewhat overzealously for an entire match statement when the borrow is only used in one of the branches. With the experimental "Polonius" borrow checker (available on the nightly compiler with the -Z polonius flag), your code is accepted as-is.
Working in the stable compiler, it's probably a good idea to redesign your data structures as Sébastien Renauld's answer also suggests, but if you need to make this work with a Vec, you can work around it by briefly using an index to end the borrow:
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(i) = vec.iter().position(|each| *each == val) {
&mut vec[i]
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
This works because the result of calling position is not a reference, so the borrow of vec is not held during the if let.
This is similar to the following questions, which manage to find the same limitation using early return from a loop:
Double mutable borrow error in a loop happens even with NLL on
"Variable does not live long enough" when returning a Result containing a reference but it does live long enough
Vec is an unordered, not very structured type. It has no way to look up the exact position of an item within it; the closest the default functions get to is contains(), which only tells you if the item is contained.
Furthermore, due to the fact that a Vec is not a Set, the behavior of "find the item or append and return " is undefined - "find the item", if there is a duplicate, needs to be defined further.
To solve this problem without changing to the correct type (HashSet is the type you really want for this. Note the existence of get_or_insert(), which is literally what you are after. It pays to use the proper structure for the job, rather than to try to make everything fit a Vec), we're going to have to build it ourselves. Keeping to your signature, it looks like this (Playground):
trait VecSeekOrAppend<T:PartialEq>:Sized {
fn get_or_insert(&mut self, item: T) -> &mut T;
}
impl<T> VecSeekOrAppend<T> for Vec<T>
where T: PartialEq + Clone {
fn get_or_insert(&mut self, item: T) -> &mut T {
if !self.contains(&item) {
self.push(item.clone());
}
for i in self.iter_mut() {
if i == &mut item {
return i;
}
}
unreachable!();
}
}
The reason your initial version does not work is due to the returned lifetime requirement; all methods returning a reference from a Vec require a lifetime validity for the duration of use. By returning such a &mut reference, if you attempt to do it in one go, the mutation of the Vec<_> will happen while there is already a mutable borrow of it.
Splitting the cycle in two, and performing the insertion (without keeping a reference) to then find the eference, allows us to sidestep this problem. Another way to perform this is to store items by a serializable or hashable identifier (the exact way HashMap and HashSet work) in order to innately provide this layer of indirection.
There is a rust feature in the works to ease some of this pain (non-lexical lifetimes), but, as you can see from the github issue, it's not in the near future.
This question already has an answer here:
Returning a reference from a HashMap or Vec causes a borrow to last beyond the scope it's in?
(1 answer)
Closed 3 years ago.
I'm trying to write a function that finds returns a mutable reference to an existing element in a Vec, or inserts it if it doesn't exist and returns a mutable reference to the new element.
I've tried a couple of times, but the borrow checker isn't convinced. I've simplified the code I was trying to write to the example below, which gives the same errors.
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(u) = vec.iter_mut().find(|u| **u == val) {
u
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
Playground link: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=cb12c38bcf3682b15a247d14aab48b6b
Rust gives me the following compiler error (full message through the playground link):
error[E0499]: cannot borrow `*vec` as mutable more than once at a time
This seems to be something that should be possible to implement in rust, however it's not clear to me how I reimplement this to avoid borrow checker errors.
The reason this doesn't work as written is because of a limitation in the current borrow checker. This is very similar to NLL case #3, in which the compiler borrows somewhat overzealously for an entire match statement when the borrow is only used in one of the branches. With the experimental "Polonius" borrow checker (available on the nightly compiler with the -Z polonius flag), your code is accepted as-is.
Working in the stable compiler, it's probably a good idea to redesign your data structures as Sébastien Renauld's answer also suggests, but if you need to make this work with a Vec, you can work around it by briefly using an index to end the borrow:
fn mut_find_or_insert<T: PartialEq>(vec: &mut Vec<T>, val: T) -> &mut T {
if let Some(i) = vec.iter().position(|each| *each == val) {
&mut vec[i]
} else {
vec.push(val);
vec.last_mut().unwrap()
}
}
This works because the result of calling position is not a reference, so the borrow of vec is not held during the if let.
This is similar to the following questions, which manage to find the same limitation using early return from a loop:
Double mutable borrow error in a loop happens even with NLL on
"Variable does not live long enough" when returning a Result containing a reference but it does live long enough
Vec is an unordered, not very structured type. It has no way to look up the exact position of an item within it; the closest the default functions get to is contains(), which only tells you if the item is contained.
Furthermore, due to the fact that a Vec is not a Set, the behavior of "find the item or append and return " is undefined - "find the item", if there is a duplicate, needs to be defined further.
To solve this problem without changing to the correct type (HashSet is the type you really want for this. Note the existence of get_or_insert(), which is literally what you are after. It pays to use the proper structure for the job, rather than to try to make everything fit a Vec), we're going to have to build it ourselves. Keeping to your signature, it looks like this (Playground):
trait VecSeekOrAppend<T:PartialEq>:Sized {
fn get_or_insert(&mut self, item: T) -> &mut T;
}
impl<T> VecSeekOrAppend<T> for Vec<T>
where T: PartialEq + Clone {
fn get_or_insert(&mut self, item: T) -> &mut T {
if !self.contains(&item) {
self.push(item.clone());
}
for i in self.iter_mut() {
if i == &mut item {
return i;
}
}
unreachable!();
}
}
The reason your initial version does not work is due to the returned lifetime requirement; all methods returning a reference from a Vec require a lifetime validity for the duration of use. By returning such a &mut reference, if you attempt to do it in one go, the mutation of the Vec<_> will happen while there is already a mutable borrow of it.
Splitting the cycle in two, and performing the insertion (without keeping a reference) to then find the eference, allows us to sidestep this problem. Another way to perform this is to store items by a serializable or hashable identifier (the exact way HashMap and HashSet work) in order to innately provide this layer of indirection.
There is a rust feature in the works to ease some of this pain (non-lexical lifetimes), but, as you can see from the github issue, it's not in the near future.
This code fails the dreaded borrow checker (playground):
struct Data {
a: i32,
b: i32,
c: i32,
}
impl Data {
fn reference_to_a(&mut self) -> &i32 {
self.c = 1;
&self.a
}
fn get_b(&self) -> i32 {
self.b
}
}
fn main() {
let mut dat = Data{ a: 1, b: 2, c: 3 };
let aref = dat.reference_to_a();
println!("{}", dat.get_b());
}
Since non-lexical lifetimes were implemented, this is required to trigger the error:
fn main() {
let mut dat = Data { a: 1, b: 2, c: 3 };
let aref = dat.reference_to_a();
let b = dat.get_b();
println!("{:?}, {}", aref, b);
}
Error:
error[E0502]: cannot borrow `dat` as immutable because it is also borrowed as mutable
--> <anon>:19:20
|
18 | let aref = dat.reference_to_a();
| --- mutable borrow occurs here
19 | println!("{}", dat.get_b());
| ^^^ immutable borrow occurs here
20 | }
| - mutable borrow ends here
Why is this? I would have thought that the mutable borrow of dat is converted into an immutable one when reference_to_a() returns, because that function only returns an immutable reference. Is the borrow checker just not clever enough yet? Is this planned? Is there a way around it?
Lifetimes are separate from whether a reference is mutable or not. Working through the code:
fn reference_to_a(&mut self) -> &i32
Although the lifetimes have been elided, this is equivalent to:
fn reference_to_a<'a>(&'a mut self) -> &'a i32
i.e. the input and output lifetimes are the same. That's the only way to assign lifetimes to a function like this (unless it returned an &'static reference to global data), since you can't make up the output lifetime from nothing.
That means that if you keep the return value alive by saving it in a variable, you're keeping the &mut self alive too.
Another way of thinking about it is that the &i32 is a sub-borrow of &mut self, so is only valid until that expires.
As #aSpex points out, this is covered in the nomicon.
Why is this an error: While a more precise explanation was already given by #Chris some 2.5 years ago, you can read fn reference_to_a(&mut self) -> &i32 as a declaration that:
“I want to exclusively borrow self, then return a shared/immutable reference which lives as long as the original exclusive borrow” (source)
Apparently it can even prevent me from shooting myself in the foot.
Is the borrow checker just not clever enough yet? Is this planned?
There's still no way to express "I want to exclusively borrow self for the duration of the call, and return a shared reference with a separate lifetime". It is mentioned in the nomicon as #aSpex pointed out, and is listed among the Things Rust doesn’t let you do as of late 2018.
I couldn't find specific plans to tackle this, as previously other borrow checker improvements were deemed higher priority. The idea about allowing separate read/write "lifetime roles" (Ref2<'r, 'w>) was mentioned in the NLL RFC, but no-one has made it into an RFC of its own, as far as I can see.
Is there a way around it? Not really, but depending on the reason you needed this in the first place, other ways of structuring the code may be appropriate:
You can return a copy/clone instead of the reference
Sometimes you can split a fn(&mut self) -> &T into two, one taking &mut self and another returning &T, as suggested by #Chris here
As is often the case in Rust, rearranging your structs to be "data-oriented" rather than "object-oriented" can help
You can return a shared reference from the method: fn(&mut self) -> (&Self, &T) (from this answer)
You can make the fn take a shared &self reference and use interior mutability (i.e. define the parts of Self that need to be mutated as Cell<T> or RefCell<T>). This may feel like cheating, but it's actually appropriate, e.g. when the reason you need mutability as an implementation detail of a logically-immutable method. After all we're making a method take a &mut self not because it mutates parts of self, but to make it known to the caller so that it's possible to reason about which values can change in a complex program.
This code fails the dreaded borrow checker (playground):
struct Data {
a: i32,
b: i32,
c: i32,
}
impl Data {
fn reference_to_a(&mut self) -> &i32 {
self.c = 1;
&self.a
}
fn get_b(&self) -> i32 {
self.b
}
}
fn main() {
let mut dat = Data{ a: 1, b: 2, c: 3 };
let aref = dat.reference_to_a();
println!("{}", dat.get_b());
}
Since non-lexical lifetimes were implemented, this is required to trigger the error:
fn main() {
let mut dat = Data { a: 1, b: 2, c: 3 };
let aref = dat.reference_to_a();
let b = dat.get_b();
println!("{:?}, {}", aref, b);
}
Error:
error[E0502]: cannot borrow `dat` as immutable because it is also borrowed as mutable
--> <anon>:19:20
|
18 | let aref = dat.reference_to_a();
| --- mutable borrow occurs here
19 | println!("{}", dat.get_b());
| ^^^ immutable borrow occurs here
20 | }
| - mutable borrow ends here
Why is this? I would have thought that the mutable borrow of dat is converted into an immutable one when reference_to_a() returns, because that function only returns an immutable reference. Is the borrow checker just not clever enough yet? Is this planned? Is there a way around it?
Lifetimes are separate from whether a reference is mutable or not. Working through the code:
fn reference_to_a(&mut self) -> &i32
Although the lifetimes have been elided, this is equivalent to:
fn reference_to_a<'a>(&'a mut self) -> &'a i32
i.e. the input and output lifetimes are the same. That's the only way to assign lifetimes to a function like this (unless it returned an &'static reference to global data), since you can't make up the output lifetime from nothing.
That means that if you keep the return value alive by saving it in a variable, you're keeping the &mut self alive too.
Another way of thinking about it is that the &i32 is a sub-borrow of &mut self, so is only valid until that expires.
As #aSpex points out, this is covered in the nomicon.
Why is this an error: While a more precise explanation was already given by #Chris some 2.5 years ago, you can read fn reference_to_a(&mut self) -> &i32 as a declaration that:
“I want to exclusively borrow self, then return a shared/immutable reference which lives as long as the original exclusive borrow” (source)
Apparently it can even prevent me from shooting myself in the foot.
Is the borrow checker just not clever enough yet? Is this planned?
There's still no way to express "I want to exclusively borrow self for the duration of the call, and return a shared reference with a separate lifetime". It is mentioned in the nomicon as #aSpex pointed out, and is listed among the Things Rust doesn’t let you do as of late 2018.
I couldn't find specific plans to tackle this, as previously other borrow checker improvements were deemed higher priority. The idea about allowing separate read/write "lifetime roles" (Ref2<'r, 'w>) was mentioned in the NLL RFC, but no-one has made it into an RFC of its own, as far as I can see.
Is there a way around it? Not really, but depending on the reason you needed this in the first place, other ways of structuring the code may be appropriate:
You can return a copy/clone instead of the reference
Sometimes you can split a fn(&mut self) -> &T into two, one taking &mut self and another returning &T, as suggested by #Chris here
As is often the case in Rust, rearranging your structs to be "data-oriented" rather than "object-oriented" can help
You can return a shared reference from the method: fn(&mut self) -> (&Self, &T) (from this answer)
You can make the fn take a shared &self reference and use interior mutability (i.e. define the parts of Self that need to be mutated as Cell<T> or RefCell<T>). This may feel like cheating, but it's actually appropriate, e.g. when the reason you need mutability as an implementation detail of a logically-immutable method. After all we're making a method take a &mut self not because it mutates parts of self, but to make it known to the caller so that it's possible to reason about which values can change in a complex program.