Borrow-checker behavior in if - else blocks [duplicate] - rust

I'm trying to write a function which pushes an element onto the end of a sorted vector only if the element is larger than the last element already in the vector, otherwise returns an error with a ref to the largest element. This doesn't seem to violate any borrowing rules as far as I cant tell, but the borrow checker doesn't like it. I don't understand why.
struct MyArray<K, V>(Vec<(K, V)>);
impl<K: Ord, V> MyArray<K, V> {
pub fn insert_largest(&mut self, k: K, v: V) -> Result<(), &K> {
{
match self.0.iter().next_back() {
None => (),
Some(&(ref lk, _)) => {
if lk > &k {
return Err(lk);
}
}
};
}
self.0.push((k, v));
Ok(())
}
}
error[E0502]: cannot borrow `self.0` as mutable because it is also borrowed as immutable
--> src/main.rs:15:9
|
6 | match self.0.iter().next_back() {
| ------ immutable borrow occurs here
...
15 | self.0.push((k, v));
| ^^^^^^ mutable borrow occurs here
16 | Ok(())
17 | }
| - immutable borrow ends here
Why doesn't this work?
In response to Paolo Falabella's answer.
We can translate any function with a return statement into one without a return statement as follows:
fn my_func() -> &MyType {
'inner: {
// Do some stuff
return &x;
}
// And some more stuff
}
Into
fn my_func() -> &MyType {
let res;
'outer: {
'inner: {
// Do some stuff
res = &x;
break 'outer;
}
// And some more stuff
}
res
}
From this, it becomes clear that the borrow outlives the scope of 'inner.
Is there any problem with instead using the following rewrite for the purpose of borrow-checking?
fn my_func() -> &MyType {
'outer: {
'inner: {
// Do some stuff
break 'outer;
}
// And some more stuff
}
panic!()
}
Considering that return statements preclude anything from happening afterwards which might otherwise violate the borrowing rules.

If we name lifetimes explicitly, the signature of insert_largest becomes fn insert_largest<'a>(&'a mut self, k: K, v: V) -> Result<(), &'a K>. So, when you create your return type &K, its lifetime will be the same as the &mut self.
And, in fact, you are taking and returning lk from inside self.
The compiler is seeing that the reference to lk escapes the scope of the match (as it is assigned to the return value of the function, so it must outlive the function itself) and it can't let the borrow end when the match is over.
I think you're saying that the compiler should be smarter and realize that the self.0.push can only ever be reached if lk was not returned. But it is not. And I'm not even sure how hard it would be to teach it that sort of analysis, as it's a bit more sophisticated than the way I understand the borrow checker reasons today.
Today, the compiler sees a reference and basically tries to answer one question ("how long does this live?"). When it sees that your return value is lk, it assigns lk the lifetime it expects for the return value from the fn's signature ('a with the explicit name we gave it above) and calls it a day.
So, in short:
should an early return end the mutable borrow on self? No. As said the borrow should extend outside of the function and follow its return value
is the borrow checker a bit too strict in the code that goes from the early return to the end of the function? Yes, I think so. The part after the early return and before the end of the function is only reachable if the function has NOT returned early, so I think you have a point that the borrow checked might be less strict with borrows in that specific area of code
do I think it's feasible/desirable to change the compiler to enable that pattern? I have no clue. The borrow checker is one of the most complex pieces of the Rust compiler and I'm not qualified to give you an answer on that. This seems related to (and might even be a subset of) the discussion on non-lexical borrow scopes, so I encourage you to look into it and possibly contribute if you're interested in this topic.
For the time being I'd suggest just returning a clone instead of a reference, if possible. I assume returning an Err is not the typical case, so performance should not be a particular worry, but I'm not sure how the K:Clone bound might work with the types you're using.
impl <K, V> MyArray<K, V> where K:Clone + Ord { // 1. now K is also Clone
pub fn insert_largest(&mut self, k: K, v: V) ->
Result<(), K> { // 2. returning K (not &K)
match self.0.iter().next_back() {
None => (),
Some(&(ref lk, _)) => {
if lk > &k {
return Err(lk.clone()); // 3. returning a clone
}
}
};
self.0.push((k, v));
Ok(())
}
}

Why does returning early not finish outstanding borrows?
Because the current implementation of the borrow checker is overly conservative.
Your code works as-is once non-lexical lifetimes are enabled, but only with the experimental "Polonius" implementation. Polonius is what enables conditional tracking of borrows.
I've also simplified your code a bit:
#![feature(nll)]
struct MyArray<K, V>(Vec<(K, V)>);
impl<K: Ord, V> MyArray<K, V> {
pub fn insert_largest(&mut self, k: K, v: V) -> Result<(), &K> {
if let Some((lk, _)) = self.0.iter().next_back() {
if lk > &k {
return Err(lk);
}
}
self.0.push((k, v));
Ok(())
}
}

Related

Taking ownership of a &String without copying

Context
Link: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=9a9ffa99023735f4fbedec09e1c7ac55
Here's a contrived repro of what I'm running into
fn main() {
let mut s = String::from("Hello World");
example(&mut s);
}
fn example(s: &mut str) -> Option<String> {
other_func(Some(s.to_owned()))
// other random mutable stuff happens
}
fn other_func(s: Option<String>) {
match s {
Some(ref s) => other_func2(*s),
None => panic!()
}
}
fn other_func2(s: String) {
println!("{}", &s)
}
and the error
Compiling playground v0.0.1 (/playground)
error[E0507]: cannot move out of `*s` which is behind a shared reference
--> src/main.rs:12:36
|
12 | Some(ref s) => other_func2(*s),
| ^^ move occurs because `*s` has type `String`, which does not implement the `Copy` trait
Question
In the following code, why can't I deference the &String without having to do some sort of clone/copy? i.e. this doesn't work
fn other_func(s: Option<String>) {
match s {
Some(ref s) => other_func2(*s),
None => panic!()
}
}
but it works if I replace *s with s.to_owned()/s.to_string()/s.clone()
As an aside, I understand this can probably be solved by refactoring to use &str, but I'm specifically interested in turning &String -> String
Why would the compiler allow you to?
s is &String. And you cannot get a String from a &String without cloning. That's obvious.
And the fact that it was created from an owned String? The compiler doesn't care, and it is right. This is not different from the following code:
let s: String = ...;
let r: &String = ...;
let s2: String = *r; // Error
Which is in turn not different from the following code, for instance, as far as the compiler is concerned:
let r: &String = ...;
let s: String = *s;
And we no longer have an owned string at the beginning. In general, the compiler doesn't track data flow. And rightfully so - when it type-checks the move it doesn't even can confirm that this reference isn't aliased. Or that the owned value is not used anymore. References are just references, they give you no right to drop the value.
Changing that will not be feasible in the general case (for example, the compiler will have to track data flow across function calls), and will require some form of manual annotation to say "this value is mine". And you already have such annotation - use an owned value, String, instead of &String: this is exactly what it's about.

How to access mutable iterables inside a nested loop

Every now and then, I run into the same issue with borrowing (or not borrowing) mutable variables inside a loop, and I finally sat down and compiled a minimal example. As a result, the code is a little silly, but it is the shortest version that I could come up with that highlights the problem:
struct Association {
used: bool,
key: usize,
value: String,
}
impl Association {
fn new(key: usize, value: &str) -> Self {
Association{used: false, key: key, value: value.to_string()}
}
}
fn find_unused<'a>(data: &'a mut Vec<Association>) -> Option<&'a String> {
for k in 0.. {
for a in data {
if a.key == k && !a.used {
a.used = true;
return Some(&a.value);
}
}
}
None
}
fn main() {
let mut assoc = vec![
Association::new(7, "Hello"),
Association::new(9, "World")
];
println!("{}", find_unused(&mut assoc).unwrap());
println!("{}", find_unused(&mut assoc).unwrap());
}
This will fail with an error because data was moved before. If I borrow it instead, it will fail because it was borrowed before. I would like to understand exactly what is happening and how to solve it in general. In particular, I do not want to change the structure of the code, even if it is silly. I do not want to implement a workaround, because this is just a minimal example: Please assume that the nesting of the loops is the "right" way to do it, even if it is utterly silly here, which it definitely is.
I would only like to know how to communicate to the borrow checker that what is happening here is actually ok. I know of one way to do this:
fn find_unused<'a>(data: &'a mut Vec<Association>) -> Option<&'a String> {
for k in 0.. {
for j in 0..data.len() {
if data[j].key == k && !data[j].used {
data[j].used = true;
return Some(&data[j].value);
}
}
}
None
}
This compiles without error and works as intended. In my naïve understanding, there should be a way to express the above with iterators instead of indexing, and I would like to know how that would be done.
The behavior you want is possible if you:
make fresh mutable borrows in each nested loop
And wait for rustc to integrate more Polonius capabilities. (Can experiment now with cargo +nightly rustc -- -Z polonius). While you wait, you could use unsafe like in this post: Borrow checker complains about multiple borrowing, when only one will ever occur
Keeping the rest of your code the same and focussing on this part, but with the return commented out:
fn find_unused<'a>(data: &'a mut Vec<Association>) -> Option<&'a String> {
for k in 0.. {
for a in data { // Error - but rustc says what to do here
if a.key == k && !a.used {
a.used = true;
// return Some(&a.value);
}
}
}
None
}
The error messages from rustc say what to do and why. The mutable borrow in the parameters is moved (used up) by the call to into_iter() in for a in data.
rustc recommends a mutable re-borrow. This way we don't try to borrow something that has already been moved. Making that change (and keeping the return commented-out for now), the following now type-checks:
fn find_unused<'a>(data: &'a mut Vec<Association>) -> Option<&'a String> {
for k in 0.. {
for a in &mut *data { // we took rustc's suggestion
if a.key == k && !a.used {
a.used = true;
// return Some(&a.value);
}
}
}
None
}
If we un-comment the return, we get an error saying that the lifetime of the value we're returning doesn't match 'a.
But if we run with cargo +nightly rustc -- -Z polonius, the code type-checks.
#stargateur suggested trying polonius in a comment
Here's my guess as to why Polonius helps:
For the current rustc borrow-checker, lifetimes in function signatures are treated in a relatively coarse and simplistic way within the body of the function. I think what's happening here is that 'a is supposed to span the entire function body, but the fresh mutable borrow for a only spans part of the function body, so Rust doesn't unify them.
Polonius tracks origins of borrows in types. So it knows that the lifetime of &a.value comes from a, which comes from data, which has lifetime 'a, so Polonius knows the return is OK.
I'm basing this on The Polonius Talk but The Book is probably more up-to-date

How to fix ".. was mutably borrowed here in the previous iteration of the loop" in Rust?

I have to iterate on keys, find the value in HashMap by key, possibly do some heavy computation in the found struct as a value (lazy => mutate the struct) and cached return it in Rust.
I'm getting the following error message:
error[E0499]: cannot borrow `*self` as mutable more than once at a time
--> src/main.rs:25:26
|
23 | fn it(&mut self) -> Option<&Box<Calculation>> {
| - let's call the lifetime of this reference `'1`
24 | for key in vec!["1","2","3"] {
25 | let result = self.find(&key.to_owned());
| ^^^^ `*self` was mutably borrowed here in the previous iteration of the loop
...
28 | return result
| ------ returning this value requires that `*self` is borrowed for `'1`
Here is the code in playground.
use std::collections::HashMap;
struct Calculation {
value: Option<i32>
}
struct Struct {
items: HashMap<String, Box<Calculation>> // cache
}
impl Struct {
fn find(&mut self, key: &String) -> Option<&Box<Calculation>> {
None // find, create, and/or calculate items
}
fn it(&mut self) -> Option<&Box<Calculation>> {
for key in vec!["1","2","3"] {
let result = self.find(&key.to_owned());
if result.is_some() {
return result
}
}
None
}
}
I can't avoid the loop as I have to check multiple keys
I have to make it mutable (self and the structure) as the possible calculation changes it
Any suggestion on how to change the design (as Rust forces to think in a bit different way that makes sense) or work around it?
PS. There are some other issues with the code, but let's split the problems and solve this one first.
You can't do caching with exclusive access. You can't treat Rust references like general-purpose pointers (BTW: &String and &Box<T> are double indirection, and very unidiomatic in Rust. Use &str or &T for temporary borrows).
&mut self means not just mutable, but exclusive and mutable, so your cache supports returning only one item, because the reference it returns has to keep self "locked" for as long as it exists.
You need to convince the borrow checker that the thing that find returns won't suddenly disappear next time you call it. Currently there's no such guarantee, because the interface doesn't stop you from calling e.g. items.clear() (borrow checker checks what function's interface allows, not what function actually does).
You can do that either by using Rc, or using a crate that implements a memory pool/arena.
struct Struct {
items: HashMap<String, Rc<Calculation>>,
}
fn find(&mut self, key: &str) -> Rc<Calculation>
This way if you clone the Rc, it will live for as long as it needs, independently of the cache.
You can also make it nicer with interior mutability.
struct Struct {
items: RefCell<HashMap<…
}
This will allow your memoizing find method to use a shared borrow instead of an exclusive one:
fn find(&self, key: &str) -> …
which is much easier to work with for the callers of the method.
Probably not the cleanest way to do that, but it compiles. The idea is not to store the value found in a temporary result, to avoid aliasing: if you store the result, self is kept borrowed.
impl Struct {
fn find(&mut self, key: &String) -> Option<&Box<Calculation>> {
None
}
fn it(&mut self) -> Option<&Box<Calculation>> {
for key in vec!["1","2","3"] {
if self.find(&key.to_owned()).is_some() {
return self.find(&key.to_owned());
}
}
None
}
}
I had similar issues, and I found a workaround by turning the for loop into a fold, which convinced the compiler that self was not mutably borrowed twice.
It works without using interior mutability or duplicated function call; the only downside is that if the result was found early, it will not short-circuit but continue iterating until the end.
Before:
for key in vec!["1","2","3"] {
let result = self.find(&key.to_owned());
if result.is_some() {
return result
}
}
After:
vec!["1", "2,", "3"]
.iter()
.fold(None, |result, key| match result {
Some(result) => Some(result),
None => self.find(&key.to_string())
})
Working playground link: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=92bc73e4bac556ce163e0790c7d3f154

Why do some while-let assignments inside a loop fail to compile with "cannot borrow as mutable more than once at a time"? [duplicate]

Suppose I have several structures like in the following example, and in the next() method I need to pull the next event using a user-provided buffer, but if this event is a comment, and ignore comments flag is set to true, I need to pull the next event again:
struct Parser {
ignore_comments: bool,
}
enum XmlEvent<'buf> {
Comment(&'buf str),
Other(&'buf str),
}
impl Parser {
fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
let result = loop {
buffer.clear();
let temp_event = self.parse_outside_tag(buffer);
match temp_event {
XmlEvent::Comment(_) if self.ignore_comments => {}
_ => break temp_event,
}
};
result
}
fn parse_outside_tag<'buf>(&mut self, _buffer: &'buf mut String) -> XmlEvent<'buf> {
unimplemented!()
}
}
This code, however, gives a double borrow error, even when I have #![feature(nll)] enabled:
error[E0499]: cannot borrow `*buffer` as mutable more than once at a time
--> src/main.rs:14:13
|
14 | buffer.clear();
| ^^^^^^ second mutable borrow occurs here
15 |
16 | let temp_event = self.parse_outside_tag(buffer);
| ------ first mutable borrow occurs here
|
note: borrowed value must be valid for the lifetime 'buf as defined on the method body at 12:5...
--> src/main.rs:12:5
|
12 | fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E0499]: cannot borrow `*buffer` as mutable more than once at a time
--> src/main.rs:16:53
|
16 | let temp_event = self.parse_outside_tag(buffer);
| ^^^^^^ mutable borrow starts here in previous iteration of loop
|
note: borrowed value must be valid for the lifetime 'buf as defined on the method body at 12:5...
--> src/main.rs:12:5
|
12 | fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: aborting due to 2 previous errors
I can (approximately at least) understand why an error could happen here with the NLL feature turned off, but I don't understand why it happens with NLL.
Anyway, my end goal is to implement this without flags, so I also tried doing this (it is recursive, which is really unfortunate, but all non-recursive versions I came up with could not possibly work without NLL):
fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
buffer.clear();
{
let temp_event = self.parse_outside_tag(buffer);
match temp_event {
XmlEvent::Comment(_) if self.ignore_comments => {}
_ => return temp_event,
}
}
self.next(buffer)
}
Here I tried to confine the borrow inside a lexical block, and nothing from this block leaks to the outside. However, I'm still getting an error:
error[E0499]: cannot borrow `*buffer` as mutable more than once at a time
--> src/main.rs:23:19
|
15 | let temp_event = self.parse_outside_tag(buffer);
| ------ first mutable borrow occurs here
...
23 | self.next(buffer)
| ^^^^^^ second mutable borrow occurs here
24 | }
| - first borrow ends here
error: aborting due to previous error
And again, NLL does not fix it.
It has been a long time since I encountered a borrow checking error which I don't understand, so I'm hoping it is actually something simple which I'm overlooking for some reason :)
I really suspect that the root cause is somehow connected with the explicit 'buf lifetime (in particular, errors with the NLL flag turned on have these notes about it), but I can't understand what exactly is wrong here.
This is a limitation of the current implementation of non-lexical lifetimes This can be shown with this reduced case:
fn next<'buf>(buffer: &'buf mut String) -> &'buf str {
loop {
let event = parse(buffer);
if true {
return event;
}
}
}
fn parse<'buf>(_buffer: &'buf mut String) -> &'buf str {
unimplemented!()
}
fn main() {}
This limitation prevents NLL case #3: conditional control flow across functions
In compiler developer terms, the current implementation of non-lexical lifetimes is "location insensitive". Location sensitivity was originally available but it was disabled in the name of performance.
I asked Niko Matsakis about this code:
In the context of your example: the value event only has to have the lifetime 'buf conditionally — at the return point which may or may not execute. But when we are "location insensitive", we just track the lifetime that event must have anywhere, without considering where that lifetime must hold. In this case, that means we make it hold everywhere, which is why you get a compilation failure.
One subtle thing is that the current analysis is location sensitive in one respect — where the borrow takes place. The length of the borrow is not.
The good news is that adding this concept of location sensitivity back is seen as an enhancement to the implementation of non-lexical lifetimes. The bad news:
That may or may not be before the [Rust 2018] edition.
(Note: it did not make it into the initial release of Rust 2018)
This hinges on a (even newer!) underlying implementation of non-lexical lifetimes that improves the performance. You can opt-in to this half-implemented version using -Z polonius:
rustc +nightly -Zpolonius --edition=2018 example.rs
RUSTFLAGS="-Zpolonius" cargo +nightly build
Because this is across functions, you can sometimes work around this by inlining the function.
I posted a question (A borrow checker problem with a loop and non-lexical lifetimes) that was answered by the answer of this question.
I'll document here a workaround that also answers the question. Let's say you have code like this, that only compiles with Polonius:
struct Inner;
enum State<'a> {
One,
Two(&'a ()),
}
fn get<'s>(_inner: &'s mut Inner) -> State<'s> {
unimplemented!()
}
struct Outer {
inner: Inner,
}
impl Outer {
pub fn read<'s>(&'s mut self) -> &'s () {
loop {
match get(&mut self.inner) {
State::One => (), // In this case nothing happens, the borrow should end and the loop should continue
State::Two(a) => return a, // self.inner ought to be borrowed for 's, that's just to be expected
}
}
}
}
As it was said in the another answer:
One subtle thing is that the current analysis is location sensitive in one respect — where the borrow takes place. The length of the borrow is not.
Indeed, borrowing the needed reference again inside the conditional branch makes it compile! Of course, this makes the assumption that get is referentially transparent, so your mileage may vary, but borrowing again seems like an easy enough workaround.
struct Inner;
enum State<'a> {
One,
Two(&'a ()),
}
fn get<'s>(_inner: &'s mut Inner) -> State<'s> {
unimplemented!()
}
struct Outer {
inner: Inner,
}
impl Outer {
pub fn read<'s>(&'s mut self) -> &'s () {
loop {
match get(&mut self.inner) {
State::One => (), // In this case nothing happens, the borrow should end and the loop should continue
State::Two(a) => {
return match get(&mut self.inner) { // Borrowing again compiles!
State::Two(a) => a,
_ => unreachable!(),
}
}, // self.inner ought to be borrowed for 's, that's just to be expected
}
}
}
}
fn main() {
println!("Hello, world!");
}

Why can I not borrow a variable as mutable more than once at a time with a &mut Box<T> while &mut T works?

I'm trying to implement a linked list in Rust and I'm having some trouble understanding the difference between these two functions:
enum List<T> {
Nil,
Cons(T, Box<List<T>>)
}
fn foo<T>(list: &mut Box<List<T>>) {
match **list {
List::Nil => return,
List::Cons(ref mut head, ref mut tail) => {
// ...
}
}
}
fn bar<T>(list: &mut List<T>) {
match *list {
List::Nil => return,
List::Cons(ref mut head, ref mut tail) => {
// ...
}
}
}
foo fails to compile, with the following error:
error[E0499]: cannot borrow `list` (via `list.1`) as mutable more than once at a time
--> src/main.rs:66:34
|
66 | List::Cons(ref mut head, ref mut rest) => {
| ------------ ^^^^^^^^^^^^ second mutable borrow occurs here (via `list.1`)
| |
| first mutable borrow occurs here (via `list.0`)
...
69 | }
| - first borrow ends here
However, bar compiles and runs perfectly. Why does bar work, but not foo? I am using Rust version 1.25.
This can be simplified to
fn foo(v: &mut Box<(i32, i32)>) {
match **v {
(ref mut head, ref mut tail) => {}
}
}
or
fn foo(v: &mut Box<(i32, i32)>) {
let (ref mut head, ref mut tail) = **v;
}
The problem is that Box is a a strange, in-between type.
Way back in Rust's history, Box was special-cased by the compiler; it knew a lot of the details of Box, but this meant that it was "magic" and no one else could implement something that worked like Box.
RFC 130 proposed changing that; making Box "just another type". Unfortunately, this still hasn't been fully transitioned.
The details are nuanced, but basically the current borrow checker handles pattern-matching syntactically, not semantically. It needs to do this to prevent some unsoundness issues.
In the future, non-lexical lifetimes (NLL) just magically fix this; you don't have to to anything (hooray!).
Until then, you can explicitly get back to a &mut T with this ugly blob:
match *&mut **list {
Or call DerefMut explicitly:
match *std::ops::DerefMut::deref_mut(list) {
However, there's very little reason to accept a &mut Box<T>.
See also:
Destructuring boxes into multiple mutable references seems broken #30104
Bad / misleading error message with auto deref and mutable borrows of multiple fields #32930
Why can I not borrow a boxed vector content as mutable?
Confused by move semantics of struct fields inside a Box
Moving out of boxed tuple

Resources