How to get this example to compile without array copying or multiple calls to b() per iteration — b() has to perform some expensive parsing?
This is not the full code that I wrote, but it illustrates the problem I had. Here, Test is attempting to perform some kind of streaming parsing work. c() is the parsing function, it returns Some when parsing was successful. b() is a function that attempts to read more data from the stream when c() can not parse using the available data yet. The returned value is a slice into the self.v containing the parsed range.
struct Test {
v: [u8; 10],
index: u8,
}
impl Test {
fn b(&mut self) {
self.index = 1
}
fn c(i: &[u8]) -> Option<&[u8]> {
Some(i)
}
fn a(&mut self) -> &[u8] {
loop {
self.b();
match Test::c(&self.v) {
Some(r) => return r,
_ => continue,
}
}
}
}
fn main() {
let mut q = Test {
v: [0; 10],
index: 0,
};
q.a();
}
When compiling, it produces the following borrow checker error:
error[E0502]: cannot borrow `*self` as mutable because `self.v` is also
borrowed as immutable
--> <anon>:17:13
|
17 | self.b();
| ^^^^ mutable borrow occurs here
18 |
19 | match Test::c(&self.v) {
| ------ immutable borrow occurs here
...
24 | }
| - immutable borrow ends here
If I change a() to:
fn a(&mut self) -> Option<&[u8]> {
loop {
self.b();
if let None = Test::c(&self.v) {
continue
}
if let Some(r) = Test::c(&self.v) {
return Some(r);
} else {
unreachable!();
}
}
}
Then it runs, but with the obvious drawback of calling the parsing function c() twice.
I kind of understand that changing self while the return value depends on it is unsafe, however, I do not understand why is the immutable borrow for self.v is still alive in the next iteration, when we attempted to call b() again.
Right now "Rustc can't "deal" with conditional borrowing returns". See this comment from Gankro on issue 21906.
It can't assign a correct lifetime to the borrow if only one execution path terminates the loop.
I can suggest this workaround, but I'm not sure it is optimal:
fn c(i: &[u8]) -> Option<(usize, usize)> {
Some((0, i.len()))
}
fn a(&mut self) -> &[u8] {
let parse_result;
loop {
self.b();
match Test::c(&self.v) {
Some(r) => {
parse_result = r;
break;
}
_ => {}
}
}
let (start, end) = parse_result;
&self.v[start..end]
}
You can construct result of parsing using array indexes and convert them into references outside of the loop.
Another option is to resort to unsafe to decouple lifetimes. I am not an expert in safe use of unsafe, so pay attention to comments of others.
fn a(&mut self) -> &[u8] {
loop {
self.b();
match Test::c(&self.v) {
Some(r) => return unsafe{
// should be safe. It decouples lifetime of
// &self.v and lifetime of returned value,
// while lifetime of returned value still
// cannot outlive self
::std::slice::from_raw_parts(r.as_ptr(), r.len())
},
_ => continue,
}
}
}
Related
I am trying to pass around a HashMap which stores values through a set of nested enums/structs. The problem of multiple mutability happens during iteration, even all references should be dropped.
The general idea is to have a vector of values, iterate through them and simplify them, keeping track of them within the HashMap. There are two stages of simplification.
The general flow looks something like
run(Vec<ComplexVal>)
-for each val->
val.fix_complex(holder)
-for each `smp` SimpleVal in val->
basicval = Simplifier::step(smp, holder)
holder.insert("name", basicval)
But the problem is that the holder is borrowed mutably in each stage, and there isn't supposed to be any reference from the ComplexVal to the holder and since the borrowchecker doesn't like multiple borrows, it fails.
Full playground snippet: here
It happens in this snippet:
pub fn run(&mut self, mut vals: Vec<ComplexVal>) {
let mut holder = Holder{hold:HashMap::new()};
// .. setup holder code omitted
let len = vals.len();
for _ in 0..len {
let mut val = vals.remove(0); // remove from vec, should drop after running
println!("Running {:?}", val);
match val {
ComplexVal::Cmplx1(mut c) => {
c.fix_complex(&mut holder)
},
//... more cases of different types of values omitted for simplicity
}
// val *should* be dropped here, and therefore the mutable borrow of holder?
}
println!("Holder: {:?}", holder);
}
}
The only thing I can think of is that it somehow is related to the BasicVal::Ref(&BasicVal) value when created.
I need to return a reference of type &BasicVal so I can't use a regular fn() -> &BasicVal as the reference would be dangling, so I pass a ret value which is to be modified and used as the storage for the return value.
I have also tried just returning the enum BasicVal::Ref(&BasicVal), but run into the same mutability issues.
The example below is a much more simple version which (sort of) demonstrates the same error, just thought I'd include this context in case someone has another idea on how to implement this which wouldn't have these issues
Code (edited)
Updated playground link
Edit: I made a mistake in not needing the lifetimes of both holder and ret to explicitly be the same, so I have made an updated example for it
use std::borrow::BorrowMut;
///////////////////////////////
use std::cell::{RefCell, RefMut};
use std::collections::HashMap;
#[derive(Debug)]
enum BasicVal<'a> {
Ref(&'a BasicVal<'a>),
Val1(BasicStruct),
}
#[derive(Debug)]
struct Holder<'b> {
hold: HashMap<String, RefCell<BasicVal<'b>>>,
}
#[derive(Debug)]
struct BasicStruct {
val: i32,
}
impl<'a> BasicVal<'a> {
pub fn empty() -> Self { BasicVal::Val1(BasicStruct { val: 0 }) }
}
// must match sig of modify_val_ref
fn modify_val<'f>(holder: &'f mut Holder<'f>, mut ret: RefMut<BasicVal<'f>>) {
*ret = BasicVal::Val1(BasicStruct { val: 5 });
}
// must match sig of modify_val
fn modify_val_ref<'f>(holder: &'f mut Holder<'f>, mut ret: RefMut<BasicVal<'f>>) {
ret = holder.hold.get("reference_val").unwrap().borrow_mut();
}
fn do_modify<'f>(holder: &'f mut Holder<'f>) {
let mut v = RefCell::new(BasicVal::empty());
println!("Original {:?}", v);
modify_val(holder, v.borrow_mut());
holder.hold.insert("Data".to_string(), v);
println!("Modified {:?}", holder.hold.get("Data"));
}
pub fn test_dropborrow() {
let mut holder = Holder { hold: HashMap::new() };
holder.hold.insert(
"reference_val".to_string(),
RefCell::new(BasicVal::Val1(BasicStruct { val: 8 })),
);
do_modify(&mut holder);
}
pub fn main() {
test_dropborrow();
}
Edit: Using just the holder for a temp return value gives me a multiple mutable borrow issue, so that workaround doesn't work. I have also tried it with a RefCell with the same issue.
fn modify_val<'f>(holder: &'f mut Holder<'f>) {
holder.hold.insert("$return".to_string(), BasicVal::Val1(BasicStruct{val: 5}));
}
fn do_modify<'f>(holder: &'f mut Holder<'f>) {
modify_val(holder);
let mut v = holder.hold.remove("$return").unwrap();
holder.hold.insert("Data".to_string(), v);
println!("Modified {:?}", v);
}
Error:
935 | fn do_modify<'f>(holder: &'f mut Holder<'f>) {
| -- lifetime `'f` defined here
936 |
937 | modify_val(holder);
| ------------------
| | |
| | first mutable borrow occurs here
| argument requires that `*holder` is borrowed for `'f`
938 | let mut v = holder.hold.remove("$return").unwrap();
| ^^^^^^^^^^^ second mutable borrow occurs here
Any help is greatly appreciated!!!
Figured it out, essentially the BasicVal<'a> was causing Holder to mutably borrow itself in successive iterations of the loop, so removing the lifetime was pretty much the only solution
I'm writing a simple tokenizer in Rust but I'm having trouble. I've simplified the code a bit for the sake of this question:
use std::iter::Peekable;
use std::str::Chars;
struct Example<'a> {
it: Peekable<Chars<'a>>,
}
impl<'a> Example<'a> {
fn tokenize_string(&mut self) {
loop {
match self.it.peek() {
None => break,
Some(_x) => self.it.next(),
};
}
}
}
The error I'm getting is:
error[E0499]: cannot borrow `self.it` as mutable more than once at a time
--> src/main.rs:13:29
|
11 | match self.it.peek() {
| ------- first mutable borrow occurs here
12 | None => break,
13 | Some(_x) => self.it.next(),
| ^^^^^^^ second mutable borrow occurs here
14 | };
| - first borrow ends here
I've been able to work around this by creating a copy of the iterator and calling peek() on that:
fn tokenize_string(&mut self) {
loop {
let mut iterator = self.it.clone();
match iterator.peek() {
None => break,
Some(_x) => self.it.next(),
};
}
}
Is this the best way to do this? It seems a little hack-ish.
Since you're working with str::chars(), and char is Copy, you can dereference to get a char instead of &char. :
fn tokenize_string(&mut self) {
loop {
let r = self.it.peek().cloned();
let n = match r {
Some(_) => self.it.next(),
None => break,
};
// whatever
}
}
If you just want to check if the iterator has returned something, use is_some():
let r = self.it.peek().is_some();
if r { ... } else { ... }
In general, however, I'm not sure if it is possible exactly in this manner without non-lexical lifetimes. You will need to put the code which checks iterator state and the code which works with the iterator based on the state with scopes, something like this:
let r = {
// work with self.it
};
if r { ... } else { ... }
Here any references into self must not escape the lexical block in r, so there is no direct match on a value which contains references into self. There's further examples of working around this in Rust borrow of a HashMap lasts beyond the scope it's in?.
I'm trying to recurse down a structure of nodes, modifying them, and then returning the last Node that I get to. I solved the problems with mutable references in the loop using an example in the non-lexical lifetimes RFC. If I try to return the mutable reference to the last Node, I get a use of moved value error:
#[derive(Debug)]
struct Node {
children: Vec<Node>,
}
impl Node {
fn new(children: Vec<Self>) -> Self {
Self { children }
}
fn get_last(&mut self) -> Option<&mut Node> {
self.children.last_mut()
}
}
fn main() {
let mut root = Node::new(vec![Node::new(vec![])]);
let current = &mut root;
println!("Final: {:?}", get_last(current));
}
fn get_last(mut current: &mut Node) -> &mut Node {
loop {
let temp = current;
println!("{:?}", temp);
match temp.get_last() {
Some(child) => { current = child },
None => break,
}
}
current
}
Gives this error
error[E0382]: use of moved value: `*current`
--> test.rs:51:5
|
40 | let temp = current;
| ---- value moved here
...
51 | current
| ^^^^^^^ value used here after move
|
= note: move occurs because `current` has type `&mut Node`, which does not implement the `Copy` trait
If I return the temporary value instead of breaking, I get the error cannot borrow as mutable more than once.
fn get_last(mut current: &mut Node) -> &mut Node {
loop {
let temp = current;
println!("{:?}", temp);
match temp.get_last() {
Some(child) => { current = child },
None => return temp,
}
}
}
error[E0499]: cannot borrow `*temp` as mutable more than once at a time
--> test.rs:47:28
|
43 | match temp.get_last() {
| ---- first mutable borrow occurs here
...
47 | None => return temp,
| ^^^^ second mutable borrow occurs here
48 | }
49 | }
| - first borrow ends here
How can I iterate through the structure with mutable references and return the last Node? I've searched, but I haven't found any solutions for this specific problem.
I can't use Obtaining a mutable reference by iterating a recursive structure because it gives me a borrowing more than once error:
fn get_last(mut current: &mut Node) -> &mut Node {
loop {
let temp = current;
println!("{:?}", temp);
match temp.get_last() {
Some(child) => current = child,
None => current = temp,
}
}
current
}
This is indeed different from Cannot obtain a mutable reference when iterating a recursive structure: cannot borrow as mutable more than once at a time. If we look at the answer there, modified a bit, we can see that it matches on a value and is able to return the value that was matched on in the terminal case. That is, the return value is an Option:
fn back(&mut self) -> &mut Option<Box<Node>> {
let mut anchor = &mut self.root;
loop {
match {anchor} {
&mut Some(ref mut node) => anchor = &mut node.next,
other => return other, // transferred ownership to here
}
}
}
Your case is complicated by two aspects:
The lack of non-lexical lifetimes.
The fact that you want to take a mutable reference and "give it up" in one case (there are children) and not in the other (no children). This is conceptually the same as this:
fn maybe_identity<T>(_: T) -> Option<T> { None }
fn main() {
let name = String::from("vivian");
match maybe_identity(name) {
Some(x) => println!("{}", x),
None => println!("{}", name),
}
}
The compiler cannot tell that the None case could (very theoretically) continue to use name.
The straight-forward solution is to encode this "get it back" action explicitly. We create an enum that returns the &mut self in the case of no children, a helper method that returns that enum, and rewrite the primary method to use the helper:
enum LastOrNot<'a> {
Last(&'a mut Node),
NotLast(&'a mut Node),
}
impl Node {
fn get_last_or_self(&mut self) -> LastOrNot<'_> {
match self.children.is_empty() {
false => LastOrNot::Last(self.children.last_mut().unwrap()),
true => LastOrNot::NotLast(self),
}
}
fn get_last(mut current: &mut Node) -> &mut Node {
loop {
match { current }.get_last_or_self() {
LastOrNot::Last(child) => current = child,
LastOrNot::NotLast(end) => return end,
}
}
}
}
Note that we are using all of the techniques exposed in both Returning a reference from a HashMap or Vec causes a borrow to last beyond the scope it's in? and Cannot obtain a mutable reference when iterating a recursive structure: cannot borrow as mutable more than once at a time.
With an in-progress reimplementation of NLL, we can simplify get_last_or_self a bit to avoid the boolean:
fn get_last_or_self(&mut self) -> LastOrNot<'_> {
match self.children.last_mut() {
Some(l) => LastOrNot::Last(l),
None => LastOrNot::NotLast(self),
}
}
The final version of Polonius should allow reducing the entire problem to a very simple form:
fn get_last(mut current: &mut Node) -> &mut Node {
while let Some(child) = current.get_last() {
current = child;
}
current
}
See also:
Returning a reference from a HashMap or Vec causes a borrow to last beyond the scope it's in?
Cannot obtain a mutable reference when iterating a recursive structure: cannot borrow as mutable more than once at a time
Why does the call self.f2() in the following code trip the borrow checker? Isn't the else block in a different scope? This is quite a conundrum!
use std::str::Chars;
struct A;
impl A {
fn f2(&mut self) {}
fn f1(&mut self) -> Option<Chars> {
None
}
fn f3(&mut self) {
if let Some(x) = self.f1() {
} else {
self.f2()
}
}
}
fn main() {
let mut a = A;
}
Playground
error[E0499]: cannot borrow `*self` as mutable more than once at a time
--> src/main.rs:16:13
|
13 | if let Some(x) = self.f1() {
| ---- first mutable borrow occurs here
...
16 | self.f2()
| ^^^^ second mutable borrow occurs here
17 | }
| - first borrow ends here
Doesn't the scope of the borrow for self begin and end with the self.f1() call? Once the call from f1() has returned f1() is not using self anymore hence the borrow checker should not have any problem with the second borrow. Note the following code fails too...
// ...
if let Some(x) = self.f1() {
self.f2()
}
// ...
Playground
I think the second borrow should be fine here since f1 and f3 are not using self at the same time as f2.
I put together an example to show off the scoping rules here:
struct Foo {
a: i32,
}
impl Drop for Foo {
fn drop(&mut self) {
println!("Foo: {}", self.a);
}
}
fn generate_temporary(a: i32) -> Option<Foo> {
if a != 0 { Some(Foo { a: a }) } else { None }
}
fn main() {
{
println!("-- 0");
if let Some(foo) = generate_temporary(0) {
println!("Some Foo {}", foo.a);
} else {
println!("None");
}
println!("-- 1");
}
{
println!("-- 0");
if let Some(foo) = generate_temporary(1) {
println!("Some Foo {}", foo.a);
} else {
println!("None");
}
println!("-- 1");
}
{
println!("-- 0");
if let Some(Foo { a: 1 }) = generate_temporary(1) {
println!("Some Foo {}", 1);
} else {
println!("None");
}
println!("-- 1");
}
{
println!("-- 0");
if let Some(Foo { a: 2 }) = generate_temporary(1) {
println!("Some Foo {}", 1);
} else {
println!("None");
}
println!("-- 1");
}
}
This prints:
-- 0
None
-- 1
-- 0
Some Foo 1
Foo: 1
-- 1
-- 0
Some Foo 1
Foo: 1
-- 1
-- 0
None
Foo: 1
-- 1
In short, it seems that the expression in the if clause lives through both the if block and the else block.
On the one hand it is not surprising since it is indeed required to live longer than the if block, but on the other hand it does indeed prevent useful patterns.
If you prefer a visual explanation:
if let pattern = foo() {
if-block
} else {
else-block
}
desugars into:
{
let x = foo();
match x {
pattern => { if-block }
_ => { else-block }
}
}
while you would prefer that it desugars into:
bool bypass = true;
{
let x = foo();
match x {
pattern => { if-block }
_ => { bypass = false; }
}
}
if not bypass {
else-block
}
You are not the first one being tripped by this, so this may be addressed at some point, despite changing the meaning of some code (guards, in particular).
It's annoying, but you can work around this by introducing an inner scope and changing the control flow a bit:
fn f3(&mut self) {
{
if let Some(x) = self.f1() {
// ...
return;
}
}
self.f2()
}
As pointed out in the comments, this works without the extra braces. This is because an if or if...let expression has an implicit scope, and the borrow lasts for this scope:
fn f3(&mut self) {
if let Some(x) = self.f1() {
// ...
return;
}
self.f2()
}
Here's a log of an IRC chat between Sandeep Datta and mbrubeck:
mbrubeck: std:tr::Chars contains a borrowed reference to the string that created it. The full type name is Chars<'a>. So f1(&mut self) -> Option<Chars> without elision is f1(&'a mut self) -> Option<Chars<'a>> which means that self remains borrowed as long as
the return value from f1 is in scope.
Sandeep Datta: Can I use 'b for self and 'a for Chars to avoid this problem?
mbrubeck: Not if you are actually returning an iterator over something from self. Though if you can make a function from &self -> Chars (instead of &mut self -> Chars) that would fix the issue.
As of Rust 2018, available in Rust 1.31, the original code will work as-is. This is because Rust 2018 enables non-lexical lifetimes.
A mutable reference is a very strong guarantee: that there's only one pointer to a particular memory location. Since you've already had one &mut borrow, you can't also have a second. That would introduce a data race in a multithreaded context, and iterator invalidation and other similar issues in a single-threaded context.
Right now, borrows are based on lexical scope, and so the first borrow lasts until the end of the function, period. Eventually, we hope to relax this restriction, but it will take some work.
Here is how you can get rid of the spurious errors. I am new to Rust so there may be serious errors in the following explanation.
use std::str::Chars;
struct A<'a> {
chars: Chars<'a>,
}
The 'a here is a lifetime parameter (just like template parameters in C++). Types can be parameterised by lifetimes in Rust.
The Chars type also takes a lifetime parameter. What this implies is that the Chars type probably has a member element which needs a lifetime parameter. Lifetime parameters only make sense on references (since lifetime here actually means "lifetime of a borrow").
We know that Chars needs to keep a reference to the string from which it was created, 'a will probably be used to denote the source string's lifetime.
Here we simply supply 'a as the lifetime parameter to Chars telling the Rust compiler that the lifetime of Chars is the same as the lifetime of the struct A. IMO "lifetime 'a of type A" should be read as "lifetime 'a of the references contained in the struct A".
I think the struct implementation can be parameterised independently from the struct itself hence we need to repeat the parameters with the impl keyword. Here we bind the name 'a to the lifetime of the struct A.
impl<'a> A<'a> {
The name 'b is introduced in the context of the function f2. Here it is used to bind with the lifetime of the reference &mut self.
fn f2<'b>(&'b mut self) {}
The name 'b is introduced in the context of the function f1.This 'b does not have a direct relationship with the 'b introduced by f2 above.
Here it is used to bind with the lifetime of the reference &mut self. Needless to say this reference also does not have any relationship with the &mut self in the previous function, this is a new independent borrow of self.
Had we not used explicit lifetime annotation here Rust would have used its lifetime elision rules to arrive at the following function signature...
//fn f1<'a>(&'a mut self) -> Option<Chars<'a>>
As you can see this binds the lifetime of the reference &mut self parameter to the lifetime of the Chars object being returned from this function (this Chars object need not be the same as self.chars) this is absurd since the returned Chars will outlive the &mut self reference. Hence we need to separate the two lifetimes as follows...
fn f1<'b>(&'b mut self) -> Option<Chars<'a>> {
self.chars.next();
Remember &mut self is a borrow of self and anything referred to by &mut self is also a borrow. Hence we cannot return Some(self.chars) here. self.chars is not ours to give (Error: cannot move out of borrowed content.).
We need to create a clone of self.chars so that it can be given out.
Some(self.chars.clone())
Note here the returned Chars has the same lifetime as the struct A.
And now here is f3 unchanged and without compilation errors!
fn f3<'b>(&'b mut self) {
if let Some(x) = self.f1() { //This is ok now
} else {
self.f2() //This is also ok now
}
}
The main function just for completeness...
fn main() {
let mut a = A { chars:"abc".chars() };
a.f3();
for c in a.chars {
print!("{}", c);
}
}
I have updated the code the make the lifetime relationships clearer.
I'm writing a simple tokenizer in Rust but I'm having trouble. I've simplified the code a bit for the sake of this question:
use std::iter::Peekable;
use std::str::Chars;
struct Example<'a> {
it: Peekable<Chars<'a>>,
}
impl<'a> Example<'a> {
fn tokenize_string(&mut self) {
loop {
match self.it.peek() {
None => break,
Some(_x) => self.it.next(),
};
}
}
}
The error I'm getting is:
error[E0499]: cannot borrow `self.it` as mutable more than once at a time
--> src/main.rs:13:29
|
11 | match self.it.peek() {
| ------- first mutable borrow occurs here
12 | None => break,
13 | Some(_x) => self.it.next(),
| ^^^^^^^ second mutable borrow occurs here
14 | };
| - first borrow ends here
I've been able to work around this by creating a copy of the iterator and calling peek() on that:
fn tokenize_string(&mut self) {
loop {
let mut iterator = self.it.clone();
match iterator.peek() {
None => break,
Some(_x) => self.it.next(),
};
}
}
Is this the best way to do this? It seems a little hack-ish.
Since you're working with str::chars(), and char is Copy, you can dereference to get a char instead of &char. :
fn tokenize_string(&mut self) {
loop {
let r = self.it.peek().cloned();
let n = match r {
Some(_) => self.it.next(),
None => break,
};
// whatever
}
}
If you just want to check if the iterator has returned something, use is_some():
let r = self.it.peek().is_some();
if r { ... } else { ... }
In general, however, I'm not sure if it is possible exactly in this manner without non-lexical lifetimes. You will need to put the code which checks iterator state and the code which works with the iterator based on the state with scopes, something like this:
let r = {
// work with self.it
};
if r { ... } else { ... }
Here any references into self must not escape the lexical block in r, so there is no direct match on a value which contains references into self. There's further examples of working around this in Rust borrow of a HashMap lasts beyond the scope it's in?.