So I'm pursuing my Rust adventures (loving it) and I'm exploring threads. As usual I stumbled upon an error that I do not understand.
Here is a minimal example:
use std::thread;
pub fn compute_something(input: &Vec<&usize>) -> usize {
input.iter().map(|v| *v).sum()
}
pub fn main() {
let items = vec![0, 1, 2, 3, 4, 5];
let mut slice: Vec<&usize> = Vec::new();
slice.push(&items[1]); // borrowed value does not live long enough
// argument requires that `items` is borrowed for `'static`
slice.push(&items[2]); // borrowed value does not live long enough
// argument requires that `items` is borrowed for `'static`
assert_eq!(3, compute_something(&slice));
let h = thread::spawn(move || compute_something(&slice));
match h.join() {
Ok(result) => println!("Result: {:?}", result),
Err(e) => println!("Nope: {:?}", e)
}
} // `items` dropped here while still borrowed
I have of course made a playground to illustrate.
If I drop the thread part (everything after the assert_eq! line) and just call compute_something(&slice) it compiles fine.
There are three main things I don't understand here:
Why is it a problem to drop items while borrowed at the end of the program, shouldn't the runtime clean-up the memory just fine? It's not like I'm gonna be able to access slice outside of main.
What is still borrowing items at the end of the program? slice? If so, why does that same program compile by just removing everything after the assert_eq! line? I can't see how it changes the borrowing pattern.
Why is calling compute_something from inside the thread's closure creating the issue and how do I solve it?
You move slice into the closure that you pass to thread::spawn(). Since the closure passed to thread::spawn() must be 'static, this implies that the vector being moved into the closure must not borrow anything that isn't 'static either. The compiler therefore deduces the type of slice to be Vec<&'static usize>.
But it does borrow something that's not 'static -- the values that you try to push into it borrow from a variable local to main(), and so the compiler complains about this.
The simplest way to fix this case is to have slice be a Vec<usize> and not borrow from items at all.
Another option is to use scoped threads from the crossbeam crate, which know how to borrow from local variables safely by enforcing that all threads are joined before a scope ends.
To directly answer the questions you posed:
Why is it a problem to drop items while borrowed at the end of the program, shouldn't the runtime clean-up the memory just fine?
When main() terminates, all threads are also terminated -- however, there is a brief window of time during which the values local to main() have been destroyed but before the threads are terminated. There can exist dangling references during this window, and that violates Rust's memory safety model. This is why thread::spawn() requires a 'static closure.
Even though you join the thread yourself, the borrow checker doesn't know that joining the thread ends the borrow. (This is the problem that crossbeam's scoped threads solve.)
What is still borrowing items at the end of the program?
The vector that was moved into the closure is still borrowing items.
Why is calling compute_something from inside the thread's closure creating the issue and how do I solve it?
Calling this function isn't creating the issue. Moving slice into the closure is creating the issue.
Here is the way I solved this issue.
I used Box::Leak: https://doc.rust-lang.org/std/boxed/struct.Box.html#method.leak
let boxed_data = data.into_boxed_slice();
let boxed_data_static_ref = Box::leak(boxed_data);
let compressed_data = &boxed_data_static_ref[start_of_data..start_of_data+zfr.compressed_size as usize];
let handles = (0..NUM_THREADS).map(|thread_nr| {
thread::spawn(move || {
main2(thread_nr, compressed_data, zfr.crc);
})
}).collect::<Vec<_>>();
for h in handles {
h.join().unwrap();
}
Related
Consider this code (Rust Playground):
#[derive(Clone, Copy, Debug)]
struct X(i32);
impl std::ops::AddAssign for X {
fn add_assign(&mut self, rhs: Self) {
self.0 += rhs.0;
}
}
fn main() {
let mut ary_i32 = [1_i32; 2];
ary_i32[0] += ary_i32[1]; // OK
let mut ary_x = [X(1); 2];
ary_x[0] += ary_x[1]; // OK
let mut vec_i32 = vec![1_i32; 2];
vec_i32[0] += vec_i32[1]; // OK
let mut vec_x = vec![X(1); 2];
vec_x[0] += vec_x[1]; // error[E0502]: cannot borrow `vec_x` as immutable because it is also borrowed as mutable
}
Why I get E0502 only on vec_x line?
I could not understand why only the operations for ary_x and vec_i32 are permitted. Does borrow checker treat builtin types (i32, array) specially?
I researched some resources and read MIR of my code, and managed to understand what is going on.
The comment by #trentcl will be the best answer.
I write the details as possible.
For array, Index and IndexMut traits are not used and compiler directly manipulates array elements (you can see this with MIR). So, borrowing problem does not exist here.
Explanating for Vec, rustc guide is useful.
First, Two-phase borrow is not applied to vec_foo[0] += vec_foo[1] statement.
And, the difference between i32 and X is caused by operator lowering.
Basically, statements like vec_user_defined[0] += vec_user_defined[1] are converted to function calls like add_assign(index_mut(...), *index(...)), and function arguments are evaluated from left to right. So, index_mut() borrows x mutably and index() tries to borrow x, and fails.
But for builtin types like i32, compound assignment operator is not converted to function call, and rhs is evaluated before lhs (you can see index() is called before index_mut() with MIR). So, for builtin types, vec_builtin[0] += vec_builtin[1] works.
I know these things from lo48576's article (Japanese).
I considered some workarounds:
Just use an intermediate variable as #sebpuetz said.
Convert Vec to slice as #trentcl said. But this doesn't work well for multidimensional Vec.
Write some macro to automatically introduce an intermediate variable. I found rhs_first_assign crate does such works.
Rust arrays live on the stack, are predictably sized, and therefore have stronger borrow checker guarantees. Vectors are smart pointers on the stack pointing at data that can grow and shrink on the Heap. Because the final example uses the Vector type, the borrow checker considers the entire Vector as a single mutably borrowed object when loading it from the Heap.
As you've observed, the borrow checker can create a mutable reference to a single element to something living on the Stack, whereas it creates a mutable reference to the Vector's smart pointer on the Stack, and then a further mutable reference to the data on the heap. This is why the immutable reference to vec_vec_x[1][1] fails.
As #sebpuetz noted in a comment, you can solve this by first copying an immutable reference to vec_vec_x[1][1], then creating an immutable reference.
I've read What are non-lexical lifetimes?. With the non-lexical borrow checker, the following code compiles:
fn main() {
let mut scores = vec![1, 2, 3];
let score = &scores[0]; // borrows `scores`, but never used
// its lifetime can end here
scores.push(4); // borrows `scores` mutably, and succeeds
}
It seems reasonable in the case above, but when it comes to a mutex lock, we don't want it to be released prematurely.
In the following code, I would like to lock a shared structure first and then execute a closure, mainly to avoid deadlock. However, I'm not sure if the lock will be released prematurely.
use lazy_static::lazy_static; // 1.3.0
use std::sync::Mutex;
struct Something;
lazy_static! {
static ref SHARED: Mutex<Something> = Mutex::new(Something);
}
pub fn lock_and_execute(f: Box<Fn()>) {
let _locked = SHARED.lock(); // `_locked` is never used.
// does its lifetime end here?
f();
}
Does Rust treat locks specially, so that their lifetimes are guaranteed to extend to the end of their scope? Must we use that variable explicitly to avoid premature dropping of the lock, like in the following code?
pub fn lock_and_execute(f: Box<Fn()>) {
let locked = SHARED.lock(); // - lifetime begins
f(); // |
drop(locked); // - lifetime ends
}
There is a misunderstanding here: NLL (non-lexical lifetimes) affects the borrow-checks, not the actual lifetime of the objects.
Rust uses RAII1 extensively, and thus the Drop implementation of a number of objects, such as locks, has side-effects which have to occur at a well-determined and predictable point in the flow of execution.
NLL did NOT change the lifetime of such objects, and therefore their destructor is executed at exactly the same point that it was before: at the end of their lexical scope, in reverse order of creation.
NLL did change the understanding of the compiler of the use of lifetimes for the purpose of borrow-checking. This does not, actually, cause any code change; this is purely analysis. This analysis was made more clever, to better recognize the actual scope in which a reference is used:
Prior to NLL, a reference was considered "in use" from the moment it was created to the moment it was dropped, generally its lexical scope (hence the name).
NLL, instead:
Tries to defer the start of the "in use" span, if possible.
Ends the "in use" span with the last use of the reference.
In the case of a Ref<'a> (from RefCell), the Ref<'a> will be dropped at the end of the lexical scope, at which point it will use the reference to RefCell to decrement the counter.
NLL does not peel away layers of abstractions, so must consider that any object containing a reference (such as Ref<'a>) may access said reference in its Drop implementation. As a result, any object that contains a reference, such as a lock, will force NLL to consider that the "in use" span of the reference extends until they are dropped.
1 Resource Acquisition Is Initialization, whose original meaning is that once a variable constructor has been executed it has acquired the resources it needed and is not in a half-baked state, and which is generally used to mean that the destruction of said variable will release any resources it owned.
Does Rust treat locks specially, so that their lifetimes are guaranteed to extend to the end of their scope?
No. This is the default for every type, and has nothing to do with the borrow checker.
Must we use that variable explicitly to avoid premature dropping of the lock
No.
All you need to do is ensure that the lock guard is bound to a variable. Your example does this (let _lock = ...), so the lock will be dropped at the end of scope. If you had used the _ pattern instead, the lock would have been dropped immediately:
You can prove this for yourself by testing if the lock has indeed been dropped:
pub fn lock_and_execute() {
let shared = Mutex::new(Something);
println!("A");
let _locked = shared.lock().unwrap();
// If `_locked` was dropped, then we can re-lock it:
println!("B");
shared.lock().unwrap();
println!("C");
}
fn main() {
lock_and_execute();
}
This code will deadlock, as the same thread attempts to acquire the lock twice.
You could also attempt to use a method that requires &mut self to see that the immutable borrow is still held by the guard, which has not been dropped:
pub fn lock_and_execute() {
let mut shared = Mutex::new(Something);
println!("A");
let _locked = shared.lock().unwrap();
// If `_locked` was dropped, then we can re-lock it:
println!("B");
shared.get_mut().unwrap();
println!("C");
}
error[E0502]: cannot borrow `shared` as mutable because it is also borrowed as immutable
--> src/main.rs:13:5
|
9 | let _locked = shared.lock().unwrap();
| ------ immutable borrow occurs here
...
13 | shared.get_mut().unwrap();
| ^^^^^^^^^^^^^^^^ mutable borrow occurs here
...
16 | }
| - immutable borrow might be used here, when `_locked` is dropped and runs the `Drop` code for type `std::sync::MutexGuard`
See also:
Where is a MutexGuard if I never assign it to a variable?
How to lock a Rust struct the way a struct is locked in Go?
Why does _ destroy at the end of statement?
In this github discussion you find this code that draws the ire of the borrow checker:
fn main() {
let mut vec = vec!();
match vec.first() {
None => vec.push(5),
Some(v) => unreachable!(),
}
}
I understand why having a mutation while immutable borrows are outstanding is problematic. I assumed that a solution was to explicitly only have one borrow (a mutable one) but it still resulted in my having two borrows, an immutable borrow and then a mutable borrow:
fn main() {
let mut vec: Vec<i32> = vec!();
let r_vec: &mut Vec<i32> = &mut vec;
match r_vec.first() {
None => r_vec.push(5),
Some(v) => unreachable!(),
}
}
The compiler is still not happy:
error[E0502]: cannot borrow `*r_vec` as mutable because it is also borrowed as immutable
--> testrust.rs:7:17
|
6 | match r_vec.first() {
| ----- immutable borrow occurs here
7 | None => r_vec.push(5),
| ^^^^^ mutable borrow occurs here
8 | Some(v) => unreachable!(),
9 | }
| - immutable borrow ends here
Why does my workaround not work, and what is the proper way to get around this issue?
You don't. Well, you "avoid" having multiple borrows by... not having multiple borrows.
fn main() {
let mut vec = vec![];
if vec.first().is_none() {
vec.push(5);
}
}
Even more idiomatically:
if vec.is_empty() {
vec.push(5);
}
In both cases, we borrow vec to make the method call, but terminate that borrow before the body of the if is executed. Compare that to the match where the borrow is made in the match head expression, and then shared with the match arms.
take one mutable borrow that can be used for both cases
That's not how it works. You have to understand how memory plays out and what a reference is. A Vec contains a pointer to memory where the data is stored.
When you get a reference to data the vector, the reference holds the address of the memory for the data, and the compiler ensures there's only one of those allowed to mutate the Vec. When you push a value, that may need to allocate new memory to store all the data. This may invalidate the reference you hold. If that were to occur, then the next time you use the reference, it would point to some other, unrelated piece of memory, your program would crash, your users data would be exposed to security vulnerabilities, etc. etc. etc.
The entire point of the issue you linked and the related pre-RFC is that this code should be able to be determined as safe:
match vec.first() {
None => vec.push(5),
Some(v) => unreachable!(),
}
In this case, the programmer can see that we never use the borrow in the None case, so the the compiler could theoretically end the borrow before executing any of the match arms or otherwise make the two arms disjoint with respect to lifetimes. It does not do that now.
However, in your version of code, it's actually worse. By explicitly taking the borrow and keeping it in a variable, you could be extending how long the borrow needs to stay around, forcing them to overlap.
Currently, the only solution is to reorder your code to artificially constrain borrows. I've not found this very annoying in practice, as usually there's a better organization of code anyway.
See also:
If let borrow conundrum
`if` condition remains borrowed in body
"cannot borrow as mutable more than once at a time" in if-else
Is there a way to release a binding before it goes out of scope?
match + RefCell = X does not live long enough
Any of the 90+ Rust questions that have the error message "mutable because it is also borrowed as immutable"
I suppose this question is about lifetimes in general, but I'm having difficulty with closures specifically because you can't write out their type.
This example is a bit contrived - I'm just starting to learn Rust, and this is something I've been hung up on.
This program won't compile:
fn main () {
let mut list: Vec<&Fn() -> i32> = Vec::new();
{
list.push(&|| 1);
}
}
Because:
src/main.rs:5:25: 5:24 error: borrowed value does not live long enough
src/main.rs:5 list.push(&|| 1);
^~~~
src/main.rs:2:50: 7:2 note: reference must be valid for the block suffix following statement 0 at 2:49...
src/main.rs:2 let mut list: Vec<&Fn() -> i32> = Vec::new();
src/main.rs:3
src/main.rs:4 {
src/main.rs:5 list.push(&move || 1);
src/main.rs:6 }
src/main.rs:7 }
src/main.rs:5:9: 5:26 note: ...but borrowed value is only valid for the statement at 5:8
src/main.rs:5 list.push(&|| 1);
^~~~~~~~~~~~~~~~~
src/main.rs:5:9: 5:26 help: consider using a `let` binding to increase its lifetime
src/main.rs:5 list.push(&|| 1);
^~~~~~~~~~~~~~~~~
What I gather from this error is that the closure's lifetime is limited to the
statement inside the block, but it needs to live for the entire body of main.
I know (or, I think) that passing the closure to push as a reference means that push is only borrowing the closure, and that ownership will be returned to the block. This code would work if I could just give the closure to push (i.e. if push took ownership of the closure), but since a closure isn't sized, I must pass it as a reference.
Is that right? How can I make this code work?
There are two things you are asking about:
specifying a typename for something that has no specifyable typename
letting a closure live longer than the block where it's defined.
The first issue is fixed by NOT specifying the typename, and letting rust's type inference do the work.
let mut list: Vec<_> = Vec::new();
The second issue is fixed by not trying to make the closure live longer, but by making it "by value" so you can move it. This enforces that your closure does not reference anything, but owns all the captured values.
for i in 0..10 {
list.push(move || i);
}
Now this gives us a new problem. If we add a different closure to the Vec, the types won't match. Therefore to achieve that, we need to box the closures.
fn main () {
let mut list: Vec<Box<Fn() -> i32>> = Vec::new();
for i in 0..10 {
list.push(Box::new(move|| i));
}
{
list.push(Box::new(move|| 42));
}
}
Borrows do not own the thing they point to. Your problem is that you're borrowing a temporary which is going to cease to exist right after it's borrowed because you haven't stored it anywhere. If it helps, consider that borrows don't borrow values, they borrow storage, and a temporary has only transient storage.
If you want a borrow to something to last for any given period, you must borrow from storage that will last at least that long. In this case, because you want to store the borrow in a Vec, this means that whatever storage you borrow from must outlive the Vec as well. Thus:
fn main () {
let closure;
let mut list: Vec<&Fn() -> i32> = Vec::new();
{
closure = || 1;
list.push(&closure);
}
}
Note that closure is defined before list is. In Rust, values are dropped in reverse lexical order at the end of their scope, so any variable defined after list will necessarily be dropped before it, thus leading to list containing invalid pointers.
If you want to push multiple closures, you will need a separate variable for each one.
To forestall a possible "my actual problem isn't this simple" addendum (:P): f you need to return list or in some way persist it beyond a single function call, note that there is no way to extend a borrow. In that case, what you need to do is change list to a vector of owned, boxed closures (i.e. Vec<Box<Fn() -> i32>>).
What is my error and how to fix it?
fn get_m() -> Vec<i8> {
vec![1, 2, 3]
}
fn main() {
let mut vals = get_m().iter().peekable();
println!("Saw a {:?}", vals.peek());
}
(playground)
The compiler's error suggests "consider using a let binding" — but I already am:
error[E0597]: borrowed value does not live long enough
--> src/main.rs:6:45
|
6 | let mut vals = get_m().iter().peekable();
| ------- ^ temporary value dropped here while still borrowed
| |
| temporary value created here
7 | println!("Saw a {:?}", vals.peek());
8 | }
| - temporary value needs to live until here
|
= note: consider using a `let` binding to increase its lifetime
This is obviously a newbie question -- though I thought I'd written enough Rust at this point that I had a handle on the borrow checker... apparently I haven't.
This question is similar to Using a `let` binding to increase value lifetime, but doesn't involve breaking down an expression into multiple statements, so I don't think the problem is identical.
The problem is that the Peekable iterator lives to the end of the function, but it holds a reference to the vector returned by get_m, which only lasts as long as the statement containing that call.
There are actually a lot of things going on here, so let's take it step by step:
get_m allocates and returns a vector, of type Vec<i8>.
We make the call .iter(). Surprisingly, Vec<i8> has no iter method, nor does it implement any trait that has one. So there are three sub-steps here:
Any method call checks whether its self value implements the Deref trait, and applies it if necessary. Vec<i8> does implement Deref, so we implicitly call its deref method. However, deref takes its self argument by reference, which means that get_m() is now an rvalue appearing in an lvalue context. In this situation, Rust creates a temporary to hold the value, and passes a reference to that. (Keep an eye on this temporary!)
We call deref, yielding a slice of type &[i8] borrowing the vector's elements.
This slice implements the SliceExt trait, which does have an iter method. Finally! This iter also takes its self argument by reference, and returns a std::slice::Iter holding a reference to the slice.
We make the call .peekable(). As before, std::slice::Iter has no peekable method, but it does implement Iterator; IteratorExt is implemented for every Iterator; and IteratorExt does have a peekable method. This takes its self by value, so the Iter is consumed, and we get a std::iter::Peekable back in return, again holding a reference to the slice.
This Peekable is then bound to the variable vals, which lives to the end of the function.
The temporary holding the original Vec<i8>, to whose elements the Peekable refers, now dies. Oops. This is the borrowed value not living long enough.
But the temporary dies there only because that's the rule for temporaries. If we give it a name, then it lasts as long as its name is in scope:
let vec = get_m();
let mut peekable = vec.iter().peekable();
println!("Saw a {:?}", vals.peek());
I think that's the story. What still confuses me, though, is why that temporary doesn't live longer, even without a name. The Rust reference says, "A temporary's lifetime equals the largest lifetime of any reference that points to it." But that's clearly not the case here.
This is happening because you are trying to run your .iter().peekable() on the actual vector inside of get_m(), which is getting re-referenced by vals.
Basically, you want something like this:
fn get_m() -> Vec<i8> {
vec![1, 2, 3]
}
fn main() {
let vals = get_m();
let mut val = vals.iter().peekable();
println!("Saw a {:?}", val.peek());
}
(Playground)
Result:
Saw a Some(1)