How to declare a closure that lives longer than its enclosing block - rust

I suppose this question is about lifetimes in general, but I'm having difficulty with closures specifically because you can't write out their type.
This example is a bit contrived - I'm just starting to learn Rust, and this is something I've been hung up on.
This program won't compile:
fn main () {
let mut list: Vec<&Fn() -> i32> = Vec::new();
{
list.push(&|| 1);
}
}
Because:
src/main.rs:5:25: 5:24 error: borrowed value does not live long enough
src/main.rs:5 list.push(&|| 1);
^~~~
src/main.rs:2:50: 7:2 note: reference must be valid for the block suffix following statement 0 at 2:49...
src/main.rs:2 let mut list: Vec<&Fn() -> i32> = Vec::new();
src/main.rs:3
src/main.rs:4 {
src/main.rs:5 list.push(&move || 1);
src/main.rs:6 }
src/main.rs:7 }
src/main.rs:5:9: 5:26 note: ...but borrowed value is only valid for the statement at 5:8
src/main.rs:5 list.push(&|| 1);
^~~~~~~~~~~~~~~~~
src/main.rs:5:9: 5:26 help: consider using a `let` binding to increase its lifetime
src/main.rs:5 list.push(&|| 1);
^~~~~~~~~~~~~~~~~
What I gather from this error is that the closure's lifetime is limited to the
statement inside the block, but it needs to live for the entire body of main.
I know (or, I think) that passing the closure to push as a reference means that push is only borrowing the closure, and that ownership will be returned to the block. This code would work if I could just give the closure to push (i.e. if push took ownership of the closure), but since a closure isn't sized, I must pass it as a reference.
Is that right? How can I make this code work?

There are two things you are asking about:
specifying a typename for something that has no specifyable typename
letting a closure live longer than the block where it's defined.
The first issue is fixed by NOT specifying the typename, and letting rust's type inference do the work.
let mut list: Vec<_> = Vec::new();
The second issue is fixed by not trying to make the closure live longer, but by making it "by value" so you can move it. This enforces that your closure does not reference anything, but owns all the captured values.
for i in 0..10 {
list.push(move || i);
}
Now this gives us a new problem. If we add a different closure to the Vec, the types won't match. Therefore to achieve that, we need to box the closures.
fn main () {
let mut list: Vec<Box<Fn() -> i32>> = Vec::new();
for i in 0..10 {
list.push(Box::new(move|| i));
}
{
list.push(Box::new(move|| 42));
}
}

Borrows do not own the thing they point to. Your problem is that you're borrowing a temporary which is going to cease to exist right after it's borrowed because you haven't stored it anywhere. If it helps, consider that borrows don't borrow values, they borrow storage, and a temporary has only transient storage.
If you want a borrow to something to last for any given period, you must borrow from storage that will last at least that long. In this case, because you want to store the borrow in a Vec, this means that whatever storage you borrow from must outlive the Vec as well. Thus:
fn main () {
let closure;
let mut list: Vec<&Fn() -> i32> = Vec::new();
{
closure = || 1;
list.push(&closure);
}
}
Note that closure is defined before list is. In Rust, values are dropped in reverse lexical order at the end of their scope, so any variable defined after list will necessarily be dropped before it, thus leading to list containing invalid pointers.
If you want to push multiple closures, you will need a separate variable for each one.
To forestall a possible "my actual problem isn't this simple" addendum (:P): f you need to return list or in some way persist it beyond a single function call, note that there is no way to extend a borrow. In that case, what you need to do is change list to a vector of owned, boxed closures (i.e. Vec<Box<Fn() -> i32>>).

Related

Borrowed value does not live long enough when used by thread

So I'm pursuing my Rust adventures (loving it) and I'm exploring threads. As usual I stumbled upon an error that I do not understand.
Here is a minimal example:
use std::thread;
pub fn compute_something(input: &Vec<&usize>) -> usize {
input.iter().map(|v| *v).sum()
}
pub fn main() {
let items = vec![0, 1, 2, 3, 4, 5];
let mut slice: Vec<&usize> = Vec::new();
slice.push(&items[1]); // borrowed value does not live long enough
// argument requires that `items` is borrowed for `'static`
slice.push(&items[2]); // borrowed value does not live long enough
// argument requires that `items` is borrowed for `'static`
assert_eq!(3, compute_something(&slice));
let h = thread::spawn(move || compute_something(&slice));
match h.join() {
Ok(result) => println!("Result: {:?}", result),
Err(e) => println!("Nope: {:?}", e)
}
} // `items` dropped here while still borrowed
I have of course made a playground to illustrate.
If I drop the thread part (everything after the assert_eq! line) and just call compute_something(&slice) it compiles fine.
There are three main things I don't understand here:
Why is it a problem to drop items while borrowed at the end of the program, shouldn't the runtime clean-up the memory just fine? It's not like I'm gonna be able to access slice outside of main.
What is still borrowing items at the end of the program? slice? If so, why does that same program compile by just removing everything after the assert_eq! line? I can't see how it changes the borrowing pattern.
Why is calling compute_something from inside the thread's closure creating the issue and how do I solve it?
You move slice into the closure that you pass to thread::spawn(). Since the closure passed to thread::spawn() must be 'static, this implies that the vector being moved into the closure must not borrow anything that isn't 'static either. The compiler therefore deduces the type of slice to be Vec<&'static usize>.
But it does borrow something that's not 'static -- the values that you try to push into it borrow from a variable local to main(), and so the compiler complains about this.
The simplest way to fix this case is to have slice be a Vec<usize> and not borrow from items at all.
Another option is to use scoped threads from the crossbeam crate, which know how to borrow from local variables safely by enforcing that all threads are joined before a scope ends.
To directly answer the questions you posed:
Why is it a problem to drop items while borrowed at the end of the program, shouldn't the runtime clean-up the memory just fine?
When main() terminates, all threads are also terminated -- however, there is a brief window of time during which the values local to main() have been destroyed but before the threads are terminated. There can exist dangling references during this window, and that violates Rust's memory safety model. This is why thread::spawn() requires a 'static closure.
Even though you join the thread yourself, the borrow checker doesn't know that joining the thread ends the borrow. (This is the problem that crossbeam's scoped threads solve.)
What is still borrowing items at the end of the program?
The vector that was moved into the closure is still borrowing items.
Why is calling compute_something from inside the thread's closure creating the issue and how do I solve it?
Calling this function isn't creating the issue. Moving slice into the closure is creating the issue.
Here is the way I solved this issue.
I used Box::Leak: https://doc.rust-lang.org/std/boxed/struct.Box.html#method.leak
let boxed_data = data.into_boxed_slice();
let boxed_data_static_ref = Box::leak(boxed_data);
let compressed_data = &boxed_data_static_ref[start_of_data..start_of_data+zfr.compressed_size as usize];
let handles = (0..NUM_THREADS).map(|thread_nr| {
thread::spawn(move || {
main2(thread_nr, compressed_data, zfr.crc);
})
}).collect::<Vec<_>>();
for h in handles {
h.join().unwrap();
}

When variables move in rust does it do variable shadowing under the hood?

I wanted to know if rust is doing variable shadowing under the hood when variables move from one scope to another.
Since when I move a variable to another function the other function's signature can be changed to make the variable mutable and but when the variable is moved/returned back from other function to main it's not immutable anymore.
I am just curious to what's happening here.
fn main() {
let vec0 = Vec::new(); // immutable by default
let mut vec1 = foreign_func(vec0); // vec0 has now moved
// Is foreign_func returning a mutable vector? Because without `mut` keyword vec1 is immutable by default.
vec1.push(10);
}
fn foreign_func(mut vec: Vec<i32>) -> Vec<i32> {
// argument passed for vec is now mutable in this scope.
// Is this like variable shadowing?
// let test = 10;
// let mut test = 20;
vec.push(20);
vec
}
There's no such thing as an immutable Vec, it's only a variable or a borrow that can be immutable (actually unique/exclusive, but in case of Vec that's the same thing).
What happens in your example is a move: the Vec first moves from main's vec0 into foreign_func's vec, and then back to main, this time to vec1. Some of these bindings are mut and some are not, but that has no bearing on the moved value. In fact, transferring a Vec from an immutable to a mutable variable can be as simple as let mut vec1 = vec0, which compiles just fine.
As pointed out by Peter Hall, shadowing is a different concept: introducing a new variable which intentionally shares the name of an existing one. Since the two variables have the same name, only the most recent one is accessible in its scope. Note that the shadowed variable and the one that shadowed it don't have to have the same type, and that a move may or may not occur when shadowing:
fn shadow1(v: Option<Vec<u32>>) {
// here shadowing moves the old value
let v = v.unwrap_or_else(Vec::new);
}
fn shadow2(v: Vec<u32>) {
// here shadowing not accompanied by move
let v = v.as_slice();
}
When there is no move, it is possible that the shadowed value will again become useful after an operation ends. In that case the shadowing variable may be introduced in a separate scope, so that the old value resurfaces after the inner scope finishes:
fn shadow3(v: Vec<u32>) {
{
let v = v.as_slice();
// temporarily working with a &[u32] slice, Vec is inaccessible
}
// here we have the Vec again
}

Cannot add-assign within `Vec` of user-defined type

Consider this code (Rust Playground):
#[derive(Clone, Copy, Debug)]
struct X(i32);
impl std::ops::AddAssign for X {
fn add_assign(&mut self, rhs: Self) {
self.0 += rhs.0;
}
}
fn main() {
let mut ary_i32 = [1_i32; 2];
ary_i32[0] += ary_i32[1]; // OK
let mut ary_x = [X(1); 2];
ary_x[0] += ary_x[1]; // OK
let mut vec_i32 = vec![1_i32; 2];
vec_i32[0] += vec_i32[1]; // OK
let mut vec_x = vec![X(1); 2];
vec_x[0] += vec_x[1]; // error[E0502]: cannot borrow `vec_x` as immutable because it is also borrowed as mutable
}
Why I get E0502 only on vec_x line?
I could not understand why only the operations for ary_x and vec_i32 are permitted. Does borrow checker treat builtin types (i32, array) specially?
I researched some resources and read MIR of my code, and managed to understand what is going on.
The comment by #trentcl will be the best answer.
I write the details as possible.
For array, Index and IndexMut traits are not used and compiler directly manipulates array elements (you can see this with MIR). So, borrowing problem does not exist here.
Explanating for Vec, rustc guide is useful.
First, Two-phase borrow is not applied to vec_foo[0] += vec_foo[1] statement.
And, the difference between i32 and X is caused by operator lowering.
Basically, statements like vec_user_defined[0] += vec_user_defined[1] are converted to function calls like add_assign(index_mut(...), *index(...)), and function arguments are evaluated from left to right. So, index_mut() borrows x mutably and index() tries to borrow x, and fails.
But for builtin types like i32, compound assignment operator is not converted to function call, and rhs is evaluated before lhs (you can see index() is called before index_mut() with MIR). So, for builtin types, vec_builtin[0] += vec_builtin[1] works.
I know these things from lo48576's article (Japanese).
I considered some workarounds:
Just use an intermediate variable as #sebpuetz said.
Convert Vec to slice as #trentcl said. But this doesn't work well for multidimensional Vec.
Write some macro to automatically introduce an intermediate variable. I found rhs_first_assign crate does such works.
Rust arrays live on the stack, are predictably sized, and therefore have stronger borrow checker guarantees. Vectors are smart pointers on the stack pointing at data that can grow and shrink on the Heap. Because the final example uses the Vector type, the borrow checker considers the entire Vector as a single mutably borrowed object when loading it from the Heap.
As you've observed, the borrow checker can create a mutable reference to a single element to something living on the Stack, whereas it creates a mutable reference to the Vector's smart pointer on the Stack, and then a further mutable reference to the data on the heap. This is why the immutable reference to vec_vec_x[1][1] fails.
As #sebpuetz noted in a comment, you can solve this by first copying an immutable reference to vec_vec_x[1][1], then creating an immutable reference.

Immutable object changing to mutable depending on function signature

Checkout the Rust code below. It compiles
fn main() {
let vec0 = Vec::new();
let mut vec1 = fill_vec(vec0);
println!("{} has length {} content `{:?}`", "vec1", vec1.len(), vec1);
vec1.push(88);
println!("{} has length {} content `{:?}`", "vec1", vec1.len(), vec1);
}
fn fill_vec(mut vec: Vec<i32>) -> Vec<i32> {
vec.push(22);
vec.push(44);
vec.push(66);
vec
}
Here I am declaring vec0 as immutable but fill_vec takes in a mutable vector. Depending on the function signature it seems Rust is changing the nature of the argument being passed.
My question is, this obviously seems like a "shot yourself in the foot" instant. Why does Rust allow this? Or, is this actually safe and I am missing something?
There are different things at play here that can all explain why this behavior make sense:
First of, mut doesn't really mean "mutable". There are such things as interior mutability, Cells, Mutexes, etc., which allow you to modify state without needing a single mut. Rather, mut means that you can get mutually exclusive references.
Second, mutability is a property of a binding. vec0 in main and vec in fill_vec are different bindings, so they can have different mutability.
See also:
What does 'let x = x' do in Rust?
Finally ownership: fill_vec takes full ownership of its parameter, which effectively doesn't exist anymore in main. Why should the function not be allowed to do whatever it wants with its owned parameters? Had the function taken the parameter as a mutable reference, you would have needed to declare the original binding as mut:
fn main() {
let mut vec0 = Vec::new();
// ^^^ now _needs_ a mutable binding
fill_vec(&mut vec0);
// ^^^^ needs an explicit `&mut` reference
}
fn fill_vec(vec: &mut Vec<i32>) {
// ^^^^ borrows rather than take ownership
// …
}
You're making the argument vec of fill_vec mutable. You are still passing the vec by value.
If you wanted a mutable reference you would have vec: &mut Vec<i32>.

Why does the compiler tell me to consider using a `let` binding" when I already am?

What is my error and how to fix it?
fn get_m() -> Vec<i8> {
vec![1, 2, 3]
}
fn main() {
let mut vals = get_m().iter().peekable();
println!("Saw a {:?}", vals.peek());
}
(playground)
The compiler's error suggests "consider using a let binding" — but I already am:
error[E0597]: borrowed value does not live long enough
--> src/main.rs:6:45
|
6 | let mut vals = get_m().iter().peekable();
| ------- ^ temporary value dropped here while still borrowed
| |
| temporary value created here
7 | println!("Saw a {:?}", vals.peek());
8 | }
| - temporary value needs to live until here
|
= note: consider using a `let` binding to increase its lifetime
This is obviously a newbie question -- though I thought I'd written enough Rust at this point that I had a handle on the borrow checker... apparently I haven't.
This question is similar to Using a `let` binding to increase value lifetime, but doesn't involve breaking down an expression into multiple statements, so I don't think the problem is identical.
The problem is that the Peekable iterator lives to the end of the function, but it holds a reference to the vector returned by get_m, which only lasts as long as the statement containing that call.
There are actually a lot of things going on here, so let's take it step by step:
get_m allocates and returns a vector, of type Vec<i8>.
We make the call .iter(). Surprisingly, Vec<i8> has no iter method, nor does it implement any trait that has one. So there are three sub-steps here:
Any method call checks whether its self value implements the Deref trait, and applies it if necessary. Vec<i8> does implement Deref, so we implicitly call its deref method. However, deref takes its self argument by reference, which means that get_m() is now an rvalue appearing in an lvalue context. In this situation, Rust creates a temporary to hold the value, and passes a reference to that. (Keep an eye on this temporary!)
We call deref, yielding a slice of type &[i8] borrowing the vector's elements.
This slice implements the SliceExt trait, which does have an iter method. Finally! This iter also takes its self argument by reference, and returns a std::slice::Iter holding a reference to the slice.
We make the call .peekable(). As before, std::slice::Iter has no peekable method, but it does implement Iterator; IteratorExt is implemented for every Iterator; and IteratorExt does have a peekable method. This takes its self by value, so the Iter is consumed, and we get a std::iter::Peekable back in return, again holding a reference to the slice.
This Peekable is then bound to the variable vals, which lives to the end of the function.
The temporary holding the original Vec<i8>, to whose elements the Peekable refers, now dies. Oops. This is the borrowed value not living long enough.
But the temporary dies there only because that's the rule for temporaries. If we give it a name, then it lasts as long as its name is in scope:
let vec = get_m();
let mut peekable = vec.iter().peekable();
println!("Saw a {:?}", vals.peek());
I think that's the story. What still confuses me, though, is why that temporary doesn't live longer, even without a name. The Rust reference says, "A temporary's lifetime equals the largest lifetime of any reference that points to it." But that's clearly not the case here.
This is happening because you are trying to run your .iter().peekable() on the actual vector inside of get_m(), which is getting re-referenced by vals.
Basically, you want something like this:
fn get_m() -> Vec<i8> {
vec![1, 2, 3]
}
fn main() {
let vals = get_m();
let mut val = vals.iter().peekable();
println!("Saw a {:?}", val.peek());
}
(Playground)
Result:
Saw a Some(1)

Resources