I am playing around with multithreaded Rust after learning about it in C++. However Rust's borrowing seems to make it difficult to have an equivalent to waitpid and returning values from child threads.
My current implementation uses crossbeam:
let half = array.len() / 2;
let whole = array.len();
let mut left_half = array[0 .. half].to_vec();
let mut right_half = array[half .. whole].to_vec();
crossbeam::scope(|scope| {
parts.push(scope.spawn(move || i32_merge_sort(&mut left_half)));
parts.push(scope.spawn(move || i32_merge_sort(&mut right_half)));
});
I get the following error due to the closure and the values in the spawned threads not being waited for properly:
error[E0382]: use of moved value: `left_half`
--> src\main.rs:39:22
|
29 | crossbeam::scope(|scope| {
| ------- value moved (into closure) here
...
39 | while left < left_half.len() && right < right_half.len() {
| ^^^^^^^^^ value used here after move
|
= note: move occurs because `left_half` has type `std::vec::Vec<i32>`, which does not implement the `Copy` trait
I have not been able to find a concise explanation on how to do this sort of multi-threading, which I for one don't think is a strange use case.
I assume you have code after the scope call, which is operating on the vector. Unfortunately, the scope is taking ownership of the vector, so it can't.
You can fix it by rebinding it as a mutable reference, which will be released at the end of the scope normally:
let mut left_half = array[0 .. half].to_vec();
let mut right_half = array[half .. whole].to_vec();
let left_half = &mut left_half;
let right_half = &mut right_half;
crossbeam::scope(|scope| {
parts.push(scope.spawn(move || i32_merge_sort(left_half)));
parts.push(scope.spawn(move || i32_merge_sort(right_half)));
});
Related
Let's say I'm using a nested closure to modify a local variable like so:
let mut i = 0;
let x = (0..).flat_map(|_| {
(0..).map(|_| {
let x = i;
i += 1;
x
})
});
This does not compile with the error:
error: captured variable cannot escape `FnMut` closure body
--> src/main.rs:35:9
|
32 | let mut i = 0;
| ----- variable defined here
33 |
34 | let x = (0..).flat_map(|_| {
| - inferred to be a `FnMut` closure
35 | / (0..).map(|_| {
36 | | let x = i;
| | - variable captured here
37 | | i += 1;
38 | | x
39 | | })
| |__________^ returns a reference to a captured variable which escapes the closure body
|
= note: `FnMut` closures only have access to their captured variables while they are executing...
= note: ...therefore, they cannot allow references to captured variables to escape
I experimented with the same code on a non-nested context, and it compiles without an error:
let mut i = 0;
let x = (0..).map(|_| {
let x = i;
i += 1;
x
});
So I guess the error comes from the fact that the closure is nested, but I can't fully figure out why the error is being triggered.
Let's strip out all of the iterator parts and consider just the closures that are being constructed.
let mut i = 0;
let f = || {
|| {
let x = i;
i += 1;
x
}
};
i is a variable being used by the inner closure, and therefore borrowed by it. Now, think about what happens if I run this program:
let c1 = f();
let c2 = f();
[c1(), c2()]
Both c1 and c2 are closures that capture i and mutate it. Therefore, having both of them allows shared mutable state without any synchronization/mutual-exclusion mechanism, which is always prohibited in Rust. Compiling the above code will produce the same error as you had.
Iterator::flat_map() won't actually call the function and keep two of it around like this, but its signature can't communicate that fact to the compiler, so the compiler has to assume the worst could happen.
Now that we know that it would be unsound to permit this pattern, why is the error described the way you saw?
When you refer to a variable in a closure, this (most often) becomes a borrow of that variable. In other words, calling || { i += 1; } constructs a value that contains an &mut i32 pointing to i. This means that || { i += 1; } itself is an expression that needs to be able to mutably borrow i, so a closure which contains || { i += 1; } itself requires a mutable borrow of i.
Whenever a closure contains a mutable borrow (or some other cases), that means that invoking the closure itself requires &mut self (on the same general principle that you can't ever mutate an &mut by way of an &). That's why the closures are inferred to be FnMuts (the type of function that requires &mut self to invoke) and the compiler is telling you about that.
Now, why do FnMut closures “only have access to their captured variables while they are executing”? I'm not sure how to explain this exact description of the problem, but I'm sure that it's fundamentally about the requirement of &mut being unique, and that if it were permitted there would end up being some way to modify the function's state while its result was still borrowed.
Finally, how do you solve your problem? Well, in simple cases, the answer is usually to make the closure a move || { closure. That means there is no longer a borrow, and the closure now owns the i counter, so it can live as long as the closure does. But in your case, this doesn't help, because you're actually trying (I assume) to have a counter which is shared between everything. Therefore you need an interior mutability tool. The simplest one for this situation is Cell, which allows reading and writing a value (as long as it's Copy) without ever borrowing it.
use std::cell::Cell;
let i = Cell::new(0);
let x = (0..).flat_map(|_| {
(0..).map(|_| {
let x: i32 = i.get();
i.set(x + 1);
x
})
});
I'm new to Rust. I'm trying to solve a problem using recursion. Within the recursive function, I clone the state of the game, then take each pair of elements from a vector and mutate them using try_add_to(), then check if the game is solved.
fn try_find_solution(game: &GameState) -> Option<GameState> {
let mut cloned_game = game.clone();
for split_index in 1..cloned_game.cups.len() - 1 {
// Need two mutable references from the vector. Use split_at_mut() to allow this.
let (a, b) = cloned_game.cups.split_at_mut(split_index);
let first_cup = a.last_mut().unwrap();
for second_cup in b.iter_mut() {
if first_cup.try_add_to(second_cup) || second_cup.try_add_to(first_cup) {
if cloned_game.is_solved() {
return Some(cloned_game);
}
else {
// let solution = try_find_solution(&cloned_game);
// #TODO.
}
}
}
}
None
}
I'm running into the following error.
error[E0502]: cannot borrow `cloned_game` as immutable because it is also borrowed as mutable
--> src/main.rs:31:20
|
26 | let (a, b) = cloned_game.cups.split_at_mut(split_index);
| ---------------- mutable borrow occurs here
...
30 | if first_cup.try_add_to(second_cup) || second_cup.try_add_to(first_cup) {
| --------- mutable borrow later used here
31 | if cloned_game.is_solved() {
| ^^^^^^^^^^^ immutable borrow occurs here
I think I understand why this error is occuring, but don't know how to fix it without a complete redesign of the solution. Is there anything I can do?
If this is your attempt at brute forcing something, I might not use recursion. You may hit the recursion limit for anything but the most trivial solutions. Also, you're going to have to do an excessive amount of cloning as each stack makes its own clone of the game state.
As for your original problem, the best way is to avoid having to hold these references at the same time.
For example, this won't compile:
fn hello_world(s: &mut [u8]){
let (a,b) = s.split_at_mut(1);
let c = s.len();
drop((a,b,c));
}
...but this does:
fn hello_world(s: &mut [u8]){
let (a,b) = s.split_at_mut(1);
drop((a,b));
let c = s.len();
drop(c);
}
...because we're not forcibly holding a and b to the end. If you can extract the .is_solved() call to outside the loop, that should work.
I have two versions of a function that are intended to do the same thing.
Version 1 - Works!
pub fn example1() {
// Make a mutable slice
let mut v = [0, 1, 2, 3];
// Make a mutable reference to said slice
let mut v_ref = &mut v[..];
let len = v_ref.len();
// Reduce slice to sub-slice -> np ;)
v_ref = {
// Create sub-slices
let (v_l, v_r) = {
// Involves some edits -> need mut
v_ref.swap(0, len - 1);
{ v_ref }.split_at_mut(len / 2)
};
// Choose slice with which to overwrite
// (involves some non-trivial condition here)
match len % 2 {
0 => v_l,
_ => v_r,
}
};
// And then we do something with v_ref
println!("{:?}", v_ref);
}
Essentially:
We start with a mutable variable, mut v_ref: &mut [i32], containing a mutable reference to a slice
We make two sub-slices from v_ref using split_at_mut*
The variable v_ref is overwritten to hold one of the sub-slices
*(Note - We avoid the problem of having two mutable references by moving v_ref, as opposed to reborrowing, by using an identity block)
(Regarding the intent of the code - this slice reduction is intended to be repeated in a loop; however this detail does not affect the problem)
Version 2 - Fails to compile
Version 2 is nearly identical to version 1, except that the sub-slice creation is moved to its own function:
fn example2_inner(v_ref: &mut [i32]) -> (&mut [i32], &mut [i32]) {
// Recreate len variable in function scope
let len = v_ref.len();
// Same mutation here
v_ref.swap(0, len - 1);
// Same slice split here
v_ref.split_at_mut(len / 2)
}
pub fn example2() {
let mut v = [0, 1, 2, 3];
let mut v_ref = &mut v[..];
let len = v_ref.len();
// This now fails to compile :(
v_ref = {
// Mutating & slice spliting moved to function
let (v_l, v_r) = example2_inner({ v_ref });
match len % 2 {
0 => v_l,
_ => v_r,
}
};
println!("{:?}", v_ref);
}
When I try to build this, I get the following errors:
error[E0506]: cannot assign to `v_ref` because it is borrowed
--> src/lib.rs:19:5
|
19 | / v_ref = {
20 | | // Mutating & slice spliting moved to function
21 | | let (v_l, v_r) = example2_inner({ v_ref });
| | ----- borrow of `v_ref` occurs here
22 | |
... |
26 | | }
27 | | };
| |_____^ assignment to borrowed `v_ref` occurs here
error[E0502]: cannot borrow `v_ref` as immutable because `*v_ref` is also borrowed as mutable
--> src/lib.rs:29:22
|
21 | let (v_l, v_r) = example2_inner({ v_ref });
| ----- mutable borrow occurs here
...
29 | println!("{:?}", v_ref);
| ^^^^^ immutable borrow occurs here
30 | }
| - mutable borrow ends here
Questions
Why don't these two examples compile the same way? Are mutable reference move semantics (i.e., the {vref} from E1) enforced for methods (i.e. split_at_mut), but not functions (i.e., example2_inner)? Why is this the case?
I would like to keep example2_inner as an independent utility function: how would I change Example 2 to accommodate that?
You can fix it like this:
v_ref = {
// move `v_ref` to a new variable which will go out of scope by the end of the block
let r = v_ref;
// Mutating & slice splitting moved to function
let (v_l, v_r) = example2_inner(r);
match len % 2 {
0 => v_l,
_ => v_r,
}
};
The reason is that v_1 and v_2 both are references to v_ref, but these references both outlive the block because they are being returned from it. Then you try to write to v_ref, which the borrow checker doesn't like because there are still these live references around.
Assigning v_ref to a new variable, r, means that v_ref is no longer valid - the variable has been moved. The live references, v_1 and v_2 are now referring to r, which in turn is a reference to v, and which only lives as long as the block. You are now free to write to v_ref because nothing else is referring to it.
Alternatively, you can just update to Rust Edition 2018, or enable non-lexical lifetimes, which can handle this situation.
Why don't these two examples compile the same way? Are mutable
reference move semantics (i.e., the {vref} from E1) enforced for
methods (i.e. split_at_mut), but not functions (i.e., example2_inner)?
Why is this the case?
Actually, it is not about methods vs functions, but about method call syntax vs function call syntax.
Every method call can be translated into the UFCS (Universal Function Call Syntax) which is generally of this form:
<Type as Trait>::method(args);
If we make a naive attempt to translate the call of { v_ref }.split_at_mut(len / 2) into UFCS in version 1, we end up with the same error as in version 2:
<[_]>::split_at_mut({v_ref}, len / 2)
The reason is that the above is equivalent to something which again does not cause v_ref to be moved into the block:
<[_]>::split_at_mut({&mut *v_ref}, len / 2)
What the method syntax actually resolves to is this working variant of the above:
<[_]>::split_at_mut(&mut *{v_ref}, len / 2)
For this variant, the compiler infers that v_ref indeed should be moved into the block, because the compiler realizes that the re-borrowing necessary for the method call is already going to happen on {v_ref}, so it doesn't insert an additional superfluous re-borrow on v_ref.
Now that we know how the method syntax solves your issue implicitly, we have an alternative solution for your issue in version 2:
example2_inner(&mut *{ v_ref });
I'm trying to read the values from a vector and use the values as indexes to perform an addition:
fn main() {
let objetive = 3126.59;
// 27
let values: Vec<f64> = vec![
2817.42, 2162.17, 3756.57, 2817.42, -2817.42, 946.9, 2817.42, 964.42, 795.43, 3756.57,
139.34, 903.58, -3756.57, 939.14, 828.04, 1120.04, 604.03, 3354.74, 2748.06, 1470.8,
4695.71, 71.11, 2391.48, 331.29, 1214.69, 863.52, 7810.01,
];
let values_number = values.len();
let values_index_max = values_number - 1;
let mut additions: Vec<usize> = vec![0];
println!("{:?}", values_number);
while additions.len() > 0 {
let mut addition: f64 = 0.0;
let mut saltar: i32 = 0;
// Sumar valores en additions
for element_index in additions {
let addition_aux = values[element_index];
addition = addition_aux + addition;
}
}
}
I get the following error. How can I solve it?
error[E0382]: use of moved value: `additions`
--> src/main.rs:18:11
|
18 | while additions.len() > 0 {
| ^^^^^^^^^ value used here after move
...
23 | for element_index in additions {
| --------- value moved here
|
= note: move occurs because `additions` has type `std::vec::Vec<usize>`, which does not implement the `Copy` trait
error[E0382]: use of moved value: `additions`
--> src/main.rs:23:30
|
23 | for element_index in additions {
| ^^^^^^^^^ value moved here in previous iteration of loop
|
= note: move occurs because `additions` has type `std::vec::Vec<usize>`, which does not implement the `Copy` trait
The fix for this particular problem is to borrow the Vec you're iterating over instead of moving it:
for element_index in &additions {
let addition_aux = values[*element_index];
addition = addition_aux + addition;
}
but your code has other problems. You never change additions by adding or removing elements, so your while additions.len() > 0 will never terminate. I hope this is because you haven't finished and wanted to work out how to fix the immediate problem before writing the rest of the function.
For now, you might benefit from re-reading the chapter of the Rust Book about ownership, moves, and borrowing.
I'd like the ability to have multiple threads evaluate the same closure. The application I have in mind is parallelized numerical integration, so a situation where the function domain can be easily split into N chunks and handed to threads.
This is a simple function that evaluates the provided closure multiple times and averages the result:
use std::sync::mpsc;
use std::thread;
const THREAD_COUNT: u64 = 4;
fn average<F: Fn(f64) -> f64>(f: F) -> f64 {
let (tx, rx) = mpsc::channel();
for id in 0..THREAD_COUNT {
let thread_tx = tx.clone();
thread::spawn(move || {
thread_tx.send(f(id as f64));
});
}
let mut total = 0.0;
for id in 0..THREAD_COUNT {
total += rx.recv().unwrap();
}
total / THREAD_COUNT as f64
}
fn main() {
average(|x: f64| -> f64 { x });
}
When I compile I get this error:
error[E0277]: `F` cannot be sent between threads safely
--> src/main.rs:10:9
|
10 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `F` cannot be sent between threads safely
|
= help: within `[closure#src/main.rs:10:23: 12:10 thread_tx:std::sync::mpsc::Sender<f64>, f:F, id:u64]`, the trait `std::marker::Send` is not implemented for `F`
= help: consider adding a `where F: std::marker::Send` bound
= note: required because it appears within the type `[closure#src/main.rs:10:23: 12:10 thread_tx:std::sync::mpsc::Sender<f64>, f:F, id:u64]`
= note: required by `std::thread::spawn`
So I add + Send to the bounds on F and get a new error:
error[E0310]: the parameter type `F` may not live long enough
--> src/main.rs:10:9
|
6 | fn average<F: Fn(f64) -> f64 + Send>(f: F) -> f64 {
| -- help: consider adding an explicit lifetime bound `F: 'static`...
...
10 | thread::spawn(move || {
| ^^^^^^^^^^^^^
|
note: ...so that the type `[closure#src/main.rs:10:23: 12:10 thread_tx:std::sync::mpsc::Sender<f64>, f:F, id:u64]` will meet its required lifetime bounds
--> src/main.rs:10:9
|
10 | thread::spawn(move || {
| ^^^^^^^^^^^^^
So I add + 'static to F and get this:
error[E0382]: capture of moved value: `f`
--> src/main.rs:11:28
|
10 | thread::spawn(move || {
| ------- value moved (into closure) here
11 | thread_tx.send(f(id as f64));
| ^ value captured here after move
|
= note: move occurs because `f` has type `F`, which does not implement the `Copy` trait
So I add + Copy to F and get:
error: the trait `core::marker::Copy` is not implemented for the type `[closure#src/test.rs:115:11: 115:26]
It seems every thread wants its own copy of the closure (because of move) but closures don't implement Copy so no luck. It seems strange to me because if the closures are never mutating state then what's the safety issue with multiple threads accessing them?
I can get the code to work by providing a regular function instead of a closure, but this makes my code non-generic, i.e. it only works for a specific function instead of for anything that's Fn(f64) -> f64. And for the type of integration I'm doing, the functions integrated often have certain fixed variables mixed with the variable of integration, so it would seem natural to capture the fixed variables with a closure.
Is there some way to make this kind of multithreaded function evaluation work in a generic manner? Am I just thinking about things wrong?
The ultimate problem revolves around who owns the closure. The code as written states that ownership of the closure is transferred to average. This function then tries to give the closure to multiple threads, which fails as you have seen, as you can't give one item to multiple children.
but closures don't implement Copy so no luck
As of Rust 1.26.0, closures do implement Clone and Copy if all of the captured variables do. This means your final example code now works as-is:
fn average<F: Fn(f64) -> f64 + Send + 'static + Copy>(f: F) -> f64 { /* ... */ }
However, it's possible that your closures won't implement Copy or Clone.
You cannot give out a reference to the closure owned by average because the thread created with thread::spawn may outlive the call to average. When average exits, any stack-allocated variables will be destroyed. Any use of them would cause memory unsafety, which Rust aims to prevent.
One solution is to use an Arc. This will allow multiple shared owners of a single resource in a multithreaded context. When the wrapped closure is cloned, only a new reference is created. When all references disappear, the object is freed.
use std::{
sync::{mpsc, Arc},
thread,
};
const THREAD_COUNT: u64 = 4;
fn average<F>(f: F) -> f64
where
F: Fn(f64) -> f64 + Send + Sync + 'static,
{
let (tx, rx) = mpsc::channel();
let f = Arc::new(f);
for id in 0..THREAD_COUNT {
let thread_tx = tx.clone();
let f = f.clone();
thread::spawn(move || {
thread_tx.send(f(id as f64)).unwrap();
});
}
let mut total = 0.0;
for _ in 0..THREAD_COUNT {
total += rx.recv().unwrap();
}
total / THREAD_COUNT as f64
}
fn main() {
average(|x| x);
}
A more standard solution is to use scoped threads. These threads are guaranteed to exit by a certain time, which allows you to pass references that outlive the threads to the threads.
See also:
How can I pass a reference to a stack variable to a thread?
How do I pass disjoint slices from a vector to different threads?