I'm trying to write a function that will initialize n threads based on their index:
use std::iter::FromIterator;
use std::marker::{Send};
use std::thread::{JoinHandle, spawn};
// spawn n threads, initialized with their index
fn nspawn_i<F, T>(n: usize, f: F) -> Vec<JoinHandle<T>>
where
F: Fn(usize) -> T,
F: Send,
T: Send {
// spawn the i'th thread
let spawn_i = |mut i: usize| -> JoinHandle<T> {
spawn(move || f(i))
};
Vec::from_iter((0..n).map(spawn_i))
}
However, I get an error
error[E0525]: expected a closure that implements the `FnMut` trait, but this closure only implements `FnOnce`
--> src/main.rs:13:19
|
13 | let spawn_i = |mut i: usize| -> JoinHandle<T> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this closure implements `FnOnce`, not `FnMut`
14 | spawn(move || f(i))
| ------- closure is `FnOnce` because it moves the variable `f` out of its environment
...
17 | Vec::from_iter((0..n).map(spawn_i))
| --- the requirement to implement `FnMut` derives from here
I don't understand why spawn_i is FnOnce, and not FnMut, since the argument is mutable.
I don't understand why spawn_i is FnOnce, and not FnMut, since the argument is mutable.
The argument is not relevant, closure classifications is based on what they close over and how.
Here, the issue is that spawn_i consumes f when called:
when spawn_i is created, f is moved into it
when spawn_i is called the first time, it creates the sub-closure and f is moved into that
the second time spawn_i would be called... it can't be because there's no f to move into the sub-closure anymore
therefore spawn_i can only be called once.
So you need f to be reusable somehow e.g. store it in an Arc, or make it a function pointer (fn(usize) -> T) instead of a trait bound, or maybe play around with lifetimes.
Related
I have a function that composes multiple functions into a single one:
fn compose<'a, T>(fns: &'a [&dyn Fn(T) -> T]) -> Box<dyn Fn(T) -> T + 'a> {
Box::new(move |mut x| {
for f in fns {
x = f(x);
}
x
})
}
Why do I need to move fns here? If I don't put move, I get the following error:
error[E0373]: closure may outlive the current function, but it borrows `fns`, which is owned by the current function
--> src/lib.rs:2:14
|
2 | Box::new(|mut x| {
| ^^^^^^^ may outlive borrowed value `fns`
3 | for f in fns {
| --- `fns` is borrowed here
I don't get why I need to move the fns into the closure when it's just a reference. What does move actually do here?
A reference is just a normal variable, like any other variable. If you don't move, your fns inside the closure will be a reference to a reference, which goes out of scope at the end of the function.
I implemented a trait to share similar logic between my structs. This trait provides a collect_responses function, which takes vector of handlers and return vector of responses from the handlers. But after I applied this trait I got an errors like:
error[E0507]: cannot move out of `input`, a captured variable in an `FnMut` closure
--> src/lib.rs:16:30
|
12 | input: HandlerInput
| ----- captured outer variable
...
16 | .map(|func| func(input))
| ------------^^^^^-
| | |
| | move occurs because `input` has type `HandlerInput<'_>`, which does not implement the `Copy` trait
| captured by this `FnMut` closure
error[E0382]: use of partially moved value: `input`
This is my trait code:
pub trait Processor {
fn process_input(input: HandlerInput) -> Result<Vec<HandlerOutput>, Error>;
fn collect_responses(
handlers: Vec<fn(HandlerInput) -> Result<HandlerOutput, Error>>,
input: HandlerInput
) -> Result<Vec<HandlerOutput>, Error> {
let responses = handlers
.iter()
.map(|func| func(input))
.filter(|r| match r {
Ok(_) => true,
_ => false,
})
.map(|r| r.unwrap())
.collect::<Vec<HandlerOutput>>();
return if responses.is_empty() {
Err(
Error::new(
ErrorKind::Other,
"No successful response"
)
)
} else {
Ok(responses)
}
}
}
This is sandbox implementation in context of my real code.
Could somebody explain, why this issue happen and how to fix it ?
fn(HandlerInput) -> Result<HandlerOutput, Error>
This is almost certainly not the type you want, for multiple reasons. First, fn is a function pointer, i.e. a literal function written at the top level in your application. It can't be a closure or some other function-like thing. fn is almost exclusively used for interfacing with C, so you, at minimum, want Fn, FnMut, or FnOnce. See that link for more details on the difference. FnOnce won't suffice because you need to call it in a map, so I'm guessing you want FnMut, but read the difference between FnMut and Fn and determine for yourself which one you need.
Second, if you write
FnMut(HandlerInput) -> Result<HandlerOutput, Error>
This is a function that takes a HandlerInput by value. It moves its argument and the caller can never use the argument again. So you certainly can't call it several times on the same handler in a .map, since the very first call moves the value and renders the variable handler unusable. If HandlerInput implements Clone, then you can write
.map(|func| func(input.clone())
But that's probably not what you want. Almost certainly, you want a reference here.
FnMut(&HandlerInput) -> Result<HandlerOutput, Error>
FnMut(&mut HandlerInput) -> Result<HandlerOutput, Error>
The first is a function which takes an immutable reference to a HandlerInput. It can read the HandlerInput but can't change, move, or drop it. If HandlerInput is a read-only data structure, then this is the best option.
The second is a function which takes a mutable reference. It can read and modify the HandlerInput but can't move or drop it. It also can't be safely used across threads. Use this if your handlers are intended to modify the input argument.
The idea is to to have one closure (change_x in this case) that captures state (x in this case) that takes a function as its parameter(alterer) that would dictate how the inner state changes.
pub fn plus17(h: & u64) -> u64 {
*h + 17
}
pub fn main() {
let x = 0; //take x by reference
let mut change_x = move |alterer: &dyn FnOnce(&u64) ->u64 | alterer(&x) ;
change_x(&mut plus17);
println!("{}", x);
}
I can't seem to get the types right however:
error[E0161]: cannot move a value of type dyn for<'r> FnOnce(&'r u64) -> u64: the size of dyn for<'r> FnOnce(&'r u64) -> u64 cannot be statically determined
--> playground/src/main.rs:19:69
|
19 | let mut increment_x = move |alterer: &dyn FnOnce(&u64) ->u64 | alterer(&x) ;
| ^^^^^^^
error[E0507]: cannot move out of `*alterer` which is behind a shared reference
--> playground/src/main.rs:19:69
|
19 | let mut increment_x = move |alterer: &dyn FnOnce(&u64) ->u64 | alterer(&x) ;
| ^^^^^^^ move occurs because `*alterer` has type `dyn for<'r> FnOnce(&'r u64) -> u64`, which does not implement the `Copy` trait
I'm not sure if i'm justified in putting the dyn where i put it, it was a compiler's suggestion and im not really sure why i have to put it there. Is it because the alterer can be of arbitrary size despite the input/return type of &u64->u64?
I have also tried to make alterer a FnMut as opposed to FnOnce, but im also pretty shaky as to their distinction and the fact that a given alterer would run only once (at the moment of invocation by outer closure change_x) seemed reasonable.
FnOnce needs an owned self. Thus alterer cannot be FnOnce, because it is not owned but a reference.
You can either make it &dyn Fn or &mut dyn FnMut (I'd recommend going with FnMut), or take Box<dyn FnOnce>.
As explained in Why is the move keyword needed when returning a closure which captures a Copy type? and How to copy instead of borrow an i64 into a closure in Rust?, if a closure captures a type that implements Copy, we need to use the move keyword to eagerly copy the value:
fn use_move_to_copy_into_closure() -> impl FnOnce(i32) -> bool {
let captured = 0;
move |value| value > captured
}
As of Rust 2021, disjoint capture in closures means that only the specific fields used in a closure are captured by the closure:
struct Wrapper(i32);
fn edition_2021_only_captures_specific_fields(captured: Wrapper) -> impl FnOnce(i32) -> bool {
let ret = move |value| value > captured.0;
drop(captured); // fails in 2015, 2018, succeeds in 2021
ret
}
If I capture a Copy field belonging to a reference, however, the field is not copied:
struct Wrapper(i32);
fn capturing_a_field_of_a_reference(captured: &Wrapper) -> impl FnOnce(i32) -> bool {
move |value| value > captured.0
}
error[E0700]: hidden type for `impl Trait` captures lifetime that does not appear in bounds
--> src/lib.rs:15:60
|
15 | fn capturing_a_field_of_a_reference(captured: &Wrapper) -> impl FnOnce(i32) -> bool {
| -------- ^^^^^^^^^^^^^^^^^^^^^^^^
| |
| hidden type `[closure#src/lib.rs:16:5: 16:36]` captures the anonymous lifetime defined here
|
help: to declare that the `impl Trait` captures `'_`, you can add an explicit `'_` lifetime bound
|
15 | fn capturing_a_field_of_a_reference(captured: &Wrapper) -> impl FnOnce(i32) -> bool + '_ {
| ++++
I expected that the field .0 would be copied in, just like in the unwrapped i32 or the owned Wrapper cases. What causes this difference?
move captures the environment by value, but things in the environment that are references remain references -- the references are captured by value.
Put another way, move closures only try to move values, because otherwise there wouldn't be a way to simultaneously capture some things by value and some things by reference. For example, it's a common pattern to do this when dealing with threads in e.g. crossbeam. Assume these structs:
#[derive(Clone)]
struct Foo;
impl Foo {
pub fn baz(&mut self, _: i32) {}
}
struct Bar(pub i32);
And this snippet:
let foo = Foo;
let bar = Bar(0);
{
let mut foo = foo.clone();
let bar = &bar;
s.spawn(move |_| {
foo.baz(bar.0);
});
}
Here the closure takes ownership of the clone of foo but references bar.0. This is true even if the type of bar.0 is Copy.
If it didn't work this way, there would be no way to express that the closure should own the foo clone, but borrow the copyable value bar.0.
Remember that implementing Copy on a type only means that an attempt to move a value of that type will instead copy. Since captured is a reference in your third example, capturing captured.0 doesn't attempt to move the i32 like it does in your second example where you have an owned value, and if a move isn't attempted then no copy can happen.
It's not a question of whether the copy occurs, but when it occurs. The copy only happens when the closure is called, not when it's defined (how could it be -- a captured hasn't been provided yet). Since the closure outlives the function call, it needs to know that its reference to captured will be valid when it's actually called precisely so that it has something to copy. Therefore it needs to be annotated with a lifetime tying it to the lifetime of captured. Thanks to lifetime inference, it's sufficient to simply add '_ to the impl instead of needing to explicitly write fn capturing_a_field_of_a_reference<'a>(captured: &'a Wrapper) -> impl 'a + FnOnce(i32) -> bool.
I'd like the ability to have multiple threads evaluate the same closure. The application I have in mind is parallelized numerical integration, so a situation where the function domain can be easily split into N chunks and handed to threads.
This is a simple function that evaluates the provided closure multiple times and averages the result:
use std::sync::mpsc;
use std::thread;
const THREAD_COUNT: u64 = 4;
fn average<F: Fn(f64) -> f64>(f: F) -> f64 {
let (tx, rx) = mpsc::channel();
for id in 0..THREAD_COUNT {
let thread_tx = tx.clone();
thread::spawn(move || {
thread_tx.send(f(id as f64));
});
}
let mut total = 0.0;
for id in 0..THREAD_COUNT {
total += rx.recv().unwrap();
}
total / THREAD_COUNT as f64
}
fn main() {
average(|x: f64| -> f64 { x });
}
When I compile I get this error:
error[E0277]: `F` cannot be sent between threads safely
--> src/main.rs:10:9
|
10 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `F` cannot be sent between threads safely
|
= help: within `[closure#src/main.rs:10:23: 12:10 thread_tx:std::sync::mpsc::Sender<f64>, f:F, id:u64]`, the trait `std::marker::Send` is not implemented for `F`
= help: consider adding a `where F: std::marker::Send` bound
= note: required because it appears within the type `[closure#src/main.rs:10:23: 12:10 thread_tx:std::sync::mpsc::Sender<f64>, f:F, id:u64]`
= note: required by `std::thread::spawn`
So I add + Send to the bounds on F and get a new error:
error[E0310]: the parameter type `F` may not live long enough
--> src/main.rs:10:9
|
6 | fn average<F: Fn(f64) -> f64 + Send>(f: F) -> f64 {
| -- help: consider adding an explicit lifetime bound `F: 'static`...
...
10 | thread::spawn(move || {
| ^^^^^^^^^^^^^
|
note: ...so that the type `[closure#src/main.rs:10:23: 12:10 thread_tx:std::sync::mpsc::Sender<f64>, f:F, id:u64]` will meet its required lifetime bounds
--> src/main.rs:10:9
|
10 | thread::spawn(move || {
| ^^^^^^^^^^^^^
So I add + 'static to F and get this:
error[E0382]: capture of moved value: `f`
--> src/main.rs:11:28
|
10 | thread::spawn(move || {
| ------- value moved (into closure) here
11 | thread_tx.send(f(id as f64));
| ^ value captured here after move
|
= note: move occurs because `f` has type `F`, which does not implement the `Copy` trait
So I add + Copy to F and get:
error: the trait `core::marker::Copy` is not implemented for the type `[closure#src/test.rs:115:11: 115:26]
It seems every thread wants its own copy of the closure (because of move) but closures don't implement Copy so no luck. It seems strange to me because if the closures are never mutating state then what's the safety issue with multiple threads accessing them?
I can get the code to work by providing a regular function instead of a closure, but this makes my code non-generic, i.e. it only works for a specific function instead of for anything that's Fn(f64) -> f64. And for the type of integration I'm doing, the functions integrated often have certain fixed variables mixed with the variable of integration, so it would seem natural to capture the fixed variables with a closure.
Is there some way to make this kind of multithreaded function evaluation work in a generic manner? Am I just thinking about things wrong?
The ultimate problem revolves around who owns the closure. The code as written states that ownership of the closure is transferred to average. This function then tries to give the closure to multiple threads, which fails as you have seen, as you can't give one item to multiple children.
but closures don't implement Copy so no luck
As of Rust 1.26.0, closures do implement Clone and Copy if all of the captured variables do. This means your final example code now works as-is:
fn average<F: Fn(f64) -> f64 + Send + 'static + Copy>(f: F) -> f64 { /* ... */ }
However, it's possible that your closures won't implement Copy or Clone.
You cannot give out a reference to the closure owned by average because the thread created with thread::spawn may outlive the call to average. When average exits, any stack-allocated variables will be destroyed. Any use of them would cause memory unsafety, which Rust aims to prevent.
One solution is to use an Arc. This will allow multiple shared owners of a single resource in a multithreaded context. When the wrapped closure is cloned, only a new reference is created. When all references disappear, the object is freed.
use std::{
sync::{mpsc, Arc},
thread,
};
const THREAD_COUNT: u64 = 4;
fn average<F>(f: F) -> f64
where
F: Fn(f64) -> f64 + Send + Sync + 'static,
{
let (tx, rx) = mpsc::channel();
let f = Arc::new(f);
for id in 0..THREAD_COUNT {
let thread_tx = tx.clone();
let f = f.clone();
thread::spawn(move || {
thread_tx.send(f(id as f64)).unwrap();
});
}
let mut total = 0.0;
for _ in 0..THREAD_COUNT {
total += rx.recv().unwrap();
}
total / THREAD_COUNT as f64
}
fn main() {
average(|x| x);
}
A more standard solution is to use scoped threads. These threads are guaranteed to exit by a certain time, which allows you to pass references that outlive the threads to the threads.
See also:
How can I pass a reference to a stack variable to a thread?
How do I pass disjoint slices from a vector to different threads?