I have a Rust struct with a method that is designed to parallelise over multiple threads. Each thread needs to access that struct. So naturally I want to put it into an Arc. However I can't work out a way to do this.
Here's a simplified example:
use std::sync::{Arc};
use std::thread;
struct Foo {
field: usize
}
impl Foo {
fn parallel(&self){
let arc = Arc::new(&self);
for i in 1..5 {
let arc = Arc::clone(&arc);
thread::spawn(move || {
i + arc.field
});
}
}
}
And the compiler says:
fn parallel(&self){
^^^^^ this data with an anonymous lifetime `'_`...
let arc = Arc::new(&self);
^^^^^ ...is captured here...
thread::spawn(move || {
^^^^^^^^^^^^^ ...and is required to live as long as `'static` here
`self` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement
So it would seem that I'm actually not allowed to put self into an Arc (not quite sure why, though).
I've also tried cloning self but that gives the same error:
impl Foo {
fn parallel(&self){
let clone = self.clone();
let arc = Arc::new(&clone);
for i in 1..5 {
let arc = Arc::clone(&arc);
thread::spawn(move || {
i + arc.field
});
}
}
}
fn parallel(&self){
^^^^^ this data with an anonymous lifetime `'_`...
let clone = self.clone();
^^^^^ ...is captured here...
thread::spawn(move || {
^^^^^^^^^^^^^ ...and is required to live as long as `'static` here
`self` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement
Similar things happen if I try to move the clone into the Arc, or if I remove the move in front of the closure.
What is going on here, and how can I fix it? Also, in general, is there any way to do this: ie putting self into an Arc, where self is a struct?
If you put self or &self in the arc, you have to take it back at the end of the function.
It means you can't let it to the other threads.
The crossbeam crate includes a facility to deal with exactly this: ensure the threads release what you gave them:
Creates a new scope for spawning threads.
All child threads that haven’t been manually joined will be
automatically joined just before this function invocation ends. If all
joined threads have successfully completed, Ok is returned with the
return value of f. If any of the joined threads has panicked, an Err
is returned containing errors from panicked threads.
Example:
use std::sync::{Arc};
use crossbeam::thread;
struct Foo {
field: usize
}
impl Foo {
fn parallel(&self){
let arc = Arc::new(&self);
for i in 1..5 {
let arc = Arc::clone(&arc);
thread::scope(|s| {
s.spawn(|_| {
i + arc.field
});
});
}
}
}
You can wrap Foo in Arc and define methods for Arc<Foo>
use std::{sync::Arc, time::Duration};
use tokio::task::JoinSet;
struct Foo {
field: usize
}
impl Foo {
async fn add(&self, n: usize) {
println!("calculation {} + {n}...", self.field);
tokio::time::sleep(Duration::from_secs(1)).await;
println!("{} + {n} = {}", self.field, self.field + n);
}
async fn parallel(self: Arc<Self>){
let mut set = JoinSet::new();
for i in 1..5 {
let foo = self.clone();
set.spawn(async move {
foo.add(i).await
});
}
// wait for tasks...
while let Some(_) = set.join_next().await {}
}
}
#[tokio::main]
async fn main() {
println!("Start");
let foo = Foo { field: 23 };
foo.add(10).await;
// foo.parallel().await; // error: Foo should be wrapped in Arc
println!("Non parallel done");
let foo = Arc::new(foo);
{
let foo = foo.clone();
tokio::spawn(async move { foo.parallel().await });
}
foo.parallel().await;
println!("End");
}
Related
My question is the following:
Would it be possible to make a deferred initialization of an object and then borrow it as mutable for a 'static lifetime?
Some context
First of all, I am going to show an example of what I mean by deferred initialization. The deferred initialization model that I have come up with is to use the crate lazy_static with a standard library Mutex holding an Option<T>. Here there is an example of a deferred initialization.
use lazy_static::lazy_static;
use std::thread::JoinHandle;
use std::sync::Mutex;
pub struct DeferredStruct {
internal_field: u32,
}
impl DeferredStruct {
pub fn new(internal_field: u32) -> Self {
Self {
internal_field,
}
}
pub fn regular_function(&mut self) {
self.internal_field += 1;
println!("{}", self.internal_field);
}
}
lazy_static! {
static ref MY_DEFERRED: Mutex<Option<DeferredStruct>> = Mutex::new(None);
}
fn main() {
// Initial processing
// ...
// ...
// The value 10 would be obtained from a configuration file on runtime.
let deferred_struct = DeferredStruct::new(10);
let mut lock = MY_DEFERRED.lock().unwrap();
lock.replace(deferred_struct);
std::mem::drop(lock); // In this example we drop the lock to avoid a deadlock when calling another_function.
// More processing
// ...
// ...
let thread_handle = another_function();
thread_handle.join().unwrap();
}
// From another part of the program and possibly from another thread we
// lock MY_DEFERRED and call regular_funcion on it.
fn another_function() -> JoinHandle<()> {
std::thread::spawn(|| {
let mut lock = MY_DEFERRED.lock().unwrap();
if let Some(deferred) = lock.as_mut() {
deferred.regular_function();
}
})
}
You can execute the above code in Rust Playground and check that it prints 11 correctly.
Introducing a struct function that requires a static lifetime
Now, let's say I add a function inside DeferredStruct that will create a worker thread in order to execute some computations that will take a long time:
pub struct DeferredStruct {
internal_field: u32,
}
impl DeferredStruct {
pub fn new(internal_field: u32) -> Self {
Self {
internal_field,
}
}
pub fn regular_function(&mut self) {
self.internal_field += 1;
println!("{}", self.internal_field);
}
pub fn static_function(&'static mut self) {
std::thread::spawn(move || {
// Do something really long.
// Finally I make some changes on self
self.internal_field += 100;
});
}
}
In this case is required for &mut self to have a 'static lifetime:
error[E0759]: `self` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement.
Due to this, the function will take a &'static mut self.
The problem comes when trying to borrow the DeferredStruct inside the Option in MY_DEFERRED as 'static and mut.
If I cannot borrow for 'static and mut then I cannot call deferred.static_function(). As the value will not live for long enough.
error[E0597]: `lock` does not live long enough
--> src/main.rs:57:33
|
57 | if let Some(deferred) = lock.as_mut() {
| ^^^^---------
| |
| borrowed value does not live long enough
| argument requires that `lock` is borrowed for `'static`
...
60 | })
| - `lock` dropped here while still borrowed
Here there is a minimal reproducible example in Rust Playground.
TL;DR
Is it possible to borrow an object created at runtime (not necessarily created at the immediate start of the program) inside a Mutex<Option<T>> as mutable and for a 'static lifetime?
Any help is appreciated.
I'm trying to have a function shared between threads, that can be called with another function on it:
fn main() {
let on_virtual_tun_write = std::sync::Arc::new(|f: &dyn FnOnce(&mut [u8]), size: usize|-> std::result::Result<(),()>{
let mut buffer = vec![0; size];
f(buffer.as_mut_slice());
Ok(())
});
}
Once it's called, it uses the f passed as an argument, to retrieve a buffer. As you can see, I do not intend to copy the function, I just use it as soon as someone calls the closure, and discards it. However, Rust thinks that I want to copy it. What can I do to tell Rust that I just want to use a reference to the function? I thought &dyn was sufficient.
Error:
error[E0161]: cannot move a value of type dyn for<'r> FnOnce(&'r mut [u8]): the size of dyn for<'r> FnOnce(&'r mut [u8]) cannot be statically determined
--> src/main.rs:4:9
|
4 | f(buffer.as_mut_slice());
| ^
error[E0507]: cannot move out of `*f` which is behind a shared reference
--> src/main.rs:4:9
|
4 | f(buffer.as_mut_slice());
| ^ move occurs because `*f` has type `dyn for<'r> FnOnce(&'r mut [u8])`, which does not implement the `Copy` trait
error: aborting due to 2 previous errors; 1 warning emitted
So I think there's a few issues that are coming together to cause you this problem.
I'm going to use a reduced example which generates essentially the same error message.
fn main() {
let ff = |f: &dyn FnOnce()| { f(); };
}
The core of the problem is that rust is stopping bad code where the FnOnce is used more than once. Like this (which doesn't compile):
fn main() {
let f = || { }; // Imagine this is a real FnOnce()
let ff = |f: &dyn FnOnce()| { f(); };
ff(&f); // Use the FnOnce in our wrapper
f(); // Use the FnOnce again - should fail to compile.
}
Firstly FnOnce() when called consumes its self - this is how it ensures it can only be called once. Which means you need to be passing the object, not a reference to it - otherwise the caller can still be holding a reference. This is not the case for Fn() and FnMut(). The first uses only a reference to self, and the second a mutable reference to self. This means the following compile OK:
fn main() {
let ff = |f: &dyn Fn()| { f(); };
let f = || {};
ff(f); // First call is OK
f(); // Second usage is OK - it's a pure function
let ff = |f: &mut dyn FnMut()| { f(); };
let mut i = 0;
let mut f = || { i += 1 };
ff(&mut f); // OK i=1
f(); // OK i=2
println!("i={}", i); // prints 2
}
So if you can reduce the restrictions on your function type so that it is either Fn or FnMut, that should solve it for you, if not ... read on.
You can pass an instance (not a reference) of a FnOnce to a generic function that uses impl. e.g.
fn ff(f: impl FnOnce()) {
f()
}
fn main() {
// Creating a closure that moves q into it ensures we get a FnOnce
let q=vec![1,2,3];
let f = move || {println!("{}", q.len())};
ff(f); // OK.
// f(); // Compile error with "use of moved value: `f`"
}
this is equivalent to
fn<F> ff(f: F)
where F: FnOnce()
{
f()
}
However, we can't mix generics and closures in a way that can help us. Trying to use |f: impl FnOnce()| generates the error : error[E0562]: impl Trait not allowed outside of function and inherent method return types.
The only way I think you can use both the FnOnce and closure is to box up the function and pass it in. That will transfer ownership of the function into the closure and so we call it. For example the following compiles:
fn main() {
let ff = |f: Box<dyn FnOnce()>| { f() };
let q=vec![1,2,3];
let f = Box::new(move || {println!("{}", q.len())});
ff(f); // OK
// f(); // Errors with "use of moved value: `f`"
}
This question already has answers here:
Cannot move out of value which is behind a shared reference when unwrapping
(2 answers)
error[E0507]: Cannot move out of borrowed content
(1 answer)
cannot move out of borrowed content when unwrapping a member variable in a &mut self method
(2 answers)
Closed 4 years ago.
Why can't I do this?
pub fn start_workers(&mut self) {
// start all the worker threads
self.dispatch_thread = Some(spawn(||{
for _i in 1..10 {
println!("Price = {}", 10);
thread::sleep(time::Duration::from_secs(1));
}
}));
self.dispatch_thread.unwrap().join();
}
I'm getting the following error,
error[E0507]: cannot move out of borrowed content
--> src/orderbook.rs:195:9
|
195 | self.dispatch_thread.unwrap().join();
| ^^^^ cannot move out of borrowed content
This is indeed a non-obvious error message. Have a look at the method signatures for unwrap:
pub fn unwrap(self) -> T
and take:
pub fn take(&mut self) -> Option<T>
unwrap consumes the Option (note the receiver is self), which would leave self.dispatch_thread in an unknown state. If you use take it is returned to the None state as you probably intended.
You probably want take in this context; as shown here:
use std::thread;
use std::time;
struct Foo {
foo: Option<thread::JoinHandle<()>>,
}
impl Foo {
fn nope(&mut self) {
self.foo = Some(thread::spawn(|| {
for _i in 1..10 {
println!("Price = {}", 10);
thread::sleep(time::Duration::from_millis(10));
}
}));
self.foo.take().unwrap().join();
}
}
fn main() {
let foo = Some(thread::spawn(|| {
for _i in 1..10 {
println!("Price = {}", 10);
thread::sleep(time::Duration::from_millis(10));
}
}));
foo.unwrap().join();
let mut foo = Foo { foo: None };
foo.foo = Some(thread::spawn(|| {
for _i in 1..10 {
println!("Price = {}", 10);
thread::sleep(time::Duration::from_millis(10));
}
}));
foo.foo.unwrap().join();
let mut foo = Foo { foo: None };
foo.nope();
}
Note that assert!(foo.foo.is_none()); would be similarly illegal; but is valid in this case because we don't violate that constraint. In the method with &self as a receiver, this isn't true, which is why it is illegal in that case.
I have a struct with a function next() (similar to iterators but not an iterator). This method returns the next state after modification (preserving the original state). So: fn next(&A) -> A.
I started with a simple struct where I didn't need a lifetime (struct A in the example) and I extended it to add a reference to a new struct (struct B).
The problem is that I now need to specify the lifetime for my struct and for some reason my method next() refuses to work anymore.
I suspect that the lifetime of the new struct of every iteration is limited to the scope where it is created and I cannot move it outside of this scope.
Is it possible to preserve the behavior of my next() method?
Try it here
#[derive(Clone)]
struct A(u32);
#[derive(Clone)]
struct B<'a>(u32, &'a u32);
impl A {
fn next(&self) -> A {
let mut new = self.clone();
new.0 = new.0 + 1;
new
}
}
impl<'a> B<'a> {
fn next(&self) -> B {
let mut new = self.clone();
new.0 = new.0 + 1;
new
}
}
fn main() {
let mut a = A(0);
for _ in 0..5 {
a = a.next();
}
let x = 0;
let mut b = B(0, &x);
for _ in 0..5 {
b = b.next();
}
}
The error is:
error[E0506]: cannot assign to `b` because it is borrowed
--> src/main.rs:31:9
|
31 | b = b.next();
| ^^^^-^^^^^^^
| | |
| | borrow of `b` occurs here
| assignment to borrowed `b` occurs here
The problem is here:
impl<'a> B<'a> {
fn next(&self) -> B {
let mut new = self.clone();
new.0 = new.0 + 1;
new
}
}
You didn't specify a lifetime for B, the return type of next. Because of Rust's lifetime elision rules, the compiler infers that you intended this:
impl<'a> B<'a> {
fn next<'c>(&'c self) -> B<'c> {
let mut new = self.clone();
new.0 = new.0 + 1;
new
}
}
Which means that the return value may not outlive self. Or, put another way, self has to live longer than the B that is returned. Given the body of the function, this is a completely unnecessary requirement because those references are independent of each other. And it causes a problem here:
for _ in 0..5 {
b = b.next();
}
You are overwriting a value that the borrow-checker thinks is still borrowed by the call to next(). Inside next we know that there is no such relationship – the lifetime annotations do not reflect the constraints of what you're actually doing.
So what are the lifetime bounds here?
The lifetimes of references to B are unrelated - each can exist without the other. So, to give the most flexibility to a caller, the lifetime of B should be different from the lifetime of the reference to self in next.
However, each B that is created with next() holds a reference to the same u32 as is held by self. So the lifetime parameter that you give to each B must be the same.
Using explicitly named lifetimes, this is the result of combining both of those things:
impl<'a> B<'a> {
fn next<'c>(&'c self) -> B<'a> {
let mut new = self.clone();
new.0 = new.0 + 1;
new
}
}
Note that — even though the reference to self here has lifetime 'c — the type of self is B<'a>, where 'a is the lifetime of the &u32 inside. Just the same as the return value.
But actually, the 'c can be elided. So it's really just the same as this:
impl<'a> B<'a> {
fn next(&self) -> B<'a> {
let mut new = self.clone();
new.0 = new.0 + 1;
new
}
}
In Rust, I want to specify a callback (e.g. a closure) to be called by a worker thread when something interesting happens. What would be the correct type signature for the callback?
This is an example of what I am trying to do:
use std::thread;
fn spawner(f: Box<FnMut()->()+Send>) -> thread::JoinHandle<()> {
thread::spawn(move || {
f();
})
}
fn main() {
let cb = || {
println!("callback");
};
spawner(Box::new(cb)).join().unwrap();
}
src/main.rs:5:9: 5:10 error: cannot borrow immutable `Box` content
`*f` as mutable
src/main.rs:5 f();
Mutability in Rust is inherited: because the variable f holding the box is immutable, the contents of the box are also immutable. An immutable FnMut closure cannot be called (FnMut requires the ability to mutate its environment).
Solution: make the variable f mutable:
fn spawner(mut f: Box<FnMut()->()+Send>) -> thread::JoinHandle<()>
The Rust compiler is not allowing you to move the immutable argument f into the closure. Changing it to mutable (adding mut before the variable) the compiler will stop complaining and you will get the desired behaviour.
use std::thread;
fn spawner(mut f: Box<FnMut()->()+Send>) -> thread::JoinHandle<()> {
thread::spawn(move || {
f();
})
}
fn main() {
let cb = || {
println!("callback");
};
spawner(Box::new(cb)).join().unwrap();
}