I'm invoking an async implemented method:
let mut safebrowsing: MutexGuard<Safebrowsing> = self.safebrowsing.lock().unwrap();
safebrowsing.is_safe(input: &message.content).await;
The is_safe-Method:
pub async fn is_safe(&mut self, input: &str) {
let links = self.finder.links(input);
for link in links {
match reqwest::get("url").await {
Ok(response) => {
println!(
"{}",
response.text().await.expect("response has no content")
);
}
Err(_) => {
println!("Unable to get safebrowsing-response")
}
}
}
}
But unfortunately by invoking the is_safe-Method asynchronously, the compiler tells me that threads cannot be sent safely. The error is about:
future cannot be sent between threads safely
within `impl std::future::Future`, the trait `std::marker::Send` is not implemented for `std::sync::MutexGuard<'_, Safebrowsing>`
required for the cast to the object type `dyn std::future::Future<Output = ()> + std::marker::Send`
handler.rs(31, 9): future is not `Send` as this value is used across an await
^-- safebrowsing.is_safe(input: &message.content).await;
---
future cannot be sent between threads safely
the trait `std::marker::Send` is not implemented for `(dyn for<'r> Fn(&'r [u8]) -> Option<usize> + 'static)`
required for the cast to the object type `dyn std::future::Future<Output = ()> + std::marker::Send`
safebrowsing.rs(22, 19): future is not `Send` as this value is used across an await
^-- match reqwest::get("url").await
I already tried to implement the Send-Trait to my Safebrowsing-Struct, but that does also not work.
Is there something I need to do to get it working? Because I have no clue why that is appearing
The key of this error is that MutexGuard<T> is not Send. This means that you are trying to do an await while the mutex is locked, and that is usually a bad idea, if you think about it: await may wait, in principle, indefinitely, but by waiting so long with the mutex held, any other thread that tries to lock the mutex will block, also indefinitely (unless you set a timeout, of course).
So, as a rule of thumb, you should never sleep with a mutex locked. For example your code could be rewritten as (totally untested):
pub async fn is_safe(this: &Mutex<Safebrowsing>, input: &str) {
//lock, find, unlock
let links = this.lock().unwrap().finder.links(input);
//now we can await safely
for link in links {
match reqwest::get("url").await {
Ok(response) => {
println!(
"{}",
response.text().await.expect("response has no content")
);
}
Err(_) => {
println!("Unable to get safebrowsing-response")
}
}
}
}
If you need to lock the Mutex later in the function, beware of the races! It may have been modified by other thread, maybe that input is no longer a thing.
Use Mutex implementation from the async runtime you're using.
Before 😭
Using mutex from standard library:
use std::sync::Mutex; // stdlib
let m = Mutex::new(...);
let v = m.lock().unwrap();
After 😁
Using mutex from tokio:
use tokio::sync::Mutex; // tokio async runtime
let m = Mutex::new(...); // the same!
let v = m.lock().await;
But why?
Roughly speaking, the native mutex forces the lock to be kept in the same thread,
but async runtime does not understand it.
If your lock does not cross with an async, then you can use mutex
from stdlib (it can be faster).
See the discussion from tokio documentation.
Related
I'm attempting to write a generic set_interval function helper:
pub fn set_interval<F, Fut>(mut f: F, dur: Duration)
where
F: Send + 'static + FnMut() -> Fut,
Fut: Future<Output = ()> + Send + 'static,
{
let mut interval = tokio::time::interval(dur);
tokio::spawn(async move {
// first tick is at 0ms
interval.tick().await;
loop {
interval.tick().await;
tokio::spawn(f());
}
});
}
This works fine until it's called from inside a class:
fn main() {}
struct Foo {}
impl Foo {
fn bar(&self) {
set_interval(|| self.task(), Duration::from_millis(1000));
}
async fn task(&self) {
}
}
self is not 'static, and we can't restrict lifetime parameter to something that is less than 'static because of tokio::task.
Is it possible to modify set_interval implementation so it works in cases like this?
Link to playground
P.S. Tried to
let instance = self.clone();
set_interval(move || instance.task(), Duration::from_millis(1000));
but I also get an error: error: captured variable cannot escape FnMut closure body
Is it possible to modify set_interval implementation so it works in cases like this?
Not really. Though spawn-ing f() really doesn't help either, as it precludes a simple "callback owns the object" solution (as you need either both callback and future to own the object, or just future).
I think that leaves two solutions:
Convert everything to shared mutability Arc, the callback owns one Arc, then on each tick it clones that and moves the clone into the future (the task method).
Have the future (task) acquire the object from some external source instead of being called on one, this way the intermediate callback doesn't need to do anything. Or the callback can do the acquiring and move that into the future, same diff.
Incidentally at this point it could make sense to just create the future directly, but allow cloning it. So instead of taking a callback set_interval would take a clonable future, and it would spawn() clones of its stored future instead of creating them anew.
As mentioned by #Masklinn, you can clone the Arc to allow for this. Note that cloning the Arc will not clone the underlying data, just the pointer, so it is generally OK to do so, and should not have a major impact on performance.
Here is an example. The following code will produce the error async block may outlive the current function, but it borrows data, which is owned by the current function:
fn main() {
// 🛑 Error: async block may outlive the current function, but it borrows data, which is owned by the current function
let data = Arc::new("Hello, World".to_string());
tokio::task::spawn(async {
println!("1: {}", data.len());
});
tokio::task::spawn(async {
println!("2: {}", data.len());
});
}
Rust unhelpfully suggests adding move to both async blocks, but that will result in a borrowing error because there would be multiple ownership.
To fix the problem, we can clone the Arc for each task and then add the move keyword to the async blocks:
fn main() {
let data = Arc::new("Hello, World".to_string());
let data_for_task_1 = data.clone();
tokio::task::spawn(async move {
println!("1: {}", data_for_task_1.len());
});
let data_for_task_2 = data.clone();
tokio::task::spawn(async move {
println!("2: {}", data_for_task_2.len());
});
}
I want to leverage Tokio's runtime to handle a variable amount of async futures. Since the count of futures is unknown at compile time, it seems FuturesUnordered is my best option (macros such as select! require specifying your branches at compile time; join_all might be possible but the docs recommend FuturesUnordered "in a lot of cases" when order doesn't matter).
The logic of this snippet is a recv() loop getting pushed to the bucket of futures, which should always run. When new data arrives, its parsing/processing gets pushed to the futures bucket too (instead of being processed immediately). This ensures the receiver maintains low latency in responding to new events, and data processing (potentially computationally expensive decryption) occurs concurrently with all other data processing async blocks (plus the listening receiver).
This thread explains why the futures get .boxed(), by the way.
The problem is this cryptic error:
error[E0277]: `dyn futures::Future<Output = ()> + std::marker::Send` cannot be shared between threads safely
--> src/main.rs:27:8
|
27 | }).boxed());
| ^^^^^ `dyn futures::Future<Output = ()> + std::marker::Send` cannot be shared between threads safely
|
= help: the trait `Sync` is not implemented for `dyn futures::Future<Output = ()> + std::marker::Send`
= note: required because of the requirements on the impl of `Sync` for `Unique<dyn futures::Future<Output = ()> + std::marker::Send>`
= note: required because it appears within the type `Box<dyn futures::Future<Output = ()> + std::marker::Send>`
= note: required because it appears within the type `Pin<Box<dyn futures::Future<Output = ()> + std::marker::Send>>`
= note: required because of the requirements on the impl of `Sync` for `FuturesUnordered<Pin<Box<dyn futures::Future<Output = ()> + std::marker::Send>>>`
= note: required because of the requirements on the impl of `std::marker::Send` for `&FuturesUnordered<Pin<Box<dyn futures::Future<Output = ()> + std::marker::Send>>>`
= note: required because it appears within the type `[static generator#src/main.rs:16:25: 27:6 _]`
= note: required because it appears within the type `from_generator::GenFuture<[static generator#src/main.rs:16:25: 27:6 _]>`
= note: required because it appears within the type `impl futures::Future`
It looks like pushing to an UnorderedFutures "recursively" (not really I guess, but what else would you call it?) doesn't work, but I'm not sure why. This error indicates some Sync trait requirement isn't met for the Box'd & Pin'd async blocks being tended to by the FuturesUnordered -- a requirement I guess is only imposed because &FuturesUnordered (used during futures.push(...) because that method borrows &self) needs it for its Send trait... or something?
use std::error::Error;
use tokio::sync::mpsc::{self, Receiver, Sender};
use futures::stream::futures_unordered::FuturesUnordered;
use futures::FutureExt;
#[tokio::main]
pub async fn main() -> Result<(), Box<dyn Error>> {
let mut futures = FuturesUnordered::new();
let (tx, rx) = mpsc::channel(32);
tokio::spawn( foo(tx) ); // Only the receiver is relevant; its transmitter is
// elsewhere, occasionally sending data.
futures.push((async { // <--- NOTE: futures.push()
loop {
match rx.recv().await {
Some(data) => {
futures.push((async move { // <--- NOTE: nested futures.push()
let _ = data; // TODO: replace with code that processes 'data'
}).boxed());
},
None => {}
}
}
}).boxed());
while let Some(_) = futures.next().await {}
Ok(())
}
I will leave the low-level error for another answer, but I believe a more idiomatic way to solve the high-level problem here would be to combine the use of FuturesUnordered with something like tokio::select! as follows:
use tokio::sync::mpsc;
use futures::stream::FuturesUnordered;
use futures::StreamExt;
#[tokio::main]
pub async fn main() {
let mut futures = FuturesUnordered::new();
let (tx, mut rx) = mpsc::channel(32);
//turn foo into something more concrete
tokio::spawn(async move {
let _ = tx.send(42i32).await;
});
loop {
tokio::select! {
Some(data) = rx.recv() => {
futures.push(async move {
data.to_string()
});
},
Some(result) = futures.next() => {
println!("{}", result)
},
else => break,
}
}
}
You can read more about the select macro here: https://tokio.rs/tokio/tutorial/select
When you box the future created by the async block with the boxed method, you are trying to coerce it to a dyn Future + Send:
pub fn boxed<'a>(
self
) -> Pin<Box<dyn Future<Output = Self::Output> + 'a + Send>>
However, the created future is not Send. Why? Because inside of it, you try to push to the FuturesUnordered, which borrows it:
pub fn push(&self, future: Fut)
This means that the async block captures a &FuturesUnordered. For a type to be Send, all it's fields must be Send, so for the generated future to be Send, &FuturesUnordered must be Send.
For a reference to be Send, the type must also be Sync:
impl<'_, T> Send for &'_ T where
T: Sync
And for FuturesUnordered to be Sync, the stored futures must also be Sync:
impl<Fut: Sync> Sync for FuturesUnordered<Fut> {}
However, the future returned by boxed is not necessarily Sync:
pub fn boxed<'a>(
self
) -> Pin<Box<dyn Future<Output = Self::Output> + 'a + Send>>
Which means that the async generator is not Send, so you cannot coerce it to a dyn Future + Send, and you get a confusing error message.
The solution is to add a Sync bound to the future, and Box::pin manually:
type BoxedFuture = Pin<Box<dyn Future<Output = ()> + Send + Sync>>;
let mut futures = FuturesUnordered::<BoxedFuture>::new();
futures.push(Box::pin(async {
loop {
match rx.recv().await {
Some(data) => {
futures.push(Box::pin(async move {
let _ = data;
}));
}
None => {}
}
}
}));
However, you will then run into a bunch of borrowing issues. A better solution would be to use tokio::select! instead of the outer push, as explained by Michael's answer.
I have an async method that should execute some futures in parallel, and only return after all futures finished. However, it is passed some data by reference that does not live as long as 'static (it will be dropped at some point in the main method). Conceptually, it's similar to this (Playground):
async fn do_sth(with: &u64) {
delay_for(Duration::new(*with, 0)).await;
println!("{}", with);
}
async fn parallel_stuff(array: &[u64]) {
let mut tasks: Vec<JoinHandle<()>> = Vec::new();
for i in array {
let task = spawn(do_sth(i));
tasks.push(task);
}
for task in tasks {
task.await;
}
}
#[tokio::main]
async fn main() {
parallel_stuff(&[3, 1, 4, 2]);
}
Now, tokio wants futures that are passed to spawn to be valid for the 'static lifetime, because I could drop the handle without the future stopping. That means that my example above produces this error message:
error[E0759]: `array` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement
--> src/main.rs:12:25
|
12 | async fn parallel_stuff(array: &[u64]) {
| ^^^^^ ------ this data with an anonymous lifetime `'_`...
| |
| ...is captured here...
...
15 | let task = spawn(do_sth(i));
| ----- ...and is required to live as long as `'static` here
So my question is: How do I spawn futures that are only valid for the current context that I can then wait until all of them completed?
It is not possible to spawn a non-'static future from async Rust. This is because any async function might be cancelled at any time, so there is no way to guarantee that the caller really outlives the spawned tasks.
It is true that there are various crates that allow scoped spawns of async tasks, but these crates cannot be used from async code. What they do allow is to spawn scoped async tasks from non-async code. This doesn't violate the problem above, because the non-async code that spawned them cannot be cancelled at any time, as it is not async.
Generally there are two approaches to this:
Spawn a 'static task by using Arc rather than ordinary references.
Use the concurrency primitives from the futures crate instead of spawning.
Generally to spawn a static task and use Arc, you must have ownership of the values in question. This means that since your function took the argument by reference, you cannot use this technique without cloning the data.
async fn do_sth(with: Arc<[u64]>, idx: usize) {
delay_for(Duration::new(with[idx], 0)).await;
println!("{}", with[idx]);
}
async fn parallel_stuff(array: &[u64]) {
// Make a clone of the data so we can shared it across tasks.
let shared: Arc<[u64]> = Arc::from(array);
let mut tasks: Vec<JoinHandle<()>> = Vec::new();
for i in 0..array.len() {
// Cloning an Arc does not clone the data.
let shared_clone = shared.clone();
let task = spawn(do_sth(shared_clone, i));
tasks.push(task);
}
for task in tasks {
task.await;
}
}
Note that if you have a mutable reference to the data, and the data is Sized, i.e. not a slice, it is possible to temporarily take ownership of it.
async fn do_sth(with: Arc<Vec<u64>>, idx: usize) {
delay_for(Duration::new(with[idx], 0)).await;
println!("{}", with[idx]);
}
async fn parallel_stuff(array: &mut Vec<u64>) {
// Swap the array with an empty one to temporarily take ownership.
let vec = std::mem::take(array);
let shared = Arc::new(vec);
let mut tasks: Vec<JoinHandle<()>> = Vec::new();
for i in 0..array.len() {
// Cloning an Arc does not clone the data.
let shared_clone = shared.clone();
let task = spawn(do_sth(shared_clone, i));
tasks.push(task);
}
for task in tasks {
task.await;
}
// Put back the vector where we took it from.
// This works because there is only one Arc left.
*array = Arc::try_unwrap(shared).unwrap();
}
Another option is to use the concurrency primitives from the futures crate. These have the advantage of working with non-'static data, but the disadvantage that the tasks will not be able to run on multiple threads at the same time.
For many workflows this is perfectly fine, as async code should spend most of its time waiting for IO anyway.
One approach is to use FuturesUnordered. This is a special collection that can store many different futures, and it has a next function that runs all of them concurrently, and returns once the first of them finished. (The next function is only available when StreamExt is imported)
You can use it like this:
use futures::stream::{FuturesUnordered, StreamExt};
async fn do_sth(with: &u64) {
delay_for(Duration::new(*with, 0)).await;
println!("{}", with);
}
async fn parallel_stuff(array: &[u64]) {
let mut tasks = FuturesUnordered::new();
for i in array {
let task = do_sth(i);
tasks.push(task);
}
// This loop runs everything concurrently, and waits until they have
// all finished.
while let Some(()) = tasks.next().await { }
}
Note: The FuturesUnordered must be defined after the shared value. Otherwise you will get a borrow error that is caused by them being dropped in the wrong order.
Another approach is to use a Stream. With streams, you can use buffer_unordered. This is a utility that uses FuturesUnordered internally.
use futures::stream::StreamExt;
async fn do_sth(with: &u64) {
delay_for(Duration::new(*with, 0)).await;
println!("{}", with);
}
async fn parallel_stuff(array: &[u64]) {
// Create a stream going through the array.
futures::stream::iter(array)
// For each item in the stream, create a future.
.map(|i| do_sth(i))
// Run at most 10 of the futures concurrently.
.buffer_unordered(10)
// Since Streams are lazy, we must use for_each or collect to run them.
// Here we use for_each and do nothing with the return value from do_sth.
.for_each(|()| async {})
.await;
}
Note that in both cases, importing StreamExt is important as it provides various methods that are not available on streams without importing the extension trait.
In case of code that uses threads for parallelism, it is possible to avoid copying by extending a lifetime with transmute. An example:
fn main() {
let now = std::time::Instant::now();
let string = format!("{now:?}");
println!(
"{now:?} has length {}",
parallel_len(&[&string, &string]) / 2
);
}
fn parallel_len(input: &[&str]) -> usize {
// SAFETY: this variable needs to be static, because it is passed into a thread,
// but the thread does not live longer than this function, because we wait for
// it to finish by calling `join` on it.
let input: &[&'static str] = unsafe { std::mem::transmute(input) };
let mut threads = vec![];
for txt in input {
threads.push(std::thread::spawn(|| txt.len()));
}
threads.into_iter().map(|t| t.join().unwrap()).sum()
}
It seems reasonable that this should also work for asynchronous code, but I do not know enough about that to say for sure.
I have a tokio core whose main task is running a websocket (client). When I receive some messages from the server, I want to execute a new task that will update some data. Below is a minimal failing example:
use tokio_core::reactor::{Core, Handle};
use futures::future::Future;
use futures::future;
struct Client {
handle: Handle,
data: usize,
}
impl Client {
fn update_data(&mut self) {
// spawn a new task that updates the data
self.handle.spawn(future::ok(()).and_then(|x| {
self.data += 1; // error here
future::ok(())
}));
}
}
fn main() {
let mut runtime = Core::new().unwrap();
let mut client = Client {
handle: runtime.handle(),
data: 0,
};
let task = future::ok::<(), ()>(()).and_then(|_| {
// under some conditions (omitted), we update the data
client.update_data();
future::ok::<(), ()>(())
});
runtime.run(task).unwrap();
}
Which produces this error:
error[E0477]: the type `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:13:51: 16:10 self:&mut &mut Client]>` does not fulfill the required lifetime
--> src/main.rs:13:21
|
13 | self.handle.spawn(future::ok(()).and_then(|x| {
| ^^^^^
|
= note: type must satisfy the static lifetime
The problem is that new tasks that are spawned through a handle need to be static. The same issue is described here. Sadly it is unclear to me how I can fix the issue. Even some attempts with and Arc and a Mutex (which really shouldn't be needed for a single-threaded application), I was unsuccessful.
Since developments occur rather quickly in the tokio landscape, I am wondering what the current best solution is. Do you have any suggestions?
edit
The solution by Peter Hall works for the example above. Sadly when I built the failing example I changed tokio reactor, thinking they would be similar. Using tokio::runtime::current_thread
use futures::future;
use futures::future::Future;
use futures::stream::Stream;
use std::cell::Cell;
use std::rc::Rc;
use tokio::runtime::current_thread::{Builder, Handle};
struct Client {
handle: Handle,
data: Rc<Cell<usize>>,
}
impl Client {
fn update_data(&mut self) {
// spawn a new task that updates the data
let mut data = Rc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
data.set(data.get() + 1);
future::ok(())
}));
}
}
fn main() {
// let mut runtime = Core::new().unwrap();
let mut runtime = Builder::new().build().unwrap();
let mut client = Client {
handle: runtime.handle(),
data: Rc::new(Cell::new(1)),
};
let task = future::ok::<(), ()>(()).and_then(|_| {
// under some conditions (omitted), we update the data
client.update_data();
future::ok::<(), ()>(())
});
runtime.block_on(task).unwrap();
}
I obtain:
error[E0277]: `std::rc::Rc<std::cell::Cell<usize>>` cannot be sent between threads safely
--> src/main.rs:17:21
|
17 | self.handle.spawn(future::ok(()).and_then(move |_x| {
| ^^^^^ `std::rc::Rc<std::cell::Cell<usize>>` cannot be sent between threads safely
|
= help: within `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<std::cell::Cell<usize>>`
= note: required because it appears within the type `[closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]`
= note: required because it appears within the type `futures::future::chain::Chain<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`
= note: required because it appears within the type `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`
So it does seem like in this case I need an Arc and a Mutex even though the entire code is single-threaded?
In a single-threaded program, you don't need to use Arc; Rc is sufficient:
use std::{rc::Rc, cell::Cell};
struct Client {
handle: Handle,
data: Rc<Cell<usize>>,
}
impl Client {
fn update_data(&mut self) {
let data = Rc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
data.set(data.get() + 1);
future::ok(())
}));
}
}
The point is that you no longer have to worry about the lifetime because each clone of the Rc acts as if it owns the data, rather than accessing it via a reference to self. The inner Cell (or RefCell for non-Copy types) is needed because the Rc can't be dereferenced mutably, since it has been cloned.
The spawn method of tokio::runtime::current_thread::Handle requires that the future is Send, which is what is causing the problem in the update to your question. There is an explanation (of sorts) for why this is the case in this Tokio Github issue.
You can use tokio::runtime::current_thread::spawn instead of the method of Handle, which will always run the future in the current thread, and does not require that the future is Send. You can replace self.handle.spawn in the code above and it will work just fine.
If you need to use the method on Handle then you will also need to resort to Arc and Mutex (or RwLock) in order to satisfy the Send requirement:
use std::sync::{Mutex, Arc};
struct Client {
handle: Handle,
data: Arc<Mutex<usize>>,
}
impl Client {
fn update_data(&mut self) {
let data = Arc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
*data.lock().unwrap() += 1;
future::ok(())
}));
}
}
If your data is really a usize, you could also use AtomicUsize instead of Mutex<usize>, but I personally find it just as unwieldy to work with.
I am attempting to write a simpler unit test runner for my Rust project. I have created a TestFixture trait that my test fixture structs will implement, similar to inheriting from the unit test base class in other testing frameworks. The trait is fairly simple. This is my test fixture
pub trait TestFixture {
fn setup(&mut self) -> () {}
fn teardown(&mut self) -> () {}
fn before_each(&mut self) -> () {}
fn after_each(&mut self) -> () {}
fn tests(&mut self) -> Vec<Box<Fn(&mut Self)>>
where Self: Sized {
Vec::new()
}
}
My test running function is as follows
pub fn test_fixture_runner<T: TestFixture>(fixture: &mut T) {
fixture.setup();
let _r = fixture.tests().iter().map(|t| {
let handle = thread::spawn(move || {
fixture.before_each();
t(fixture);
fixture.after_each();
});
if let Err(_) = handle.join() {
println!("Test failed!")
}
});
fixture.teardown();
}
I get the error
src/tests.rs:73:22: 73:35 error: the trait `core::marker::Send` is not implemented for the type `T` [E0277]
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
src/tests.rs:69:41: 84:6 note: expansion site
src/tests.rs:73:22: 73:35 note: `T` cannot be sent between threads safely
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
src/tests.rs:69:41: 84:6 note: expansion site
src/tests.rs:73:22: 73:35 error: the trait `core::marker::Sync` is not implemented for the type `for<'r> core::ops::Fn(&'r mut T)` [E0277]
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
src/tests.rs:69:41: 84:6 note: expansion site
src/tests.rs:73:22: 73:35 note: `for<'r> core::ops::Fn(&'r mut T)` cannot be shared between threads safely
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
I have tried adding Arcs around the types being sent to the thread, no dice, same error.
pub fn test_fixture_runner<T: TestFixture>(fixture: &mut T) {
fixture.setup();
let fix_arc = Arc::new(Mutex::new(fixture));
let _r = fixture.tests().iter().map(|t| {
let test_arc = Arc::new(Mutex::new(t));
let fix_arc_clone = fix_arc.clone();
let test_arc_clone = test_arc.clone();
let handle = thread::spawn(move || {
let thread_test = test_arc_clone.lock().unwrap();
let thread_fix = fix_arc_clone.lock().unwrap();
(*thread_fix).before_each();
(*thread_test)(*thread_fix);
(*thread_fix).after_each();
});
if let Err(_) = handle.join() {
println!("Test failed!")
}
});
fixture.teardown();
}
A sample test fixture would be something like
struct BuiltinTests {
pwd: PathBuf
}
impl TestFixture for BuiltinTests {
fn setup(&mut self) {
let mut pwd = env::temp_dir();
pwd.push("pwd");
fs::create_dir(&pwd);
self.pwd = pwd;
}
fn teardown(&mut self) {
fs::remove_dir(&self.pwd);
}
fn tests(&mut self) -> Vec<Box<Fn(&mut BuiltinTests)>> {
vec![Box::new(BuiltinTests::cd_with_no_args)]
}
}
impl BuiltinTests {
fn new() -> BuiltinTests {
BuiltinTests {
pwd: PathBuf::new()
}
}
}
fn cd_with_no_args(&mut self) {
let home = String::from("/");
env::set_var("HOME", &home);
let mut cd = Cd::new();
cd.run(&[]);
assert_eq!(env::var("PWD"), Ok(home));
}
#[test]
fn cd_tests() {
let mut builtin_tests = BuiltinTests::new();
test_fixture_runner(&mut builtin_tests);
}
My whole intention of using threads is isolation from the test runner. If a test fails an assertion it causes a panic which kills the runner. Thanks for any insight, I'm willing to change my design if that will fix the panic problem.
There are several problems with your code, I'll show you how to fix them one by one.
The first problem is that you're using map() to iterate over an iterator. It won't work correctly because map() is lazy - unless you consume the iterator, the closure you passed to it won't run. The correct way is to use for loop:
for t in fixture().tests().iter() {
Second, you're iterating the vector of closures by reference:
fixture.tests().iter().map(|t| {
iter() on a Vec<T> returns an iterator yielding items of type &T, so your t will be of type &Box<Fn(&mut Self)>. However, Box<Fn(&mut T)> does not implement Sync by default (it is a trait object which have no information about the underlying type except that you specified explicitly), so &Box<Fn(&mut T)> can't be used across multiple threads. That's what the second error you see is about.
Most likely you don't want to use these closures by reference; you probably want to move them to the spawned thread entirely. For this you need to use into_iter() instead of iter():
for t in fixture.tests().into_iter() {
Now t will be of type Box<Fn(&mut T)>. However, it still can't be sent across threads. Again, it is a trait object, and the compiler does not know if the type contained inside is Send. For this you need to add Send bound to the type of the closure:
fn tests(&mut self) -> Vec<Box<Fn(&mut Self)+Send>>
Now the error about Fn is gone.
The last error is about Send not being implemented for T. We need to add a Send bound on T:
pub fn test_fixture_runner<T: TestFixture+Send>(fixture: &mut T) {
And now the error becomes more comprehensible:
test.rs:18:22: 18:35 error: captured variable `fixture` does not outlive the enclosing closure
test.rs:18 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
test.rs:18:5: 28:6 note: expansion site
test.rs:15:66: 31:2 note: captured variable is valid for the anonymous lifetime #1 defined on the block at 15:65
test.rs:15 pub fn test_fixture_runner<T: TestFixture+Send>(fixture: &mut T) {
test.rs:16 fixture.setup();
test.rs:17
test.rs:18 for t in fixture.tests().into_iter() {
test.rs:19 let handle = thread::spawn(move || {
test.rs:20 fixture.before_each();
...
note: closure is valid for the static lifetime
This error happens because you're trying to use a reference in a spawn()ed thread. spawn() requires its closure argument to have 'static bound, that is, its captured environment must not contain references with non-'static lifetimes. But that's exactly what happens here - &mut T is not 'static. spawn() design does not prohibit avoiding joining, so it is explicitly written to disallow passing non-'static references to the spawned thread.
Note that while you're using &mut T, this error is unavoidable, even if you put &mut T in Arc, because then the lifetime of &mut T would be "stored" in Arc and so Arc<Mutex<&mut T>> also won't be 'static.
There are two ways to do what you want.
First, you can use the unstable thread::scoped() API. It is unstable because it is shown to allow memory unsafety in safe code, and the plan is to provide some kind of replacement for it in the future. However, you can use it in nightly Rust (it won't cause memory unsafety by itself, only in specifically crafted situations):
pub fn test_fixture_runner<T: TestFixture+Send>(fixture: &mut T) {
fixture.setup();
let tests = fixture.lock().unwrap().tests();
for t in tests.into_iter() {
let f = &mut *fixture;
let handle = thread::scoped(move || {
f.before_each();
t(f);
f.after_each();
});
handle.join();
}
fixture.teardown();
}
This code compiles because scoped() is written in such a way that it guarantees (in most cases) that the thread won't outlive all captured references. I had to reborrow fixture because otherwise (because &mut references aren't copyable) it would be moved into the thread and fixture.teardown() would be prohibited. Also I had to extract tests variable because otherwise the mutex will be locked by the main thread for the duration of the for loop which would naturally disallow locking it in the child threads.
However, with scoped() you can't isolate the panic in the child thread. If the child thread panics, this panic will be rethrown from join() call. This may or may not be a problem in general, but I think it is a problem for your code.
Another way is to refactor your code to hold the fixture in Arc<Mutex<..>> from the beginning:
pub fn test_fixture_runner<T: TestFixture + Send + 'static>(fixture: Arc<Mutex<T>>) {
fixture.lock().unwrap().setup();
for t in fixture.lock().unwrap().tests().into_iter() {
let fixture = fixture.clone();
let handle = thread::spawn(move || {
let mut fixture = fixture.lock().unwrap();
fixture.before_each();
t(&mut *fixture);
fixture.after_each();
});
if let Err(_) = handle.join() {
println!("Test failed!")
}
}
fixture.lock().unwrap().teardown();
}
Note that now T has also to be 'static, again, because otherwise it couldn't be used with thread::spawn() as it requires 'static. fixture inside the inner closure is not &mut T but a MutexGuard<T>, and so it has to be explicitly converted to &mut T in order to pass it to t.
This may seem overly and unnecessarily complex, however, such design of a programming language does prevent you from making many errors in multithreaded programming. Each of the above errors we have seen is valid - each of them would be a potential cause of memory unsafety or data races if it was ignored.
As stated in the Rust HandBook's Concurrency section:
When a type T implements Send, it indicates to the compiler that something of this type is able to have ownership transferred safely between threads.
If you do not implement Send, ownership cannot be transfered between threads.