Box<dyn> and Arc<dyn> in multiple threads situation - rust

i have code below, it is a threadpool implementation, type Job is my problem:
pub struct ThreadPool {
workers:Vec<Worker>,
sender: mpsc::Sender<Job>
}
type Job = Box<dyn FnOnce() + Send + 'static>;
impl ThreadPool {
pub fn new (size: usize) -> ThreadPool {
let mut workers = Vec::with_capacity(size);
let (sender, receiver) = mpsc::channel();
let receiver = Arc::new(Mutex::new(receiver));
for id in 0..size {
workers.push(Worker::new(id, Arc::clone(&receiver)));
}
ThreadPool{ workers, sender }
}
pub fn execute<F>(&self, f: F)
where
F: FnOnce()->() + Send + 'static,
{
&self.sender.send(Box::new(f)).unwrap();
}
}
struct Worker {
id:usize,
thread: thread::JoinHandle<Arc<Mutex<Receiver<Job>>>>
}
impl Worker {
fn new(id:usize, reciever:Arc<Mutex<mpsc::Receiver<Job>>>) -> Worker {
let thread = thread::spawn( move || loop {
let job = reciever.lock().unwrap().recv().unwrap();
println!("Worker {id} got a job; executing.");
job()
});
Worker { id, thread }
}
}
the code above can work well and type Job is defined by Box.
then i tried to define type Job by Arc,
type Job = Arc<dyn FnOnce() + Send + 'static>;
the code shows error below:
error[E0277]: `(dyn FnOnce() + Send + 'static)` cannot be shared between threads safely
--> src\lib.rs:36:22
|
36 | let thread = thread::spawn( move || loop {
| ^^^^^^^^^^^^^ `(dyn FnOnce() + Send + 'static)` cannot be shared between threads safely
|
= help: the trait `Sync` is not implemented for `(dyn FnOnce() + Send + 'static)`
= note: required for `Arc<(dyn FnOnce() + Send + 'static)>` to implement `Send`
= note: 1 redundant requirement hidden
= note: required for `std::sync::mpsc::Receiver<Arc<(dyn FnOnce() + Send + 'static)>>` to implement `Send`
= note: required for `Mutex<std::sync::mpsc::Receiver<Arc<(dyn FnOnce() + Send + 'static)>>>` to implement `Sync`
= note: 1 redundant requirement hidden
= note: required for `Arc<Mutex<std::sync::mpsc::Receiver<Arc<(dyn FnOnce() + Send + 'static)>>>>` to implement `Send`
note: required by a bound in `spawn`
i think Arc is a thread safe smart pointer, why i can not replace Arc with Box?

Arc is not always thread-safe, it is only Send and Sync if the type inside is Send and Sync. This is because it can be used to access data from threads other than the thread created it. dyn FnOnce() + Send is obviously Send, but not Sync. Replace it with dyn FnOnce() + Send + Sync.

Related

Cannot find type `F` in this scope

This might seem obvious, but I want to learn something from it. I am doing some exercises, reimplementing the 'multi-thread http server' of the book, chapter 20. It's not completely the same (like I didn't use a Box to hold the task closure), but that's not important.
use std::{thread};
use std::sync::{mpsc, Arc, Mutex};
pub struct Executor {
threads: Vec<thread::JoinHandle<()>>,
sender: mpsc::Sender<FnOnce() + Send + 'static>,
}
impl Executor {
pub fn new(thread_num: usize) -> Executor {
let (tx, rx) = mpsc::channel();
let rx = Arc::new(Mutex::new(rx));
let mut threads = Vec::with_capacity(thread_num);
for i in 0..thread_num {
let handle = thread::spawn(|| {
loop {
let rx = rx.lock();
let f = rx.recv().unwrap();
f();
}
});
threads.push(handle);
}
Executor { threads, sender: tx }
}
pub fn run(&mut self, f: F)
where
F: FnOnce() + Send + 'static, { //*** the compile complains here***
// let bf = Box::new(f);
self.sender.send(f);
}
}
However, the compiler says:
error[E0412]: cannot find type `F` in this scope
--> src\lib.rs:31:13
|
31 | F: FnOnce() + Send + 'static, {
| ^ help: a trait with a similar name exists: `Fn`
|
::: C:\Users\ynx\.rustup\toolchains\stable-x86_64-pc-windows-gnu\lib/rustlib/src/rust\library\core\src\ops\function.rs:67:1
|
67 | pub trait Fn<Args>: FnMut<Args> {
| ------------------------------- similarly named trait `Fn` defined here
Where am I wrong?
You need to declare your generic with <F>:
pub fn run<F>(&mut self, f: F)
where
F: FnOnce() + Send + 'static, {

What is the difference between `future` and `async move { future }`?

I'm wondering why changing the code from using a Future directly to using it in an async move block makes a difference. Passing it to tokio::spawn directly yields an error:
use std::error::Error;
use tokio::sync::{mpsc::{self, Receiver}, oneshot::Sender};
struct Msg {
resp: Sender<Vec<u8>>,
}
async fn client_thread(
mut rx: Receiver<Msg>,
) -> Result<(), Box<dyn Error>> {
while let Some(msg) = rx.recv().await {
msg.resp
.send(vec![]) // return some data
.unwrap();
}
Ok(())
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let (_tx, rx) = mpsc::channel(5);
tokio::spawn(client_thread(rx));
Ok(())
}
error[E0277]: `(dyn std::error::Error + 'static)` cannot be sent between threads safely
--> src/main.rs:23:5
|
23 | tokio::spawn(client_thread(rx)); // <--- Difference is here
| ^^^^^^^^^^^^ `(dyn std::error::Error + 'static)` cannot be sent between threads safely
|
= help: the trait `Send` is not implemented for `(dyn std::error::Error + 'static)`
= note: required because of the requirements on the impl of `Send` for `Unique<(dyn std::error::Error + 'static)>`
= note: required because it appears within the type `Box<(dyn std::error::Error + 'static)>`
= note: required because it appears within the type `Result<(), Box<(dyn std::error::Error + 'static)>>`
note: required by a bound in `tokio::spawn`
--> /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/task/spawn.rs:128:20
|
128 | T::Output: Send + 'static,
| ^^^^ required by this bound in `tokio::spawn`
But changing it to an async block makes it compile:
tokio::spawn(async move { client_thread(rx) });
The return type of client_thread is exactly the same as the main function, however, which runs with Tokio without any problems. Moreover, the error type from reqwest implements Send.
Wrapping the Future in an async {} block as you've done makes it compile but doesn't actually run the Future. You need to .await it:
use futures::executor::block_on; // 0.3.12
async fn f(s: &str) {
println!("ran {}", s);
}
fn main() {
block_on(f("first"));
block_on(async { f("second") }); // this doesn't print
block_on(async { f("third").await });
}
ran first
ran third
Since the future is not ran, it doesn't end up affecting the traits for the async {} block and thus can can satisfy the Send constraint. You'll have the same issue again with .await.
In your case, all you need to do is ensure that the future implements Send so it can be ran with tokio::spawn. The fix is to dictate that the Error trait being returned implements Send:
async fn client_thread(
mut rx: Receiver<ClientMsg>,
) -> Result<(), Box<dyn Error + Send>> {
// ... ^^^^^^
}

How to duplicate Rust Stream

I have a program similar to the following:
struct MyEvent { /* some fields */ }
struct MyStruct { /* some fields */ }
struct MyStreamer { /* holds some state */ }
impl MyStreamer {
pub fn stream_objects<'a, 'b: 'a>(
&'a self,
event_stream: Pin<Box<dyn Stream<Item = MyEvent> + Send + 'b>>,
) -> Pin<Box<dyn Stream<Item = Arc<MyStruct>> + Send + 'a>> { /* implementation */ }
}
The goal is to process events and build a stream of MyStruct. I then have two consumers for the stream of MyStruct and I'm struggling to duplicate it.
I'm trying to write the following function (also see my attempted implementation):
pub fn duplicate_stream<'a, 'b: 'a>(
&'a self,
struct_stream: Pin<Box<dyn Stream<Item = Arc<MyStruct>> + Send + 'b>>,
) -> (
Pin<Box<dyn Stream<Item = Arc<MyStruct>> + Send + 'b>>,
Pin<Box<dyn Stream<Item = Arc<MyStruct>> + Send + 'b>>
) {
let (s1, r1) = mpsc::unbounded::<Arc<MyStruct>>();
let (s2, r2) = mpsc::unbounded::<Arc<MyStruct>>();
let s = s1.fanout(s2);
let _handle = tokio::spawn(async move { struct_stream.map(Ok).forward(s).await });
(r1.boxed(), r2.boxed())
}
At this point, I'm told the following:
|
155 | struct_stream: Pin<Box<dyn Stream<Item = Arc<MyStruct>> + Send + 'b>>,
| ----------------------------------------------- this data with lifetime `'b`...
...
165 | let _handle = tokio::spawn(async move { struct_stream.map(Ok).forward(s).await });
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...is captured here...
|
note: ...and is required to live as long as `'static` here
--> *file name here*
|
165 | let _handle = tokio::spawn(async move { struct_stream.map(Ok).forward(s).await });
| ^^^^^^^^^^^^
I can remove the lifetimes, but then static will be inferred and I get errors in the callers.
I'm curious to understand what the best way is to clone all the elements of a stream and obtain two identical streams. Using tokio::spawn and mpsc channels seems to require changing a lot of lifetimes to static.
I think this is the XY problem
Compiler is correct and you probably should think about the lifetimes because the fanout is ok.
tokio::spawn requires 'static and you have 'b lifetime specified for your struct_stream. Maybe wrap the struct_stream in an Arc/Rc?

How to pass data struct with trait over multiple threads in rust? [duplicate]

This question already has answers here:
Sharing a struct with trait objects as properties across threads
(1 answer)
Sending trait objects between threads in Rust
(1 answer)
Closed 2 years ago.
I am trying to store the state of my app which includes a list of traits that are initialized in the beginning and pass the state between multiple threads but I am getting an error.
use std::sync::{Arc, RwLock};
use std::thread;
use std::time::Duration;
trait TestTrait {
fn test(&self);
}
struct MyTestStruct {}
impl TestTrait for MyTestStruct {
fn test(&self) {
println!("Test trait called");
}
}
struct DataStore {
t: Box<dyn TestTrait>,
}
fn main() {
let s = Arc::new(RwLock::new(DataStore {
t: Box::new(MyTestStruct {}),
}));
let s_clone = s.clone();
thread::spawn(move || {
for i in 1..10 {
s_clone.read().unwrap().t.test();
println!("hi number {} from the spawned thread!", i);
thread::sleep(Duration::from_millis(1));
}
});
for i in 1..5 {
println!("hi number {} from the main thread!", i);
thread::sleep(Duration::from_millis(1));
}
}
When I run this, I get the following error:
error[E0277]: `(dyn TestTrait + 'static)` cannot be sent between threads safely
--> src/main.rs:26:5
|
26 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `(dyn TestTrait + 'static)` cannot be sent between threads safely
|
= help: the trait `std::marker::Send` is not implemented for `(dyn TestTrait + 'static)`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<(dyn TestTrait + 'static)>`
= note: required because it appears within the type `std::boxed::Box<(dyn TestTrait + 'static)>`
= note: required because it appears within the type `DataStore`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::RwLock<DataStore>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<std::sync::RwLock<DataStore>>`
= note: required because it appears within the type `[closure#src/main.rs:26:19: 32:6 s_clone:std::sync::Arc<std::sync::RwLock<DataStore>>]`
error[E0277]: `(dyn TestTrait + 'static)` cannot be shared between threads safely
--> src/main.rs:26:5
|
26 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `(dyn TestTrait + 'static)` cannot be shared between threads safely
|
= help: the trait `std::marker::Sync` is not implemented for `(dyn TestTrait + 'static)`
= note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<(dyn TestTrait + 'static)>`
= note: required because it appears within the type `std::boxed::Box<(dyn TestTrait + 'static)>`
= note: required because it appears within the type `DataStore`
= note: required because of the requirements on the impl of `std::marker::Sync` for `std::sync::RwLock<DataStore>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<std::sync::RwLock<DataStore>>`
= note: required because it appears within the type `[closure#src/main.rs:26:19: 32:6 s_clone:std::sync::Arc<std::sync::RwLock<DataStore>>]`
How do I fix this?
thread::spawn requires a closure that implements Send. Arc<T> only implements Send if T implements both Send and Sync.
Send and Sync are auto traits. You can add auto traits to a dyn Trait type to provide more information about the erased type:
struct DataStore {
t: Box<dyn TestTrait + Send + Sync>,
}
If you are going to write dyn TestTrait + Send + Sync everywhere, then another option is to declare Send and Sync as supertraits of TestTrait:
trait TestTrait: Send + Sync {
fn test(&self);
}

Sharing a struct with trait objects as properties across threads

I have the code below. With the commented out parts, it's working. When I uncomment the parts it does not compile anymore.
How can I adjust the commented parts to make them work, i.e., I want to make threads access the expression tree simultaneously.
When I try it, the compiler starts with errors about thread safeness.
I read the Rust book and know C/C++, but didn't understood everything about Rust type system and semantics yet.
use std::thread;
use std::sync::Arc;
pub trait Expr {
fn run(&self) -> i32;
}
pub struct ConstantExpr {
n: i32,
}
impl ConstantExpr {
pub fn new(n: i32) -> Self {
Self { n }
}
}
impl Expr for ConstantExpr {
fn run(&self) -> i32 {
self.n
}
}
pub struct AddExpr {
expr1: Box<Expr>,
expr2: Box<Expr>,
}
impl AddExpr {
pub fn new(expr1: Box<Expr>, expr2: Box<Expr>) -> Self {
Self { expr1, expr2 }
}
}
impl Expr for AddExpr {
fn run(&self) -> i32 {
self.expr1.run() + self.expr2.run()
}
}
struct Container {
x: i32,
cached_expr: Arc<Expr>,
}
impl Container {
fn new() -> Self {
Self {
x: 0,
cached_expr: Arc::new(AddExpr::new(
Box::new(ConstantExpr::new(10)),
Box::new(ConstantExpr::new(1)),
)),
}
}
}
fn main() {
let container = Arc::new(Container::new());
let container1 = Arc::clone(&container);
/*
let thread1 = thread::spawn(move || {
println!("thread1: {}", container1.x);
println!("thread1: {}", container1.cached_expr.run());
});
*/
println!("main: {}", container.x);
println!("main: {}", container.cached_expr.run());
//thread1.join().unwrap();
}
The error:
error[E0277]: the trait bound `Expr + 'static: std::marker::Send` is not satisfied
--> src/main.rs:64:19
|
64 | let thread1 = thread::spawn(move || {
| ^^^^^^^^^^^^^ `Expr + 'static` cannot be sent between threads safely
|
= help: the trait `std::marker::Send` is not implemented for `Expr + 'static`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<Expr + 'static>`
= note: required because it appears within the type `Container`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<Container>`
= note: required because it appears within the type `[closure#src/main.rs:64:33: 67:6 container1:std::sync::Arc<Container>]`
= note: required by `std::thread::spawn`
error[E0277]: the trait bound `Expr + 'static: std::marker::Sync` is not satisfied
--> src/main.rs:64:19
|
64 | let thread1 = thread::spawn(move || {
| ^^^^^^^^^^^^^ `Expr + 'static` cannot be shared between threads safely
|
= help: the trait `std::marker::Sync` is not implemented for `Expr + 'static`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<Expr + 'static>`
= note: required because it appears within the type `Container`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<Container>`
= note: required because it appears within the type `[closure#src/main.rs:64:33: 67:6 container1:std::sync::Arc<Container>]`
= note: required by `std::thread::spawn`
I find the error message pretty straightforward:
the trait std::marker::Send is not implemented for Expr + 'static
required because of the requirements on the impl of std::marker::Send for std::sync::Arc<Expr + 'static>
required because it appears within the type Container
required because of the requirements on the impl of std::marker::Send for std::sync::Arc<Container>
required because it appears within the type [closure#src/main.rs:64:33: 67:6 container1:std::sync::Arc<Container>]
required by std::thread::spawn
You are trying to move your Arc<Container> to another thread, but it contains an Arc<Expr + 'static>, which cannot be guaranteed to be safely sent (Send) or shared (Sync) across threads.
Either add Send and Sync as supertraits to Expr:
pub trait Expr: Send + Sync { /* ... */ }
Or add them as trait bounds to your trait objects:
pub struct AddExpr {
expr1: Box<Expr + Send + Sync>,
expr2: Box<Expr + Send + Sync>,
}
impl AddExpr {
pub fn new(expr1: Box<Expr + Send + Sync>, expr2: Box<Expr + Send + Sync>) -> Self {
Self { expr1, expr2 }
}
}
struct Container {
x: i32,
cached_expr: Arc<Expr + Send + Sync>,
}
See also:
How can I share references across threads?
Multithreaded application fails to compile with error-chain
Is there any way to implement the Send trait for ZipFile?
How do I share a generic struct between threads using Arc<Mutex<MyStruct<T>>>?

Resources