I am attempting to implement a queue where producers can add a struct in that implements QueueTrait and when dequeued the invoke method of the trait will be called on a different thread. This does not compile due to the following error:
self.queue.enqueue(self.data);
^^^^^^^^^ expected trait object `dyn QueueTrait`, found struct `ProducerData`
expected struct `std::sync::Arc<std::boxed::Box<dyn QueueTrait + std::marker::Send>>`
found struct `std::sync::Arc<std::boxed::Box<ProducerData>>
Is it possible to cast ProducerData to QueueTrait? Is there any way to avoid splitting Producer into ProducerData altogether and simply add Producer itself into the queue?
use std::collections::VecDeque;
use std::sync::{Arc, Condvar, Mutex};
use std::thread;
pub struct Queue<'a> {
items: Mutex<VecDeque<Arc<Box<dyn QueueTrait + Send + 'a>>>>,
condvar: Condvar,
}
impl<'a> Queue<'a> {
pub fn new() -> Self {
Self {
condvar: Condvar::new(),
items: Mutex::new(VecDeque::new()),
}
}
pub fn enqueue(&self, qt: Arc<Box<dyn QueueTrait + Send + 'a>>) {
self.items.lock().unwrap().push_back(qt);
self.condvar.notify_one();
}
fn dequeue(&self) -> Arc<Box<dyn QueueTrait + Send + 'a>> {
let mut q = self.items.lock().unwrap();
while q.is_empty() {
q = self.condvar.wait(q).unwrap();
}
q.pop_front().unwrap()
}
pub fn dispatch(&self) {
loop {
let qt = self.dequeue();
qt.invoke();
}
}
}
struct ProducerData {
invoke_count: Mutex<u32>
}
struct Producer {
queue: Arc<Queue<'static>>,
data: Arc<Box<ProducerData>>
}
impl Producer {
fn new(queue: Arc<Queue<'static>>) -> Self {
Self { queue, data: Arc::new(Box::new(ProducerData{ invoke_count: Mutex::new(0)}) ) }
}
fn produce(&self) {
self.queue.enqueue(self.data);
}
}
pub trait QueueTrait {
fn invoke(&self);
fn as_trait(&self) -> &dyn QueueTrait;
}
impl QueueTrait for ProducerData {
fn invoke(&self) {
self.invoke_count.lock().unwrap() +=1;
println!("Invoke called!")
}
fn as_trait(&self) -> &dyn QueueTrait {
self as &dyn QueueTrait
}
}
fn main() {
let q = Arc::new(Queue::new());
let p1 = Producer::new(Arc::clone(&q));
// Consumer thread
let c = thread::Builder::new().name("consumer".to_string()).spawn(move
|| q.dispatch() ).unwrap();
// Produce on main thread
p1.produce();
c.join().unwrap();
}
You used casting already in as_trait but just to make it clearer. Casting an object to a trait can be done with the as keyword:
use std::sync::Arc;
struct MyStruct;
trait MyTrait {}
impl MyTrait for MyStruct {}
fn main() {
let my_struct: MyStruct = MyStruct;
// behind a reference
let trait_object: &dyn MyTrait = &MyStruct as &dyn MyTrait;
// behind a box
let trait_object: Box<dyn MyTrait> = Box::new(MyStruct) as Box<dyn MyTrait>;
// behind an Arc
let trait_object: Arc<dyn MyTrait> = Arc::new(MyStruct) as Arc<dyn MyTrait>;
// behind an Arc Box
// DOESN'T COMPILE!
let trait_object: Arc<Box<dyn MyTrait>> = Arc::new(Box::new(MyStruct)) as Arc<Box<dyn MyTrait>>;
}
But it doesn't work behind two indirections for example Arc and Box.
This whole setup you have seems very overcomplicted to me. As pointed out by #kmdreko using just Arc seems to work. You should re-think what your program is trying to do and think of a simpler way.
Related
I am building a Rocket app and want it to manage some objects for me. For that, they need to be Send and Sync. Here's a minimal contrived example that shows the error I am getting (Playground):
trait MyTrait {
fn foo(&self) {}
}
struct TraitThing {}
impl MyTrait for TraitThing {
fn foo(&self) {
println!("Foo!");
}
}
struct Thing {
tt: &'static (dyn MyTrait + Send + Sync),
}
impl Thing {
fn with_trait(tt: &'static (dyn MyTrait + Send + Sync)) -> Self {
Self { tt }
}
}
fn main() {
let tt = TraitThing {};
let thing = Thing::with_trait(&tt);
thing.tt.foo();
}
I probably don't understand lifetimes of 'static. Ideally, I want Thing to own tt, but as far as I understand, since my TraitThing isn't Sized (and probably won't ever be), it cannot be a trait object, so it must be passed by reference.
So, how do I solve this while keeping the Send and Sync traits?
Thanks!
Additional note: I've been reading a lot about lifetimes and traits in the Rust book and other places, but I'd appreciate further reading material.
You need to use Box:
trait MyTrait {
fn foo(&self) {}
}
struct TraitThing {}
impl MyTrait for TraitThing {
fn foo(&self) {
println!("Foo!");
}
}
struct Thing {
tt: Box<dyn MyTrait + Send + Sync>,
}
impl Thing {
fn with_trait(tt: Box<dyn MyTrait + Send + Sync>) -> Self {
Self { tt }
}
}
fn main() {
let tt = TraitThing {};
let thing = Thing::with_trait(Box::new(tt));
thing.tt.foo();
}
Playground
You can use a Box but it will allocate to the Heap and use dynamic dispatch which is not useful in your case. It is useful when you want a list of different type that implements a similar Trait like :
struct Thing {
tt: Vec<Box<dyn MyTrait + Send + Sync>>,
}
But in your case you only have one element so you can use Generics in this setup:
trait MyTrait {
fn foo(&self) {}
}
struct TraitThing {}
impl MyTrait for TraitThing {
fn foo(&self) {
println!("Foo!");
}
}
struct Thing<T>
where
T: MyTrait + Send + Sync
{
tt: T,
}
impl<T> Thing<T>
where
T: MyTrait + Send + Sync
{
fn with_trait(tt: T) -> Self {
Self { tt }
}
}
fn main() {
let tt = TraitThing {};
let thing = Thing::with_trait(tt);
thing.tt.foo();
}
I want to have a background worker which uses a trait implementation / object for some time. The background worker owns this object as long as it is used. After the background worker is "destroyed", the object should be free to be used again.
I tried to make all the things with async/await, but it produced some more problems. Therefore, I use plain threads to create kind of a minimal example. First I also used Box<&dyn mut...> to pass the object to the background worker, but I think that is not even needed.
My minimal example contains a MyWriter-trait which can write string to somewhere. There exists one implementation which writes strings to stdout. A background-worker uses this writer for a background-job. The worker has a start-method to start the job and a stop-method to join it (in my real code I would use a channel to send a stop-info to the worker and joining then).
I'll post the code and then a description with my problems:
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=a01745c15ba1088acd2e3d287d60e270
use std::sync::Arc;
use std::sync::Mutex;
use std::thread::{spawn, JoinHandle};
/* Trait + an implementation */
trait MyWriter {
fn write(&mut self, text: &str);
}
struct StdoutWriter {}
impl StdoutWriter {
pub fn new() -> Self {
Self {}
}
}
impl MyWriter for StdoutWriter {
fn write(&mut self, text: &str) {
println!("{}", text);
}
}
/* A background job which uses a "MyWriter" */
struct BackgroundJob<'a> {
writer: Arc<Mutex<&'a dyn MyWriter>>,
job: Option<JoinHandle<()>>,
}
impl<'a> BackgroundJob<'a> {
pub fn new(writer: &'a mut dyn MyWriter) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
pub fn start(&mut self) {
assert!(self.job.is_none());
let writer = &self.writer;
self.job = Some(std::thread::spawn(move || {
// this background job uses "writer"
let mut my_writer = writer.lock().unwrap();
my_writer.write("x");
// do something
my_writer.write("y");
}));
}
pub fn stop(&mut self) {
if let Some(job) = self.job {
job.join().unwrap();
self.job = None;
}
}
}
/* Using BackgroundJob */
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
{
let mut job = BackgroundJob::new(&mut writer);
// inside this block, writer is owned by "job"
job.start();
job.stop();
}
// writer should be usable again
writer.write("b");
}
The desired output on stdout is a\nx\ny\nz\n, but the program does not even compile. My main problem is that (dyn MyWriter + 'a) cannot be shared between threads safely (compiler error).
How can I implement Send / Sync for a trait? It does not seem to be possible. Actually, I assumed it should be ok if the object (or a ref.) is inside Arc<Mutex<...>>, but that does not seem to be sufficient. Why not?
Maybe someone has an idea how this can be fixed or even more important what exactly is the underlying issue?
Putting a reference in an Arc doesn't work. Since the Arc can be kept alive indefinitely simply by cloning it, the reference could easily outlive whatever it was borrowed from, so that can't compile. You need to put an owned object in the Arc, such as Box<dyn MyWriter>. (Ideally you'd just use Arc<dyn MyWriter>, but that would conflict with returning the writer from the BackgroundJob, as shown below.)
Since you can't borrow from writer in main, you must move it into the BackgroundJob. But at this point you've relinquished ownership over writer, having moved the value to BackgroundJob, so your only option is to have BackgroundJob return the writer. However, since BackgroundJob keeps its writer behind a trait object, it can only give back the Box<dyn MyWriter> it stores, not the original StdoutWriter.
Here is the version that works that way, retaining type erasure and giving back the type-erased writer:
// Trait definition and StdoutWriter implementation unchanged
struct BackgroundJob {
writer: Arc<Mutex<Box<dyn MyWriter + Send>>>,
job: Option<JoinHandle<()>>,
}
impl BackgroundJob {
pub fn new(writer: Box<dyn MyWriter + Send>) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
pub fn start(&mut self) {
assert!(self.job.is_none());
let writer = Arc::clone(&self.writer);
self.job = Some(std::thread::spawn(move || {
// this background job uses "writer"
let mut my_writer = writer.lock().unwrap();
my_writer.write("x");
// do something
my_writer.write("y");
}));
}
pub fn stop(&mut self) {
if let Some(job) = self.job.take() {
job.join().unwrap();
}
}
pub fn into_writer(self) -> Box<dyn MyWriter> {
Arc::try_unwrap(self.writer)
.unwrap_or_else(|_| panic!())
.into_inner()
.unwrap()
}
}
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
let mut writer = {
let mut job = BackgroundJob::new(Box::new(writer));
job.start();
job.stop();
job.into_writer()
};
writer.write("b");
}
Playground
A version that gave back the writer of the same type would have to give up on type erasure and be generic over the writer type. Though a bit more complex, its ownership semantics would be very close (at least conceptually) to what you originally envisioned:
struct BackgroundJob<W> {
writer: Arc<Mutex<W>>,
job: Option<JoinHandle<()>>,
}
impl<W: MyWriter + Send + 'static> BackgroundJob<W> {
pub fn new(writer: W) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
// start() and stop() are unchanged
pub fn into_writer(self) -> W {
Arc::try_unwrap(self.writer)
.unwrap_or_else(|_| panic!())
.into_inner()
.unwrap()
}
}
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
{
// inside this block, writer is moved into "job"
let mut job = BackgroundJob::new(writer);
job.start();
job.stop();
// reclaim the writer
writer = job.into_writer();
}
writer.write("b");
}
Playground
The main issue is that you want to pass a reference to the thread. The problem with that approach is that the thread can outlive the referenced object. Obviously this does not happen in your case, but the rust compiler cannot reason about that.
The solution to that problem is to use Arc<Mutex<dyn MyType>> instead of Arc<Mutex<&dyn MyType>> - no lifetimes - no problems.
The next issue is with Mutex<T> - it can be send across threads only if T can. So you have to make T, in your case dyn MyType, implement Send. This can be done in two ways:
Make MyType require Send - in that case that trait can be implemented only by Send objects:
trait MyWriter : Send{
fn write(&mut self, text: &str);
}
Or use an additional trait bound - in that case your trait is less restrictive, but you must always specify MyTrait + Send when you want to send it across threads:
Arc<Mutex<dyn MyWriter + Send>>
So far so good, but now your new() method does not work, because dyn MyWriter is not Sized. In order to fix that you have to make your method generic:
pub fn new<T: MyWriter + Send>(writer: T) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
or directly pass an Arc<Mutex<dyn MyWriter + Send>>:
pub fn new(writer: Arc<Mutex<dyn MyWriter + Send>>) -> Self {
Self { writer, job: None }
}
Full working code
use std::sync::Arc;
use std::sync::Mutex;
use std::thread::JoinHandle;
trait MyWriter {
fn write(&mut self, text: &str);
}
struct StdoutWriter {}
impl StdoutWriter {
pub fn new() -> Self {
Self {}
}
}
impl MyWriter for StdoutWriter {
fn write(&mut self, text: &str) {
println!("{}", text);
}
}
/* A background job which uses a "MyWriter" */
struct BackgroundJob {
writer: Arc<Mutex<dyn MyWriter + Send>>,
job: Option<JoinHandle<()>>,
}
impl BackgroundJob {
pub fn new(writer: Arc<Mutex<dyn MyWriter + Send>>) -> Self {
Self { writer, job: None }
}
pub fn start(&mut self) {
assert!(self.job.is_none());
let writer = self.writer.clone();
self.job = Some(std::thread::spawn(move || {
let mut my_writer = writer.lock().unwrap();
my_writer.write("x");
// do something
my_writer.write("y");
}));
}
pub fn stop(&mut self) {
if let Some(job) = self.job.take() {
job.join().unwrap();
}
}
}
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
let writer = Arc::new(Mutex::new(writer));
{
let mut job = BackgroundJob::new(writer.clone());
// inside this block, writer is owned by "job"
job.start();
job.stop();
}
// you have to acquire the lock in order to use the writer
writer.lock().unwrap_or_else(|e| e.into_inner()).write("b");
}
pub struct ForesterViewModel {
m_tree_lines: Arc<Mutex<Vec<TreeLine>>>,
}
impl ForesterViewModel {
pub fn new() -> ForesterViewModel {
ForesterViewModel {
m_tree_lines: Arc::new(Mutex::new(vec![])),
}
}
pub fn get_the_forest(&mut self) -> &mut Vec<TreeLine> {
???????????????????????????????
}
}
I need help writing the get_the_forest function. I've tried many various things but they all return compilation errors. I need to return a mutable reference to Vec<TreeLine> which is wrapped behind an Arc and a Mutex in self.m_tree_lines.
There is no way of doing this.
You create a concrete MutexGuard object that releases the mutex when it dropped when you call lock; you cannot move a reference out of the scope that contains the guard:
pub fn as_mut(&mut self) -> &Whatever {
let mut guard = self.data.lock().unwrap();
Ok(guard.deref())
drop(guard) // <--- implicitly added here, which would invalidate the ref
}
You also cannot return both the mutex guard and a reference, for more complex reasons (basically rust cannot express that), for the same reason it cannot have a reference and an object in a single structure; see the discussion on Why can't I store a value and a reference to that value in the same struct?
...so basically your best bet is one of two things:
/// Return the mutex guard itself
pub fn get_the_forest(&mut self) -> Result<MutexGuard<Vec<TreeLine>>, TreeLockError> {
Ok(self.m_tree_lines.lock()?)
}
/// Pass a function in, which patches the mutable internal value
pub fn patch_forest(&mut self, patch: impl Fn(&mut Vec<TreeLine>)) -> Result<(), TreeLockError>{
let mut guard = self.m_tree_lines.lock()?;
patch(&mut guard); // <-- patch happens while guard is still alive
Ok(())
}
Full code:
use std::sync::{Arc, Mutex, MutexGuard};
use std::sync::PoisonError;
use std::error::Error;
use std::fmt;
use std::fmt::Formatter;
use std::ops::Deref;
#[derive(Debug, Copy, Clone)]
pub enum TreeLockError {
FailedToLock
}
impl Error for TreeLockError {}
impl fmt::Display for TreeLockError {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
write!(f, "{:?}", self)
}
}
impl<T> From<PoisonError<T>> for TreeLockError {
fn from(_: PoisonError<T>) -> Self {
TreeLockError::FailedToLock
}
}
// ---
#[derive(Debug)]
pub struct TreeLine {
pub value: &'static str
}
pub struct ForesterViewModel {
m_tree_lines: Arc<Mutex<Vec<TreeLine>>>,
}
impl ForesterViewModel {
pub fn new() -> ForesterViewModel {
ForesterViewModel {
m_tree_lines: Arc::new(Mutex::new(vec![])),
}
}
pub fn get_the_forest(&mut self) -> Result<MutexGuard<Vec<TreeLine>>, TreeLockError> {
Ok(self.m_tree_lines.lock()?)
}
pub fn patch_forest(&mut self, patch: impl Fn(&mut Vec<TreeLine>)) -> Result<(), TreeLockError>{
let mut guard = self.m_tree_lines.lock()?;
patch(&mut guard);
Ok(())
}
}
fn main() -> Result<(), Box<dyn Error>> {
let mut vm = ForesterViewModel::new();
{
let mut trees = vm.get_the_forest()?;
trees.push(TreeLine{ value: "one"});
trees.push(TreeLine{ value: "two"});
} // <--- Drop the mutable reference here so you can get it again later
// Patch
vm.patch_forest(|trees| {
trees.push(TreeLine{ value: "three"});
});
// ...
let trees = vm.get_the_forest()?;
println!("{:?}", trees.deref());
Ok(())
}
I'm trying to send a closure which will generate a structure to a thread, however when I try to do it I get a Sized error. I understand the error (the size is indeed not known at compile time), however adding Boxes and other such tricks does not seem to solve it.
I've tried to look into how to implement the Sized trait, however it seems to be quite special and honestly above my understanding.
I've written a minimal reproducible example:
use std::thread;
trait DataProcess {
fn start(&self);
fn run(&self);
fn stop(&self);
}
struct SomeDP {
name: String,
}
impl DataProcess for SomeDP {
fn start(&self) {
println!("Started");
}
fn run(&self) {
println!("Running");
}
fn stop(&self) {
println!("Stopped");
}
}
fn thread_maker(builder: Box<dyn Fn() -> (dyn DataProcess + Send)>) {
let thread_builder = thread::Builder::new();
let handle = thread_builder.spawn(move || {
let dp = builder();
dp.start();
});
}
fn main() {
let dp_builder = || SomeDP {
name: "nice".to_string(),
};
thread_maker(Box::new(dp_builder));
}
Which you can also find on the playground here
This works
use std::thread;
trait DataProcess{
fn start(&self);
fn run(&self);
fn stop(&self);
}
struct SomeDP{
name: String
}
impl DataProcess for SomeDP{
fn start(&self){println!("Started");}
fn run(&self){println!("Running");}
fn stop(&self){println!("Stopped");}
}
fn thread_maker<F>(builder: F)
where
F: Fn() -> Box<dyn DataProcess>,
F: Send + 'static {
let thread_builder = thread::Builder::new();
let handle = thread_builder.spawn(
move ||{
let dp = builder();
dp.start();
}
);
}
fn main(){
let dp_builder = || -> Box<dyn DataProcess> {
Box::new(SomeDP{name: "nice".to_string()})
};
thread_maker(dp_builder);
}
I am trying to store structs in a HashMap keyed by string so that I can later create new objects by string. Think of a REST API where clients can get the server to instantiate a specific object by supplying a name.
use std::collections::HashMap;
struct MyStruct;
impl MyStruct {
pub fn new() -> Self {
Self {}
}
}
struct MyOtherStruct;
impl MyOtherStruct {
pub fn new() -> Self {
Self {}
}
}
fn main() {
let mut h = HashMap::new();
h.insert("MyStruct", MyStruct);
h.insert("MyOtherStruct", MyOtherStruct);
// This is pseudo-code
let obj = h.get("MyStruct").unwrap()::new();
}
As I expected, this doesn't work due to syntax errors:
error: expected one of `.`, `;`, `?`, or an operator, found `::`
--> src/main.rs:25:41
|
25 | let obj = h.get("MyStruct").unwrap()::new();
| ^^ expected one of `.`, `;`, `?`, or an operator here
My second attempt was to store a reference to the new method of each struct instead of the types themselves.
use std::collections::HashMap;
struct MyStruct;
impl MyStruct {
pub fn new() -> Self {
Self {}
}
}
struct MyOtherStruct;
impl MyOtherStruct {
pub fn new() -> Self {
Self {}
}
}
fn main() {
let mut h = HashMap::new();
h.insert("MyStruct", &MyStruct::new);
h.insert("MyOtherStruct", &MyOtherStruct::new);
let obj = h.get("MyStruct").unwrap()();
}
This fails because the fn items have different types and can't be stored in the same HashMap:
error[E0308]: mismatched types
--> src/main.rs:22:31
|
22 | h.insert("MyOtherStruct", &MyOtherStruct::new);
| ^^^^^^^^^^^^^^^^^^^ expected fn item, found a different fn item
|
= note: expected type `&fn() -> MyStruct {MyStruct::new}`
found type `&fn() -> MyOtherStruct {MyOtherStruct::new}`
Since I'm pretty new to Rust, I'm out of ideas. How can I solve this problem?
This is ultimately fundamentally impossible. In Rust, local variables are stored on the stack, which means that they have to have a fixed size, known at compile time. Your construction requires the size of the value on the stack to be determined at runtime.
The closest alternative is to move to trait objects, which introduce a layer of indirection:
use std::collections::HashMap;
trait NewThing {
fn new(&self) -> Box<Thing>;
}
trait Thing {}
struct MyStruct;
impl NewThing for MyStruct {
fn new(&self) -> Box<Thing> {
Box::new(Self {})
}
}
impl Thing for MyStruct {}
struct MyOtherStruct;
impl NewThing for MyOtherStruct {
fn new(&self) -> Box<Thing> {
Box::new(Self {})
}
}
impl Thing for MyOtherStruct {}
fn main() {
let mut h: HashMap<_, Box<NewThing>> = HashMap::new();
h.insert("MyStruct", Box::new(MyStruct));
h.insert("MyOtherStruct", Box::new(MyOtherStruct));
let obj = h["MyStruct"].new();
}
You will find this pattern out in the world, such as in hyper's NewService.
what is [the value of &self of method new] when calling h["MyStruct"].new()
It's an instance of MyStruct or MyOtherStruct. The only reason that the same type can implement both traits is because there's no real unique state for the "factory" and the "instance". In more complicated implementations, these would be two different types.
Using the same type is common for such cases as sharing a reference-counted value.
See also:
Is it possible to have a constructor function in a trait?
Here is a more complex example of #Shepmaster's solution, using different types for Factories and the objects themselves:
use std::collections::HashMap;
trait NewThing {
fn new(&self) -> Box<Thing>;
}
trait Thing {
fn execute(&mut self);
}
// MyStruct
struct MyStructFactory;
impl NewThing for MyStructFactory {
fn new(&self) -> Box<Thing> {
Box::new(MyStruct {test: 12, name: "Test".into()})
}
}
struct MyStruct {
test: i32,
name: String
}
impl Thing for MyStruct {
fn execute(&mut self) {
self.test+=1;
println!("MyStruct {} {}", self.test, self.name);
}
}
// MyOtherStruct
struct MyOtherStructFactory;
impl NewThing for MyOtherStructFactory {
fn new(&self) -> Box<Thing> {
Box::new(MyOtherStruct {my_member: 1})
}
}
struct MyOtherStruct {
my_member: u32
}
impl Thing for MyOtherStruct {
fn execute(&mut self) { println!("MyOtherStruct.my_member: {}", self.my_member); }
}
fn main() {
let mut h: HashMap<_, Box<NewThing>> = HashMap::new();
h.insert("MyStruct", Box::new(MyStructFactory));
h.insert("MyOtherStruct", Box::new(MyOtherStructFactory));
h["MyStruct"].new().execute();
h["MyOtherStruct"].new().execute();
}
You could use std::any::Any to erase the type of the entry. They use Any::downcast<T> to check if the entry at the location matches your type, and get a Ok(Box<T>)