Using a trait object in a background job (different thread) - multithreading

Question

I want to have a background worker which uses a trait implementation / object for some time. The background worker owns this object as long as it is used. After the background worker is "destroyed", the object should be free to be used again.
I tried to make all the things with async/await, but it produced some more problems. Therefore, I use plain threads to create kind of a minimal example. First I also used Box<&dyn mut...> to pass the object to the background worker, but I think that is not even needed.
My minimal example contains a MyWriter-trait which can write string to somewhere. There exists one implementation which writes strings to stdout. A background-worker uses this writer for a background-job. The worker has a start-method to start the job and a stop-method to join it (in my real code I would use a channel to send a stop-info to the worker and joining then).
I'll post the code and then a description with my problems:
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=a01745c15ba1088acd2e3d287d60e270
use std::sync::Arc;
use std::sync::Mutex;
use std::thread::{spawn, JoinHandle};
/* Trait + an implementation */
trait MyWriter {
fn write(&mut self, text: &str);
}
struct StdoutWriter {}
impl StdoutWriter {
pub fn new() -> Self {
Self {}
}
}
impl MyWriter for StdoutWriter {
fn write(&mut self, text: &str) {
println!("{}", text);
}
}
/* A background job which uses a "MyWriter" */
struct BackgroundJob<'a> {
writer: Arc<Mutex<&'a dyn MyWriter>>,
job: Option<JoinHandle<()>>,
}
impl<'a> BackgroundJob<'a> {
pub fn new(writer: &'a mut dyn MyWriter) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
pub fn start(&mut self) {
assert!(self.job.is_none());
let writer = &self.writer;
self.job = Some(std::thread::spawn(move || {
// this background job uses "writer"
let mut my_writer = writer.lock().unwrap();
my_writer.write("x");
// do something
my_writer.write("y");
}));
}
pub fn stop(&mut self) {
if let Some(job) = self.job {
job.join().unwrap();
self.job = None;
}
}
}
/* Using BackgroundJob */
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
{
let mut job = BackgroundJob::new(&mut writer);
// inside this block, writer is owned by "job"
job.start();
job.stop();
}
// writer should be usable again
writer.write("b");
}
The desired output on stdout is a\nx\ny\nz\n, but the program does not even compile. My main problem is that (dyn MyWriter + 'a) cannot be shared between threads safely (compiler error).
How can I implement Send / Sync for a trait? It does not seem to be possible. Actually, I assumed it should be ok if the object (or a ref.) is inside Arc<Mutex<...>>, but that does not seem to be sufficient. Why not?
Maybe someone has an idea how this can be fixed or even more important what exactly is the underlying issue?

Answer 1

Putting a reference in an Arc doesn't work. Since the Arc can be kept alive indefinitely simply by cloning it, the reference could easily outlive whatever it was borrowed from, so that can't compile. You need to put an owned object in the Arc, such as Box<dyn MyWriter>. (Ideally you'd just use Arc<dyn MyWriter>, but that would conflict with returning the writer from the BackgroundJob, as shown below.)
Since you can't borrow from writer in main, you must move it into the BackgroundJob. But at this point you've relinquished ownership over writer, having moved the value to BackgroundJob, so your only option is to have BackgroundJob return the writer. However, since BackgroundJob keeps its writer behind a trait object, it can only give back the Box<dyn MyWriter> it stores, not the original StdoutWriter.
Here is the version that works that way, retaining type erasure and giving back the type-erased writer:
// Trait definition and StdoutWriter implementation unchanged
struct BackgroundJob {
writer: Arc<Mutex<Box<dyn MyWriter + Send>>>,
job: Option<JoinHandle<()>>,
}
impl BackgroundJob {
pub fn new(writer: Box<dyn MyWriter + Send>) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
pub fn start(&mut self) {
assert!(self.job.is_none());
let writer = Arc::clone(&self.writer);
self.job = Some(std::thread::spawn(move || {
// this background job uses "writer"
let mut my_writer = writer.lock().unwrap();
my_writer.write("x");
// do something
my_writer.write("y");
}));
}
pub fn stop(&mut self) {
if let Some(job) = self.job.take() {
job.join().unwrap();
}
}
pub fn into_writer(self) -> Box<dyn MyWriter> {
Arc::try_unwrap(self.writer)
.unwrap_or_else(|_| panic!())
.into_inner()
.unwrap()
}
}
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
let mut writer = {
let mut job = BackgroundJob::new(Box::new(writer));
job.start();
job.stop();
job.into_writer()
};
writer.write("b");
}
Playground
A version that gave back the writer of the same type would have to give up on type erasure and be generic over the writer type. Though a bit more complex, its ownership semantics would be very close (at least conceptually) to what you originally envisioned:
struct BackgroundJob<W> {
writer: Arc<Mutex<W>>,
job: Option<JoinHandle<()>>,
}
impl<W: MyWriter + Send + 'static> BackgroundJob<W> {
pub fn new(writer: W) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
// start() and stop() are unchanged
pub fn into_writer(self) -> W {
Arc::try_unwrap(self.writer)
.unwrap_or_else(|_| panic!())
.into_inner()
.unwrap()
}
}
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
{
// inside this block, writer is moved into "job"
let mut job = BackgroundJob::new(writer);
job.start();
job.stop();
// reclaim the writer
writer = job.into_writer();
}
writer.write("b");
}
Playground

Answer 2

The main issue is that you want to pass a reference to the thread. The problem with that approach is that the thread can outlive the referenced object. Obviously this does not happen in your case, but the rust compiler cannot reason about that.
The solution to that problem is to use Arc<Mutex<dyn MyType>> instead of Arc<Mutex<&dyn MyType>> - no lifetimes - no problems.
The next issue is with Mutex<T> - it can be send across threads only if T can. So you have to make T, in your case dyn MyType, implement Send. This can be done in two ways:
Make MyType require Send - in that case that trait can be implemented only by Send objects:
trait MyWriter : Send{
fn write(&mut self, text: &str);
}
Or use an additional trait bound - in that case your trait is less restrictive, but you must always specify MyTrait + Send when you want to send it across threads:
Arc<Mutex<dyn MyWriter + Send>>
So far so good, but now your new() method does not work, because dyn MyWriter is not Sized. In order to fix that you have to make your method generic:
pub fn new<T: MyWriter + Send>(writer: T) -> Self {
Self {
writer: Arc::new(Mutex::new(writer)),
job: None,
}
}
or directly pass an Arc<Mutex<dyn MyWriter + Send>>:
pub fn new(writer: Arc<Mutex<dyn MyWriter + Send>>) -> Self {
Self { writer, job: None }
}
Full working code
use std::sync::Arc;
use std::sync::Mutex;
use std::thread::JoinHandle;
trait MyWriter {
fn write(&mut self, text: &str);
}
struct StdoutWriter {}
impl StdoutWriter {
pub fn new() -> Self {
Self {}
}
}
impl MyWriter for StdoutWriter {
fn write(&mut self, text: &str) {
println!("{}", text);
}
}
/* A background job which uses a "MyWriter" */
struct BackgroundJob {
writer: Arc<Mutex<dyn MyWriter + Send>>,
job: Option<JoinHandle<()>>,
}
impl BackgroundJob {
pub fn new(writer: Arc<Mutex<dyn MyWriter + Send>>) -> Self {
Self { writer, job: None }
}
pub fn start(&mut self) {
assert!(self.job.is_none());
let writer = self.writer.clone();
self.job = Some(std::thread::spawn(move || {
let mut my_writer = writer.lock().unwrap();
my_writer.write("x");
// do something
my_writer.write("y");
}));
}
pub fn stop(&mut self) {
if let Some(job) = self.job.take() {
job.join().unwrap();
}
}
}
fn main() {
let mut writer = StdoutWriter::new();
writer.write("a");
let writer = Arc::new(Mutex::new(writer));
{
let mut job = BackgroundJob::new(writer.clone());
// inside this block, writer is owned by "job"
job.start();
job.stop();
}
// you have to acquire the lock in order to use the writer
writer.lock().unwrap_or_else(|e| e.into_inner()).write("b");
}