Why isnt Send implemented for a struct containing Arc? - multithreading

I'm using a crate to interact with Postgres with simply writing sql queries by hands (Diesel seems for my simple case) and got stuck about the multithreaded access to the database client. Here is the code:
use postgres::Client;
pub struct Database{
connection: Arc<Client>
}
impl Database {
pub fn from_config(url: &str) -> Database {
//...
}
}
fn main() {
let url: String = //...
let db = db::Database::from_config(&url);
let db_ref = Arc::new(db);
consume(future(Arc::clone(&db_ref))); // <------------------- compile error
}
async fn future(db_ref: Arc<db::Database>){ }
fn consume<F>(f: F)
where F: Send{ }
The postgres::Client is defined as
/// A synchronous PostgreSQL client.
pub struct Client {
connection: Connection,
client: tokio_postgres::Client,
}
When compiling this code the compile I got somewhat crazy error message:
error[E0277]: `(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)` cannot be shared between threads safely
--> src/main.rs:17:5
|
17 | consume(future(Arc::clone(&db_ref)));
| ^^^^^^^ `(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)` cannot be shared between threads safely
...
24 | fn consume<F>(f: F)
| ------- required by a bound in this
25 | where F: Send{ }
| ---- required by this bound in `consume`
|
= help: the trait `std::marker::Sync` is not implemented for `(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)`
= note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)>`
= note: required because it appears within the type `std::boxed::Box<(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)>`
= note: required because it appears within the type `std::pin::Pin<std::boxed::Box<(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)>>`
= note: required because it appears within the type `postgres::connection::Connection`
= note: required because it appears within the type `postgres::client::Client`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<postgres::client::Client>`
= note: required because it appears within the type `db::Database`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<db::Database>`
= note: required because it appears within the type `[static generator#src/main.rs:22:43: 22:46 db_ref:std::sync::Arc<db::Database> {}]`
= note: required because it appears within the type `std::future::from_generator::GenFuture<[static generator#src/main.rs:22:43: 22:46 db_ref:std::sync::Arc<db::Database> {}]>`
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because it appears within the type `impl core::future::future::Future`
Which seems to mean that Database does not implement Send. Is there a way to make implement Send? Maybe Mutex or something else should be used instead of Arc?
UPD:
Replacing the struct definition with
pub struct Database{
connection: Mutex<Client>
}
makes the error disappear, but it is completely unclear why...

Arc<T> allows multiple threads to access the same value at the same time. The Sync trait is what verifies that the access will not cause memory unsafety, hence why Arc requires it.
On the other hand, Mutex<T> controls access to T via locking, so that only one thread may access T at a time (in a sense, 'sending' it to the thread that has the lock). So Mutex<T> is Sync even if T is not (though it still must be Send).
However Mutex<T> on its own isn't useful since only one thread will have access to the mutex anyway. You usually combine it with a way of sharing ownership (i.e. Arc) to allow multiple threads to have access the mutex.

The reason for this was that the trait implementation Send for Arc was defined as
impl<T> Send for Arc<T>
where
T: Send + Sync + ?Sized,
So Sync is required to be implemented as well. And that's what the error message was saying. For Mutex in turns Send is defined as
impl<T: ?Sized + Send> Send for Mutex<T>
not requiring Sync to be implemented.

Related

How to wrap std::io::Write in Mutex for tracing-subscriber?

In the documentation for Mutex, it says that it implements Send and Sync -- which makes sense, because a Mutex is designed to be accessed from multiple threads that are locking, using the resource it protects, then unlocking.
However, in my code below, I get a compiler error that, as far as I can tell, complains that the Mutex doesn't implement Send/Sync:
error[E0599]: the method `try_init` exists for struct `SubscriberBuilder<DefaultFields, Format, tracing::level_filters::LevelFilter, std::sync::Mutex<MultiWriter>>`, but its trait bounds were not satisfied
--> src/main.rs:131:10
|
131 | .try_init().expect("setting default subscriber failed");
| ^^^^^^^^ method cannot be called on `SubscriberBuilder<DefaultFields, Format, tracing::level_filters::LevelFilter, std::sync::Mutex<MultiWriter>>` due to unsatisfied trait bounds
|
::: /Users/sean/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-subscriber-0.3.16/src/fmt/fmt_layer.rs:62:1
|
62 | / pub struct Layer<
63 | | S,
64 | | N = format::DefaultFields,
65 | | E = format::Format<format::Full>,
66 | | W = fn() -> io::Stdout,
67 | | > {
| | -
| | |
| |_doesn't satisfy `_: std::marker::Send`
| doesn't satisfy `_: std::marker::Sync`
|
= note: the following trait bounds were not satisfied:
`tracing_subscriber::fmt::Layer<Registry, DefaultFields, Format, std::sync::Mutex<MultiWriter>>: std::marker::Send`
`tracing_subscriber::fmt::Layer<Registry, DefaultFields, Format, std::sync::Mutex<MultiWriter>>: std::marker::Sync`
If I remove the line .with_writer(mw) from my code below, the error goes away. Clearly the problem is related to the writer, but I'm not sure how to do this correctly.
The goal of the code is to write the logs from the tracing framework to both stderr and a file specified from dotenvy if a file name is specified (it's optional).
NB: I'm using the latest stable Rust and the released version of each crate used below, and compiling with std, libc, alloc, etc. (full Rust, not embedded) on MacOS, but the code is expected to work on the "multi-platform x86(_64) desktop" environment (Windows/MacOS/desktop Linux). For Tokio I have features = ["full"].
use std::fs::File;
use std::io::Write;
use std::sync::Mutex;
use dotenvy::var;
se std::sync::Arc;
use tracing::Level;
struct MultiWriter {
writers: Vec<Arc<dyn Write>>,
}
impl Write for MultiWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
for writer in self.writers.iter_mut() {
writer.write(buf)?;
}
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
for writer in self.writers.iter_mut() {
writer.flush()?;
}
Ok(())
}
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let mut writers: Vec<Arc<dyn Write>> = vec![(Arc::new(std::io::stderr()))];
if let Some(log_file) = var("log_file").ok() {
writers.push(Arc::new(File::create(log_file).unwrap()));
}
let mw = Mutex::new(MultiWriter { writers });
let tsb = tracing_subscriber::FmtSubscriber::builder()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env()).with_ansi(false)
.with_writer(mw);
if let Ok(log_level) = var("log_level") {
match log_level.to_uppercase().as_str() {
"TRACE" => tsb.with_max_level(Level::TRACE),
"DEBUG" => tsb.with_max_level(Level::DEBUG),
"INFO" => tsb.with_max_level(Level::INFO),
"WARN" => tsb.with_max_level(Level::WARN),
"ERROR" => tsb.with_max_level(Level::ERROR),
_ => tsb.with_max_level(Level::INFO)
}
.try_init().expect("setting default subscriber failed");
}
}
In the documentation for Mutex, it says that it implements Send and Sync
That's not completely true:
impl<T: ?Sized + Send> Send for Mutex<T>
impl<T: ?Sized + Send> Sync for Mutex<T>
This means that a Mutex is Send and Sync only if T is Send (the reason for this is described in this question.
However, T isn't Send here:
T is a struct MultiWriter
struct MultiWriter contains a dyn Write
dyn Write is not Send (at least not always)
in turn, struct MultiWriter isn't either.
To fix this, replace dyn Write by dyn Write + Send, and it should work.
I think it needs to be emphasized that the OP is doing Mutex<Arc<MyType>> that this is different than Arc<Mutex<MyType>>. If a type is Send and wrapped in a Mutex, the whole thing is Send + Sync, but since Arc needs both, that's important here. But if Mutex is on the outside (like it is here) then Arc isn't satisfied by what's inside of it.
As Elias cites (correctly), Mutex only requires it's contained value to be Send, not both Send + Sync. The OP's problem (one of them) is that Mutex is on the outside, and Arc demands both Sync + Send which dyn Write isn't.
Arc:
#[stable(feature = "rust1", since = "1.0.0")]
unsafe impl<T: ?Sized + Sync + Send> Send for Arc<T> {}
#[stable(feature = "rust1", since = "1.0.0")]
unsafe impl<T: ?Sized + Sync + Send> Sync for Arc<T> {}
Mutex:
#[stable(feature = "rust1", since = "1.0.0")]
unsafe impl<T: ?Sized + Send> Send for Mutex<T> {}
#[stable(feature = "rust1", since = "1.0.0")]
unsafe impl<T: ?Sized + Send> Sync for Mutex<T> {}
Ultimately #ChayimFriedman is right that for the OP's block of code, they need both (dyn + Write + Send + Sync), but I wanted to put down here even more as to why.

Sharing an immutable reference to a Send resource in Rust

I am trying to get some async Rust going, to allow concurrent access to a database. I understand futures need to have 'Send' implemented for all the variables that can be held whilst it is awaited, but I am specifically importing Send for the Database itself, as well as declaring it on the input type.
#[async_trait]
pub trait Database: Send {
async fn get_something(&self) -> Option<Vec<f32>>;
}
async fn calculate_something(&mut self, db: Arc<(dyn Database + Send)>, var: i16) -> Option<f32> {
let something = db.get_something(var).await;
// Do stuff with the something.
Some()
}
This gives me the error:
^ future created by async block is not `Send`
help: the trait `Sync` is not implemented for `dyn database::Database + Send`
note: captured value is not `Send`
--> src/alarms/something_calculator.rs:31:47
|
31 | async fn calculate_something(&mut self, db: Arc<(dyn Database + Send)>, var: i16) -> Option<f32> {
| ^^ has type `Arc<dyn database::Database + Send>` which is not `Send`
= note: required for the cast to the object type `dyn Future<Output = Option<f32>> + Send`
I am new to rust so I'm sure I am doing something silly, but I can't wrap my head around it not implementing Send when it is defined as a Send in its type.
Thanks in advance.
The async function requires all captured values to be Send such that itself is Send. For the Arc type to be Send its type argument must be Sync + Send. You can write db: Arc<(dyn Database + Send + Sync)> to make the async Future send.
It mentions this in the error message:
help: the trait `Sync` is not implemented for `dyn database::Database + Send
This change might have further consequences for your code if the db argument is not already Sync, but you are not providing enough context to determine that.

Send FnMut to other thread by wrapping it into a Weak<Mutex<FnMut>> complains about missing Send [duplicate]

This question already has answers here:
Sending trait objects between threads in Rust
(1 answer)
Sharing a struct with trait objects as properties across threads
(1 answer)
Closed 3 years ago.
I'm trying to write a simple Observer-pattern-ish thing with events and callbacks. The idea is that when the thing owning the mutating function dies (struct Obj {cb: Arc<Mutex<FnMut>>, .. }), it drops the Arc. After that, any Weak references will be (effectively) dead and can be detected on access. I was ALSO hoping that this would decouple the FnMut from Thread's 'static lifetime. I might need to wrap the whole thing in another Arc<Rwlock<T>> to prevent the Weak references from doing thread unsafe stuff, but that's a different problem.
The closest solution I've gotten was this: How can I send a function to another thread?
However, I seem to have the Arc<Mutex<T>> and adding lifetimes (I may have done that wrong, though) didn't seem to help. I'm kind of lost as to what is actually wrong.
I wrote a minimal example:
use std::{
collections::HashMap,
sync::{
mpsc::{self, Receiver},
Arc, Mutex, Weak,
},
thread::{self, JoinHandle},
};
struct Callback {
cb_w: Weak<Mutex<FnMut()>>,
}
type FnMapping = Arc<HashMap<String, Callback>>;
fn start_thread<'a>(rx: Receiver<String>, mapping: FnMapping) -> JoinHandle<()> {
thread::spawn(move || {
match rx.recv() {
Ok(v) => {
if let Some(o) = mapping.get(&v) {
match o.cb_w.upgrade() {
Some(cb_m) => match cb_m.lock() {
Ok(cb_lg) => (*cb_lg)(),
Err(e) => (),
},
None => { /* TODO owner is gone, mark for delete */ }
}
}
}
Err(e) => (),
}
})
}
fn main() {
let mapping: FnMapping = Arc::new(HashMap::new());
let (tx, rx) = mpsc::channel();
drop(tx);
start_thread(rx, mapping)
.join()
.expect("Could not join thread -- failed to terminate?");
println!("Leaving the test bay.");
}
This fails to compile with the following error:
error[E0277]: `(dyn std::ops::FnMut() + 'static)` cannot be sent between threads safely
--> src/main.rs:17:5
|
17 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `(dyn std::ops::FnMut() + 'static)` cannot be sent between threads safely
|
= help: the trait `std::marker::Send` is not implemented for `(dyn std::ops::FnMut() + 'static)`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<(dyn std::ops::FnMut() + 'static)>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Weak<std::sync::Mutex<(dyn std::ops::FnMut() + 'static)>>`
= note: required because it appears within the type `Callback`
= note: required because it appears within the type `(std::string::String, Callback)`
= note: required because it appears within the type `std::marker::PhantomData<(std::string::String, Callback)>`
= note: required because it appears within the type `std::collections::hash::table::RawTable<std::string::String, Callback>`
= note: required because it appears within the type `std::collections::HashMap<std::string::String, Callback>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<std::collections::HashMap<std::string::String, Callback>>`
= note: required because it appears within the type `[closure#src/main.rs:17:19: 32:6 rx:std::sync::mpsc::Receiver<std::string::String>, mapping:std::sync::Arc<std::collections::HashMap<std::string::String, Callback>>]`
= note: required by `std::thread::spawn`

Compiler says that data cannot be shared between threads safely even though the data is wrapped within a Mutex

I'm using Rocket which has a State that it passes to the HTTP requests. This struct contains a Mutex<DatastoreInstance> which gives access to a SQLite database and is locked with a mutex to make read and writes safe.
pub struct DatastoreInstance {
conn: Connection,
}
When the DatastoreInstance struct looked like this, with only a SQLite connection everything worked fine, but I then also wanted to add a transaction object within this struct:
pub struct DatastoreInstance {
conn: Connection,
events_transaction: Transaction,
}
This did not compile because the Transaction object needs to reference a Connection object which should have a lifetime which it is aware of.
The Connection and Transaction objects within rusqlite which I am using are defined as following:
pub struct Connection {
db: RefCell<InnerConnection>,
cache: StatementCache,
path: Option<PathBuf>,
}
pub struct Transaction<'conn> {
conn: &'conn Connection,
drop_behavior: DropBehavior,
}
To solve the lifetime issues I had to add these lifetime parameters to get it working:
pub struct DatastoreInstance<'a> {
conn: Connection,
events_transaction: Transaction<'a>,
}
This was the result and was supposed to work according to my understanding of both lifetimes and mutexes, but I now get a compiler error telling me:
`std::cell::RefCell<lru_cache::LruCache<std::string::String, rusqlite::raw_statement::RawStatement>>` cannot be shared between threads safely
|
= help: within `rusqlite::Connection`, the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<lru_cache::LruCache<std::string::String, rusqlite::raw_statement::RawStatement>>`
= note: required because it appears within the type `rusqlite::cache::StatementCache`
= note: required because it appears within the type `rusqlite::Connection`
= note: required because of the requirements on the impl of `std::marker::Send` for `&rusqlite::Connection`
= note: required because it appears within the type `datastore::DatastoreInstance<'_>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<datastore::DatastoreInstance<'_>>`
= note: required because it appears within the type `endpoints::ServerState<'_>`
= note: required by `rocket::State`
According to my understanding of mutexes, this code should be valid because the whole DatastoreInstance struct is wrapped within a Mutex which should guarantee that only one thread is referencing this object at a time.
What am I missing?
Why doesn't the compiler find RefCell to be safe anymore after being within a Connection referenced within a Transaction instead of solely within a Connection?
Do I have a bad understanding of how mutexes work? Are my lifetimes invalid and somehow break read/write safety? Is the design of having the Connection and Transaction within the same struct a bad design which breaks read/write safety? Do I need to redesign my data structures somehow to make this safe? Or am I just missing something very obvious?
A Mutex is only Send or Sync if the value it contains is itself Send:
impl<T: ?Sized + Send> Send for Mutex<T>
impl<T: ?Sized + Send> Sync for Mutex<T>
A &T is only Send when T is Sync:
impl<'a, T> Send for &'a T
where
T: Sync + ?Sized,
And a RefCell is never Sync
impl<T> !Sync for RefCell<T>
where
T: ?Sized,
As the error message states, your transaction contains a reference to a RefCell. It doesn't matter that there's a mutex, it's inherently not memory-safe to share it across threads. A simple reproduction:
use std::{cell::RefCell, sync::Mutex};
struct Connection(RefCell<i32>);
struct Transaction<'a>(&'a Connection);
fn is_send<T: Send>(_: T) {}
fn main() {
let c = Connection(RefCell::new(42));
let t = Transaction(&c);
let m = Mutex::new(t);
is_send(m);
}
error[E0277]: `std::cell::RefCell<i32>` cannot be shared between threads safely
--> src/main.rs:13:5
|
13 | is_send(m);
| ^^^^^^^ `std::cell::RefCell<i32>` cannot be shared between threads safely
|
= help: within `Connection`, the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<i32>`
= note: required because it appears within the type `Connection`
= note: required because of the requirements on the impl of `std::marker::Send` for `&Connection`
= note: required because it appears within the type `Transaction<'_>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<Transaction<'_>>`
note: required by `is_send`
--> src/main.rs:6:1
|
6 | fn is_send<T: Send>(_: T) {}
| ^^^^^^^^^^^^^^^^^^^^^^^^^
Why doesn't the compiler find RefCell to be safe anymore after being within a Connection referenced within a Transaction instead of solely within a Connection?
The RefCell is fine, it's the reference to a RefCell that is not.
Is the design of having the Connection and Transaction within the same struct a bad design [...] Do I need to redesign my data structures
Yes.
How to store rusqlite Connection and Statement objects in the same struct in Rust?
Why can't I store a value and a reference to that value in the same struct?

Type constructor in trait declaration

I want to pass a reference to a Rusoto SQS client to a WebSocket server struct to push messages onto an SQS queue.
To that end, I have a (naive) struct as follows:
struct MyHandler {
sqsclient: &SqsClient<ProvideAwsCredentials, DispatchSignedRequest>,
}
This produces the error:
error[E0106]: missing lifetime specifier
--> src/main.rs:29:13
|
29 | sqsclient: &SqsClient<ProvideAwsCredentials, DispatchSignedRequest>,
| ^ expected lifetime parameter
I've attempted all sorts of type signatures and lifetime trickery which all fail to varying levels, but I keep coming up against the same error:
error[E0277]: the trait bound `rusoto_core::ProvideAwsCredentials + 'static: std::marker::Sized` is not satisfied
--> src/main.rs:29:2
|
29 | sqsclient: &'static SqsClient<ProvideAwsCredentials, DispatchSignedRequest>,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `rusoto_core::ProvideAwsCredentials + 'static` does not have a constant size known at compile-time
|
= help: the trait `std::marker::Sized` is not implemented for `rusoto_core::ProvideAwsCredentials + 'static`
= note: required by `rusoto_sqs::SqsClient`
error[E0277]: the trait bound `rusoto_core::DispatchSignedRequest + 'static: std::marker::Sized` is not satisfied
--> src/main.rs:29:2
|
29 | sqsclient: &'static SqsClient<ProvideAwsCredentials, DispatchSignedRequest>,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `rusoto_core::DispatchSignedRequest + 'static` does not have a constant size known at compile-time
|
= help: the trait `std::marker::Sized` is not implemented for `rusoto_core::DispatchSignedRequest + 'static`
= note: required by `rusoto_sqs::SqsClient`
I've also tried wrapping it in Rc and Box to no avail. I feel like I'm looking in the wrong place with these though.
This happens when giving MyHandler an <'a> lifetime too. What am I misunderstanding about the Rust type system here, and what can I do to be able to pass a reference to an SqsClient<...> (and eventually other stuff from Rusoto) into my own structs? It would be useful to know if what I'm trying to do above is idiomatic Rust. If it's not, what pattern should I use instead?
This is a follow up from How do I pass a struct with type parameters as a function argument?.
Solved it! DispatchSignedRequest and ProvideAwsCredentials (from Rusoto) are traits. I needed to use impls of those traits, i.e. structs, so now the code looks like this:
extern crate rusoto_core;
extern crate hyper;
use hyper::Client;
use rusoto_sqs::{ Sqs, SqsClient };
use rusoto_core::{ DefaultCredentialsProvider };
struct MyHandler<'a> {
sqsclient: &'a SqsClient<DefaultCredentialsProvider, Client>,
}
impl<'a> Handler for MyHandler<'a> {
// ...
}
DefaultCredentialsProvider and Client (from Hyper) are both structs, so now this code compiles fine.
I'm using Hyper 0.10 here. Hyper 0.11 requires a different type signature for Client.

Resources