I am trying to get some async Rust going, to allow concurrent access to a database. I understand futures need to have 'Send' implemented for all the variables that can be held whilst it is awaited, but I am specifically importing Send for the Database itself, as well as declaring it on the input type.
#[async_trait]
pub trait Database: Send {
async fn get_something(&self) -> Option<Vec<f32>>;
}
async fn calculate_something(&mut self, db: Arc<(dyn Database + Send)>, var: i16) -> Option<f32> {
let something = db.get_something(var).await;
// Do stuff with the something.
Some()
}
This gives me the error:
^ future created by async block is not `Send`
help: the trait `Sync` is not implemented for `dyn database::Database + Send`
note: captured value is not `Send`
--> src/alarms/something_calculator.rs:31:47
|
31 | async fn calculate_something(&mut self, db: Arc<(dyn Database + Send)>, var: i16) -> Option<f32> {
| ^^ has type `Arc<dyn database::Database + Send>` which is not `Send`
= note: required for the cast to the object type `dyn Future<Output = Option<f32>> + Send`
I am new to rust so I'm sure I am doing something silly, but I can't wrap my head around it not implementing Send when it is defined as a Send in its type.
Thanks in advance.
The async function requires all captured values to be Send such that itself is Send. For the Arc type to be Send its type argument must be Sync + Send. You can write db: Arc<(dyn Database + Send + Sync)> to make the async Future send.
It mentions this in the error message:
help: the trait `Sync` is not implemented for `dyn database::Database + Send
This change might have further consequences for your code if the db argument is not already Sync, but you are not providing enough context to determine that.
Related
Sending a struct containing a Sender through a channel and using the question mark to handle errors, Rust complains about Sender<..> not implementing Sync.
Without the question mark, it doesn't.
So, is the Sync trait expected from the Sender?
Not understanding the question mark operator in detail, I assume an anyhow::Error is being constructed - but why would the Sender end up in the error object?
(My application has one database thread that handles all queries and communicates via channels)
Here's the code:
|db: Db| -> anyhow::Result<()> {
let (res_tx, res_rx) = mpsc::channel::<Result<Vec<Row>>>();
db.send(database_handler::Request::Query {
query: "select name from actions",
params: Vec::new(),
result: res_tx,
})?; // dis be line 35
}
Here's the error:
error[E0277]: `Sender<Result<Vec<postgres::Row>, anyhow::Error>>` cannot be shared between threads safely
--> src/events.rs:35:6
|
35 | })?;
| ^ `Sender<Result<Vec<postgres::Row>, anyhow::Error>>` cannot be shared between threads safely
|
= help: within `SendError<Request<'_>>`, the trait `Sync` is not implemented for `Sender<Result<Vec<postgres::Row>, anyhow::Error>>`
= help: the following other types implement trait `FromResidual<R>`:
<Result<T, F> as FromResidual<Result<Infallible, E>>>
<Result<T, F> as FromResidual<Yeet<E>>>
note: required because it appears within the type `Request<'_>`
--> src/database.rs:39:10
|
39 | pub enum Request<'a> {
| ^^^^^^^
= note: required because it appears within the type `SendError<Request<'_>>`
= note: required because of the requirements on the impl of `From<SendError<Request<'_>>>` for `anyhow::Error`
= note: required because of the requirements on the impl of `FromResidual<Result<Infallible, SendError<Request<'_>>>>` for `Result<(), anyhow::Error>`
And these are the involved types:
type Db<'a> = Sender<database_handler::Request<'a>>;
type ParamsType<'a> = Vec<&'a (dyn ToSql + Sync)>;
pub enum Request<'a> {
Query { query: &'static str, params: ParamsType<'a>, result: Sender<Result<Vec<Row>>> },
QueryOne { query: &'static str, params: ParamsType<'a>, result: Sender<Result<Option<Row>>> },
Execute { query: &'static str, params: ParamsType<'a>, result: Sender<Result<u64>> },
}
You are correct that an anyhow::Error is being constructed, but - from what? Since Sender::send() takes ownership of the value you give it, the error it returns when it fails to send includes the original value, allowing you to retry sending later or in a different manner. So your anyhow::Error gets constructed from SendError<Query> that contains a Query.
The Sync requirement comes from anyhow::Error itself being Send and Sync (because it's useful to move error to a different thread), so its From implementation must require the same of the source error. In your case the source error is SendError<Query> which is not Sync because it contains Query whose result field contains a (different) Sender, which not Sync.
Once aware of this, you can see the error message telling you this, too. The fix is to replace db.send(...)? with something like db.send(...).map_err(|_| anyhow!("receiver is gone")). Also, you'll need an explicit Ok(()) at the end of the closure.
In Rust I have a callback defined as Box<dyn Fn(MyType) + Send>. I want to be able to convert a C callback provided by FFI into that signature.
To do that, I created a Rust closure that calls the C callback with a context pointer:
let rust_closure = move |data: u32| {
callback(obj, data);
};
The problem is obj in this case is a c_void. According to the docs (https://doc.rust-lang.org/core/ffi/enum.c_void.html) it should have auto-implemented Send and Sync. However the compiler tells me:
the trait Send is not implemented for *const c_void
How can I convince Rust that my c_void is indeed Send? I've tried wrapping it in a unit struct and impl'ing Send for the struct, but that did not change the error.
I've created a minimal example in the playground.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=5737c7449b925ab83e35d2404a6bd9f8
Pointers are automatically !Send so the solution is to wrap the pointer in a custom type, that hopefully better describes its use and reasons to guarantee that the pointer is safe to use across threads. I'm assuming you made a solution similar to this:
struct UserObj(*const c_void);
// SAFETY: Any user data object must be safe to send between threads.
unsafe impl Send for UserObj {}
let obj = UserObj(obj);
let rust_closure = move |data: u32| {
callback(obj.0, data);
};
ch.add_callback(Box::new(rust_closure));
Which unfortunately, does yield the same error:
error[E0277]: `*const c_void` cannot be sent between threads safely
--> src/lib.rs:33:25
|
29 | let rust_closure = move |data: u32| {
| ____________________________-
30 | | callback(obj.0, data);
31 | | };
| |_________- within this `[closure#src/lib.rs:29:28: 31:10]`
32 |
33 | ch.add_callback(Box::new(rust_closure));
| ^^^^^^^^^^^^^^^^^^^^^^ `*const c_void` cannot be sent between threads safely
|
= help: within `[closure#src/lib.rs:29:28: 31:10]`, the trait `Send` is not implemented for `*const c_void`
= note: required because it appears within the type `[closure#src/lib.rs:29:28: 31:10]`
= note: required for the cast to the object type `dyn Fn(u32) + Send`
But why? We're moving a UserObj into the closure and we've asserted to the compiler that it is Send. Why does it still complain about *const c_void?
As of Rust 2021 edition, closures can capture disjoint fields rather than the whole object, meaning the above closure would capture obj.0 instead of obj itself, causing the error. My suggestion would be to add a method to get the pointer instead of using field access in this case:
struct UserObj(*const c_void);
impl UserObj {
fn as_ptr(&self) -> *const c_void {
self.0
}
}
// SAFETY: Any user data object must be safe to send between threads.
unsafe impl Send for UserObj {}
let obj = UserObj(obj);
let rust_closure = move |data: u32| {
callback(obj.as_ptr(), data);
};
ch.add_callback(Box::new(rust_closure));
Playground
I'm using a crate to interact with Postgres with simply writing sql queries by hands (Diesel seems for my simple case) and got stuck about the multithreaded access to the database client. Here is the code:
use postgres::Client;
pub struct Database{
connection: Arc<Client>
}
impl Database {
pub fn from_config(url: &str) -> Database {
//...
}
}
fn main() {
let url: String = //...
let db = db::Database::from_config(&url);
let db_ref = Arc::new(db);
consume(future(Arc::clone(&db_ref))); // <------------------- compile error
}
async fn future(db_ref: Arc<db::Database>){ }
fn consume<F>(f: F)
where F: Send{ }
The postgres::Client is defined as
/// A synchronous PostgreSQL client.
pub struct Client {
connection: Connection,
client: tokio_postgres::Client,
}
When compiling this code the compile I got somewhat crazy error message:
error[E0277]: `(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)` cannot be shared between threads safely
--> src/main.rs:17:5
|
17 | consume(future(Arc::clone(&db_ref)));
| ^^^^^^^ `(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)` cannot be shared between threads safely
...
24 | fn consume<F>(f: F)
| ------- required by a bound in this
25 | where F: Send{ }
| ---- required by this bound in `consume`
|
= help: the trait `std::marker::Sync` is not implemented for `(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)`
= note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)>`
= note: required because it appears within the type `std::boxed::Box<(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)>`
= note: required because it appears within the type `std::pin::Pin<std::boxed::Box<(dyn futures_core::stream::Stream<Item = std::result::Result<tokio_postgres::AsyncMessage, tokio_postgres::error::Error>> + std::marker::Send + 'static)>>`
= note: required because it appears within the type `postgres::connection::Connection`
= note: required because it appears within the type `postgres::client::Client`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<postgres::client::Client>`
= note: required because it appears within the type `db::Database`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<db::Database>`
= note: required because it appears within the type `[static generator#src/main.rs:22:43: 22:46 db_ref:std::sync::Arc<db::Database> {}]`
= note: required because it appears within the type `std::future::from_generator::GenFuture<[static generator#src/main.rs:22:43: 22:46 db_ref:std::sync::Arc<db::Database> {}]>`
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because it appears within the type `impl core::future::future::Future`
Which seems to mean that Database does not implement Send. Is there a way to make implement Send? Maybe Mutex or something else should be used instead of Arc?
UPD:
Replacing the struct definition with
pub struct Database{
connection: Mutex<Client>
}
makes the error disappear, but it is completely unclear why...
Arc<T> allows multiple threads to access the same value at the same time. The Sync trait is what verifies that the access will not cause memory unsafety, hence why Arc requires it.
On the other hand, Mutex<T> controls access to T via locking, so that only one thread may access T at a time (in a sense, 'sending' it to the thread that has the lock). So Mutex<T> is Sync even if T is not (though it still must be Send).
However Mutex<T> on its own isn't useful since only one thread will have access to the mutex anyway. You usually combine it with a way of sharing ownership (i.e. Arc) to allow multiple threads to have access the mutex.
The reason for this was that the trait implementation Send for Arc was defined as
impl<T> Send for Arc<T>
where
T: Send + Sync + ?Sized,
So Sync is required to be implemented as well. And that's what the error message was saying. For Mutex in turns Send is defined as
impl<T: ?Sized + Send> Send for Mutex<T>
not requiring Sync to be implemented.
I'm using Rocket which has a State that it passes to the HTTP requests. This struct contains a Mutex<DatastoreInstance> which gives access to a SQLite database and is locked with a mutex to make read and writes safe.
pub struct DatastoreInstance {
conn: Connection,
}
When the DatastoreInstance struct looked like this, with only a SQLite connection everything worked fine, but I then also wanted to add a transaction object within this struct:
pub struct DatastoreInstance {
conn: Connection,
events_transaction: Transaction,
}
This did not compile because the Transaction object needs to reference a Connection object which should have a lifetime which it is aware of.
The Connection and Transaction objects within rusqlite which I am using are defined as following:
pub struct Connection {
db: RefCell<InnerConnection>,
cache: StatementCache,
path: Option<PathBuf>,
}
pub struct Transaction<'conn> {
conn: &'conn Connection,
drop_behavior: DropBehavior,
}
To solve the lifetime issues I had to add these lifetime parameters to get it working:
pub struct DatastoreInstance<'a> {
conn: Connection,
events_transaction: Transaction<'a>,
}
This was the result and was supposed to work according to my understanding of both lifetimes and mutexes, but I now get a compiler error telling me:
`std::cell::RefCell<lru_cache::LruCache<std::string::String, rusqlite::raw_statement::RawStatement>>` cannot be shared between threads safely
|
= help: within `rusqlite::Connection`, the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<lru_cache::LruCache<std::string::String, rusqlite::raw_statement::RawStatement>>`
= note: required because it appears within the type `rusqlite::cache::StatementCache`
= note: required because it appears within the type `rusqlite::Connection`
= note: required because of the requirements on the impl of `std::marker::Send` for `&rusqlite::Connection`
= note: required because it appears within the type `datastore::DatastoreInstance<'_>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<datastore::DatastoreInstance<'_>>`
= note: required because it appears within the type `endpoints::ServerState<'_>`
= note: required by `rocket::State`
According to my understanding of mutexes, this code should be valid because the whole DatastoreInstance struct is wrapped within a Mutex which should guarantee that only one thread is referencing this object at a time.
What am I missing?
Why doesn't the compiler find RefCell to be safe anymore after being within a Connection referenced within a Transaction instead of solely within a Connection?
Do I have a bad understanding of how mutexes work? Are my lifetimes invalid and somehow break read/write safety? Is the design of having the Connection and Transaction within the same struct a bad design which breaks read/write safety? Do I need to redesign my data structures somehow to make this safe? Or am I just missing something very obvious?
A Mutex is only Send or Sync if the value it contains is itself Send:
impl<T: ?Sized + Send> Send for Mutex<T>
impl<T: ?Sized + Send> Sync for Mutex<T>
A &T is only Send when T is Sync:
impl<'a, T> Send for &'a T
where
T: Sync + ?Sized,
And a RefCell is never Sync
impl<T> !Sync for RefCell<T>
where
T: ?Sized,
As the error message states, your transaction contains a reference to a RefCell. It doesn't matter that there's a mutex, it's inherently not memory-safe to share it across threads. A simple reproduction:
use std::{cell::RefCell, sync::Mutex};
struct Connection(RefCell<i32>);
struct Transaction<'a>(&'a Connection);
fn is_send<T: Send>(_: T) {}
fn main() {
let c = Connection(RefCell::new(42));
let t = Transaction(&c);
let m = Mutex::new(t);
is_send(m);
}
error[E0277]: `std::cell::RefCell<i32>` cannot be shared between threads safely
--> src/main.rs:13:5
|
13 | is_send(m);
| ^^^^^^^ `std::cell::RefCell<i32>` cannot be shared between threads safely
|
= help: within `Connection`, the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<i32>`
= note: required because it appears within the type `Connection`
= note: required because of the requirements on the impl of `std::marker::Send` for `&Connection`
= note: required because it appears within the type `Transaction<'_>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<Transaction<'_>>`
note: required by `is_send`
--> src/main.rs:6:1
|
6 | fn is_send<T: Send>(_: T) {}
| ^^^^^^^^^^^^^^^^^^^^^^^^^
Why doesn't the compiler find RefCell to be safe anymore after being within a Connection referenced within a Transaction instead of solely within a Connection?
The RefCell is fine, it's the reference to a RefCell that is not.
Is the design of having the Connection and Transaction within the same struct a bad design [...] Do I need to redesign my data structures
Yes.
How to store rusqlite Connection and Statement objects in the same struct in Rust?
Why can't I store a value and a reference to that value in the same struct?
I have some mutable state I need to share between threads. I followed the concurrency section of the Rust book, which shares a vector between threads and mutates it.
Instead of a vector, I need to share a generic struct that is ultimately monomorphized. Here is a distilled example of what I'm trying:
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
use std::marker::PhantomData;
trait Memory {}
struct SimpleMemory;
impl Memory for SimpleMemory {}
struct SharedData<M: Memory> {
value: usize,
phantom: PhantomData<M>,
}
impl<M: Memory> SharedData<M> {
fn new() -> Self {
SharedData {
value: 0,
phantom: PhantomData,
}
}
}
fn main() {
share(SimpleMemory);
}
fn share<M: Memory>(memory: M) {
let data = Arc::new(Mutex::new(SharedData::<M>::new()));
for i in 0..3 {
let data = data.clone();
thread::spawn(move || {
let mut data = data.lock().unwrap();
data.value += i;
});
}
thread::sleep(Duration::from_millis(50));
}
The compiler complains with the following error:
error[E0277]: the trait bound `M: std::marker::Send` is not satisfied
--> src/main.rs:37:9
|
37 | thread::spawn(move || {
| ^^^^^^^^^^^^^
|
= help: consider adding a `where M: std::marker::Send` bound
= note: required because it appears within the type `std::marker::PhantomData<M>`
= note: required because it appears within the type `SharedData<M>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<SharedData<M>>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<std::sync::Mutex<SharedData<M>>>`
= note: required because it appears within the type `[closure#src/main.rs:37:23: 40:10 data:std::sync::Arc<std::sync::Mutex<SharedData<M>>>, i:usize]`
= note: required by `std::thread::spawn`
I'm trying to understand why M would need to implement Send, and what the appropriate way to accomplish this is.
I'm trying to understand why M would need to implement Send, ...
Because, as stated by the Send documentation:
Types that can be transferred across thread boundaries.
If it's not Send, it is by definition not safe to send to another thread.
Almost all of the information you need is right there in the documentation:
thread::spawn requires the callable you give it to be Send.
You're using a closure, which is only Send if all the values it captures are Send. This is true in general of most types (they are Send if everything they're made of is Send, and similarly for Sync).
You're capturing data, which is an Arc<T>, which is only Send if T is Send.
T is a Mutex<U>, which is only Send if U is Send.
U is M. Thus, M must be Send.
In addition, note that thread::spawn also requires that the callable be 'static, so you need that too. It needs that because if it didn't require that, it'd have no guarantee that the value will continue to exist for the entire lifetime of the thread (which may or may not outlive the thread that spawned it).
..., and what the appropriate way to accomplish this is.
Same way as any other constraints: M: 'static + Send + Memory.