Call a C callback from a Send closure - rust

In Rust I have a callback defined as Box<dyn Fn(MyType) + Send>. I want to be able to convert a C callback provided by FFI into that signature.
To do that, I created a Rust closure that calls the C callback with a context pointer:
let rust_closure = move |data: u32| {
callback(obj, data);
};
The problem is obj in this case is a c_void. According to the docs (https://doc.rust-lang.org/core/ffi/enum.c_void.html) it should have auto-implemented Send and Sync. However the compiler tells me:
the trait Send is not implemented for *const c_void
How can I convince Rust that my c_void is indeed Send? I've tried wrapping it in a unit struct and impl'ing Send for the struct, but that did not change the error.
I've created a minimal example in the playground.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=5737c7449b925ab83e35d2404a6bd9f8

Pointers are automatically !Send so the solution is to wrap the pointer in a custom type, that hopefully better describes its use and reasons to guarantee that the pointer is safe to use across threads. I'm assuming you made a solution similar to this:
struct UserObj(*const c_void);
// SAFETY: Any user data object must be safe to send between threads.
unsafe impl Send for UserObj {}
let obj = UserObj(obj);
let rust_closure = move |data: u32| {
callback(obj.0, data);
};
ch.add_callback(Box::new(rust_closure));
Which unfortunately, does yield the same error:
error[E0277]: `*const c_void` cannot be sent between threads safely
--> src/lib.rs:33:25
|
29 | let rust_closure = move |data: u32| {
| ____________________________-
30 | | callback(obj.0, data);
31 | | };
| |_________- within this `[closure#src/lib.rs:29:28: 31:10]`
32 |
33 | ch.add_callback(Box::new(rust_closure));
| ^^^^^^^^^^^^^^^^^^^^^^ `*const c_void` cannot be sent between threads safely
|
= help: within `[closure#src/lib.rs:29:28: 31:10]`, the trait `Send` is not implemented for `*const c_void`
= note: required because it appears within the type `[closure#src/lib.rs:29:28: 31:10]`
= note: required for the cast to the object type `dyn Fn(u32) + Send`
But why? We're moving a UserObj into the closure and we've asserted to the compiler that it is Send. Why does it still complain about *const c_void?
As of Rust 2021 edition, closures can capture disjoint fields rather than the whole object, meaning the above closure would capture obj.0 instead of obj itself, causing the error. My suggestion would be to add a method to get the pointer instead of using field access in this case:
struct UserObj(*const c_void);
impl UserObj {
fn as_ptr(&self) -> *const c_void {
self.0
}
}
// SAFETY: Any user data object must be safe to send between threads.
unsafe impl Send for UserObj {}
let obj = UserObj(obj);
let rust_closure = move |data: u32| {
callback(obj.as_ptr(), data);
};
ch.add_callback(Box::new(rust_closure));
Playground

Related

Pin vs Box: Why is Box not enough?

I would like to know examples where keeping a T type within Box would be unsafe, while within Pin it would be safe.
Initially, I thought that std::marker::PhantomPinned prevents an instance from being moved around (by forbidding it), but seemingly it does not. Since:
use std::pin::Pin;
use std::marker::PhantomPinned;
#[derive(Debug)]
struct MyStruct {
field: u32,
_pin: PhantomPinned
}
impl MyStruct {
fn new(field: u32) -> Self {
Self {
field,
_pin: PhantomPinned,
}
}
}
fn func(x: MyStruct) {
println!("{:?}", x);
func2(x);
}
fn func2(x: MyStruct) {
println!("{:?}", x);
}
fn main() {
let x = MyStruct::new(5);
func(x);
}
this code is compilable, despite the fact that it moves MyStruct from main to func and etc.
as for Box and Pin they both keep their contents on the heap, so it does not seem to be subjected to motions.
Thus, I would appreciate if someone elaborated this topic on these questions. Since it is not covered in other questions and docs, what is inherently wrong with just getting by with Box.
I think you misunderstand.
PhantomPinned does not make data immovable. It just says that once the data is pinned, it will never be able to be unpinned again.
Therefore, to make data with PhantomPinned unmovable, you have to Pin it first.
For example, if you create a pinned version of your MyStruct variable, you cannot unpin it:
fn main() {
let pinned_x = Box::pin(MyStruct::new(5));
let unpinned_x = Pin::into_inner(pinned_x);
}
error[E0277]: `PhantomPinned` cannot be unpinned
--> src/main.rs:20:38
|
20 | let unpinned_x = Pin::into_inner(pinned_x);
| --------------- ^^^^^^^^ within `MyStruct`, the trait `Unpin` is not implemented for `PhantomPinned`
| |
| required by a bound introduced by this call
|
= note: consider using `Box::pin`
note: required because it appears within the type `MyStruct`
--> src/main.rs:4:8
|
4 | struct MyStruct {
| ^^^^^^^^
note: required by a bound in `Pin::<P>::into_inner`
--> /home/martin/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/pin.rs:482:23
|
482 | impl<P: Deref<Target: Unpin>> Pin<P> {
| ^^^^^ required by this bound in `Pin::<P>::into_inner`
While with a normal struct, you can unpin it without a problem:
struct MyUnpinnableStruct;
fn main() {
let pinned_x = Box::pin(MyUnpinnableStruct);
let unpinned_x = Pin::into_inner(pinned_x);
}
Difference between Pin and Box
They are both completely different concepts. Pin makes sure that the data it points to cannot be moved. Box puts something on the heap.
As you can see from the previous examples, both are often used in conjunction, as the easiest way to prevent something from moving is to put it on the heap.
PhantomPin causes classes to be !Unpin, meaning once they are pinned, they can no longer be unpinned.
You can try to use Pin on values on the stack, but you will run into problems quickly. While it works for unpin-able structs:
struct MyUnpinnableStruct(u32);
fn main() {
let y = MyUnpinnableStruct(7);
{
let pinned_y = Pin::new(&y);
}
// This moves y into the `drop` function
drop(y);
}
It fails for structs that contain PhantomPinned:
fn main() {
let x = MyStruct::new(5);
{
// This fails; pinning a reference to a stack object
// will fail, because once we drop that reference the
// object will be movable again. So we cannot `Pin` stack objects
let pinned_x = Pin::new(&x);
}
// This moves x into the `drop` function
drop(x);
}
error[E0277]: `PhantomPinned` cannot be unpinned
--> src/main.rs:24:33
|
24 | let pinned_x = Pin::new(&x);
| -------- ^^ within `MyStruct`, the trait `Unpin` is not implemented for `PhantomPinned`
| |
| required by a bound introduced by this call
|
= note: consider using `Box::pin`
note: required because it appears within the type `MyStruct`
--> src/main.rs:4:8
|
4 | struct MyStruct {
| ^^^^^^^^
note: required by a bound in `Pin::<P>::new`
--> /home/martin/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/pin.rs:482:23
|
482 | impl<P: Deref<Target: Unpin>> Pin<P> {
| ^^^^^ required by this bound in `Pin::<P>::new`
Box without Pin
While the content of Box is on the heap and therefore has a constant address, you can still move it back from the heap to the stack, which wouldn't be possible with a Pin object:
// Note that MyData does not implement Clone or Copy
struct MyData(u32);
impl MyData {
fn print_addr(&self) {
println!("Address: {:p}", self);
}
}
fn main() {
// On the heap
let x_heap = Box::new(MyData(42));
x_heap.print_addr();
// Moved back on the stack
let x_stack = *x_heap;
x_stack.print_addr();
}
Address: 0x557452040ad0
Address: 0x7ffde8f7f0d4
Enforcing Pin
To make sure that an object is pinned in a member function, you can use the following syntax:
fn print_addr(self: Pin<&Self>)
Together with PhantomPinned, you now can be 100% sure that print_addr will always print the same address for the same object:
use std::{marker::PhantomPinned, pin::Pin};
struct MyData(u32, PhantomPinned);
impl MyData {
fn print_addr(self: Pin<&Self>) {
println!("Address: {:p}", self);
}
}
fn main() {
// On the heap
let x_pinned = Box::pin(MyData(42, PhantomPinned));
x_pinned.as_ref().print_addr();
// Moved back on the stack
let x_unpinned = Pin::into_inner(x_pinned); // FAILS!
let x_stack = *x_unpinned;
let x_pinned_again = Box::pin(x_stack);
x_pinned_again.as_ref().print_addr();
}
In this example, there is absolutely no way to ever unpin x_pinned again, and print_addr can only be called on the pinned object.
Why is this useful? For example because you can now work with raw pointers, as is required in the Future trait.
But in general, Pin is only really useful if paired with unsafe code. Without unsafe code, the borrow checker is sufficient to keep track of your objects.

"cannot infer an appropriate lifetime" when attempting to return a chunked response with hyper

I would like to return binary data in chunks of specific size. Here is a minimal example.
I made a wrapper struct for hyper::Response to hold my data like status, status text, headers and the resource to return:
pub struct Response<'a> {
pub resource: Option<&'a Resource>
}
This struct has a build method that creates the hyper::Response:
impl<'a> Response<'a> {
pub fn build(&mut self) -> Result<hyper::Response<hyper::Body>, hyper::http::Error> {
let mut response = hyper::Response::builder();
match self.resource {
Some(r) => {
let chunks = r.data
.chunks(100)
.map(Result::<_, std::convert::Infallible>::Ok);
response.body(hyper::Body::wrap_stream(stream::iter(chunks)))
},
None => response.body(hyper::Body::from("")),
}
}
}
There is also another struct holding the database content:
pub struct Resource {
pub data: Vec<u8>
}
Everything works until I try to create a chunked response. The Rust compiler gives me the following error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:14:15
|
14 | match self.resource {
| ^^^^^^^^^^^^^
|
note: first, the lifetime cannot outlive the lifetime `'a` as defined on the impl at 11:6...
--> src/main.rs:11:6
|
11 | impl<'a> Response<'a> {
| ^^
note: ...so that the types are compatible
--> src/main.rs:14:15
|
14 | match self.resource {
| ^^^^^^^^^^^^^
= note: expected `Option<&Resource>`
found `Option<&'a Resource>`
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the types are compatible
--> src/main.rs:19:31
|
19 | response.body(hyper::Body::wrap_stream(stream::iter(chunks)))
| ^^^^^^^^^^^^^^^^^^^^^^^^
= note: expected `From<&[u8]>`
found `From<&'static [u8]>`
I don't know how to fulfill these lifetime requirements. How can I do this correctly?
The problem is not in the 'a itself, but in the fact that the std::slice::chunks() function returns an iterator that borrows the original slice. You are trying to create a stream future from this Chunks<'_, u8> value, but the stream requires it to be 'static. Even if your Resource did not have the 'a lifetime, you would still have the r.data borrowed, and it would still fail.
Remember that here 'static does not mean that the value lives forever, but that it can be made to live as long as necessary. That is, the future must not hold any (non-'static) borrows.
You could clone all the data, but if it is very big, it can be costly. If so, you could try using Bytes, that is just like Vec<u8> but reference counted.
It looks like there is no Bytes::chunks() function that returns an iterator of Bytes. Fortunately it is easy to do it by hand.
Lastly, remember that iterators in Rust are lazy, so they keep the original data borrowed, even if it is a Bytes. So we need to collect them into a Vec to actually own the data (playground):
pub struct Resource {
pub data: Bytes,
}
impl<'a> Response<'a> {
pub fn build(&mut self) -> Result<hyper::Response<hyper::Body>, hyper::http::Error> {
let mut response = hyper::Response::builder();
match self.resource {
Some(r) => {
let len = r.data.len();
let chunks = (0..len)
.step_by(100)
.map(|x| {
let range = x..len.min(x + 100);
Ok(r.data.slice(range))
})
.collect::<Vec<Result<Bytes, std::convert::Infallible>>>();
response.body(hyper::Body::wrap_stream(stream::iter(chunks)))
}
None => response.body(hyper::Body::from("")),
}
}
}
UPDATE: We can avoid the call to collect() if we notice that stream::iter() takes ownership of an IntoIterator that can be evaluated lazily, as long as we make it 'static. It can be done if we do a (cheap) clone of r.data and move it into the lambda (playground):
let data = r.data.clone();
let len = data.len();
let chunks = (0..len).step_by(100)
.map(move |x| {
let range = x .. len.min(x + 100);
Result::<_, std::convert::Infallible>::Ok(data.slice(range))
});
response.body(hyper::Body::wrap_stream(stream::iter(chunks)))

Send FnMut to other thread by wrapping it into a Weak<Mutex<FnMut>> complains about missing Send [duplicate]

This question already has answers here:
Sending trait objects between threads in Rust
(1 answer)
Sharing a struct with trait objects as properties across threads
(1 answer)
Closed 3 years ago.
I'm trying to write a simple Observer-pattern-ish thing with events and callbacks. The idea is that when the thing owning the mutating function dies (struct Obj {cb: Arc<Mutex<FnMut>>, .. }), it drops the Arc. After that, any Weak references will be (effectively) dead and can be detected on access. I was ALSO hoping that this would decouple the FnMut from Thread's 'static lifetime. I might need to wrap the whole thing in another Arc<Rwlock<T>> to prevent the Weak references from doing thread unsafe stuff, but that's a different problem.
The closest solution I've gotten was this: How can I send a function to another thread?
However, I seem to have the Arc<Mutex<T>> and adding lifetimes (I may have done that wrong, though) didn't seem to help. I'm kind of lost as to what is actually wrong.
I wrote a minimal example:
use std::{
collections::HashMap,
sync::{
mpsc::{self, Receiver},
Arc, Mutex, Weak,
},
thread::{self, JoinHandle},
};
struct Callback {
cb_w: Weak<Mutex<FnMut()>>,
}
type FnMapping = Arc<HashMap<String, Callback>>;
fn start_thread<'a>(rx: Receiver<String>, mapping: FnMapping) -> JoinHandle<()> {
thread::spawn(move || {
match rx.recv() {
Ok(v) => {
if let Some(o) = mapping.get(&v) {
match o.cb_w.upgrade() {
Some(cb_m) => match cb_m.lock() {
Ok(cb_lg) => (*cb_lg)(),
Err(e) => (),
},
None => { /* TODO owner is gone, mark for delete */ }
}
}
}
Err(e) => (),
}
})
}
fn main() {
let mapping: FnMapping = Arc::new(HashMap::new());
let (tx, rx) = mpsc::channel();
drop(tx);
start_thread(rx, mapping)
.join()
.expect("Could not join thread -- failed to terminate?");
println!("Leaving the test bay.");
}
This fails to compile with the following error:
error[E0277]: `(dyn std::ops::FnMut() + 'static)` cannot be sent between threads safely
--> src/main.rs:17:5
|
17 | thread::spawn(move || {
| ^^^^^^^^^^^^^ `(dyn std::ops::FnMut() + 'static)` cannot be sent between threads safely
|
= help: the trait `std::marker::Send` is not implemented for `(dyn std::ops::FnMut() + 'static)`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<(dyn std::ops::FnMut() + 'static)>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Weak<std::sync::Mutex<(dyn std::ops::FnMut() + 'static)>>`
= note: required because it appears within the type `Callback`
= note: required because it appears within the type `(std::string::String, Callback)`
= note: required because it appears within the type `std::marker::PhantomData<(std::string::String, Callback)>`
= note: required because it appears within the type `std::collections::hash::table::RawTable<std::string::String, Callback>`
= note: required because it appears within the type `std::collections::HashMap<std::string::String, Callback>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<std::collections::HashMap<std::string::String, Callback>>`
= note: required because it appears within the type `[closure#src/main.rs:17:19: 32:6 rx:std::sync::mpsc::Receiver<std::string::String>, mapping:std::sync::Arc<std::collections::HashMap<std::string::String, Callback>>]`
= note: required by `std::thread::spawn`

Why can't I reuse a &mut reference after passing it to a function that accepts a generic type?

Why doesn't this code compile:
fn use_cursor(cursor: &mut io::Cursor<&mut Vec<u8>>) {
// do some work
}
fn take_reference(data: &mut Vec<u8>) {
{
let mut buf = io::Cursor::new(data);
use_cursor(&mut buf);
}
data.len();
}
fn produce_data() {
let mut data = Vec::new();
take_reference(&mut data);
data.len();
}
The error in this case is:
error[E0382]: use of moved value: `*data`
--> src/main.rs:14:5
|
9 | let mut buf = io::Cursor::new(data);
| ---- value moved here
...
14 | data.len();
| ^^^^ value used here after move
|
= note: move occurs because `data` has type `&mut std::vec::Vec<u8>`, which does not implement the `Copy` trait
The signature of io::Cursor::new is such that it takes ownership of its argument. In this case, the argument is a mutable reference to a Vec.
pub fn new(inner: T) -> Cursor<T>
It sort of makes sense to me; because Cursor::new takes ownership of its argument (and not a reference) we can't use that value later on. At the same time it doesn't make sense: we essentially only pass a mutable reference and the cursor goes out of scope afterwards anyway.
In the produce_data function we also pass a mutable reference to take_reference, and it doesn't produce a error when trying to use data again, unlike inside take_reference.
I found it possible to 'reclaim' the reference by using Cursor.into_inner(), but it feels a bit weird to do it manually, since in normal use-cases the borrow-checker is perfectly capable of doing it itself.
Is there a nicer solution to this problem than using .into_inner()? Maybe there's something else I don't understand about the borrow-checker?
Normally, when you pass a mutable reference to a function, the compiler implicitly performs a reborrow. This produces a new borrow with a shorter lifetime.
When the parameter is generic (and is not of the form &mut T), the compiler doesn't do this reborrowing automatically1. However, you can do it manually by dereferencing your existing mutable reference and then referencing it again:
fn take_reference(data: &mut Vec<u8>) {
{
let mut buf = io::Cursor::new(&mut *data);
use_cursor(&mut buf);
}
data.len();
}
1 — This is because the current compiler architecture only allows a chance to do a coercion if both the source and target types are known at the coercion site.

How do I share a generic struct between threads using Arc<Mutex<MyStruct<T>>>?

I have some mutable state I need to share between threads. I followed the concurrency section of the Rust book, which shares a vector between threads and mutates it.
Instead of a vector, I need to share a generic struct that is ultimately monomorphized. Here is a distilled example of what I'm trying:
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
use std::marker::PhantomData;
trait Memory {}
struct SimpleMemory;
impl Memory for SimpleMemory {}
struct SharedData<M: Memory> {
value: usize,
phantom: PhantomData<M>,
}
impl<M: Memory> SharedData<M> {
fn new() -> Self {
SharedData {
value: 0,
phantom: PhantomData,
}
}
}
fn main() {
share(SimpleMemory);
}
fn share<M: Memory>(memory: M) {
let data = Arc::new(Mutex::new(SharedData::<M>::new()));
for i in 0..3 {
let data = data.clone();
thread::spawn(move || {
let mut data = data.lock().unwrap();
data.value += i;
});
}
thread::sleep(Duration::from_millis(50));
}
The compiler complains with the following error:
error[E0277]: the trait bound `M: std::marker::Send` is not satisfied
--> src/main.rs:37:9
|
37 | thread::spawn(move || {
| ^^^^^^^^^^^^^
|
= help: consider adding a `where M: std::marker::Send` bound
= note: required because it appears within the type `std::marker::PhantomData<M>`
= note: required because it appears within the type `SharedData<M>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Mutex<SharedData<M>>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<std::sync::Mutex<SharedData<M>>>`
= note: required because it appears within the type `[closure#src/main.rs:37:23: 40:10 data:std::sync::Arc<std::sync::Mutex<SharedData<M>>>, i:usize]`
= note: required by `std::thread::spawn`
I'm trying to understand why M would need to implement Send, and what the appropriate way to accomplish this is.
I'm trying to understand why M would need to implement Send, ...
Because, as stated by the Send documentation:
Types that can be transferred across thread boundaries.
If it's not Send, it is by definition not safe to send to another thread.
Almost all of the information you need is right there in the documentation:
thread::spawn requires the callable you give it to be Send.
You're using a closure, which is only Send if all the values it captures are Send. This is true in general of most types (they are Send if everything they're made of is Send, and similarly for Sync).
You're capturing data, which is an Arc<T>, which is only Send if T is Send.
T is a Mutex<U>, which is only Send if U is Send.
U is M. Thus, M must be Send.
In addition, note that thread::spawn also requires that the callable be 'static, so you need that too. It needs that because if it didn't require that, it'd have no guarantee that the value will continue to exist for the entire lifetime of the thread (which may or may not outlive the thread that spawned it).
..., and what the appropriate way to accomplish this is.
Same way as any other constraints: M: 'static + Send + Memory.

Resources