I'm wrapping a C API which allows the caller to set/get an arbitrary pointer via function calls. In this way, the C API allows a caller to associate arbitrary data with one of the C API objects. This data is not used in any callbacks, it's just a pointer that a user can stash away and get at later.
My wrapper struct implements the Drop trait for the C object that contains this pointer. What I'd like to be able to do, but am not sure it's possible, is have the data dropped correctly if the pointer is not null when the wrapper struct drops. I'm not sure how I would recover the correct type though from a raw c_void pointer.
Two alternatives I'm thinking of are
Implement the behavior of these two calls in the wrapper. Don't make any calls to the C API.
Don't attempt to offer any kind of safer interface to these functions. Document that the pointer must be managed by the caller of the wrapper.
Is what I want to do possible? If not, is there a generally accepted practice for these kinds of situations?
A naive + fully automatic approach is NOT possible for the following reasons:
freeing memory does not call drop/deconstructors/...: the C API can be used from languages which can have objects which should be deconstructed properly, e.g. C++ or Rust itself. So when you only store a memory pointer you do not know you to call the proper function (you neither know which function not how the calling conventions look like).
which memory allocator?: memory allocation and deallocation isn't a trivial thing. your program needs to request memory from the OS and then manage this resources in an intelligent way to be efficient and correct. This is usually done by a library. In case of Rust, jemalloc is used (but can be changed). So even when you ask the API caller to only pass Plain Old Data (which should be easier to destruct) you still don't know which library function to call to deallocate memory. Just using libc::free won't work (it can but it could horrible fail).
Solutions:
dealloc callback: you can ask the API user to set an additional pointer to, let's say a void destruct(void* ptr) function. If this one is not NULL, you call that function during your drop. You could also use int as an return type to signal when the destruction went wrong. In that case you could for example panic!.
global callback: let's assume you requested your user to only pass POD (plain old data). To know which free function of the memory allocator to call, you could request the user to register a global void (*free)(void* ptr) pointer which is called during drop. You could also make that one optional.
Although I was able to follow the advice in this thread, I wasn't entirely satisfied with my results, so I asked the question on the Rust forums and found the answer I was really looking for. (play)
use std::any::Any;
static mut foreign_ptr: *mut () = 0 as *mut ();
unsafe fn api_set_fp(ptr: *mut ()) {
foreign_ptr = ptr;
}
unsafe fn api_get_fp() -> *mut() {
foreign_ptr
}
struct ApiWrapper {}
impl ApiWrapper {
fn set_foreign<T: Any>(&mut self, value: Box<T>) {
self.free_foreign();
unsafe {
let raw = Box::into_raw(Box::new(value as Box<Any>));
api_set_fp(raw as *mut ());
}
}
fn get_foreign_ref<T: Any>(&self) -> Option<&T> {
unsafe {
let raw = api_get_fp() as *const Box<Any>;
if !raw.is_null() {
let b: &Box<Any> = &*raw;
b.downcast_ref()
} else {
None
}
}
}
fn get_foreign_mut<T: Any>(&mut self) -> Option<&mut T> {
unsafe {
let raw = api_get_fp() as *mut Box<Any>;
if !raw.is_null() {
let b: &mut Box<Any> = &mut *raw;
b.downcast_mut()
} else {
None
}
}
}
fn free_foreign(&mut self) {
unsafe {
let raw = api_get_fp() as *mut Box<Any>;
if !raw.is_null() {
Box::from_raw(raw);
}
}
}
}
impl Drop for ApiWrapper {
fn drop(&mut self) {
self.free_foreign();
}
}
struct MyData {
i: i32,
}
impl Drop for MyData {
fn drop(&mut self) {
println!("Dropping MyData with value {}", self.i);
}
}
fn main() {
let p1 = Box::new(MyData {i: 1});
let mut api = ApiWrapper{};
api.set_foreign(p1);
{
let p2 = api.get_foreign_ref::<MyData>().unwrap();
println!("i is {}", p2.i);
}
api.set_foreign(Box::new("Hello!"));
{
let p3 = api.get_foreign_ref::<&'static str>().unwrap();
println!("payload is {}", p3);
}
}
Related
I would like to call the C function libusb1_sys::libusb_set_pollfd_notifiers() from Rust, providing callbacks and context. The C function is:
void libusb_set_pollfd_notifiers(
libusb_context * ctx,
libusb_pollfd_added_cb added_cb,
libusb_pollfd_removed_cb removed_cb,
void * user_data
);
This follows the traditional method for "closures" in C where user_data provides context for the callback.
The following code compiles and runs, but as far as I can tell is wrong:
use libusb1_sys::libusb_pollfd;
use rusb::UsbContext;
extern "system" fn usb_source_added_cb(
_fd: libc::c_int,
_events: libc::c_short,
_user_data: *mut libc::c_void,
) {
println!("USB source added");
}
extern "system" fn usb_source_removed_cb(
_fd: libc::c_int,
_user_data: *mut libc::c_void,
) {
println!("USB source removed");
}
struct UsbFdChannel {
sender: std::sync::mpsc::Sender<libc::c_int>,
context: rusb::Context,
}
impl UsbFdChannel {
fn new() -> (Self, std::sync::mpsc::Receiver<libc::c_int>) {
let context = rusb::Context::new().unwrap();
// This gives me a `*mut libusb_context`
let raw_context = context.as_raw();
let (mut sender, receiver) = std::sync::mpsc::channel();
unsafe {
libusb1_sys::libusb_set_pollfd_notifiers(
raw_context,
usb_source_added_cb,
usb_source_removed_cb,
&mut sender as *mut _ as *mut libc::c_void,
);
}
(Self { sender, context }, receiver)
}
}
fn main() {
let _usb_channel = UsbFdChannel::new();
}
I think it's wrong because once UsbFdChannel::new() returns, the mut * to sender is no longer valid, but libusb's context is still able to use it. Am I correct?
What I would like is for this incorrect code to not compile. If possible, I'd like this entire class of incorrect code to not compile. I think what I want is to associate a lifetime with sender, or possibly with &mut sender, that the compiler will then enforce. Or maybe I want to "trick" rust into thinking that the lifetime of *mut sender is bounded by that of &mut sender...?
If I achieve that, then I can resolve the incorrectness however I can - by having UsbFdChannel take a reference, or create a reference counted/boxed/both thing, or whatever. The important thing to me (for both solving this particular problem and for learning how to write better Rust) is that I can identify and declare the ownership problem first, and then solve that problem second.
I considered using std::marker::PhantomData, but since UsbFdChannel is not parameterised by a type, I don't see how to tell it what member the PhantomData's lifetime would correspond to. I also considered using some kind of non-stack allocation and reference counting, but I still want to know (inasmuch as Rust can tell me) that I've solved the problem. How do I do that?
Is it possible to allocate self-referential structs on stack? The pin documentation shows an example of pinning a struct on the heap. I followed it to write corresponding code:
pub struct Cont {
pub f: i32,
// shall point to 'f' field
ptr_f: NonNull<i32>,
}
impl Cont {
pub fn read(&self) -> i32 {
unsafe { *self.ptr_f.as_ref() }
}
pub fn pinned(value: i32) -> Pin<Self> {
// ensures ptr_f = &f
}
}
fn main() {
let a = Cont::pinned(5);
let b = Cont::pinned(12);
assert_eq!(a.f, a.read());
assert_eq!(b.f, b.read());
}
but I don't know how to write the Cont::pinned function, or even if it's the right signature (if even possible).
but I don't know how to write the Cont::pinned function, or even if it's the right signature (if even possible).
The type parameter to Pin<P> is always a pointer or reference; a Pin never owns its data, except via an owning pointer type such as Box. Given that you want to keep the original value on the stack, the analogue of the function in the example is:
fn pinned(value: i32) -> Pin<&mut Self>;
But this isn't possible because a function can't return a reference to something it created - unless that something is stored on the heap or in static memory. So, if you were to construct a Pin of a stack-allocated value, you'd have to create the unpinned value first, so that that the pin can reference it.
Perhaps you might try to design an API that creates a Pin by mutating some data that is already on the stack:
let a: Option<Cont> = None;
let b: Option<Cont> = None;
let a = Cont::pinned(&mut a, 5);
let b = Cont::pinned(&mut b, 12);
But then the value would live longer than the pin and you can only enforce the pin guarantees while the pin is live, making this unsound.
To make it sound, you would need to somehow enforce that the original values cannot be accessed after the pin is dropped. This would result in a very constrained API. For example:
fn main() {
// setup_conts creates the pinned Cont values and calls the closure
setup_conts(5, 12, |a: Pin<&mut Cont>, b: Pin<&mut Cont>| {
// a and b can only be used inside this closure
})
}
As part of binding a C API to Rust, I have a mutable reference ph: &mut Ph, a struct struct EnsureValidContext<'a> { ph: &'a mut Ph }, and some methods:
impl Ph {
pub fn print(&mut self, s: &str) {
/*...*/
}
pub fn with_context<F, R>(&mut self, ctx: &Context, f: F) -> Result<R, InvalidContextError>
where
F: Fn(EnsureValidContext) -> R,
{
/*...*/
}
/* some others */
}
impl<'a> EnsureValidContext<'a> {
pub fn print(&mut self, s: &str) {
self.ph.print(s)
}
pub fn close(self) {}
/* some others */
}
I don't control these. I can only use these.
Now, the closure API is nice if you want the compiler to force you to think about performance (and the tradeoffs you have to make between performance and the behaviour you want. Context validation is expensive). However, let's say you just don't care about that and want it to just work.
I was thinking of making a wrapper that handles it for you:
enum ValidPh<'a> {
Ph(&'a mut Ph),
Valid(*mut Ph, EnsureValidContext<'a>),
Poisoned,
}
impl<'a> ValidPh<'a> {
pub fn print(&mut self) {
/* whatever the case, just call .print() on the inner object */
}
pub fn set_context(&mut self, ctx: &Context) {
/*...*/
}
pub fn close(&mut self) {
/*...*/
}
/* some others */
}
This would work by, whenever necessary, checking if we're a Ph or a Valid, and if we're a Ph we'd upgrade to a Valid by going:
fn upgrade(&mut self) {
if let Ph(_) = self { // don't call mem::replace unless we need to
if let Ph(ph) = mem::replace(self, Poisoned) {
let ptr = ph as *mut _;
let evc = ph.with_context(ph.get_context(), |evc| evc);
*self = Valid(ptr, evc);
}
}
}
Downgrading is different for each method, as it has to call the target method, but here's an example close:
pub fn close(&mut self) {
if let Valid(_, _) = self {
/* ok */
} else {
self.upgrade()
}
if let Valid(ptr, evc) = mem::replace(self, Invalid) {
evc.close(); // consume the evc, dropping the borrow.
// we can now use our original borrow, but since we don't have it anymore, bring it back using our trusty ptr
*self = unsafe { Ph(&mut *ptr) };
} else {
// this can only happen due to a bug in our code
unreachable!();
}
}
You get to use a ValidPh like:
/* given a &mut vph */
vph.print("hello world!");
if vph.set_context(ctx) {
vph.print("closing existing context");
vph.close();
}
vph.print("opening new context");
vph.open("context_name");
vph.print("printing in new context");
Without vph, you'd have to juggle &mut Ph and EnsureValidContext around on your own. While the Rust compiler makes this trivial (just follow the errors), you may want to let the library handle it automatically for you. Otherwise you might end up just calling the very expensive with_context for every operation, regardless of whether the operation can invalidate the context or not.
Note that this code is rough pseudocode. I haven't compiled or tested it yet.
One might argue I need an UnsafeCell or a RefCell or some other Cell. However, from reading this it appears UnsafeCell is only a lang item because of interior mutability — it's only necessary if you're mutating state through an &T, while in this case I have &mut T all the way.
However, my reading may be flawed. Does this code invoke UB?
(Full code of Ph and EnsureValidContext, including FFI bits, available here.)
Taking a step back, the guarantees upheld by Rust are:
&T is a reference to T which is potentially aliased,
&mut T is a reference to T which is guaranteed not to be aliased.
The crux of the question therefore is: what does guaranteed not to be aliased means?
Let's consider a safe Rust sample:
struct Foo(u32);
impl Foo {
fn foo(&mut self) { self.bar(); }
fn bar(&mut self) { *self.0 += 1; }
}
fn main() { Foo(0).foo(); }
If we take a peek at the stack when Foo::bar is being executed, we'll see at least two pointers to Foo: one in bar and one in foo, and there may be further copies on the stack or in other registers.
So, clearly, there are aliases in existence. How come! It's guaranteed NOT to be aliased!
Take a deep breath: how many of those aliases can you access at the time?
Only 1. The guarantee of no aliasing is not spatial but temporal.
I would think, therefore, that at any point in time, if a &mut T is accessible, then no other reference to this instance must be accessible.
Having a raw pointer (*mut T) is perfectly fine, it requires unsafe to access; however forming a second reference may or may not be safe, even without using it, so I would avoid it.
Rust's memory model is not rigorously defined yet, so it's hard to say for sure, but I believe it's not undefined behavior to:
carry a *mut Ph around while a &'a mut Ph is also reachable from another path, so long as you don't dereference the *mut Ph, even just for reading, and don't convert it to a &Ph or &mut Ph, because mutable references grant exclusive access to the pointee.
cast the *mut Ph back to a &'a mut Ph once the other &'a mut Ph falls out of scope.
I'm trying to implement a method that looks like:
fn concretify<T: Any>(rc: Rc<Any>) -> Option<T> {
Rc::try_unwrap(rc).ok().and_then(|trait_object| {
let b: Box<Any> = unimplemented!();
b.downcast().ok().map(|b| *b)
})
}
However, try_unwrap doesn't work on trait objects (which makes sense, as they're unsized). My next thought was to try to find some function that unwraps Rc<Any> into Box<Any> directly. The closest thing I could find would be
if Rc::strong_count(&rc) == 1 {
Some(unsafe {
Box::from_raw(Rc::into_raw(rc))
})
} else {
None
}
However, Rc::into_raw() appears to require that the type contained in the Rc to be Sized, and I'd ideally not like to have to use unsafe blocks.
Is there any way to implement this?
Playground Link, I'm looking for an implementation of rc_to_box here.
Unfortunately, it appears that the API of Rc is lacking the necessary method to be able to get ownership of the wrapped type when it is !Sized.
The only method which may return the interior item of a Rc is Rc::try_unwrap, however it returns Result<T, Rc<T>> which requires that T be Sized.
In order to do what you wish, you would need to have a method with a signature: Rc<T> -> Result<Box<T>, Rc<T>>, which would allow T to be !Sized, and from there you could extract Box<Any> and perform the downcast call.
However, this method is impossible due to how Rc is implemented. Here is a stripped down version of Rc:
struct RcBox<T: ?Sized> {
strong: Cell<usize>,
weak: Cell<usize>,
value: T,
}
pub struct Rc<T: ?Sized> {
ptr: *mut RcBox<T>,
_marker: PhantomData<T>,
}
Therefore, the only Box you can get out of Rc<T> is Box<RcBox<T>>.
Note that the design is severely constrained here:
single-allocation mandates that all 3 elements be in a single struct
T: ?Sized mandates that T be the last field
so there is little room for improvement in general.
However, in your specific case, it is definitely possible to improve on the generic situation. It does, of course, require unsafe code. And while it works fairly well with Rc, implementing it with Arc would be complicated by the potential data-races.
Oh... and the code is provided as is, no warranty implied ;)
use std::any::Any;
use std::{cell, mem, ptr};
use std::rc::Rc;
struct RcBox<T: ?Sized> {
strong: cell::Cell<usize>,
_weak: cell::Cell<usize>,
value: T,
}
fn concretify<T: Any>(rc: Rc<Any>) -> Option<T> {
// Will be responsible for freeing the memory if there is no other weak
// pointer by the end of this function.
let _guard = Rc::downgrade(&rc);
unsafe {
let killer: &RcBox<Any> = {
let killer: *const RcBox<Any> = mem::transmute(rc);
&*killer
};
if killer.strong.get() != 1 { return None; }
// Do not forget to decrement the count if we do take ownership,
// as otherwise memory will not get released.
let result = killer.value.downcast_ref().map(|r| {
killer.strong.set(0);
ptr::read(r as *const T)
});
// Do not forget to destroy the content of the box if we did not
// take ownership
if result.is_none() {
let _: Rc<Any> = mem::transmute(killer as *const RcBox<Any>);
}
result
}
}
fn main() {
let x: Rc<Any> = Rc::new(1);
println!("{:?}", concretify::<i32>(x));
}
I don't think it's possible to implement your concretify function if you're expecting it to move the original value back out of the Rc; see this question for why.
If you're willing to return a clone, it's straightforward:
fn concretify<T: Any+Clone>(rc: Rc<Any>) -> Option<T> {
rc.downcast_ref().map(Clone::clone)
}
Here's a test:
#[derive(Debug,Clone)]
struct Foo(u32);
#[derive(Debug,Clone)]
struct Bar(i32);
fn main() {
let rc_foo: Rc<Any> = Rc::new(Foo(42));
let rc_bar: Rc<Any> = Rc::new(Bar(7));
let foo: Option<Foo> = concretify(rc_foo);
println!("Got back: {:?}", foo);
let bar: Option<Foo> = concretify(rc_bar);
println!("Got back: {:?}", bar);
}
This outputs:
Got back: Some(Foo(42))
Got back: None
Playground
If you want something more "movey", and creating your values is cheap, you could also make a dummy, use downcast_mut() instead of downcast_ref(), and then std::mem::swap with the dummy.
I want to have a structure on the heap with two references; one for me and another from a closure. Note that the code is for the single-threaded case:
use std::rc::Rc;
#[derive(Debug)]
struct Foo {
val: u32,
}
impl Foo {
fn set_val(&mut self, val: u32) {
self.val = val;
}
}
impl Drop for Foo {
fn drop(&mut self) {
println!("we drop {:?}", self);
}
}
fn need_callback(mut cb: Box<FnMut(u32)>) {
cb(17);
}
fn create() -> Rc<Foo> {
let rc = Rc::new(Foo { val: 5 });
let weak_rc = Rc::downgrade(&rc);
need_callback(Box::new(move |x| {
if let Some(mut rc) = weak_rc.upgrade() {
if let Some(foo) = Rc::get_mut(&mut rc) {
foo.set_val(x);
}
}
}));
rc
}
fn main() {
create();
}
In the real code, need_callback saves the callback to some place, but before that may call cb as need_callback does.
The code shows that std::rc::Rc is not suitable for this task because foo.set_val(x) is never called; I have two strong references and Rc::get_mut gives None in this case.
What smart pointer with reference counting should I use instead of std::rc::Rc to make it possible to call foo.set_val? Maybe it is possible to fix my code and still use std::rc::Rc?
After some thinking, I need something like std::rc::Rc, but weak references should prevent dropping. I can have two weak references and upgrade them to strong when I need mutability.
Because it is a singled-threaded program, I will have only strong reference at a time, so everything will work as expected.
Rc (and its multithreaded counterpart Arc) only concern themselves with ownership. Instead of a single owner, there is now joint ownership, tracked at runtime.
Mutability is a different concept, although closely related to ownership: if you own a value, then you have the ability to mutate it. This is why Rc::get_mut only works when there is a single strong reference - it's the same as saying there is a single owner.
If you need the ability to divide mutability in a way that doesn't match the structure of the program, you can use tools like Cell or RefCell for single-threaded programs:
use std::cell::RefCell;
fn create() -> Rc<RefCell<Foo>> {
let rc = Rc::new(RefCell::new(Foo { val: 5 }));
let weak_rc = Rc::downgrade(&rc);
need_callback(move |x| {
if let Some(rc) = weak_rc.upgrade() {
rc.borrow_mut().set_val(x);
}
});
rc
}
Or Mutex, RwLock, or an atomic type in multithreaded contexts:
use std::sync::Mutex;
fn create() -> Rc<Mutex<Foo>> {
let rc = Rc::new(Mutex::new(Foo { val: 5 }));
let weak_rc = Rc::downgrade(&rc);
need_callback(move |x| {
if let Some(rc) = weak_rc.upgrade() {
if let Ok(mut foo) = rc.try_lock() {
foo.set_val(x);
}
}
});
rc
}
These tools all defer the check that there is only a single mutable reference to runtime, instead of compile time.