I have many lib crates and only one bin crate.
In the lib crates I have this simple code (different for each one):
use async_graphql::{Context, Object, Result};
use app::Services;
use std::sync::Arc;
#[derive(Default)]
pub struct PlayersQueryRoot;
#[Object]
impl PlayersQueryRoot {
async fn players_query_root(&self, ctx: &Context<'_>) -> Result<String> {
let res = ctx.data_unchecked::<Arc<Services>>(); // Services here is declared in `app` and imported with Cargo.toml, hence the issue
let player = res.player_by_id.handle(1).await?;
Ok(player.name)
}
}
In the bin crate (called app) I have this code:
use async_graphql::{EmptyMutation, EmptySubscription, Schema};
use graphql::{GraphqlSchema, QueryRoot};
use std::sync::Arc;
pub struct Services {
pub players: Arc<players::Service>,
pub user: Arc<user::Service>,
}
impl Services {
pub async fn new() -> Self {
Self {
user: Arc::new(user::new(/*some needed injection*/)),
players: Arc::new(players::new(/*some needed injection*/)),
}
}
}
//...
pub fn create_schema(services: Arc<Services>) -> GraphqlSchema {
Schema::build(QueryRoot::default(), EmptyMutation, EmptySubscription)
.data(services)
.finish()
}
It doesn't work because there is a cyclic dependency between app (bin crate) which has the lib crates as dependencies:
[dependencies]
players = { path = "../crates/players" }
users = { path = "../crates/users" }
Is there a way to fix this?
Related
I have a library that is used through its rust interface by rust programs, as well as through C/C++ programs through generated cbindgen bindings, so I implemented a free function to free the string once the ffi function has used the string. However I want rust also to control the memory when it is used as a rust lib. How do I achieve this? is it even possible? or is calling the free function manually in rust the only option?
I also tried implementing drop, but that lead to this:
free(): double free detected in tcache 2
[1] 11097 IOT instruction cargo run
This block allows the string to be freed from C/C++, but the string is not freed in rust (valgrind shows definitely lost block). data is assigned using CString::into_raw()
use std::{ffi::CString, os::raw::c_char};
pub struct SomeData {
pub data: *const c_char
}
impl SomeData {
#[no_mangle] pub extern fn free_shared_string(&mut self) {
if !self.data.is_null() {
unsafe { CString::from_raw(self.data.cast_mut()); }
}
}
}
The docs for from_raw warn against doing exactly this.
Safety
This should only ever be called with a pointer that was earlier obtained by calling CString::into_raw. Other usage (e.g., trying to take ownership of a string that was allocated by foreign code) is likely to lead to undefined behavior or allocator corruption.
So do not use from_raw to pretend that a foreign string was allocated using Rust. If you just need to borrow it and let C free it, you should use the CStr type for borrowed strings. If you want to take ownership, you should copy it into a new string, or wrap it in a custom structure that has a Drop implementation capable of freeing the original memory.
You cannot have two different languages owning that memory. Rust is fundamentally built on a single-ownership model, so every piece of memory has a unique owner. There are some (intra-Rust) workarounds for that like Rc, but none of that will translate to C. So pick an owner, and make that language responsible for freeing the data.
The best solution for me was to have a separate feature, used when building the library to be used through C/C++ applications (ie. .a/.so) vs .rlib which cargo will build when included in a rust project through Cargo.toml.
This lets me use the same API from both possible application languages, call free from C/C++ on my string, and drop will free it in rust.
Note: the null character at the end is because the majority of the time my lib is used with C apps, hence storing with null for faster returns for them.
Add default-features = false when adding in Cargo.toml of a rust app.
lib.rs
use std::{ffi::{c_char, CStr, FromBytesWithNulError, CString}, mem::forget, str::Utf8Error, string::FromUtf8Error};
#[cfg(feature = "c-str")]
#[repr(C)]
pub struct SharedString {
str: *const c_char
}
#[cfg(not(feature = "c-str"))]
pub struct SharedString {
str: Vec<u8>
}
#[cfg(feature = "c-str")]
impl SharedString {
pub fn from_bytes(buf: &[u8]) -> Self {
let mut buf = buf.to_vec();
if let Some(c) = buf.last() {
if *c != 0 {
buf.push(0);
}
}
let s = Self { str: buf.as_ptr().cast() };
forget(buf);
s
}
pub unsafe fn get_string(&self) -> Result<String, SharedStringError> {
Ok(CStr::from_ptr(self.str).to_str()?.to_owned())
}
pub unsafe fn free(&self) {
if !self.str.is_null() {
let _ = CString::from_raw(self.str.cast_mut());
}
}
}
#[cfg(not(feature = "c-str"))]
impl SharedString {
pub fn from_bytes(buf: &[u8]) -> Self {
let mut buf = buf.to_vec();
if let Some(c) = buf.last() {
if *c != 0 {
buf.push(0);
}
}
Self { str: buf }
}
pub fn get_string(&self) -> Result<String, SharedStringError> {
let mut s = self.str.clone();
if let Some(c) = s.last() {
if *c == 0 {
s.pop();
}
}
String::from_utf8(s).map_err(|e| e.into())
}
// do nothing because rust vec will get dropped automatically
pub fn free(&self) {}
}
// Just for proof of concept
#[derive(Debug)]
pub enum SharedStringError {
NullError,
Utf8Error
}
impl From<FromBytesWithNulError> for SharedStringError {
fn from(_: FromBytesWithNulError) -> Self {
Self::NullError
}
}
impl From<Utf8Error> for SharedStringError {
fn from(_: Utf8Error) -> Self {
Self::Utf8Error
}
}
impl From<FromUtf8Error> for SharedStringError {
fn from(_: FromUtf8Error) -> Self {
Self::Utf8Error
}
}
Cargo.toml
[package]
name = "mylib"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
[features]
default = ["c-str"]
c-str = []
I have the following code in src/
main.rs
a.rs
b.rs
Here's the code:
main.rs
mod a;
mod b;
use crate::a::Summary;
use crate::b::Person;
fn main() {
let p = Person{ first: "John".to_string(), last: "Doe".to_string() } ;
sum(p) ;
}
fn sum(summary: impl Summary) {
println!("{}", summary.summarize()) ;
}
a.rs
pub trait Summary {
fn summarize(&self) -> String ;
}
b.rs
use crate::Summary;
pub struct Person {
pub first: String,
pub last: String,
}
impl Summary for Person {
fn summarize(&self) -> String {
format!("{}, {}.", self.last, self.first)
}
}
What I don't understand is how does "use crate::Summary;" not cause a problem in b.rs? It should be "use crate::a::Summary;" or even "use super::a::Summary;", but for some reason use crate::Summary works. Is there some kind of funky search logic being applied here under the hood?
Items defined without a visibility specifier are available to the module that they're defined in and all of its sub-modules.
Since a and b are submodules of the crate root module, they can access the Summary object that was imported via a use declaration in main into the crate root module.
I want to write a FFI wrapper for sn_api library, which contains async functions. It will be used in single-threaded non-async code written in Red.
I found, that the easy way is to use Runtime::new().unwrap().block_on(...) in every exported function, although it involves a lot of creating new Tokio runtimes and seem to be too heavy to be run on every call:
use std::os::raw::c_char;
use std::ffi::{CString, CStr};
use sn_api::{BootstrapConfig, Safe};
use tokio::runtime::Runtime;
#[no_mangle]
pub extern "C" _safe_connect(ptr: *const Safe, bootstrap_contact: *const c_char) {
assert!(!ptr.is_null());
let _safe = unsafe {
&*ptr
};
let bootstrap_contact = unsafe {
CStr::from_ptr(bootstrap_contact)
}
let mut bootstrap_contacts = BootstrapConfig::default();
bootstrap_contacts.insert(bootstrap_contact.parse().expect("Invalid bootstrap address"));
// how to reuse the Runtime in other functions?
Runtime::new().unwrap().block_on(_safe.connect(None, None, Some(bootstrap_contacts)));
}
Is it possible to run all async functions on a common Runtime? I imagine it would require creating some singleton / global, but my library is compiled with crate-type = ["cdylib"], which seems not a good place for globals. What would be the best approach?
I've decided for an approach, where I create a Tokio Runtime, and then pass it to every FFI function call containing async code:
use std::os::raw::c_char;
use std::ffi::{CString, CStr};
use sn_api::{BootstrapConfig, Safe};
use tokio::runtime::Runtime;
#[no_mangle]
pub extern "C" fn init_runtime() -> *mut Runtime {
Box::into_raw(Box::new(Runtime::new().unwrap()))
}
#[no_mangle]
pub extern "C" _safe_connect(rt_ptr: *mut Runtime, safe_ptr: *mut Safe, bootstrap_contact: *const c_char) {
assert!(!safe_ptr.is_null());
assert!(!rt_ptr.is_null());
let bootstrap_contact = unsafe {
CStr::from_ptr(bootstrap_contact)
}
let mut bootstrap_contacts = BootstrapConfig::default();
bootstrap_contacts.insert(bootstrap_contact.parse().expect("Invalid bootstrap address"));
unsafe {
let _safe = &mut *safe_ptr;
let rt = &mut *rt_ptr;
rt.block_on(_safe.connect(None, None, Some(bootstrap_contacts))).unwrap();
}
}
I faced the same issue. Here is my cut: export-tokio-to-lib.
plugin.rs:
use async_ffi::{FfiFuture, FutureExt};
use tokio::runtime::Handle;
/// # Safety
#[no_mangle]
pub unsafe extern "C" fn test(arg: f32, handle: *const Handle) -> FfiFuture<safer_ffi::String> {
let handle = &*handle;
async move {
let _enter = handle.enter();
tokio::time::sleep(std::time::Duration::from_secs_f32(arg)).await;
format!("slept {arg} secs").into()
}
.into_ffi()
}
Try this.
From this:
#[tokio::main]
async fn main() {
println!("hello");
}
Transformed into:
fn main() {
let mut rt = tokio::runtime::Runtime::new().unwrap();
rt.block_on(async {
println!("hello");
})
}
Reference: https://tokio.rs/tokio/tutorial/hello-tokio#async-main-function
I try to create a simple application parsing command line arguments using clap library and converting them to a Config custom structure. I implemented From trait for my structure, however, when I try to call from function, I receive the following error:
the trait bound `minimal_example::Config: std::convert::From<cli::Opts>` is not satisfied
the following implementations were found:
<minimal_example::Config as std::convert::From<minimal_example::cli::Opts>>
required by `std::convert::From::from`
Here is the code:
main.rs:
mod cli;
use clap::Clap;
use minimal_example::Config;
fn main() {
println!("Hello, world!");
let opts = cli::Opts::parse();
let config = Config::from(opts);
}
cli.rs:
use clap::{Clap, crate_version};
/// This doc string acts as a help message when the user runs '--help'
/// as do all doc strings on fields
#[derive(Clap)]
#[clap(version = crate_version!(), author = "Yury")]
pub struct Opts {
/// Simple option
pub opt: String,
}
lib.rs:
mod cli;
pub struct Config {
pub opt: String,
}
impl From<cli::Opts> for Config {
fn from(opts: cli::Opts) -> Self {
Config {
opt: opts.opt,
}
}
}
cargo.toml:
[package]
name = "minimal_example"
version = "0.1.0"
authors = ["Yury"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
clap = {version="3.0.0-beta.2", features=["wrap_help"]}
What am I doing wrong?
You have added mod cli to both lib.rs and main.rs.
They are different from the standpoint of each other.
Rust modules confusion when there is main.rs and lib.rs
may help in understanding that.
That's what the error says. It's satisfied for std::convert::From<minimal_example::cli::Opts> but not for std::convert::From<cli::Opts>.
A simple fix:
main.rs
mod cli;
use clap::Clap;
use minimal_example::Config;
impl From<cli::Opts> for Config {
fn from(opts: cli::Opts) -> Self {
Config {
opt: opts.opt,
}
}
}
fn main() {
println!("Hello, world!");
let opts = cli::Opts::parse();
let config = Config::from(opts);
}
Now std::convert::From<cli::Opts> is implemented for Config.
How you actually want to place all this depends on your package architecture.
I have an object that I know that is inside an Arc because all the instances are always Arced. I would like to be able to pass a cloned Arc of myself in a function call. The thing I am calling will call me back later on other threads.
In C++, there is a standard mixin called enable_shared_from_this. It enables me to do exactly this
class Bus : public std::enable_shared_from_this<Bus>
{
....
void SetupDevice(Device device,...)
{
device->Attach(shared_from_this());
}
}
If this object is not under shared_ptr management (the closest C++ has to Arc) then this will fail at run time.
I cannot find an equivalent.
EDIT:
Here is an example of why its needed. I have a timerqueue library. It allows a client to request an arbitrary closure to be run at some point in the future. The code is run on a dedicated thread. To use it you must pass a closure of the function you want to be executed later.
use std::time::{Duration, Instant};
use timerqueue::*;
use parking_lot::Mutex;
use std::sync::{Arc,Weak};
use std::ops::{DerefMut};
// inline me keeper cos not on github
pub struct MeKeeper<T> {
them: Mutex<Weak<T>>,
}
impl<T> MeKeeper<T> {
pub fn new() -> Self {
Self {
them: Mutex::new(Weak::new()),
}
}
pub fn save(&self, arc: &Arc<T>) {
*self.them.lock().deref_mut() = Arc::downgrade(arc);
}
pub fn get(&self) -> Arc<T> {
match self.them.lock().upgrade() {
Some(arc) => return arc,
None => unreachable!(),
}
}
}
// -----------------------------------
struct Test {
data:String,
me: MeKeeper<Self>,
}
impl Test {
pub fn new() -> Arc<Test>{
let arc = Arc::new(Self {
me: MeKeeper::new(),
data: "Yo".to_string()
});
arc.me.save(&arc);
arc
}
fn task(&self) {
println!("{}", self.data);
}
// in real use case the TQ and a ton of other status data is passed in the new call for Test
// to keep things simple here the 'container' passes tq as an arg
pub fn do_stuff(&self, tq: &TimerQueue) {
// stuff includes a async task that must be done in 1 second
//.....
let me = self.me.get().clone();
tq.queue(
Box::new(move || me.task()),
"x".to_string(),
Instant::now() + Duration::from_millis(1000),
);
}
}
fn main() {
// in real case (PDP11 emulator) there is a Bus class owning tons of objects thats
// alive for the whole duration
let tq = Arc::new(TimerQueue::new());
let test = Test::new();
test.do_stuff(&*tq);
// just to keep everything alive while we wait
let mut input = String::new();
std::io::stdin().read_line(&mut input).unwrap();
}
cargo toml
[package]
name = "tqclient"
version = "0.1.0"
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
timerqueue = { git = "https://github.com/pm100/timerqueue.git" }
parking_lot = "0.11"
There is no way to go from a &self to the Arc that self is stored in. This is because:
Rust references have additional assumptions compared to C++ references that would make such a conversion undefined behavior.
Rust's implementation of Arc does not even expose the information necessary to determine whether self is stored in an Arc or not.
Luckily, there is an alternative approach. Instead of creating a &self to the value inside the Arc, and passing that to the method, pass the Arc directly to the method that needs to access it. You can do that like this:
use std::sync::Arc;
struct Shared {
field: String,
}
impl Shared {
fn print_field(self: Arc<Self>) {
let clone: Arc<Shared> = self.clone();
println!("{}", clone.field);
}
}
Then the print_field function can only be called on an Shared encapsulated in an Arc.
having found that I needed this three times in recent days I decided to stop trying to come up with other designs. Maybe poor data design as far as rust is concerned but I needed it.
Works by changing the new function of the types using it to return an Arc rather than a raw self. All my objects are arced anyway, before they were arced by the caller, now its forced.
mini util library called mekeeper
use parking_lot::Mutex;
use std::sync::{Arc,Weak};
use std::ops::{DerefMut};
pub struct MeKeeper<T> {
them: Mutex<Weak<T>>,
}
impl<T> MeKeeper<T> {
pub fn new() -> Self {
Self {
them: Mutex::new(Weak::new()),
}
}
pub fn save(&self, arc: &Arc<T>) {
*self.them.lock().deref_mut() = Arc::downgrade(arc);
}
pub fn get(&self) -> Arc<T> {
match self.them.lock().upgrade() {
Some(arc) => return arc,
None => unreachable!(),
}
}
}
to use it
pub struct Test {
me: MeKeeper<Self>,
foo:i8,
}
impl Test {
pub fn new() -> Arc<Self> {
let arc = Arc::new(Test {
me: MeKeeper::new(),
foo:42
});
arc.me.save(&arc);
arc
}
}
now when an instance of Test wants to call a function that requires it to pass in an Arc it does:
fn nargle(){
let me = me.get();
Ooddle::fertang(me,42);// fertang needs an Arc<T>
}
the weak use is what the shared_from_this does so as to prevent refcount deadlocks, I stole that idea.
The unreachable path is safe because the only place that can call MeKeeper::get is the instance of T (Test here) that owns it and that call can only happen if the T instance is alive. Hence no none return from weak::upgrade