I am trying to implement a cache in Rust using DelayQueue based on an this example provided in the tokio_util documentation. This is my code:
# Cargo.toml
[dependencies]
futures = "0.3"
tokio-util = { version = "0.7.3", features = ["full"] }
tokio = { version = "1.19.2", features = ["full"] }
use tokio_util::time::{delay_queue, DelayQueue};
use futures::ready;
use std::collections::HashMap;
use std::task::{Context, Poll};
use std::time::Duration;
type CacheKey = String;
type Value = String;
struct Cache {
entries: HashMap<CacheKey, (Value, delay_queue::Key)>,
expirations: DelayQueue<CacheKey>,
}
const TTL_SECS: u64 = 10;
impl Cache {
fn new() -> Cache {
Cache {
entries: HashMap::new(),
expirations: DelayQueue::new(),
}
}
fn insert(&mut self, key: CacheKey, value: Value) {
let delay = self
.expirations
.insert(key.clone(), Duration::from_secs(TTL_SECS));
self.entries.insert(key, (value, delay));
}
fn _get(&self, key: &CacheKey) -> Option<&Value> {
self.entries.get(key).map(|&(ref v, _)| v)
}
fn remove(&mut self, key: &CacheKey) {
if let Some((_, cache_key)) = self.entries.remove(key) {
self.expirations.remove(&cache_key);
}
}
fn poll_purge(&mut self, cx: &mut Context<'_>) -> Poll<()> {
while let Some(entry) = ready!(self.expirations.poll_expired(cx)) {
self.entries.remove(entry.get_ref());
}
Poll::Ready(())
}
}
#[tokio::main]
async fn main() {
let mut cache = Cache::new();
cache.insert("k1".to_string(), "v1".to_string());
cache.insert("k2".to_string(), "v2".to_string());
cache.insert("k3".to_string(), "v3".to_string());
cache.insert("k4".to_string(), "v4".to_string());
cache.insert("k5".to_string(), "v5".to_string());
// code for consuming from the queue
}
I want to call poll_purge() method on my cache object but it has a parameter that I don't know what to pass:
cx: &mut Context<'_>
How should I change my code?
You can add an async fn purge to the Cache impl and use poll_fn from the futures crate:
async fn purge(&mut self) {
futures::future::poll_fn(|cx| self.poll_purge(cx)).await;
}
That will create a Future that wraps poll_purge and ensures it runs until completion.
Related
As the title says, we have an application which runs wasm plugins. Each plugin can register their own State in the StateRegistry. Then whenever the plugins are executed, they will modify their respective State.
The code below illustrates this:
use std::{collections::HashMap, any::Any};
type ContractId = String;
type GenericContractState = Box<dyn Any>;
// This will be WASM code
mod foo_contract {
use super::StateRegistry;
pub struct State {
}
pub fn update(states: &mut StateRegistry) {
let state = states.states.get_mut(&"foo_contract".to_string()).unwrap();
let state = state.downcast_mut::<State>().unwrap();
// I'd prefer something like:
// let state = state.lookup::<State>().unwrap()
}
}
pub struct StateRegistry {
pub states: HashMap<ContractId, GenericContractState>,
}
impl StateRegistry {
fn new() -> Self {
Self { states: HashMap::new() }
}
fn register(&mut self, contract_id: ContractId, state: GenericContractState) {
self.states.insert(contract_id, state);
}
/*
fn lookup<'a, S>(&'a mut self, contract_id: &ContractId) -> Option<StateRefWrapper<'a, S>> {
match self.states.get_mut(contract_id) {
Some(state) => {
let ptr = state.downcast_mut::<S>();
match ptr {
Some(ptr) => Some(StateRefWrapper { _mut: state, ptr }),
None => None,
}
}
None => None,
}
}
*/
}
/*
struct StateRefWrapper<'a, S> {
_mut: &'a mut Box<dyn Any>,
ptr: &'a mut S,
}
*/
fn main() {
let mut states = StateRegistry::new();
let foo_state = Box::new(foo_contract::State {});
states.register("foo_contract".to_string(), foo_state);
foo_contract::update(&mut states);
}
The part I want to improve is that currently the plugin developer has to use this code to lookup their State in the registry:
let state = states.states.get_mut(&"foo_contract".to_string()).unwrap();
let state = state.downcast_mut::<State>().unwrap();
I want to create a respective method in StateRegistry that is usable like this:
let state = state.lookup::<State>().unwrap()
My attempt (commented above) was something like this:
impl StateRegistry {
// ...
fn lookup<'a, S>(&'a mut self, contract_id: &ContractId) -> Option<StateRefWrapper<'a, S>> {
match self.states.get_mut(contract_id) {
Some(state) => {
let ptr = state.downcast_mut::<S>();
match ptr {
Some(ptr) => Some(StateRefWrapper { _mut: state, ptr }),
None => None,
}
}
None => None,
}
}
}
struct StateRefWrapper<'a, S> {
_mut: &'a mut Box<dyn Any>,
ptr: &'a mut S,
}
How can I get this lookup() function working?
Thanks
You can't construct your StateRefWrapper in the way you want to, because it would contain two mutable borrows of the same value. You can, however, return just a &mut S which seems sufficient:
fn lookup<'a, S: 'static>(&'a mut self, contract_id: &ContractId) -> Option<&'a mut S> {
self.states
.get_mut(contract_id)
.and_then(|state| state.downcast_mut())
}
And use it like this:
pub fn update(states: &mut StateRegistry) {
let state = states.lookup::<State>(&"foo_contract".to_string()).unwrap();
// state.modify();
}
I'm currently trying to wrap a C library in rust that has a few requirements. The C library can only be run on a single thread, and can only be initialized / cleaned up once on the same thread. I want something something like the following.
extern "C" {
fn init_lib() -> *mut c_void;
fn cleanup_lib(ctx: *mut c_void);
}
// This line doesn't work.
static mut CTX: Option<(ThreadId, Rc<Context>)> = None;
struct Context(*mut c_void);
impl Context {
fn acquire() -> Result<Rc<Context>, Error> {
// If CTX has a reference on the current thread, clone and return it.
// Otherwise initialize the library and set CTX.
}
}
impl Drop for Context {
fn drop(&mut self) {
unsafe { cleanup_lib(self.0); }
}
}
Anyone have a good way to achieve something like this? Every solution I try to come up with involves creating a Mutex / Arc and making the Context type Send and Sync which I don't want as I want it to remain single threaded.
A working solution I came up with was to just implement the reference counting myself, removing the need for Rc entirely.
#![feature(once_cell)]
use std::{error::Error, ffi::c_void, fmt, lazy::SyncLazy, sync::Mutex, thread::ThreadId};
extern "C" {
fn init_lib() -> *mut c_void;
fn cleanup_lib(ctx: *mut c_void);
}
#[derive(Debug)]
pub enum ContextError {
InitOnOtherThread,
}
impl fmt::Display for ContextError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match *self {
ContextError::InitOnOtherThread => {
write!(f, "Context already initialized on a different thread")
}
}
}
}
impl Error for ContextError {}
struct StaticPtr(*mut c_void);
unsafe impl Send for StaticPtr {}
static CTX: SyncLazy<Mutex<Option<(ThreadId, usize, StaticPtr)>>> =
SyncLazy::new(|| Mutex::new(None));
pub struct Context(*mut c_void);
impl Context {
pub fn acquire() -> Result<Context, ContextError> {
let mut ctx = CTX.lock().unwrap();
if let Some((id, ref_count, ptr)) = ctx.as_mut() {
if *id == std::thread::current().id() {
*ref_count += 1;
return Ok(Context(ptr.0));
}
Err(ContextError::InitOnOtherThread)
} else {
let ptr = unsafe { init_lib() };
*ctx = Some((std::thread::current().id(), 1, StaticPtr(ptr)));
Ok(Context(ptr))
}
}
}
impl Drop for Context {
fn drop(&mut self) {
let mut ctx = CTX.lock().unwrap();
let (_, ref_count, ptr) = ctx.as_mut().unwrap();
*ref_count -= 1;
if *ref_count == 0 {
unsafe {
cleanup_lib(ptr.0);
}
*ctx = None;
}
}
}
I think the most 'rustic' way to do this is with std::sync::mpsc::sync_channel and an enum describing library operations.
The only public-facing elements of this module are launch_lib(), the SafeLibRef struct (but not its internals), and the pub fn that are part of the impl SafeLibRef.
Also, this example strongly represents the philosophy that the best way to deal with global state is to not have any.
I have played fast and loose with the Result::unwrap() calls. It would be more responsible to handle error conditions better.
use std::sync::{ atomic::{ AtomicBool, Ordering }, mpsc::{ SyncSender, Receiver, sync_channel } };
use std::ffi::c_void;
extern "C" {
fn init_lib() -> *mut c_void;
fn do_op_1(ctx: *mut c_void, a: u16, b: u32, c: u64) -> f64;
fn do_op_2(ctx: *mut c_void, a: f64) -> bool;
fn cleanup_lib(ctx: *mut c_void);
}
enum LibOperation {
Op1(u16,u32,u64,SyncSender<f64>),
Op2(f64, SyncSender<bool>),
Terminate(SyncSender<()>),
}
#[derive(Clone)]
pub struct SafeLibRef(SyncSender<LibOperation>);
fn lib_thread(rx: Receiver<LibOperation>) {
static LIB_INITIALIZED: AtomicBool = AtomicBool::new(false);
if LIB_INITIALIZED.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst).is_err() {
panic!("Tried to double-initialize library!");
}
let libptr = unsafe { init_lib() };
loop {
let op = rx.recv();
if op.is_err() {
unsafe { cleanup_lib(libptr) };
break;
}
match op.unwrap() {
LibOperation::Op1(a,b,c,tx_res) => {
let res: f64 = unsafe { do_op_1(libptr, a, b, c) };
tx_res.send(res).unwrap();
},
LibOperation::Op2(a, tx_res) => {
let res: bool = unsafe { do_op_2(libptr, a) };
tx_res.send(res).unwrap();
}
LibOperation::Terminate(tx_res) => {
unsafe { cleanup_lib(libptr) };
tx_res.send(()).unwrap();
break;
}
}
}
}
/// This needs to be called no more than once.
/// The resulting SafeLibRef can be cloned and passed around.
pub fn launch_lib() -> SafeLibRef {
let (tx,rx) = sync_channel(0);
std::thread::spawn(|| lib_thread(rx));
SafeLibRef(tx)
}
// This is the interface that most of your code will use
impl SafeLibRef {
pub fn op_1(&self, a: u16, b: u32, c: u64) -> f64 {
let (res_tx, res_rx) = sync_channel(1);
self.0.send(LibOperation::Op1(a, b, c, res_tx)).unwrap();
res_rx.recv().unwrap()
}
pub fn op_2(&self, a: f64) -> bool {
let (res_tx, res_rx) = sync_channel(1);
self.0.send(LibOperation::Op2(a, res_tx)).unwrap();
res_rx.recv().unwrap()
}
pub fn terminate(&self) {
let (res_tx, res_rx) = sync_channel(1);
self.0.send(LibOperation::Terminate(res_tx)).unwrap();
res_rx.recv().unwrap();
}
}
I am writing a server that allocates some compressed data on startup. Now when I serve a hyper response I do not want to copy these bytes, but I cannot figure out a way to do this with hyper.
I have tried implementing HttpBody for my own type, but the lifetime restriction on the trait blocks me from doing this.
Am I missing something? Here is a minimal example of what I am trying to do:
use hyper::{service::Service, Body, Request, Response, Server};
use std::net::SocketAddr;
use std::sync::Arc;
use std::{
future::Future,
pin::Pin,
task::{Context, Poll},
};
fn main() {
let addr = SocketAddr::new("0.0.0.0".parse().unwrap(), 8080);
println!("Server startup...");
let rt = tokio::runtime::Runtime::new().unwrap();
rt.block_on(async {
let app = MakeSvc::new().await;
let ret = app.clone();
let server = Server::bind(&addr).serve(app);
println!("Running on {}", &addr);
server.await.unwrap();
})
}
#[derive(Debug, Clone)]
pub struct WrapperApp {
pub cache: Arc<Vec<u8>>,
}
impl WrapperApp {
//Let's say I allocate some bytes here.
async fn new() -> Self {
Self {
cache: Arc::new(Vec::new()),
}
}
}
impl Service<Request<Body>> for WrapperApp {
type Response = Response<Body>;
type Error = hyper::Error;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, _: &mut Context) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, req: Request<Body>) -> Self::Future {
let a = Arc::clone(&self.cache);
return Box::pin(async { Ok(Response::builder().body(Body::from(a)).unwrap()) });
}
}
#[derive(Debug, Clone)]
pub struct MakeSvc {
app: WrapperApp,
}
impl MakeSvc {
pub async fn new() -> Self {
Self {
app: WrapperApp::new().await,
}
}
}
impl<T> Service<T> for MakeSvc {
type Response = WrapperApp;
type Error = hyper::Error;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, _: &mut Context) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, _: T) -> Self::Future {
let app = self.app.clone();
let fut = async move { Ok(app) };
Box::pin(fut)
}
}
error[E0277]: the trait bound `Body: From<Arc<Vec<u8>>>` is not satisfied
--> src/main.rs:50:61
|
50 | return Box::pin(async { Ok(Response::builder().body(Body::from(a)).unwrap()) });
| ^^^^^^^^^^ the trait `From<Arc<Vec<u8>>>` is not implemented for `Body`
|
= help: the following implementations were found:
<Body as From<&'static [u8]>>
<Body as From<&'static str>>
<Body as From<Box<(dyn futures_core::stream::Stream<Item = std::result::Result<hyper::body::Bytes, Box<(dyn std::error::Error + Send + Sync + 'static)>>> + Send + 'static)>>>
<Body as From<Cow<'static, [u8]>>>
and 4 others
= note: required by `from`
The Cargo.toml that goes with this. The example breaks at the place where I am trying to use the ref:
[package]
name = "hyper-ptr"
version = "0.1.0"
authors = ["Pierre Laas <lanklaas123#gmail.com>"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
hyper={version="0.14", features=["server","http1","tcp"]}
tokio={version="1.0.0", features=["macros","io-util","rt-multi-thread"]}
serde={version="*", features=["derive","rc"]}
serde_json="*"
flate2="*"
openssl="*"
rand="*"
From the documentation, there is no From implementation that doesn't require either 'static references or ownership of the body.
Instead, you can clone the cache. Body::From's implementation requires ownership of the Vec<u8> or static data (if that is useful for your case).
Notice that you need to dereference it first:
use hyper::Body; // 0.14.2
use std::sync::Arc;
const FOO: [u8; 1] = [8u8];
fn main() {
let cache = Arc::new(vec![0u8]);
let body = Body::from((*cache).clone());
let other_body = Body::from(&FOO[..]);
println!("{:?}", cache);
}
Playground
I want to write a vscode extension that displays the content of a large binary file, written with bincode:
#[macro_use]
extern crate serde_derive;
use std::collections::HashMap;
use std::fs::File;
use std::io::{BufReader, BufWriter};
#[derive(Serialize, Deserialize)]
pub struct MyValue {
pub name: String,
}
#[derive(Serialize, Deserialize)]
pub struct MyStruct {
pub data: HashMap<String, MyValue>,
}
impl MyStruct {
pub fn dump(&self, filename: &str) -> Result<(), String> {
let file = File::create(filename).map_err(|msg| msg.to_string())?;
let writer = BufWriter::new(file);
bincode::serialize_into(writer, self).map_err(|msg| msg.to_string())
}
pub fn load(filename: &str) -> Result<Self, String> {
let file = File::open(filename).map_err(|msg| msg.to_string())?;
let reader = BufReader::new(file);
bincode::deserialize_from::<BufReader<_>, Self>(reader).map_err(|msg| msg.to_string())
}
}
Therefore there is a wasm binding:
#[wasm_bindgen]
#[derive(Clone)]
pub struct PyMyStruct {
inner: Arc<MyStruct>,
}
#[wasm_bindgen]
impl PyMyStruct {
pub fn new(filename: &str) -> Self {
Self {
inner: Arc::new(MyStruct::load(filename).unwrap()),
}
}
pub fn header(self) -> Array {
let keys = Array::new();
for key in self.inner.data.keys() {
keys.push(&JsValue::from_str(key));
}
keys
}
pub fn value(&self, name: &str) -> JsValue {
if let Some(value) = self.inner.data.get(name) {
JsValue::from_serde(value).unwrap_or(JsValue::NULL)
} else {
JsValue::NULL
}
}
}
which provides a simple interface to the JavaScript world in order to access the content of that file.
Using Arc in order to prevent expensive unintended memory copy when handling on the JavaScript side.
(It might look strange that keys is not marked as mutable but the rust compiler recomended that way)
When running the test code:
const {PyMyStruct} = require("./corejs.js");
let obj = new PyMyStruct("../../dump.spb")
console.log(obj.header())
you get the error message:
Error: null pointer passed to rust
Does someone know how to handle this use case?
Thank you!
The issue here is that you are using new PyMyStruct() instead of PyMyStruct.new(). In wasm-bindgen's debug mode you will get an error about this at runtime. Using .new() will fix your issue:
let obj = PyMyStruct.new("../../dump.spb")
If you add the #[wasm_bindgen(constructor)] annotation to the new method, then new PyMyStruct() will work as well:
#[wasm_bindgen]
impl PyMyStruct {
#[wasm_bindgen(constructor)]
pub fn new(filename: &str) -> Self {
Self {
inner: 1,
}
}
}
Now this is fine:
let obj = new PyMyStruct("../../dump.spb")
I solved that problem by using https://neon-bindings.com instead of compiling the API to web-assembly.
The binding here looks as follow:
use core;
use std::rc::Rc;
use neon::prelude::*;
#[derive(Clone)]
pub struct MyStruct {
inner: Rc<core::MyStruct>,
}
declare_types! {
pub class JsMyStruct for MyStruct {
init(mut cx) {
let filename = cx.argument::<JsString>(0)?.value();
match core::MyStruct::load(&filename) {
Ok(inner) => return Ok(MyStruct{
inner: Rc::new(inner)
}),
Err(msg) => {
panic!("{}", msg)
}
}
}
method header(mut cx) {
let this = cx.this();
let container = {
let guard = cx.lock();
let this = this.borrow(&guard);
(*this).clone()
};
let keys = container.inner.data.keys().collect::<Vec<_>>();
let js_array = JsArray::new(&mut cx, keys.len() as u32);
for (i, obj) in keys.into_iter().enumerate() {
let js_string = cx.string(obj);
js_array.set(&mut cx, i as u32, js_string).unwrap();
}
Ok(js_array.upcast())
}
method value(mut cx) {
let key = cx.argument::<JsString>(0)?.value();
let this = cx.this();
let container = {
let guard = cx.lock();
let this = this.borrow(&guard);
(*this).clone()
};
if let Some(data) = container.inner.data.get(&key) {
return Ok(neon_serde::to_value(&mut cx, data)?);
} else {
panic!("No value for key \"{}\" available", key);
}
}
}
}
register_module!(mut m, {
m.export_class::<JsMyStruct>("MyStruct")?;
Ok(())
});
I try to use panic::catch_unwind to catch some errors, but it seems no use, and there is an example:
rust:
use std::{sync::Mutex};
use wasm_bindgen::prelude::*;
use std::sync::PoisonError;
use std::panic;
pub struct CurrentStatus {
pub index: i32,
}
#[wasm_bindgen]
extern {
pub fn alert(s: &str);
}
impl CurrentStatus {
fn new() -> Self {
CurrentStatus {
index: 1,
}
}
fn get_index(&mut self) -> i32 {
self.index += 1;
self.index.clone()
}
fn add_index(&mut self) {
self.index += 2;
}
}
lazy_static! {
pub static ref FOO: Mutex<CurrentStatus> = Mutex::new(CurrentStatus::new());
}
unsafe impl Send for CurrentStatus {}
#[wasm_bindgen]
pub fn add_index() {
FOO.lock().unwrap_or_else(PoisonError::into_inner).add_index();
}
#[wasm_bindgen]
pub fn get_index() -> i32 {
let mut foo = FOO.lock().unwrap_or_else(PoisonError::into_inner);
let result = panic::catch_unwind(|| {
panic!("error happen!"); // original panic! code
});
if result.is_err() {
alert("panic!!!!!panic");
}
return foo.get_index();
}
js:
const js = import("../pkg/hello_wasm.js");
js.then(js => {
window.js = js;
console.log(js.get_index());
js.add_index();
});
I think it should catch the panic and I can call add_index then.
But In fact I can call neither after the panic.
I wish I can catch the panic from one function so when the users call other functions just all right..
Thanks very much