Can't get async closure to work with Warp::Filter - rust

I am trying to get an async closure working in the and_then filter from Warp.
This is the smallest example I could come up with where I am reasonably sure I didn't leave any important details out:
use std::{convert::Infallible, sync::Arc, thread, time};
use tokio::sync::RwLock;
use warp::Filter;
fn main() {
let man = Manifest::new();
let check = warp::path("updates").and_then(|| async move { GetAvailableBinaries(&man).await });
}
async fn GetAvailableBinaries(man: &Manifest) -> Result<impl warp::Reply, Infallible> {
Ok(warp::reply::json(&man.GetAvailableBinaries().await))
}
pub struct Manifest {
binaries: Arc<RwLock<Vec<i32>>>,
}
impl Manifest {
pub fn new() -> Manifest {
let bins = Arc::new(RwLock::new(Vec::new()));
thread::spawn(move || async move {
loop {
thread::sleep(time::Duration::from_millis(10000));
}
});
Manifest { binaries: bins }
}
pub async fn GetAvailableBinaries(&self) -> Vec<i32> {
self.binaries.read().await.to_vec()
}
}
I am using:
[dependencies]
tokio = { version = "0.2", features = ["full"] }
warp = { version = "0.2", features = ["tls"] }
The error is:
error[E0525]: expected a closure that implements the `Fn` trait, but this closure only implements `FnOnce`
--> src/main.rs:9:48
|
9 | let check = warp::path("updates").and_then(|| async move { GetAvailableBinaries(&man).await });
| -------- ^^^^^^^^^^^^^ ------------------------------------ closure is `FnOnce` because it moves the variable `man` out of its environment
| | |
| | this closure implements `FnOnce`, not `Fn`
| the requirement to implement `Fn` derives from here

After making Manifest implement Clone, you can fix the error by balancing when the manifest object is cloned:
fn main() {
let man = Manifest::new();
let check = warp::path("updates").and_then(move || {
let man = man.clone();
async move { get_available_binaries(&man).await }
});
warp::serve(check);
}
This moves man into the closure passed to and_then, then provides a clone of man to the async block each time the closure is executed. The async block then owns that data and can take a reference to it without worrying about executing the future after the data has been deallocated.

I'm not sure this is what you're going for, but this solution builds for me:
use std::{convert::Infallible, sync::Arc, thread, time};
use tokio::sync::RwLock;
use warp::Filter;
fn main() {
let man = Manifest::new();
let check = warp::path("updates").and_then(|| async { GetAvailableBinaries(&man).await });
}
async fn GetAvailableBinaries(man: &Manifest) -> Result<impl warp::Reply, Infallible> {
Ok(warp::reply::json(&man.GetAvailableBinaries().await))
}
#[derive(Clone)]
pub struct Manifest {
binaries: Arc<RwLock<Vec<i32>>>,
}
impl Manifest {
pub fn new() -> Manifest {
let bins = Arc::new(RwLock::new(Vec::new()));
thread::spawn(move || async {
loop {
thread::sleep(time::Duration::from_millis(10000));
//mutate bins here
}
});
Manifest { binaries: bins }
}
pub async fn GetAvailableBinaries(&self) -> Vec<i32> {
self.binaries.read().await.to_vec()
}
}
The move here is the reason the compiler gave a warning regarding the signature: let check = warp::path("updates").and_then(|| async move { GetAvailableBinaries(&man).await });. This means that everything referenced in this closure will be moved into the context of the closure. In this case, the compiler can't guarantee the closure to be Fn but only FnOnce meaning that the closure can only be guaranteed to execute once.

Related

still trying to reimplement a c++ timerqueue in rust

more on reimplementing c++ timerqueue in rust
This code tries to create a vec of tasks and run them on a different thread. Still cannot compile it. I am at the point of typing words at random into random locations to see if it helps , 'static here, move there dyn sync ,....
I think if I were to see a working version I would be able to work out why I was stuck. But I cannot make it go.
heres the complete code
use std::thread;
use std::time::Instant;
fn main() {
let x = || {
println!("hello");
};
let y = || {
println!("hello2");
};
let mut tq = TimerQueue::new();
tq.set(Box::new(x), String::from("yo"), Instant::now());
tq.set(Box::new(y), String::from("yo"), Instant::now());
tq.thr();
}
pub struct TimerQueueItem {
when: Instant,
name: String,
what: QIFunc,
}
type QIFunc = Box<dyn Fn() -> () + Send>;
struct TimerQueue {
queue: Vec<TimerQueueItem>,
}
impl TimerQueue {
fn thr(&mut self) -> () {
thread::spawn(move || {
self.runq();
});
}
fn set(&mut self, f: QIFunc, n: String, when: Instant) {
let qi = TimerQueueItem {
what: f,
name: n,
when: when,
};
self.queue.push(qi);
}
fn new() -> TimerQueue {
TimerQueue { queue: Vec::new() }
}
fn runq(&mut self) -> () {
for qi in self.queue.iter() {
(qi.what)();
}
}
}
all sorts of errors all pointing at the same place
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src\main.rs:31:23
|
31 | thread::spawn(move || {
| _______________________^
32 | | self.runq();
33 | | });
| |_________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 30:5...
--> src\main.rs:30:5
this one seems particularly designed to confuse
= note: expected `&mut TimerQueue`
found `&mut TimerQueue`
Your TimerQueue object is stored on the stack of the main thread. Because Rust cannot guarantee that the new thread will finish before the main one, it has imposed some restrictions on how you can pull off recreating your timerqueue.
I actually found a simple fix. Basically, I removed the thr method and moved its body into main.
use std::thread;
use std::time::Instant;
fn main() {
let x = || {
println!("hello");
};
let y = || {
println!("hello2");
};
let mut tq = TimerQueue::new();
tq.set(Box::new(x), String::from("yo"), Instant::now());
tq.set(Box::new(y), String::from("yo"), Instant::now());
thread::spawn(move || {
tq.runq();
});
thread::sleep_ms(1000);
}
pub struct TimerQueueItem {
when: Instant,
name: String,
what: QIFunc,
}
type QIFunc = Box<dyn Fn() -> () + Send>;
struct TimerQueue {
queue: Vec<TimerQueueItem>,
}
impl TimerQueue {
fn set(&mut self, f: QIFunc, n: String, when: Instant) {
let qi = TimerQueueItem {
what: f,
name: n,
when: when,
};
self.queue.push(qi);
}
fn new() -> TimerQueue {
TimerQueue { queue: Vec::new() }
}
fn runq(&mut self) -> () {
for qi in self.queue.iter() {
(qi.what)();
}
}
}
Note: I added the
thread::sleep_ms(1000);
which ensures that the main thread will last as long as the spawned thread - at least in this example, other scenarios might have the new thread require more time. If you want to pause the main thread until the spawned one completes (regardless of how long the spawned one takes), you can replace
thread::spawn(move || {
tq.runq();
});
thread::sleep_ms(1000);
with
let thread_handle = thread::spawn(move || {
tq.runq();
});
thread_handle.join();
For reference, my cargo version is 1.41.0
Instead of thr(&mut self), write thr(mut self).
(mut because runq wants a mutable reference. It might not need to though.)
You can't transfer ownership of the TimerQueue to the new thread with just a reference. When you "move" something into a closure, you transfer ownership there. On the other hand, if you didn't move (i.e. left the move keyword out), you couldn't ensure that self would live long enough.
A C++ analogy to this would be
// BAD: v may go out of scope in the main
// thread before the new thread completes.
// Also facilitates data races.
std::vector<...> v;
std::thread([&v] () {
// use/modify v
});
// BETTER: v is moved to be "owned" by the
// closure, so it will certainly live as long as the
// thread is alive.
std::vector<...> v;
std::thread([v=std::move(v)] () mutable {
// use/modify v
})
There's really no such thing as a member function that takes this as moved in C++, but there is in Rust. You can think of it as a static class method that looks like this instead:
static void thr(TimerQueue &&tq);
// called as
TimerQueue::thr(std::move(tq));
Of course, that's not identical to the Rust version in many ways, but hopefully this helps you understand what's going on.
That error message is certainly unhelpful. Not sure what's going on there.

How do I use PickleDB with Rocket/Juniper Context?

I'm trying to write a Rocket / Juniper / Rust based GraphQL Server using PickleDB - an in-memory key/value store.
The pickle db is created / loaded at the start and given to rocket to manage:
fn rocket() -> Rocket {
let pickle_path = var_os(String::from("PICKLE_PATH")).unwrap_or(OsString::from("pickle.db"));
let pickle_db_dump_policy = PickleDbDumpPolicy::PeriodicDump(Duration::from_secs(120));
let pickle_serialization_method = SerializationMethod::Bin;
let pickle_db: PickleDb = match Path::new(&pickle_path).exists() {
false => PickleDb::new(pickle_path, pickle_db_dump_policy, pickle_serialization_method),
true => PickleDb::load(pickle_path, pickle_db_dump_policy, pickle_serialization_method).unwrap(),
};
rocket::ignite()
.manage(Schema::new(Query, Mutation))
.manage(pickle_db)
.mount(
"/",
routes![graphiql, get_graphql_handler, post_graphql_handler],
)
}
And I want to retrieve the PickleDb instance from the Rocket State in my Guard:
pub struct Context {
pickle_db: PickleDb,
}
impl juniper::Context for Context {}
impl<'a, 'r> FromRequest<'a, 'r> for Context {
type Error = ();
fn from_request(_request: &'a Request<'r>) -> request::Outcome<Context, ()> {
let pickle_db = _request.guard::<State<PickleDb>>()?.inner();
Outcome::Success(Context { pickle_db })
}
}
This does not work because the State only gives me a reference:
26 | Outcome::Success(Context { pickle_db })
| ^^^^^^^^^ expected struct `pickledb::pickledb::PickleDb`, found `&pickledb::pickledb::PickleDb`
When I change my Context struct to contain a reference I get lifetime issues which I'm not yet familiar with:
15 | pickle_db: &PickleDb,
| ^ expected named lifetime parameter
I tried using 'static which does make rust quite unhappy and I tried to use the request lifetime (?) 'r of the FromRequest, but that does not really work either...
How do I get this to work? As I'm quite new in rust, is this the right way to do things?
I finally have a solution, although the need for unsafe indicates it is sub-optimal :)
#![allow(unsafe_code)]
use pickledb::{PickleDb, PickleDbDumpPolicy, SerializationMethod};
use serde::de::DeserializeOwned;
use serde::Serialize;
use std::env;
use std::path::Path;
use std::time::Duration;
pub static mut PICKLE_DB: Option<PickleDb> = None;
pub fn cache_init() {
let pickle_path = env::var(String::from("PICKLE_PATH")).unwrap_or(String::from("pickle.db"));
let pickle_db_dump_policy = PickleDbDumpPolicy::PeriodicDump(Duration::from_secs(120));
let pickle_serialization_method = SerializationMethod::Json;
let pickle_db = match Path::new(&pickle_path).exists() {
false => PickleDb::new(
pickle_path,
pickle_db_dump_policy,
pickle_serialization_method,
),
true => PickleDb::load(
pickle_path,
pickle_db_dump_policy,
pickle_serialization_method,
)
.unwrap(),
};
unsafe {
PICKLE_DB = Some(pickle_db);
}
}
pub fn cache_get<V>(key: &str) -> Option<V>
where
V: DeserializeOwned + std::fmt::Debug,
{
unsafe {
let pickle_db = PICKLE_DB
.as_ref()
.expect("cache uninitialized - call cache_init()");
pickle_db.get::<V>(key)
}
}
pub fn cache_set<V>(key: &str, value: &V) -> Result<(), pickledb::error::Error>
where
V: Serialize,
{
unsafe {
let pickle_db = PICKLE_DB
.as_mut()
.expect("cache uninitialized - call cache_init()");
pickle_db.set::<V>(key, value)?;
Ok(())
}
}
This can be simply imported and used as expected, but I think I'll run into issues when the load gets to high...

How to write an asynchronous recursive walkdir function with an asynchronous callback

I'm trying to write an async function that will traverse the filesystem tree, recursively, and calls an asynchronous callback for each file found.
This is for a learning effort, I have no real use case.
Here is what I have so far:
use async_std::{
fs::{self, *},
path::*,
prelude::*,
}; // 1.5.0, features = ["unstable"]
use futures::{
executor::block_on,
future::{BoxFuture, FutureExt},
}; // 0.3.4
use std::{marker::Sync, pin::Pin};
fn main() {
fn walkdir<F>(path: String, cb: &'static F) -> BoxFuture<'static, ()>
where
F: Fn(&DirEntry) -> BoxFuture<()> + Sync + Send,
{
async move {
let mut entries = fs::read_dir(&path).await.unwrap();
while let Some(path) = entries.next().await {
let entry = path.unwrap();
let path = entry.path().to_str().unwrap().to_string();
if entry.path().is_file().await {
cb(&entry).await
} else {
walkdir(path, cb).await
}
}
}
.boxed()
}
let foo = async {
walkdir(".".to_string(), &|entry: &DirEntry| async {
async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
})
.await
};
block_on(foo);
}
I get this far by some sort of trial and error, but now I'm stuck on async closure callback with this error
warning: unused import: `path::*`
--> src/main.rs:3:5
|
3 | path::*,
| ^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused import: `pin::Pin`
--> src/main.rs:10:25
|
10 | use std::{marker::Sync, pin::Pin};
| ^^^^^^^^
error[E0308]: mismatched types
--> src/main.rs:33:54
|
33 | walkdir(".".to_string(), &|entry: &DirEntry| async {
| ______________________________________________________^
34 | | async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
35 | | })
| |_________^ expected struct `std::pin::Pin`, found opaque type
|
= note: expected struct `std::pin::Pin<std::boxed::Box<dyn core::future::future::Future<Output = ()> + std::marker::Send>>`
found opaque type `impl core::future::future::Future`
use async_std::{
fs::{self, *},
path::*,
prelude::*,
}; // 1.5.0
use futures::{future::{Future, FutureExt, LocalBoxFuture}, executor}; // 0.3.4
fn main() {
async fn walkdir<R>(path: impl AsRef<Path>, mut cb: impl FnMut(DirEntry) -> R)
where
R: Future<Output = ()>,
{
fn walkdir_inner<'a, R>(path: &'a Path, cb: &'a mut dyn FnMut(DirEntry) -> R) -> LocalBoxFuture<'a, ()>
where
R: Future<Output = ()>,
{
async move {
let mut entries = fs::read_dir(path).await.unwrap();
while let Some(path) = entries.next().await {
let entry = path.unwrap();
let path = entry.path();
if path.is_file().await {
cb(entry).await
} else {
walkdir_inner(&path, cb).await
}
}
}.boxed_local()
}
walkdir_inner(path.as_ref(), &mut cb).await
}
executor::block_on({
walkdir(".", |entry| async move {
async_std::println!(">> {}", entry.path().display()).await
})
});
}
Notable changes:
Take in AsRef<Path> instead of a String and a generic closure instead of a trait object reference
Change the closure type to be FnMut as it's more permissive
The closure returns any type that is a future.
There's an inner implementation function that hides the ugly API required for recursive async functions.
The callback takes the DirEntry by value instead of by reference.
See also:
How to asynchronously explore a directory and its sub-directories?
How to using async fn callback in rust

How to use wirefilter over an infinite stream of data

I am writing a program to use wirefilter in order to filter data from an infinite stream.
But it seems that I cannot use a compiled ast in a loop because of lifetimes and when I try to compile, this is the output:
error: borrowed data cannot be stored outside of its closure
--> src/main.rs:34:33
|
31 | let filter = ast.compile();
| ------ ...so that variable is valid at time of its declaration
32 |
33 | for my_struct in data.filter(|my_struct| {
| ----------- borrowed data cannot outlive this closure
34 | let execution_context = my_struct.execution_context();
| ^^^^^^^^^ ----------------- cannot infer an appropriate lifetime...
| |
| cannot be stored outside of its closure
error: aborting due to previous error
error: Could not compile `wirefilter_playground`.
To learn more, run the command again with --verbose.
main.rs
use wirefilter::{ExecutionContext, Scheme};
lazy_static::lazy_static! {
static ref SCHEME: Scheme = Scheme! {
port: Int
};
}
#[derive(Debug)]
struct MyStruct {
port: i32,
}
impl MyStruct {
fn scheme() -> &'static Scheme {
&SCHEME
}
fn execution_context(&self) -> ExecutionContext {
let mut ctx = ExecutionContext::new(Self::scheme());
ctx.set_field_value("port", self.port).unwrap();
ctx
}
}
fn main() -> Result<(), failure::Error> {
let data = expensive_data_iterator();
let scheme = MyStruct::scheme();
let ast = scheme.parse("port in {2 5}")?;
let filter = ast.compile();
for my_struct in data.filter(|my_struct| {
let execution_context = my_struct.execution_context();
filter.execute(&execution_context).unwrap()
}).take(10) {
println!("{:?}", my_struct);
}
Ok(())
}
fn expensive_data_iterator() -> impl Iterator<Item=MyStruct> {
(0..).map(|port| MyStruct { port })
}
Cargo.toml
[package]
name = "wirefilter_playground"
version = "0.1.0"
edition = "2018"
[dependencies]
wirefilter-engine = "0.6.1"
failure = "0.1.5"
lazy_static = "1.3.0"
is it possible to make it work? I would like to yield only the filtered data for the final user otherwise the amount of data would be huge in memory.
Thank you in advance!
It looks like the problem is with the lifetime elision in return structs. In particular this code:
fn execution_context(&self) -> ExecutionContext {
//...
}
is equivalent to this one:
fn execution_context<'s>(&'s self) -> ExecutionContext<'s> {
//...
}
Which becomes obvious once you realize that ExecutionContext has an associated lifetime.
The lifetime of ExecutionContext does not have to match that of the MyStruct so you probably want to write:
fn execution_context<'e>(&self) -> ExecutionContext<'e> {
//...
}
or maybe:
fn execution_context<'s, 'e>(&'s self) -> ExecutionContext<'e>
where 'e: 's {
//...
}
depending on whether your context will eventually refer to any content of MyStruct.

Closure may outlive the current function

I am just starting to learn Rust. For this purpose I am rewriting my C++ project in Rust, but the biggest problems are lifetimes of closures and such.
I created a absolute minimal scenario of my problem seen here and below:
use std::sync::Arc;
use std::cell::{RefCell, Cell};
struct Context {
handler: RefCell<Option<Arc<Handler>>>,
}
impl Context {
pub fn new() -> Arc<Context> {
let context = Arc::new(Context{
handler: RefCell::new(None),
});
let handler = Handler::new(context.clone());
(*context.handler.borrow_mut()) = Some(handler);
context
}
pub fn get_handler(&self) -> Arc<Handler> {
self.handler.borrow().as_ref().unwrap().clone()
}
}
struct Handler {
context: Arc<Context>,
clickables: RefCell<Vec<Arc<Clickable>>>,
}
impl Handler {
pub fn new(context: Arc<Context>) -> Arc<Handler> {
Arc::new(Handler{
context: context,
clickables: RefCell::new(Vec::new()),
})
}
pub fn add_clickable(&self, clickable: Arc<Clickable>) {
self.clickables.borrow_mut().push(clickable);
}
pub fn remove_clickable(&self, clickable: Arc<Clickable>) {
// remove stuff ...
}
}
struct Clickable {
context: Arc<Context>,
callback: RefCell<Option<Box<Fn()>>>,
}
impl Clickable {
pub fn new(context: Arc<Context>) -> Arc<Clickable> {
let clickable = Arc::new(Clickable{
context: context.clone(),
callback: RefCell::new(None),
});
context.get_handler().add_clickable(clickable.clone());
clickable
}
pub fn remove(clickable: Arc<Clickable>) {
clickable.context.get_handler().remove_clickable(clickable);
}
pub fn set_callback(&self, callback: Option<Box<Fn()>>) {
(*self.callback.borrow_mut()) = callback;
}
pub fn click(&self) {
match *self.callback.borrow() {
Some(ref callback) => (callback)(),
None => (),
}
}
}
struct Button {
context: Arc<Context>,
clickable: Arc<Clickable>,
}
impl Button {
pub fn new(context: Arc<Context>) -> Arc<Button> {
let clickable = Clickable::new(context.clone());
let button = Arc::new(Button{
context: context,
clickable: clickable.clone(),
});
let tmp_callback = Box::new(|| {
button.do_stuff();
});
clickable.set_callback(Some(tmp_callback));
button
}
pub fn do_stuff(&self) {
// doing crazy stuff
let mut i = 0;
for j in 0..100 {
i = j*i;
}
}
pub fn click(&self) {
self.clickable.click();
}
}
impl Drop for Button {
fn drop(&mut self) {
Clickable::remove(self.clickable.clone());
}
}
fn main() {
let context = Context::new();
let button = Button::new(context.clone());
button.click();
}
I just don't know how to pass references in closures.
Another ugly thing is that my Handler and my Context need each other. Is there a nicer way to to create this dependency?
Going off your initial code
pub fn new(context: Arc<Context>) -> Arc<Button> {
let clickable = Clickable::new(context.clone());
let button = Arc::new(Button{
context: context,
clickable: clickable.clone(),
});
let tmp_callback = Box::new(|| {
button.do_stuff();
});
clickable.set_callback(Some(tmp_callback));
button
}
First off, let's note the error you're getting
error[E0373]: closure may outlive the current function, but it borrows `button`, which is owned by the current function
--> src/main.rs:101:37
|
101 | let tmp_callback = Box::new(|| {
| ^^ may outlive borrowed value `button`
102 | button.do_stuff();
| ------ `button` is borrowed here
|
help: to force the closure to take ownership of `button` (and any other referenced variables), use the `move` keyword, as shown:
| let tmp_callback = Box::new(move || {
Noting the help block at the bottom, you need to use a move closure, because when the new function ends, the button variable on the stack will go out of scope. The only way to avoid that is to move ownership of it to the callback itself. Thus you'd change
let tmp_callback = Box::new(|| {
to
let tmp_callback = Box::new(move || {
Now, you'd get a second error:
error[E0382]: use of moved value: `button`
--> src/main.rs:107:9
|
102 | let tmp_callback = Box::new(move || {
| ------- value moved (into closure) here
...
107 | button
| ^^^^^^ value used here after move
|
= note: move occurs because `button` has type `std::sync::Arc<Button>`, which does not implement the `Copy` trait
And the error here may be a little clearer. You're trying to move ownership of the button value into the callback closure, but you also use it inside the body of the new function when you return it, and you can't have two different things trying to own the value.
The solution to that is hopefully what you'd guess. You have to make a copy that you can take ownership of. You'll want to then change
let tmp_callback = Box::new(move || {
button.do_stuff();
to
let button_clone = button.clone();
let tmp_callback = Box::new(move || {
button_clone.do_stuff();
Now you've created a new Button object, and returned an Arc for the object itself, while also giving ownership of a second Arc to the callback itself.
Update
Given your comment, there is indeed an issue here of cyclic dependencies, since your Clickable object holds ownership of a reference to Button, while Button holds ownership of a reference to Clickable. The easiest way to fix this here would be to update that code a third time, from
let button_clone = button.clone();
let tmp_callback = Box::new(move || {
button_clone.do_stuff();
to
let button_weak = Arc::downgrade(&button);
let tmp_callback = Box::new(move || {
if let Some(button) = button_weak.upgrade() {
button.do_stuff();
}
});
so the Clickable will only hold a weak reference to the Button, and if the Button is no longer referenced, the callback will be a no-op.
You'd also probably want to consider making clickables a list of Weak references instead of strong references, so you can remove items from it when the item they reference is removed.

Resources