I'm trying to write a couple of recursive async functions in Rust. I've created a minimal example of the problem I'm facing:
use std::collections::HashMap;
use anyhow::Result;
use async_recursion::async_recursion;
use futures::StreamExt;
#[async_recursion]
async fn foo() -> Result<()> {
let map: HashMap<String, String> = HashMap::new();
futures::stream::iter(map.keys().map(|it: &String| tokio::spawn(bar(it.clone()))))
.buffer_unordered(2)
.collect::<Vec<_>>()
.await;
Ok(())
}
#[async_recursion]
async fn bar(it: String) -> Result<()> {
foo().await;
Ok(())
}
When using an array of string references mapped to the same tokio::spawn call it works fine:
futures::stream::iter(
[&String::from("hi")].map(|it: &String| tokio::spawn(bar(it.clone()))),
)
When using the HashMap's keys, it gives me the following compiler error at the #[async_recursion] macro:
implementation of `FnOnce` is not general enough
closure with signature `fn(&'0 std::string::String) -> tokio::task::JoinHandle<Result<(), anyhow::Error>>` must implement `FnOnce<(&std::string::String,)>`, for any lifetime `'0`...
...but it actually implements `FnOnce<(&std::string::String,)>`
I'm not very informed about lifetimes, my guess is this has something to do with the HashMap's keys not living long enough or something. How can I fix this error?
If you switch to
futures::stream::iter(
map.into_keys()
.map(|it: String| tokio::spawn(bar(it.to_owned()))),
)
your code will work.
The reason your original code doesn't work is that rust doesn't know that
.buffer_unordered(2)
.collect::<Vec<_>>()
.await;
forces the spawned futures to finish before your HashMap goes out of scope.
Related
Consider the following Rust code:
use std::future::Future;
use std::pin::Pin;
fn main() {
let mut v: Vec<_> = Vec::new();
for _ in 1..10 {
v.push(wrap_future(Box::pin(async {})));
}
}
fn wrap_future<T>(a: Pin<Box<dyn Future<Output=T>>>) -> impl Future<Output=T> {
async {
println!("doing stuff before awaiting");
let result=a.await;
println!("doing stuff after awaiting");
result
}
}
As you can see, the futures I'm putting into the Vec don't need to be boxed, since they are all the same type and the compiler can infer what that type is.
I would like to create a struct that has this Vec<...> type as one of its members, so that I could add a line at the end of main():
let thing = MyStruct {myvec: v};
without any additional overhead (i.e. boxing).
Type inference and impl Trait syntax aren't allowed on struct members, and since the future type returned by an async block exists entirely within the compiler and is exclusive to that exact async block, there's no way to reference it by name. It seems to me that what I want to do is impossible. Is it? If so, will it become possible in a future version of Rust?
I am aware that it would be easy to sidestep this problem by simply boxing all the futures in the Vec as I did the argument to wrap_future() but I'd prefer not to do this if I can avoid it.
I am well aware that doing this would mean that there could be only one async block in my entire codebase whose result values could possibly be added to such a Vec, and thus that there could be only one function in my entire codebase that could create values that could possibly be pushed to it. I am okay with this limitation.
Nevermind, I'm stupid. I forgot that structs could have type parameters.
struct MyStruct<F> where F: Future<Output=()> {
myvec: Vec<F>,
}
I'm learning Rust's async/await feature, and stuck with the following task. I would like to:
Create an async closure (or better to say async block) at runtime;
Pass created closure to constructor of some struct and store it;
Execute created closure later.
Looking through similar questions I wrote the following code:
use tokio;
use std::pin::Pin;
use std::future::Future;
struct Services {
s1: Box<dyn FnOnce(&mut Vec<usize>) -> Pin<Box<dyn Future<Output = ()>>>>,
}
impl Services {
fn new(f: Box<dyn FnOnce(&mut Vec<usize>) -> Pin<Box<dyn Future<Output = ()>>>>) -> Self {
Services { s1: f }
}
}
enum NumberOperation {
AddOne,
MinusOne
}
#[tokio::main]
async fn main() {
let mut input = vec![1,2,3];
let op = NumberOperation::AddOne;
let s = Services::new(Box::new(|numbers: &mut Vec<usize>| Box::pin(async move {
for n in numbers {
match op {
NumberOperation::AddOne => *n = *n + 1,
NumberOperation::MinusOne => *n = *n - 1,
};
}
})));
(s.s1)(&mut input).await;
assert_eq!(input, vec![2,3,4]);
}
But above code won't compile, because of invalid lifetimes.
How to specify lifetimes to make above example compile (so Rust will know that async closure should live as long as input). As I understand in provided example Rust requires closure to have static lifetime?
Also it's not clear why do we have to use Pin<Box> as return type?
Is it possible somehow to refactor code and eliminate: Box::new(|arg: T| Box::pin(async move {}))? Maybe there is some crate?
Thanks
Update
There is similar question How can I store an async function in a struct and call it from a struct instance?
. Although that's a similar question and actually my example is based on one of the answers from that question. Second answer contains information about closures created at runtime, but seems it works only when I pass an owned variable, but in my example I would like to pass to closure created at runtime mutable reference, not owned variable.
How to specify lifetimes to make above example compile (so Rust will know that async closure should live as long as input). As I understand in provided example Rust requires closure to have static lifetime?
Let's take a closer look at what happens when you invoke the closure:
(s.s1)(&mut input).await;
// ^^^^^^^^^^^^^^^^^^
// closure invocation
The closure immediately returns a future. You could assign that future to a variable and hold on to it until later:
let future = (s.s1)(&mut input);
// do some other stuff
future.await;
The problem is, because the future is boxed, it could be held around for the rest of the program's life without ever being driven to completion; that is, it could have 'static lifetime. And input must obviously remain borrowed until the future resolves: else imagine, for example, what would happen if "some other stuff" above involved modifying, moving or even dropping input—consider what would then happen when the future is run?
One solution would be to pass ownership of the Vec into the closure and then return it again from the future:
let s = Services::new(Box::new(move |mut numbers| Box::pin(async move {
for n in &mut numbers {
match op {
NumberOperation::AddOne => *n = *n + 1,
NumberOperation::MinusOne => *n = *n - 1,
};
}
numbers
})));
let output = (s.s1)(input).await;
assert_eq!(output, vec![2,3,4]);
See it on the playground.
#kmdreko's answer shows how you can instead actually tie the lifetime of the borrow to that of the returned future.
Also it's not clear why do we have to use Pin as return type?
Let's look at a stupidly simple async block:
async {
let mut x = 123;
let r = &mut x;
some_async_fn().await;
*r += 1;
x
}
Notice that execution may pause at the await. When that happens, the incumbent values of x and r must be stored temporarily (in the Future object: it's just a struct, in this case with fields for x and r). But r is a reference to another field in the same struct! If the future were then moved from its current location to somewhere else in memory, r would still refer to the old location of x and not the new one. Undefined Behaviour. Bad bad bad.
You may have observed that the future can also hold references to things that are stored elsewhere, such as the &mut input in #kmdreko's answer; because they are borrowed, those also cannot be moved for the duration of the borrow. So why can't the immovability of the future similarly be enforced by r's borrowing of x, without pinning? Well, the future's lifetime would then depend on its content—and such circularities are impossible in Rust.
This, generally, is the problem with self-referential data structures. Rust's solution is to prevent them from being moved: that is, to "pin" them.
Is it possible somehow to refactor code and eliminate: Box::new(|arg: T| Box::pin(async move {}))? Maybe there is some crate?
In your specific example, the closure and future can reside on the stack and you can simply get rid of all the boxing and pinning (the borrow-checker can ensure stack items don’t move without explicit pinning). However, if you want to return the Services from a function, you'll run into difficulties stating its type parameters: impl Trait would normally be your go-to solution for this type of problem, but it's limited and does not (currently) extend to associated types, such as that of the returned future.
There are work-arounds, but using boxed trait objects is often the most practical solution—albeit it introduces heap allocations and an additional layer of indirection with commensurate runtime cost. Such trait objects are however unavoidable where a single instance of your Services structure may hold different closures in s1 over the course of its life, where you're returning them from trait methods (which currently can’t use impl Trait), or where you're interfacing with a library that does not provide any alternative.
If you want your example to work as is, the missing component is communicating to the compiler what lifetime associations are allowed. Trait objects like dyn Future<...> are constrained to be 'static by default, which means it cannot have references to non-static objects. This is a problem because your closure returns a Future that needs to keep a reference to numbers in order to work.
The direct fix is to annotate that the dyn FnOnce can return a Future that can be bound to the life of the first parameter. This requires a higher-ranked trait bound and the syntax looks like for<'a>:
struct Services {
s1: Box<dyn for<'a> FnOnce(&'a mut Vec<usize>) -> Pin<Box<dyn Future<Output = ()> + 'a>>>,
}
impl Services {
fn new(f: Box<dyn for<'a> FnOnce(&'a mut Vec<usize>) -> Pin<Box<dyn Future<Output = ()> + 'a>>>) -> Self {
Services { s1: f }
}
}
The rest of your code now compiles without modification, check it out on the playground.
I am using async-tungstenite to listen to a websocket, and async-std's StreamExt to operate on the resulting stream.
I want to use a HashMap to accumulate the latest Ticker values from the websocket. These Ticker values will be looked up later to be used in calculations. I'm using the symbol (String) value of the Ticker struct as the key for the HashMap. I'm using the .scan StreamExt method to perform the accumulation.
However, I get a compilation error related to lifetimes. Here's some stripped-down code:
let tickers = HashMap::new();
let mut stream = ws.
.scan(tickers, accumulate_tickers);
while let msg = stream.next().await {
println!("{:?}", msg)
}
...and the accumulate_tickers function:
fn accumulate_tickers(tps: &mut HashMap<String, Ticker>, bt: Ticker) -> Option<&HashMap<String, Ticker>> {
tps.insert((*bt.symbol).to_string(), bt);
Some(tps)
}
The compilation error I receive is as follows:
error[E0271]: type mismatch resolving `for<'r> <for<'s> fn(&'s mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker) -> std::option::Option<&'s std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>> {accumulate_tickers} as std::ops::FnOnce<(&'r mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker)>>::Output == std::option::Option<_>`
--> examples/async_std-ws.rs:64:4
|
64 | .scan(tickers, accumulate_tickers);
| ^^^^ expected bound lifetime parameter, found concrete lifetime
I'm unaware of a way to provide a lifetime parameter to the scan method.
I wonder whether the issue may be related to the fact that I modify the HashMap and then try to return it (is it a move issue?). How could I resolve this, or at least narrow down the cause?
I was able to get this working. Working my way through the compiler errors, I ended up with this signature for the accumulate_tickers function:
fn accumulate_tickers<'a>(tps: &'a mut &'static HashMap<String, Ticker>, bt: Ticker) -> Option<&'static HashMap<String, Ticker>>
I do want the accumulator HashMap to have a static lifetime so that makes sense. tps: &'a mut &'static HashMap... does look a bit strange, but it works.
Then, this issue was was that tickers also had to have a static lifetime (it's the initial value for the accumulator. I tried declaring it as static outside the main but it wouldn't let me set it to the result of a function - HashMap::new().
I then turned to lazy_static which allows me to create a static value which does just that:
lazy_static! {
static ref tickers: HashMap<String, Ticker> = HashMap::new();
}
This gave me a HashMap accumulator that had a static lifetime. However, like normal static values declared in the root scope, it was immutable. To fix that, I read some hints from the lazy_static team and then found https://pastebin.com/YES8dsHH. This showed me how to make my static accumulator mutable by wrapping it in Arc<Mutex<_>>.
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
This does mean that I have to retrieve the accumulator from the Mutex (and lock it) before reading or modifying it but, again, it works.
Putting it all together, the stripped-down code now looks like this:
#[macro_use]
extern crate lazy_static;
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
// SNIP
// Inside main()
let mut ticks = ws
.scan(&tickers, accumulate_tickers);
while let Some(msg) = ticks.next().await {
println!("{:?}", msg.lock().unwrap());
}
// SNIP
fn accumulate_tickers<'a>(tps: &'a mut &'static tickers, bt: Ticker) -> Option<&'static tickers> {
tps.lock().unwrap().insert((*bt.symbol).to_string(), bt);
Some(tps)
}
I'd be happy to hear suggestions for ways in which this could be made simpler or more elegant.
I've got the following piece of code (see playground):
use futures::{stream, Future, Stream}; // 0.1.25
use std::num::ParseIntError;
fn into_many(i: i32) -> impl Stream<Item = i32, Error = ParseIntError> {
stream::iter_ok(0..i)
}
fn convert_to_string(number: i32) -> Result<String, ParseIntError> {
Ok(number.to_string())
}
fn main() {
println!("start:");
let vec = into_many(10)
.map(|number| convert_to_string(number))
.collect()
.wait()
.unwrap();
println!("vec={:#?}", vec);
println!("finish:");
}
It outputs the following (i.e., Vec<Result<i32, ParseIntError>>):
start:
vec=[
Ok(
"0"
),
Ok(
"1"
),
Ok(
"2"
), ...
Is there any way to make it output a Vec<i32> and if any error happens than immediately stop execution and return from the function (e.g., like this example)?
Note: I do want to use use futures::Stream; // 0.1.25 even if it doesn't make sense for this particular example.
The following code (playground link) as a modification of your current code in your question gets the result you want:
use futures::{stream, Future, Stream}; // 0.1.25
use std::num::ParseIntError;
fn into_many(i: i32) -> impl Stream<Item = i32, Error = ParseIntError> {
stream::iter_ok(0..i)
}
fn convert_to_string(number: i32) -> Result<String, ParseIntError> {
Ok(number.to_string())
}
fn main() {
println!("start:");
let vec: Result<Vec<String>, ParseIntError> = into_many(10)
.map(|number| convert_to_string(number))
.collect()
.wait()
.unwrap()
.into_iter()
.collect();
println!("vec={:#?}", vec);
println!("finish:");
}
Since your current code returned a Vec, we can turn that into an iterator and collect that into the type you want. Type annotations are needed so that collect knows what type to collect the iterator into.
Note that the collect method on the Iterator trait isn't to be confused with the collect method on a Stream.
Finally, while this works, it may not be exactly what you want, since it still waits for all results from the stream to be collected into a vector, before using collect to transform the vector. I don't have experience with futures so not sure how possible this is (it probably is but may require a less neat functional programming style solution).
map with a function that returns Result
Don't do this, that's not when you should use map. Instead, use and_then:
let vec = into_many(10)
.and_then(|number| convert_to_string(number))
.collect()
.wait()
.unwrap();
You should practice with simpler Rust concepts like Option, Result, and iterators before diving into futures. Many concepts transfer over.
See also:
How do I unwrap an arbitrary number of nested Option types?
What is the idiomatic way to handle/unwrap nested Result types?
I'm trying to put in one structure information about inotify events and hashmap with inotify watch id as key and name of the file as value.
extern crate inotify;
use inotify::INotify;
use std::sync::{Arc, Mutex};
use std::collections::HashMap;
struct Notificator {
inotify: INotify,
watch_service: Arc<Mutex<HashMap<inotify::wrapper::Watch, Arc<String>>>>,
}
impl Notificator {
pub fn new() -> Notificator {
Notificator {
inotify: INotify::init().unwrap(),
watch_service: Arc::new(Mutex::new(HashMap::new())),
}
}
pub fn resolving_events(&mut self) {
{
let mut events = self.check_for_events();
self.events_execution(events);
}
}
fn check_for_events(&mut self) -> &[inotify::wrapper::Event] {
self.inotify.available_events().unwrap()
}
fn events_execution(&self, events: &[inotify::wrapper::Event]) {
for event in events.iter() {
}
}
}
During compilation I am receiving an error
src/main.rs:204:13: 204:17 error: cannot borrow `*self` as immutable because it is also borrowed as mutable [E0502]
src/main.rs:204 self.events_execution(events);
I thought the best solution would be to separate somehow inotify variable in Notificator structure with watch_service, but I can't dereference self.check_for_events(); because I receive
src/main.rs:203:17: 203:27 error: the trait bound `[inotify::wrapper::Event]: std::marker::Sized` is not satisfied [E0277]
src/main.rs:203 let mut events = *self.check_for_events();
I understand the core of the problem: I'm trying to borrow reference by check_for_events and then using it as parameter in events_execution which also requires self as parameter, but I have no idea how to resolve it.
One thing you could do, although it's not very elegant, is to have your mutable method consume its mutable borrow and return an immutable one that you can then use:
pub fn resolving_events(&mut self) {
let (slf, events) = self.check_for_events();
slf.events_execution(events);
}
fn check_for_events(&mut self) -> (&Self, &[inotify::wrapper::Event]) {
let events = self.inotify.available_events().unwrap();
(&self, events)
}
I've made a small proof-of-concept on the playground (using vecs of u64 as the mutable state, but the principle is similar). It might be cleaner to refactor your code so that some external client can (mutably) borrow the Notifier, produce the events, release the borrow, and borrow it (immutably) to process them...
This is a known issue in the borrow checker (more discussion on internals). You cannot have a function taking a &mut T to an object and returning a &T without losing the ability to access the object again until the &T goes out of scope. You can't work around it due to the way inotify is implemented.
But you can ask the inotify authors to create a get_available_notifications method that doesn't fetch new data. This way you can call available_notifications once and drop the returned value. Then call get_available_notifications (which doesn't take &mut INotify, but just &INotify) and work from there.