What is the difference between `future` and `async move { future }`? - rust

I'm wondering why changing the code from using a Future directly to using it in an async move block makes a difference. Passing it to tokio::spawn directly yields an error:
use std::error::Error;
use tokio::sync::{mpsc::{self, Receiver}, oneshot::Sender};
struct Msg {
resp: Sender<Vec<u8>>,
}
async fn client_thread(
mut rx: Receiver<Msg>,
) -> Result<(), Box<dyn Error>> {
while let Some(msg) = rx.recv().await {
msg.resp
.send(vec![]) // return some data
.unwrap();
}
Ok(())
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let (_tx, rx) = mpsc::channel(5);
tokio::spawn(client_thread(rx));
Ok(())
}
error[E0277]: `(dyn std::error::Error + 'static)` cannot be sent between threads safely
--> src/main.rs:23:5
|
23 | tokio::spawn(client_thread(rx)); // <--- Difference is here
| ^^^^^^^^^^^^ `(dyn std::error::Error + 'static)` cannot be sent between threads safely
|
= help: the trait `Send` is not implemented for `(dyn std::error::Error + 'static)`
= note: required because of the requirements on the impl of `Send` for `Unique<(dyn std::error::Error + 'static)>`
= note: required because it appears within the type `Box<(dyn std::error::Error + 'static)>`
= note: required because it appears within the type `Result<(), Box<(dyn std::error::Error + 'static)>>`
note: required by a bound in `tokio::spawn`
--> /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.0/src/task/spawn.rs:128:20
|
128 | T::Output: Send + 'static,
| ^^^^ required by this bound in `tokio::spawn`
But changing it to an async block makes it compile:
tokio::spawn(async move { client_thread(rx) });
The return type of client_thread is exactly the same as the main function, however, which runs with Tokio without any problems. Moreover, the error type from reqwest implements Send.

Wrapping the Future in an async {} block as you've done makes it compile but doesn't actually run the Future. You need to .await it:
use futures::executor::block_on; // 0.3.12
async fn f(s: &str) {
println!("ran {}", s);
}
fn main() {
block_on(f("first"));
block_on(async { f("second") }); // this doesn't print
block_on(async { f("third").await });
}
ran first
ran third
Since the future is not ran, it doesn't end up affecting the traits for the async {} block and thus can can satisfy the Send constraint. You'll have the same issue again with .await.
In your case, all you need to do is ensure that the future implements Send so it can be ran with tokio::spawn. The fix is to dictate that the Error trait being returned implements Send:
async fn client_thread(
mut rx: Receiver<ClientMsg>,
) -> Result<(), Box<dyn Error + Send>> {
// ... ^^^^^^
}

Related

Cannot relax lifetime of Trait Object

I have the following code:
use tokio; // 1.7.1
use futures::future::Future;
use std::pin::Pin;
use futures::FutureExt;
use std::marker::PhantomData;
use std::marker::Send;
use std::sync::Arc;
use async_trait::async_trait;
struct Orchestrator {
worker: Box<dyn WorkerT<Box<dyn Fn(i32) -> Pin<Box<dyn Future<Output = i32> + Send>> + Send + Sync>> + Send + Sync>
}
impl Orchestrator {
async fn other_things(&self, num: i32) -> i32{
// Do some async stuff in here
num+1
}
async fn main_stuff(self: Arc<Self>) {
let func = |num: i32| {
let slf = self.clone();
async move {
slf.other_things(num).await
}.boxed()
};
self.worker.do_work(Box::new(func)).await;
}
}
#[async_trait]
trait WorkerT<F: Fn(i32) -> Pin<Box<dyn Future<Output = i32> + Send>> + Send + Sync> {
async fn do_work(& self, func: F);
}
struct Worker<F: Fn(i32) -> Pin<Box<dyn Future<Output = i32> + Send>> + Send + Sync> {
phantom: std::marker::PhantomData<F>
}
#[async_trait]
impl<F: Fn(i32) -> Pin<Box<dyn Future<Output = i32> + Send>> + Send + Sync> WorkerT<F> for Worker<F> {
async fn do_work(& self, func: F) {
for i in 0..5 {
func(i).await;
}
}
}
#[tokio::main]
async fn main() {
let orchestrator = Arc::new(Orchestrator { worker: Box::new(Worker { phantom: PhantomData }) });
orchestrator.main_stuff().await;
}
With the following error:
error[E0597]: `self` does not live long enough
--> src/main.rs:22:23
|
21 | let func = |num: i32| {
| ---------- value captured here
22 | let slf = self.clone();
| ^^^^ borrowed value does not live long enough
...
28 | self.worker.do_work(Box::new(func)).await;
| -------------- cast requires that `self` is borrowed for `'static`
29 | }
| - `self` dropped here while still borrowed
Currently, because the default lifetime of dyn WorkerT... is 'static, it requires that the borrow of self in main_stuff be 'static. I need to relax the lifetime of the worker field but I don't know how. If I change worker to be of type 'Workerinstead ofdyn WorkerT...` this works, but I would much prefer keep it as the trait.
I have tried adding a lifetime to the Orchestrator struct and then giving that lifetime to the worker field, but then it says that my created lifetime needs to outlive 'static.
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=7f34fa394d706409942659f1d8ce36c0
You need to clone the Arc before creating the closure so that you can give an Arc to the closure. What you have in your code tries to borrow self, which won't outlive the closure. Cloning the Arc solves this issue as the closure can now own its own Arc pointing at the same value.
For example:
async fn main_stuff(self: Arc<Self>) {
let self2 = self.clone();
let func = move |num: i32| {
let self3 = self2.clone();
async move {
self3.other_things(num).await
}.boxed()
};
self.worker.do_work(Box::new(func)).await;
}
The inner clone is still required so that the type of func implements Fn (otherwise it will implement FnOnce due to moving a captured variable into the future).
Solution
async fn main_stuff(self: Arc<Self>) {
let slf = self.clone();
let func = move |num: i32| {
let slf = slf.clone();
async move { slf.other_things(num).await }.boxed()
};
self.worker.do_work(Box::new(func)).await;
}
Explanation
You are passing a closure that in turns creates async routine. First, the closure it self is static, so it has to take ownership of self. We clone self to slf and add move to the closure, thus closure moved it. Then we have to clone the slf each time and let the async routine own it when it exits the closure.

Size unknown when storing a struct parameterized with a future

I am trying to store a tokio::Timeout struct within a structure. My intent is for the structure to own it. I see that Timeout<T> is bounded with Sized, but my thought was Timeout would have already satisfied the compiler.
use std::future::Future;
use tokio::time::{error::Elapsed, timeout, Timeout};
struct SockTimeout {
pub timeout: Timeout<dyn Future<Output = ()>>,
}
#[tokio::main]
async fn main() {
let when = Duration::from_millis(2000);
let my_struct = SockTimeout {
timeout: timeout(when, async {}),
};
}
error[E0277]: the size for values of type `(dyn Future<Output = ()> + 'static)` cannot be known at compilation time
--> src/main.rs:30:18
|
30 | pub timeout: Timeout<dyn Future<Output = ()>>,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: the trait `Sized` is not implemented for `(dyn Future<Output = ()> + 'static)`
note: required by a bound in `tokio::time::Timeout`
--> /Users/steve/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/time/timeout.rs:135:24
|
135 | pub struct Timeout<T> {
| ^ required by this bound in `tokio::time::Timeout`
The only way I have found around this issue is to turn SockTimeout into a template. I have tried Box'ing Timeout, but the same issue persists. What am I fundamentally missing here?
Boxing the Timeout won't help because Timeout<T> requires that T: Sized, not just that Timeout<T>: Sized. Instead you need to box the future.
You may also need to pin it:
struct SockTimeout {
pub timeout: Timeout<Pin<Box<dyn Future<Output = ()>>>>,
}
#[tokio::main]
async fn main() {
let when = Duration::from_millis(2000);
let my_struct = SockTimeout {
timeout: timeout(when, Box::pin(async {})),
};
}

How to tell the compiler a value is not used after awaiting in an async_trait method?

In the example code below, a non-send value, Vec<T>, is moved into a function that returns something else. At this point, I no longer care about that vector. The returned object stores no reference to it, it no longer exists.
However, when I .await on the next line I get the error "captured value is not Send". Which it isn't, but since it should have been destroyed when vector_as_string exited, it doesn't need to send it across threads when the future restarts, because that variable is never used again.
use async_trait::async_trait;
async fn write_value(value: Vec<u8>) {
println!("something")
}
fn vector_as_string<T>(vec: Vec<T>) -> Vec<u8> {
Vec::new()
}
#[async_trait]
trait Writer {
async fn write_array<T>(&mut self, value: Vec<T>);
}
pub struct WriterImplementation {}
#[async_trait]
impl Writer for WriterImplementation {
async fn write_array<T>(&mut self, value: Vec<T>) {
let string = vector_as_string(value);
write_value(string).await
}
}
#[tokio::main]
async fn main() {
println!("Hi");
}
Dependencies:
[dependencies]
tokio = { version = "1.9.0", features = ["full"]}
async-trait = "0.1.51"
Error:
error: future cannot be sent between threads safely
--> src/main.rs:20:55
|
20 | async fn write_array<T>(&mut self, value: Vec<T>) {
| _______________________________________________________^
21 | | let string = vector_as_string(value);
22 | |
23 | | write_value(string).await
24 | | }
| |_____^ future created by async block is not `Send`
|
note: captured value is not `Send`
--> src/main.rs:20:40
|
20 | async fn write_array<T>(&mut self, value: Vec<T>) {
| ^^^^^ has type `Vec<T>` which is not `Send`
= note: required for the cast to the object type `dyn Future<Output = ()> + Send`
help: consider further restricting this bound
|
20 | async fn write_array<T + std::marker::Send>(&mut self, value: Vec<T>) {
| ^^^^^^^^^^^^^^^^^^^
Adding T: Send as it suggests allows it to compile, but why does T need to be Send if we aren't holding any T's across the await, as it has already been moved?
From the async_trait documentation:
Async fns get transformed into methods that return Pin<Box<dyn Future + Send + 'async>> and delegate to a private async freestanding function.
Not all async traits need futures that are dyn Future + Send. To avoid having Send and Sync bounds placed on the async trait methods, invoke the async trait macro as #[async_trait(?Send)] on both the trait and the impl blocks.
Applied to your case:
#[async_trait(?Send)]
trait Writer {
async fn write_array<T>(&mut self, value: Vec<T>);
}
#[async_trait(?Send)]
impl Writer for WriterImplementation {
async fn write_array<T>(&mut self, value: Vec<T>) {
let string = vector_as_string(value);
write_value(string).await
}
}

Why usage of async block in then makes my Stream Unpin?

I'm new to Rust and I'm sorry if I'm using terms incorrect. Maybe my choice of words for question is incorrect.
I was playing with streams and I needed to have some delay between stream elements. So I wrote this:
use futures::stream;
use futures::StreamExt;
use tokio::time;
#[tokio::main]
async fn main() {
let mut stream = stream::iter(0..1000).then(|x| async move {
time::delay_for(std::time::Duration::from_millis(500)).await;
x + 1
});
while let Some(x) = stream.next().await {
println!("{:?}", x)
}
}
I get a lot of compilation errors, but the most important errors are connected with pinning. Here they are:
error[E0277]: `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>` cannot be unpinned
--> src/main.rs:11:32
|
11 | while let Some(x) = stream.next().await {
| ^^^^ within `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`, the trait `std::marker::Unpin` is not implemented for `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>`
|
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because it appears within the type `std::option::Option<impl core::future::future::Future>`
= note: required because it appears within the type `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::then::Then<futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
error[E0277]: `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>` cannot be unpinned
--> src/main.rs:11:25
|
11 | while let Some(x) = stream.next().await {
| ^^^^^^^^^^^^^^^^^^^ within `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`, the trait `std::marker::Unpin` is not implemented for `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>`
|
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because it appears within the type `std::option::Option<impl core::future::future::Future>`
= note: required because it appears within the type `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::then::Then<futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
= note: required because of the requirements on the impl of `core::future::future::Future` for `futures_util::stream::stream::next::Next<'_, futures_util::stream::stream::then::Then<futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>>`
If I change my code to this:
use futures::stream;
use futures::StreamExt;
use tokio::time;
#[tokio::main]
async fn main() {
let mut stream = stream::iter(0..1000).then(|x| {
futures::future::ready(x + 1)
});
while let Some(x) = stream.next().await {
println!("{:?}", x)
}
}
Or to this:
use futures::stream;
use futures::StreamExt;
use tokio::time;
#[tokio::main]
async fn main() {
stream::iter(0..1000)
.then(|x| async move {
time::delay_for(std::time::Duration::from_millis(500)).await;
x + 1
})
.for_each(|x| async move { println!("{:?}", x) })
.await;
}
It compiles.
I assume it has something to do with pinning and simultaneous usage of then combinator and while, but I can't wrap my head around it.
I think that the issue boils down to the fact that async blocks are not Unpin. It can be proven with this code:
fn check_unpin<F: Unpin>(_: F) { }
fn main() {
check_unpin(async {});
}
that fails with the rather cryptic message:
error[E0277]: `std::future::from_generator::GenFuture<[static generator#src/main.rs:4:23: 4:25 _]>` cannot be unpinned
--> src/main.rs:4:5
|
1 | fn check_unpin<F: Unpin>(_: F) { }
| ----- required by this bound in `check_unpin`
...
4 | check_unpin(async {});
| ^^^^^^^^^^^ within `impl std::future::Future`, the trait `std::marker::Unpin` is not implemented for `std::future::from_generator::GenFuture<[static generator#src/main.rs:4:23: 4:25 _]>`
|
= note: required because it appears within the type `impl std::future::Future`
I reckon that GenFuture is the internal type that converts an async block into a impl Future.
Now back to your issue, the combinator then() returns an Unpin value if both the stream and the Future returned by the closure are Unpin (it is not totally clear in the documentation, but I've inferred that from the source code). The stream::iter is Unpin, but when you write |x| async move { x + 1} you are returning an async block that is not Unpin, thus your error.
If you use futures::future::ready(x + 1) it works simply because that Future implements Unpin.
If you use StreamExt::for_each it works because it does not require Self to be Unpin. It is not Unpin itself, but it does not matter because you are sending it up to tokio::main that pins everything internally before polling.
If you want your original code to work you just have to manually pin your stream (playground):
use futures::stream;
use futures::StreamExt;
use tokio::time;
use pin_utils::pin_mut;
#[tokio::main]
async fn main() {
let stream = stream::iter(0..1000).then(|x| async move {
time::delay_for(std::time::Duration::from_millis(500)).await;
x + 1
});
pin_mut!(stream); //<---- here, pinned!
while let Some(x) = stream.next().await {
println!("{:?}", x)
}
}

How do I resolve the compilation error "Unpin not implemented for GenFuture" resulting from adapting a stream using a combinator?

I'm unable to work out how to resolve the compilation error resulting from adapting a stream using a combinator.
The following Rust Playground demonstrates a fairly minimal example:
use futures::prelude::*;
use futures::StreamExt;
#[derive(Debug)]
pub enum Input {
A,
B(i32),
C(u16),
}
#[derive(Debug)]
enum Output {
Int(i32),
Short(u16),
}
pub struct StreamMaker;
impl StreamMaker {
/// make a stream with a series of inputs
pub fn create(self) -> impl Stream<Item = Input> {
stream::iter(vec![Input::A, Input::C(1u16), Input::B(2)])
}
}
/// consume the creator, and make output messages for a subset
pub fn adapt_stream(creator: StreamMaker) -> impl Stream<Item = Output> {
let mut upstream = creator.create();
upstream.filter_map(|message| async move {
match message {
Input::A => None,
Input::B(v) => Some(Output::Int(v)),
Input::C(v) => Some(Output::Short(v)),
}
})
}
#[tokio::main]
async fn main() -> Result<(), ()> {
let creator = StreamMaker {};
let mut stream = adapt_stream(creator);
while let Some(message) = stream.next().await {
println!("message: {:?}", message)
}
Ok(())
}
Compilation fails with:
error[E0277]: the trait bound `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>: std::marker::Unpin` is not satisfied in `impl core::future::future::Future`
--> src/main.rs:43:38
|
43 | while let Some(message) = stream.next().await {
| ^^^^ within `impl core::future::future::Future`, the trait `std::marker::Unpin` is not implemented for `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>`
|
= help: the following implementations were found:
<std::future::GenFuture<T> as std::marker::Unpin>
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::filter_map::FilterMap<impl futures_core::stream::Stream, impl core::future::future::Future, [closure#src/main.rs:29:25: 35:6]>`
= note: required because it appears within the type `impl futures_core::stream::Stream`
error[E0277]: the trait bound `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>: std::marker::Unpin` is not satisfied in `impl core::future::future::Future`
--> src/main.rs:43:31
|
43 | while let Some(message) = stream.next().await {
| ^^^^^^^^^^^^^^^^^^^ within `impl core::future::future::Future`, the trait `std::marker::Unpin` is not implemented for `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>`
|
= help: the following implementations were found:
<std::future::GenFuture<T> as std::marker::Unpin>
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::filter_map::FilterMap<impl futures_core::stream::Stream, impl core::future::future::Future, [closure#src/main.rs:29:25: 35:6]>`
= note: required because it appears within the type `impl futures_core::stream::Stream`
= note: required because of the requirements on the impl of `core::future::future::Future` for `futures_util::stream::stream::next::Next<'_, impl futures_core::stream::Stream>`
I can put a pin_mut!(stream); in the main, but I'd like to be able to push that upstream.
If you do not want the consumer of your stream to have to pin it themselves, you need to return a stream that implements the Unpin trait, meaning that it is safe to move around in memory even after is has been pinned.
pub fn adapt_stream(creator: StreamMaker) -> impl Stream<Item = Output> + Unpin {
// add Unpin trait --^
Adding this, your compiler should complain that the return value doesn't implement Unpin. This is because async move { ... } blocks don't implement Unpin, since they may be self-referential (e.g. contain references to variables they own). The most general way to work around this is to pin the stream to the heap with a Pin<Box<_>>, using the Box::pin constructor:
pub fn adapt_stream(creator: StreamMaker) -> impl Stream<Item = Output> + Unpin {
let mut upstream = creator.create();
Box::pin(upstream.filter_map(|message| async move {
// ^-- pin stream to heap
match message {
Input::A => None,
Input::B(v) => Some(Output::Int(v)),
Input::C(v) => Some(Output::Short(v)),
}
}))
}
Since we're now returning a Pin<Box<_>> pointer to the stream, that pointer can be safely moved around in memory while the inner stream is kept at the same location.
Full playground example

Resources