how to pin RefCell contents? - rust

I have a struct MyAsyncStream and tokio::io::AsyncWrite implementation for it:
impl<S: AsyncRead + AsyncWrite + Unpin> AsyncWrite for MyAsyncStream<S> {
fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context, buf: &[u8]) -> Poll<Result<usize>> {
....
}
I also have MyAsyncStreamWrapper:
struct MyAsyncStreamWrapper{inner: MyAsyncStream}
Now I want to implement AnotherTrait trait for MyAsyncStreamWrapper, with the following method:
impl AnotherTrait for MyAsyncStreamWrapper {
fn poll_send_to<B>(self: Pin<&Self>, cx: &mut Context<'_>, buf: &[u8], addr: B,) -> Poll<Result<usize, Self::Error>> {
Pin::new(&mut self.inner).poll_write(cx, buf)
}
....
}
In this method implementation, I want to call poll_write on the inner. But, unfortunately they are different on self mutability: Pin<&mut Self> vs Pin<&Self>. As expected it does not compile.
Is there an idiomatic "workaround" for such case? My idea is to wrap inner into Mutex so I could have mutable MyAsyncStream in the non-mutable context:
MyAsyncStreamWrapper{inner: Mutex<RefCell<MyAsyncStream>>}
...
fn poll_send_to<B>(mut self: Pin<&Self>, cx: &mut Context<'_>, buf: &[u8], addr: B,) -> Poll<Result<usize, Self::Error>> {
let rc = self.stream.lock().unwrap();
let ref mut inner = rc.borrow();
let pin = Pin::new(inner);
pin.poll_write(cx, buf);
}
...
But, unfortunately, it also does not compile, with the following error:
pin.poll_write(cx, buf);
^^^^^^^^^^ method not found in `std::pin::Pin<&mut std::cell::RefMut<'_, MyAsyncStream>>`
What is the right way to go?

Dereference then re-borrow seems to work:
use std::cell::RefCell;
use std::pin::Pin;
use std::sync::Mutex;
fn main() {
let a = Mutex::new(RefCell::new(42));
let rc = a.lock().unwrap();
let mut inner = rc.borrow_mut();
let pinned = Pin::new(&mut *inner);
print_type_name(pinned);
}
fn print_type_name<T>(_: T) {
println!("{}", std::any::type_name::<T>());
}
It outputs that the type is core::pin::Pin<&mut i32>.
That being said, using blocking synchronization primitives like Mutex in asynchronous context is probably not a good idea. If possible it is better to let poll_send_to take a Pin<&mut Self> parameter.

Related

How can I wrap a dynamically typed stream for API convenience?

I'm looking to implement a wrapper struct for any stream that returns a certain type, to cut down on the dynamic keywords littering my application. I've come across BoxStream, but have no idea how to make use of it in Stream::poll_next. Here's what I have so far:
use std::pin::Pin;
use std::task::{Context, Poll};
use futures::prelude::stream::BoxStream;
use futures::Stream;
pub struct Row;
pub struct RowCollection<'a> {
stream: BoxStream<'a, Row>,
}
impl RowCollection<'_> {
pub fn new<'a>(stream: BoxStream<Row>) -> RowCollection {
RowCollection { stream }
}
}
impl Stream for RowCollection<'_> {
type Item = Row;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
// I have no idea what to put here, but it needs to get the information from self.stream and return appropriate value
}
}
Dependencies:
futures = "0.3"
Since Box implements Unpin, then BoxStream implements Unpin, and so will RowCollection.
Because of this, you can make use of Pin::get_mut which will give you a &mut RowCollection. From that, you can get a &mut BoxStream. You can re-pin that via Pin::new and then call poll_next on it. This is called pin-projection.
impl Stream for RowCollection<'_> {
type Item = Row;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
Pin::new(&mut self.get_mut().stream).poll_next(cx)
}
}
See also:
No method named `poll` found for a type that implements `Future`

try_lock on futures::lock::Mutex outside of async?

I'm trying to implement Async read for a struct that has a futures::lock::Mutex:
pub struct SmolSocket<'a> {
stack: Arc<futures::lock::Mutex<SmolStackWithDevice<'a>>>,
}
impl<'a> AsyncRead for SmolSocket<'a> {
fn poll_read(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>
) -> Poll<std::io::Result<()>> {
block_on(self.stack).read(...)
}
}
The problem is that, since poll_read is not async, I cannot call await. But I also don't want to, as it'd block. I could call try_lock to try and if not, I'd register a Waker to be called by SmolSocket in the future.
Since I cannot do that either because it's not async, is there a version of block_on that does the same as try_lock for futures::lock::Mutex outside of async?
You probably mean to poll the MutexLockFuture instead, this can for example be done with the core::task::ready! macro, which desugars as following:
let num = match fut.poll(cx) {
Poll::Ready(t) => t,
Poll::Pending => return Poll::Pending,
};
To poll a future, you also need to pin it (ensure it doesn't get moved). This can be done on the stack with tokio::pin!, or Pin::new if the type is already Unpin (MutexLockFuture is), or by moving onto the heap with Box::pin.
Below is a runnable example.
⚠️ KEEP READING TO SEE WHY YOU DON'T WANT TO DO THIS!
#![feature(ready_macro)]
use core::{
future::Future,
pin::Pin,
task::{ready, Context, Poll},
};
use std::sync::Arc;
use tokio::io::{AsyncRead, AsyncReadExt};
pub struct SmolStackWithDevice<'a> {
counter: usize,
data: &'a [u8],
}
impl<'a> AsyncRead for SmolStackWithDevice<'a> {
fn poll_read(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
if self.counter % 2 == 0 {
self.counter += 1;
cx.waker().wake_by_ref();
println!("read nothing");
return Poll::Pending;
}
buf.put_slice(&[self.data[self.counter / 2]]);
self.counter += 1;
println!("read something");
Poll::Ready(Ok(()))
}
}
pub struct SmolSocket<'a> {
stack: Arc<futures::lock::Mutex<SmolStackWithDevice<'a>>>,
}
impl<'a> AsyncRead for SmolSocket<'a> {
fn poll_read(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
let mut lock_fut = self.stack.lock();
let pinned_lock_fut = Pin::new(&mut lock_fut);
let mut guard = ready!(pinned_lock_fut.poll(cx));
println!("acquired lock");
let pinned_inner = Pin::new(&mut *guard);
pinned_inner.poll_read(cx, buf)
}
}
#[tokio::main(flavor = "current_thread")]
async fn main() {
let data = b"HORSE";
let mut buf = [0; 5];
let mut s = SmolSocket {
stack: Arc::new(
SmolStackWithDevice {
counter: 0,
data: &data[..],
}
.into(),
),
};
s.read_exact(&mut buf).await.unwrap();
println!("{}", String::from_utf8_lossy(&buf));
}
Look at it go! (in Rust Playground)
⚠️ KEEP READING TO SEE WHY YOU DON'T WANT TO DO THIS!
So, what is the problem?
Well, as you can see from the output, whenever we succeed in acquiring the lock, but the underlying source is not ready to read, or only gives us a small read, we drop the lock, and on the next poll we will have to acquire it again.
This is a good point to remember that async flavors of Mutex are only recommended over std or parking_lot when it is expected that the Guard from a successful locking will be held across an await, or explicitly stored in a Future data structure.
We are not doing that here, we are only ever exercising the fast path equivalent to Mutex::try_lock, because whenever the lock is not immediately available, we drop the MutexLockFuture instead of waiting to be waked to poll it again.
However, storing the lock in the data structure would make it easy to accidentally deadlock. So a good design might be creating an awkward-to-store(borrowing) AsyncRead adapter that wraps the lock:
pub struct SmolSocket<'a> {
stack: Arc<futures::lock::Mutex<SmolStackWithDevice<'a>>>,
}
impl<'a> SmolSocket<'a> {
fn read(&'a self) -> Reader<'a> {
Reader::Locking(self.stack.lock())
}
}
pub enum Reader<'a> {
Locking(futures::lock::MutexLockFuture<'a, SmolStackWithDevice<'a>>),
Locked(futures::lock::MutexGuard<'a, SmolStackWithDevice<'a>>),
}
impl<'a> AsyncRead for Reader<'a> {
fn poll_read(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &mut tokio::io::ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
let this = self.get_mut();
match this {
Reader::Locking(f) => {
*this = Reader::Locked(ready!(Pin::new(f).poll(cx)));
println!("acquired lock");
Pin::new(this).poll_read(cx, buf)
}
Reader::Locked(l) => Pin::new(&mut **l).poll_read(cx, buf),
}
}
}
#[tokio::main(flavor = "current_thread")]
async fn main() {
let data = b"HORSE";
let mut buf = [0; 5];
let s = SmolSocket {
stack: Arc::new(
SmolStackWithDevice {
counter: 0,
data: &data[..],
}
.into(),
),
};
s.read().read_exact(&mut buf).await.unwrap();
println!("{}", String::from_utf8_lossy(&buf));
}
Look at it go! (executable Playground link)
This works out, because both the LockFuture and our SmolStackWithDevice are Unpin (non-self-referential) and so we don't have to guarantee we aren't moving them.
In a general case, for example if your SmolStackWithDevice is not Unpin, you'd have to project the Pin like this:
unsafe {
let this = self.get_unchecked_mut();
match this {
Reader::Locking(f) => {
*this = Reader::Locked(ready!(Pin::new_unchecked(f).poll(cx)));
println!("acquired lock");
Pin::new_unchecked(this).poll_read(cx, buf)
}
Reader::Locked(l) => Pin::new_unchecked(&mut **l).poll_read(cx, buf),
}
}
Not sure how to encapsulate the unsafety, pin_project isn't enough here, as we also need to dereference the guard.
But this only acquires the lock once, and drops it when the Reader is dropped, so, great success.
You can also see that it doesn't deadlock if you do
let mut r1 = s.read();
let mut r2 = s.read();
r1.read_exact(&mut buf[..3]).await.unwrap();
drop(r1);
r2.read_exact(&mut buf[3..]).await.unwrap();
println!("{}", String::from_utf8_lossy(&buf));
This is only possible because we deferred locking until polling.

Unable to use impl Sink<T>::start_send

I wanted to implement the futures::Sink trait in my struct and kept getting yelled at by the compiler telling me to import and implement the trait. So I came up with this example which surprisingly (to me at least) still doesn't work:
use futures::Sink;
use std::pin::Pin;
use std::task::{Context, Poll};
struct Dummy;
impl Sink<tungstenite::Message> for Dummy {
type Error = tungstenite::Error;
fn poll_ready(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
fn start_send(self: Pin<&mut Self>, item: tungstenite::Message) -> Result<(), Self::Error> {
todo!()
}
fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
fn poll_close(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
}
#[tokio::main]
async fn main() {
let mut dummy = Dummy;
dummy.start_send(tungstenite::Message::Text("".to_owned()));
}
I get:
error[E0599]: no method named `start_send` found for struct `Dummy` in the current scope
--> src/main.rs:29:11
|
5 | struct Dummy;
| ------------- method `start_send` not found for this
...
29 | dummy.start_send(tungstenite::Message::Text("".to_owned()));
| ^^^^^^^^^^ method not found in `Dummy`
|
= help: items from traits can only be used if the trait is implemented and in scope
= note: the following trait defines an item `start_send`, perhaps you need to implement it:
candidate #1: `futures::Sink`
error: aborting due to previous error
I'm not making a generic struct, there are no bounds issues, no lifetimes. It's just a direct trait impl which the compiler refuses to recognize.
cargo tree | grep futures says all versions are v0.3.13, and here's the docs
I even tried Pin::new(&mut dummy).start_send because of the signature, but still nothing<. I have no idea what's going on here, or what kind of silly mistake did I make...
(By the way, I had a go at the similar impl Stream before this one and had no issues whatsoever)
I even tried Pin::new(&mut dummy).start_send because of the signature, but still nothing<. I have no idea what's going on here, or what kind of silly mistake did I make...
I also tried wrapping it in a Pin, since the start_send() takes in a Pin<&mut Self> as self, and this does seem to work for me.
use futures::sink::Sink;
use std::pin::Pin;
use std::task::{Context, Poll};
struct Message {}
struct Dummy;
impl Sink<Message> for Dummy {
type Error = ();
fn poll_ready(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
fn start_send(self: Pin<&mut Self>, item: Message) -> Result<(), Self::Error> {
todo!()
}
fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
fn poll_close(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
}
#[tokio::main]
async fn main() {
let mut dummy = Dummy;
let pin_dummy = Pin::new(&mut dummy);
pin_dummy.start_send(Message {});
}
Rust Playground

How do I pin project an element of a vector?

Following code does not compile because the element pulled from the vector does not implement the Pin<> type.
The error is in ele.poll() call
#[pin_project]
pub struct FifoCompletions<T, F>
where
F: Future<Output = Result<T, Error>>
{
#[pin]
pending: VecDeque<F>,
}
impl <T, F> FifoCompletions<T, F>
where
F: Future<Output = Result<T, Error>>
{
pub fn push(&mut self, f: F) {
self.pending.push_back(f);
}
}
impl <T, F> Future for FifoCompletions<T, F>
where
F: Future<Output = Result<T, Error>>
{
type Output = Result<T, Error>;
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
let this = self.project();
while !this.pending.is_empty() {
match this.pending.front_mut() {
None => unreachable!(),
Some(ele) => {
let output = ready!(ele.poll(cx));
}
}
}
Poll::Pending
}
}
Error message is
no method named `poll` found for mutable reference `&mut F` in the current scope
method not found in `&mut F`
help: items from traits can only be used if the type parameter is bounded by the traitrustc(E0599)
sm.rs(66, 45): method not found in `&mut F`
Here's the literal answer to your question:
use std::{
future::Future,
pin::Pin,
task::{Context, Poll},
};
pub struct Completions<F>
where
F: Future<Output = ()>,
{
pending: Vec<F>,
}
impl<F> Future for Completions<F>
where
F: Future<Output = ()>,
{
type Output = ();
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
// I copied this from Stack Overflow without reading the prose
// that describes why this is or is not safe.
// ... It's probably not safe.
let first = unsafe { self.map_unchecked_mut(|this| &mut this.pending[0]) };
first.poll(cx)
}
}
However, this is very likely to be unsafe and introduce undefined behavior. The problem is that the requirements for pinning are complex and nuanced. Specifically, once pinned, a value may never be moved in memory, including when it is dropped. From the docs, emphasis mine:
This can be tricky, as witnessed by VecDeque<T>: the destructor of VecDeque<T> can fail to call drop on all elements if one of the destructors panics. This violates the Drop guarantee, because it can lead to elements being deallocated without their destructor being called. (VecDeque<T> has no pinning projections, so this does not cause unsoundness.)
I checked with taiki-e, the author of the pin-project crate. A lightly-edited quote:
Operations such as Vec(Deque)'s push, insert, remove, etc. can move elements. If you want to pin elements, these methods should not be called without Unpin after they have been pinned. The pin API actually blocks this.
If the destructor moves elements, it is unsound to pin the element returned by accessors like get_mut, front_mut without Unpin.
Make the element Unpin or use a collection that can handle !Unpin futures like FutureUnordered instead of VecDeque.
Applied directly, the safest way to do this is:
use pin_project::pin_project; // 1.0
use std::{
future::Future,
pin::Pin,
task::{Context, Poll},
};
#[pin_project]
pub struct Completions<F>
where
F: Unpin,
F: Future<Output = ()>,
{
#[pin]
pending: Vec<F>,
}
impl<F> Future for Completions<F>
where
F: Unpin,
F: Future<Output = ()>,
{
type Output = ();
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
let this = self.project();
let pending = this.pending.get_mut();
let first = Pin::new(&mut pending[0]);
first.poll(cx)
}
}
See also:
No method named `poll` found for a type that implements `Future`
When is it safe to move a member value out of a pinned future?

Creating a stream of values while calling async fns?

I can't figure out how to provide a Stream where I await async functions to get the data needed for the values of the stream.
I've tried to implement the the Stream trait directly, but I run into issues because I'd like to use async things like awaiting, the compiler does not want me to call async functions.
I assume that I'm missing some background on what the goal of Stream is and I'm just attacking this incorrectly and perhaps I shouldn't be looking at Stream at all, but I don't know where else to turn. I've seen the other functions in the stream module that could be useful, but I'm unsure how I could store any state and use these functions.
As a slightly simplified version of my actual goal, I want to provide a stream of 64-byte Vecs from an AsyncRead object (i.e. TCP stream), but also store a little state inside whatever logic ends up producing values for the stream, in this example, a counter.
pub struct Receiver<T>
where
T: AsyncRead + Unpin,
{
readme: T,
num: u64,
}
// ..code for a simple `new() -> Self` function..
impl<T> Stream for Receiver<T>
where
T: AsyncRead + Unpin,
{
type Item = Result<Vec<u8>, io::Error>;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let mut buf: [u8; 64] = [0; 64];
match self.readme.read_exact(&mut buf).await {
Ok(()) => {
self.num += 1;
Poll::Ready(Some(Ok(buf.to_vec())))
}
Err(e) => Poll::Ready(Some(Err(e))),
}
}
}
This fails to build, saying
error[E0728]: `await` is only allowed inside `async` functions and blocks
I'm using rustc 1.36.0-nightly (d35181ad8 2019-05-20) and my Cargo.toml looks like this:
[dependencies]
futures-preview = { version = "0.3.0-alpha.16", features = ["compat", "io-compat"] }
pin-utils = "0.1.0-alpha.4"
Answer copy/pasted from the reddit post by user Matthias247:
It's unfortunately not possible at the moment - Streams have to be implemented by hand and can not utilize async fn. Whether it's possible to change this in the future is unclear.
You can work around it by defining a different Stream trait which makes use of Futures like:
trait Stream<T> {
type NextFuture: Future<Output=T>;
fn next(&mut self) -> Self::NextFuture;
}
This article and this futures-rs issue have more information around it.
You can do it with gen-stream crate:
#![feature(generators, generator_trait, gen_future)]
use {
futures::prelude::*,
gen_stream::{gen_await, GenTryStream},
pin_utils::unsafe_pinned,
std::{
io,
marker::PhantomData,
pin::Pin,
sync::{
atomic::{AtomicU64, Ordering},
Arc,
},
task::{Context, Poll},
},
};
pub type Inner = Pin<Box<dyn Stream<Item = Result<Vec<u8>, io::Error>> + Send>>;
pub struct Receiver<T> {
inner: Inner,
pub num: Arc<AtomicU64>,
_marker: PhantomData<T>,
}
impl<T> Receiver<T> {
unsafe_pinned!(inner: Inner);
}
impl<T> From<T> for Receiver<T>
where
T: AsyncRead + Unpin + Send + 'static,
{
fn from(mut readme: T) -> Self {
let num = Arc::new(AtomicU64::new(0));
Self {
inner: Box::pin(GenTryStream::from({
let num = num.clone();
static move || loop {
let mut buf: [u8; 64] = [0; 64];
match gen_await!(readme.read_exact(&mut buf)) {
Ok(()) => {
num.fetch_add(1, Ordering::Relaxed);
yield Poll::Ready(buf.to_vec())
}
Err(e) => return Err(e),
}
}
})),
num,
_marker: PhantomData,
}
}
}
impl<T> Stream for Receiver<T>
where
T: AsyncRead + Unpin,
{
type Item = Result<Vec<u8>, io::Error>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
self.inner().poll_next(cx)
}
}

Resources