I have a trait that I own:
trait Reducer {
fn reduce(&self) -> Res;
}
I want to implement for the following cases:
struct MyStruct {... fields ...}
impl Reducer for Vec<MyStruct> {
fn reduce(&self) -> Res { ... some implementation ... }
}
I want to get for as cheap as possible (not duplicating code etc.) for the refs of objects in Vec:
for instance:
// I want this please:
impl Reducer for Vec<&MyStruct> {
// it's the same implementation as before
}
How should I change my impl signature to satisfy both cases?
Using the Borrow trait you can generalize for all types that can be borrowed as a &MyStruct (which in particular includes MyStruct and &MyStruct):
use std::borrow::Borrow;
trait Reducer {
fn reduce(&self) -> Res;
}
struct MyStruct;
struct Res;
impl<T: Borrow<MyStruct>> Reducer for Vec<T> {
fn reduce(&self) -> Res {
for t in self {
let _t: &MyStruct = t.borrow();
// do something with `_t`
}
Res
}
}
fn main() {
let v: Vec<MyStruct> = vec![];
let v_ref: Vec<&MyStruct> = vec![];
// fully qualified because it gets mangled with `Iterator::reduce` otherwise.
Reducer::reduce(&v);
Reducer::reduce(&v_ref);
}
If your reducer only needs to iterate over the items in the Vec, then there is no difference between Vec<T> and Vec<&T> because both can easily produce an iterator over &T. For example:
// actual implementation
fn reduce_it<'a>(_items: impl Iterator<Item = &'a MyStruct>) -> Res {
todo!()
}
impl Reducer for Vec<MyStruct> {
fn reduce(&self) -> Res {
reduce_it(self.iter())
}
}
impl Reducer for Vec<&MyStruct> {
fn reduce(&self) -> Res {
reduce_it(self.iter().copied())
}
}
Playground
So I'm getting a Response from the reqwest crate and passing it to a HttpResponseBuilder from the actix_web create. However I've tried and failed to understand how to implement the Stream trait from the futures create on a custom struct to act as a middleman and copy the contents down to a file.
I've tried doing this so far, but I'm not sure what to put inside that poll_next function to make it all work.
struct FileCache {
stream: Box<dyn futures::Stream<Item = reqwest::Result<bytes::Bytes>>>,
}
impl FileCache {
fn new(stream: Box<dyn futures::Stream<Item = reqwest::Result<bytes::Bytes>>>) -> Self {
FileCache { stream }
}
}
impl Stream for FileCache {
type Item = reqwest::Result<bytes::Bytes>;
fn poll_next(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Self::Item>> {
}
}
This is possible but requires you to understand what pinning is and how to use it safely.
Basically, we just need to proxy to self.stream.poll_next(), but this method accepts Pin<&mut Self> (as you can see in your own implementation). Storing the box as Pin<Box<T>> instead of Box<T> will give us a way to obtain this Pin relatively easily, without requiring unsafe. Making this change is straightforward, since there is a From implementation allowing conversion of Box<T> to Pin<Box<T>> directly:
use std::pin::Pin;
use std::task::{Context, Poll};
use futures::Stream;
struct FileCache {
stream: Pin<Box<dyn Stream<Item = reqwest::Result<bytes::Bytes>>>>,
}
impl FileCache {
fn new(stream: Box<dyn Stream<Item = reqwest::Result<bytes::Bytes>>>) -> FileCache {
FileCache { stream: stream.into() }
}
}
Now we have to figure out how to go from Pin<&mut FileCache> to Pin<&mut dyn Stream<...>>. The correct incantation here is self.get_mut().stream.as_mut():
impl Stream for FileCache {
type Item = reqwest::Result<bytes::Bytes>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
match self.get_mut().stream.as_mut().poll_next(cx) {
Poll::Pending => Poll::Pending,
Poll::Ready(v) => {
// Do what you need to do with v here.
Poll::Ready(v)
}
}
}
}
The catch is that poll_next isn't async and so you can't asynchronously wait for whatever you're doing with v. bytes::Bytes is atomically-refcounted, though, so you could clone the inner bytes::Bytes value and spawn a separate task on your executor, which is probably what you want to do anyway so that whoever is waiting for FileCache doesn't have to wait for that task to complete before using the data. So you'd do something like:
Poll::Ready(v) => {
if let Some(Ok(ref bytes)) = &v {
let bytes = bytes.clone();
spawn_new_task(async move {
// Do something with bytes
});
}
Poll::Ready(v)
}
Where spawn_new_task() is the function your executor provides, e.g. tokio::spawn().
Now that we can see what we're doing here, we can simplify this down and eliminate the match by pushing Poll::Ready into our pattern, and unconditionally returning whatever the inner poll_next() call did:
impl Stream for FileCache {
type Item = reqwest::Result<bytes::Bytes>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let r = self.get_mut().stream.as_mut().poll_next(cx);
if let Poll::Ready(Some(Ok(ref bytes))) = &r {
let bytes = bytes.clone();
spawn_new_task(async move {
// Do something with bytes
});
}
r
}
}
Filtering via async predicate, the "easy" way
One way would be to join_all!() the Futures that compute the filters on every item. And then filters synchronously based on those:
let arr = vec![...]
let filters = join_all!(arr.iter().map(|it| async { predicate(it).await })
let filtered = arr.enumerate().filter(|index, item| filters[index]).collect::<Vec<_>>();
However, exploring Rust, there's a cleaner way via futures::stream::iter iterators:
let filtered = futures::stream::iter(vec![...])
.filter(|item| async { predicate(item).await })
.collect::<Vec<_>>
.await
All good up to now.
Configurable filter: trouble begins
What if we want to use a functional API to make the predicate easily configurable?
In that case, our calls will look like:
let filtered = futures::stream::iter(vec![...])
.filter(by_length(4)) // neat!
.collect::<Vec<_>>
.await
And the predicate:
fn by_length(min_length: usize) -> impl FnMut(&i32) -> Future<Output = bool> {
|n| async { query_length(n).await > min_length }
}
async fn query_length(n: &i32) -> usize {
// pretend we're making a network request to fetch `len`...
// for easy reproducibility's sake this will work here
n.to_string().len()
}
Unfortunately compiler is not happy anymore: it complains the Future needs dyn keyword. And, after adding dyn, it complains it's not Sized, as in this minimal reproduction:
use futures::future::Future;
#[tokio::main]
async fn main() {
let arr = vec![10, 100, 1000];
let filtered = futures::stream::iter(arr.into_iter())
.filter(by_length(3))
.collect::<Vec<_>>()
.await;
println!("{:?}", filtered); // should print [100, 1000]
}
fn by_length(min_length: usize) -> impl FnMut(&i32) -> Future<Output = bool> {
|n| async { query_length(n).await > min_length }
}
// yeah it doesn't need to be async in this case, but let's pretend
async fn query_length(n: &i32) -> usize {
n.to_string().len()
}
The error:
Compiling playground v0.0.1 (/playground)
error[E0277]: the size for values of type `(dyn futures::Future<Output = bool> + 'static)` cannot be known at compilation time
--> src/main.rs:16:9
|
16 | |n| async { query_length(n).await > min_length }
| ^ doesn't have a size known at compile-time
|
= help: the trait `Sized` is not implemented for `(dyn futures::Future<Output = bool> + 'static)`
= note: the return type of a function must have a statically known size
Questions
So how can the filter predicate be made configurable?
From what I gather, the dyn keyword requested by the compiler here is to make it explicit it will rely automatic dispatch, incurring in additional overhead.
However we only need a single typed case here, for which we should be able to generate inline-able machine code. How can Rust be instructed to do that?
While you can't return an impl Future from an impl FnMut, you can return a boxed future, i.e. a dyn Future which must be boxed because it's in return position. After a bit of borrow checker tetris, we arrive to this:
fn by_length(min_length: usize) -> impl FnMut(&i32) -> Pin<Box<dyn Future<Output = bool>>> {
move |&n| Box::pin(async move { query_length(&n).await > min_length })
}
Playground
However we only need a single typed case here, for which we should be able to generate inline-able machine code. How can Rust be instructed to do that?
I don't think that's currently possible, at least not if you're invoking an async fn like query_length(). Consider a manually written implementation:
fn by_length(min_length: usize) -> impl FnMut(&i32) -> ByLength {
move |&n| ByLength { n }
}
Now, how do we define ByLength? It must implement Future, and its poll() simply transmits the result of polling query_length(n). But the future returned by query_length(n) could suspend multiple times, so ByLength must store the future so it can poll it as many times as needed - for example:
struct ByLength {
n: i32,
query_fut: Option<???>,
}
impl Future for ByLength {
type Output = usize;
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<usize> {
if self.query_fut.is_none() {
self.query_fut = Some(query_length(self.n));
}
self.query_fut.unwrap().poll(cx)
}
}
But now the problem becomes apparent: there is no type to substitute for ??? because query_length() is an async function which returns a future of an anonymous type. Making ByLength generic doesn't work because then we're back at the problem that a closure can't return a generic type that it provides. The signature we'd like would require higher-kinded types:
fn by_length(min_length: usize) -> impl for<T> FnMut(&i32) -> ByLength<T> {
move |&n| ByLength { n }
}
...but if we had that, we could just use query_length() directly:
fn by_length(min_length: usize) -> impl for<T: Future<Output = usize>> FnMut(&i32) -> T {
move |&n| async move { by_length(&n) }
}
I am having trouble expressing the lifetime of the return value of an Iterator implementation. How can I compile this code without changing the return value of the iterator? I'd like it to return a vector of references.
It is obvious that I am not using the lifetime parameter correctly but after trying various ways I just gave up, I have no idea what to do with it.
use std::iter::Iterator;
struct PermutationIterator<T> {
vs: Vec<Vec<T>>,
is: Vec<usize>,
}
impl<T> PermutationIterator<T> {
fn new() -> PermutationIterator<T> {
PermutationIterator {
vs: vec![],
is: vec![],
}
}
fn add(&mut self, v: Vec<T>) {
self.vs.push(v);
self.is.push(0);
}
}
impl<T> Iterator for PermutationIterator<T> {
type Item = Vec<&'a T>;
fn next(&mut self) -> Option<Vec<&T>> {
'outer: loop {
for i in 0..self.vs.len() {
if self.is[i] >= self.vs[i].len() {
if i == 0 {
return None; // we are done
}
self.is[i] = 0;
self.is[i - 1] += 1;
continue 'outer;
}
}
let mut result = vec![];
for i in 0..self.vs.len() {
let index = self.is[i];
result.push(self.vs[i].get(index).unwrap());
}
*self.is.last_mut().unwrap() += 1;
return Some(result);
}
}
}
fn main() {
let v1: Vec<_> = (1..3).collect();
let v2: Vec<_> = (3..5).collect();
let v3: Vec<_> = (1..6).collect();
let mut i = PermutationIterator::new();
i.add(v1);
i.add(v2);
i.add(v3);
loop {
match i.next() {
Some(v) => {
println!("{:?}", v);
}
None => {
break;
}
}
}
}
(Playground link)
error[E0261]: use of undeclared lifetime name `'a`
--> src/main.rs:23:22
|
23 | type Item = Vec<&'a T>;
| ^^ undeclared lifetime
As far as I understand, you want want the iterator to return a vector of references into itself, right? Unfortunately, it is not possible in Rust.
This is the trimmed down Iterator trait:
trait Iterator {
type Item;
fn next(&mut self) -> Option<Item>;
}
Note that there is no lifetime connection between &mut self and Option<Item>. This means that next() method can't return references into the iterator itself. You just can't express a lifetime of the returned references. This is basically the reason that you couldn't find a way to specify the correct lifetime - it would've looked like this:
fn next<'a>(&'a mut self) -> Option<Vec<&'a T>>
except that this is not a valid next() method for Iterator trait.
Such iterators (the ones which can return references into themselves) are called streaming iterators. You can find more here, here and here, if you want.
Update. However, you can return a reference to some other structure from your iterator - that's how most of collection iterators work. It could look like this:
pub struct PermutationIterator<'a, T> {
vs: &'a [Vec<T>],
is: Vec<usize>
}
impl<'a, T> Iterator for PermutationIterator<'a, T> {
type Item = Vec<&'a T>;
fn next(&mut self) -> Option<Vec<&'a T>> {
...
}
}
Note how lifetime 'a is now declared on impl block. It is OK to do so (required, in fact) because you need to specify the lifetime parameter on the structure. Then you can use the same 'a both in Item and in next() return type. Again, that's how most of collection iterators work.
#VladimirMatveev's answer is correct in how it explains why your code cannot compile. In a nutshell, it says that an Iterator cannot yield borrowed values from within itself.
However, it can yield borrowed values from something else. This is what is achieved with Vec and Iter: the Vec owns the values, and the the Iter is just a wrapper able to yield references within the Vec.
Here is a design which achieves what you want. The iterator is, like with Vec and Iter, just a wrapper over other containers who actually own the values.
use std::iter::Iterator;
struct PermutationIterator<'a, T: 'a> {
vs : Vec<&'a [T]>,
is : Vec<usize>
}
impl<'a, T> PermutationIterator<'a, T> {
fn new() -> PermutationIterator<'a, T> { ... }
fn add(&mut self, v : &'a [T]) { ... }
}
impl<'a, T> Iterator for PermutationIterator<'a, T> {
type Item = Vec<&'a T>;
fn next(&mut self) -> Option<Vec<&'a T>> { ... }
}
fn main() {
let v1 : Vec<i32> = (1..3).collect();
let v2 : Vec<i32> = (3..5).collect();
let v3 : Vec<i32> = (1..6).collect();
let mut i = PermutationIterator::new();
i.add(&v1);
i.add(&v2);
i.add(&v3);
loop {
match i.next() {
Some(v) => { println!("{:?}", v); }
None => {break;}
}
}
}
(Playground)
Unrelated to your initial problem. If this were just me, I would ensure that all borrowed vectors are taken at once. The idea is to remove the repeated calls to add and to pass directly all borrowed vectors at construction:
use std::iter::{Iterator, repeat};
struct PermutationIterator<'a, T: 'a> {
...
}
impl<'a, T> PermutationIterator<'a, T> {
fn new(vs: Vec<&'a [T]>) -> PermutationIterator<'a, T> {
let n = vs.len();
PermutationIterator {
vs: vs,
is: repeat(0).take(n).collect(),
}
}
}
impl<'a, T> Iterator for PermutationIterator<'a, T> {
...
}
fn main() {
let v1 : Vec<i32> = (1..3).collect();
let v2 : Vec<i32> = (3..5).collect();
let v3 : Vec<i32> = (1..6).collect();
let vall: Vec<&[i32]> = vec![&v1, &v2, &v3];
let mut i = PermutationIterator::new(vall);
}
(Playground)
(EDIT: Changed the iterator design to take a Vec<&'a [T]> rather than a Vec<Vec<&'a T>>. It's easier to take a ref to container than to build a container of refs.)
As mentioned in other answers, this is called a streaming iterator and it requires different guarantees from Rust's Iterator. One crate that provides such functionality is aptly called streaming-iterator and it provides the StreamingIterator trait.
Here is one example of implementing the trait:
extern crate streaming_iterator;
use streaming_iterator::StreamingIterator;
struct Demonstration {
scores: Vec<i32>,
position: usize,
}
// Since `StreamingIterator` requires that we be able to call
// `advance` before `get`, we have to start "before" the first
// element. We assume that there will never be the maximum number of
// entries in the `Vec`, so we use `usize::MAX` as our sentinel value.
impl Demonstration {
fn new() -> Self {
Demonstration {
scores: vec![1, 2, 3],
position: std::usize::MAX,
}
}
fn reset(&mut self) {
self.position = std::usize::MAX;
}
}
impl StreamingIterator for Demonstration {
type Item = i32;
fn advance(&mut self) {
self.position = self.position.wrapping_add(1);
}
fn get(&self) -> Option<&Self::Item> {
self.scores.get(self.position)
}
}
fn main() {
let mut example = Demonstration::new();
loop {
example.advance();
match example.get() {
Some(v) => {
println!("v: {}", v);
}
None => break,
}
}
example.reset();
loop {
example.advance();
match example.get() {
Some(v) => {
println!("v: {}", v);
}
None => break,
}
}
}
Unfortunately, streaming iterators will be limited until generic associated types (GATs) from RFC 1598 are implemented.
I wrote this code not long ago and somehow stumbled on this question here. It does exactly what the question asks: it shows how to implement an iterator that passes its callbacks a reference to itself.
It adds an .iter_map() method to IntoIterator instances. Initially I thought it should be implemented for Iterator itself, but that was a less flexible design decision.
I created a small crate for it and posted my code to GitHub in case you want to experiment with it, you can find it here.
WRT the OP's trouble with defining lifetimes for the items, I didn't run into any such trouble implementing this while relying on the default elided lifetimes.
Here's an example of usage. Note the parameter the callback receives is the iterator itself, the callback is expected to pull the data from it and either pass it along as is or do whatever other operations.
use iter_map::IntoIterMap;
let mut b = true;
let s = "hello world!".chars().peekable().iter_map(|iter| {
if let Some(&ch) = iter.peek() {
if ch == 'o' && b {
b = false;
Some('0')
} else {
b = true;
iter.next()
}
} else { None }
}).collect::<String>();
assert_eq!(&s, "hell0o w0orld!");
Because the IntoIterMap generic trait is implemented for IntoIterator, you can get an "iter map" off anything that supports that interface. For instance, one can be created directly off an array, like so:
use iter_map::*;
fn main()
{
let mut i = 0;
let v = [1, 2, 3, 4, 5, 6].iter_map(move |iter| {
i += 1;
if i % 3 == 0 {
Some(0)
} else {
iter.next().copied()
}
}).collect::<Vec<_>>();
assert_eq!(v, vec![1, 2, 0, 3, 4, 0, 5, 6, 0]);
}
Here's the full code - it was amazing it took such little code to implement, and everything just seemed to work smoothly while putting it together. It gave me a new appreciation for the flexibility of Rust itself and its design decisions.
/// Adds `.iter_map()` method to all IntoIterator classes.
///
impl<F, I, J, R, T> IntoIterMap<F, I, R, T> for J
//
where F: FnMut(&mut I) -> Option<R>,
I: Iterator<Item = T>,
J: IntoIterator<Item = T, IntoIter = I>,
{
/// Returns an iterator that invokes the callback in `.next()`, passing it
/// the original iterator as an argument. The callback can return any
/// arbitrary type within an `Option`.
///
fn iter_map(self, callback: F) -> ParamFromFnIter<F, I>
{
ParamFromFnIter::new(self.into_iter(), callback)
}
}
/// A trait to add the `.iter_map()` method to any existing class.
///
pub trait IntoIterMap<F, I, R, T>
//
where F: FnMut(&mut I) -> Option<R>,
I: Iterator<Item = T>,
{
/// Returns a `ParamFromFnIter` iterator which wraps the iterator it's
/// invoked on.
///
/// # Arguments
/// * `callback` - The callback that gets invoked by `.next()`.
/// This callback is passed the original iterator as its
/// parameter.
///
fn iter_map(self, callback: F) -> ParamFromFnIter<F, I>;
}
/// Implements an iterator that can be created from a callback.
/// does pretty much the same thing as `std::iter::from_fn()` except the
/// callback signature of this class takes a data argument.
pub struct ParamFromFnIter<F, D>
{
callback: F,
data: D,
}
impl<F, D, R> ParamFromFnIter<F, D>
//
where F: FnMut(&mut D) -> Option<R>,
{
/// Creates a new `ParamFromFnIter` iterator instance.
///
/// This provides a flexible and simple way to create new iterators by
/// defining a callback.
/// # Arguments
/// * `data` - Data that will be passed to the callback on each
/// invocation.
/// * `callback` - The callback that gets invoked when `.next()` is invoked
/// on the returned iterator.
///
pub fn new(data: D, callback: F) -> Self
{
ParamFromFnIter { callback, data }
}
}
/// Implements Iterator for ParamFromFnIter.
///
impl<F, D, R> Iterator for ParamFromFnIter<F, D>
//
where F: FnMut(&mut D) -> Option<R>,
{
type Item = R;
/// Iterator method that returns the next item.
/// Invokes the client code provided iterator, passing it `&mut self.data`.
///
fn next(&mut self) -> Option<Self::Item>
{
(self.callback)(&mut self.data)
}
}
I can't figure out how to provide a Stream where I await async functions to get the data needed for the values of the stream.
I've tried to implement the the Stream trait directly, but I run into issues because I'd like to use async things like awaiting, the compiler does not want me to call async functions.
I assume that I'm missing some background on what the goal of Stream is and I'm just attacking this incorrectly and perhaps I shouldn't be looking at Stream at all, but I don't know where else to turn. I've seen the other functions in the stream module that could be useful, but I'm unsure how I could store any state and use these functions.
As a slightly simplified version of my actual goal, I want to provide a stream of 64-byte Vecs from an AsyncRead object (i.e. TCP stream), but also store a little state inside whatever logic ends up producing values for the stream, in this example, a counter.
pub struct Receiver<T>
where
T: AsyncRead + Unpin,
{
readme: T,
num: u64,
}
// ..code for a simple `new() -> Self` function..
impl<T> Stream for Receiver<T>
where
T: AsyncRead + Unpin,
{
type Item = Result<Vec<u8>, io::Error>;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let mut buf: [u8; 64] = [0; 64];
match self.readme.read_exact(&mut buf).await {
Ok(()) => {
self.num += 1;
Poll::Ready(Some(Ok(buf.to_vec())))
}
Err(e) => Poll::Ready(Some(Err(e))),
}
}
}
This fails to build, saying
error[E0728]: `await` is only allowed inside `async` functions and blocks
I'm using rustc 1.36.0-nightly (d35181ad8 2019-05-20) and my Cargo.toml looks like this:
[dependencies]
futures-preview = { version = "0.3.0-alpha.16", features = ["compat", "io-compat"] }
pin-utils = "0.1.0-alpha.4"
Answer copy/pasted from the reddit post by user Matthias247:
It's unfortunately not possible at the moment - Streams have to be implemented by hand and can not utilize async fn. Whether it's possible to change this in the future is unclear.
You can work around it by defining a different Stream trait which makes use of Futures like:
trait Stream<T> {
type NextFuture: Future<Output=T>;
fn next(&mut self) -> Self::NextFuture;
}
This article and this futures-rs issue have more information around it.
You can do it with gen-stream crate:
#![feature(generators, generator_trait, gen_future)]
use {
futures::prelude::*,
gen_stream::{gen_await, GenTryStream},
pin_utils::unsafe_pinned,
std::{
io,
marker::PhantomData,
pin::Pin,
sync::{
atomic::{AtomicU64, Ordering},
Arc,
},
task::{Context, Poll},
},
};
pub type Inner = Pin<Box<dyn Stream<Item = Result<Vec<u8>, io::Error>> + Send>>;
pub struct Receiver<T> {
inner: Inner,
pub num: Arc<AtomicU64>,
_marker: PhantomData<T>,
}
impl<T> Receiver<T> {
unsafe_pinned!(inner: Inner);
}
impl<T> From<T> for Receiver<T>
where
T: AsyncRead + Unpin + Send + 'static,
{
fn from(mut readme: T) -> Self {
let num = Arc::new(AtomicU64::new(0));
Self {
inner: Box::pin(GenTryStream::from({
let num = num.clone();
static move || loop {
let mut buf: [u8; 64] = [0; 64];
match gen_await!(readme.read_exact(&mut buf)) {
Ok(()) => {
num.fetch_add(1, Ordering::Relaxed);
yield Poll::Ready(buf.to_vec())
}
Err(e) => return Err(e),
}
}
})),
num,
_marker: PhantomData,
}
}
}
impl<T> Stream for Receiver<T>
where
T: AsyncRead + Unpin,
{
type Item = Result<Vec<u8>, io::Error>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
self.inner().poll_next(cx)
}
}