I'm trying to design a simple serialization function:
pub fn serialize(buffer: &[u8]) -> std::result::Result<(), ()> {
Ok(())
}
the above is not ok because the buffer passed to it could not handle the size for serialization.
What would be a good signature for a serialization function? Should I use a vec?
I'm trying to serialize an RTSP message. It's like HTTP. Not much text, it should be simple.
Use a writer, designated by the std::io::Write trait.
pub fn serialize<W: Write>(writer: &mut W) -> Result<()> {
writer.write_all(&[1, 2, 3, 4])?;
Ok(())
}
// or
pub fn serialize(writer: &mut dyn Write) -> Result<()> {
writer.write_all(&[1, 2, 3, 4])?;
Ok(())
}
This makes it generic over many types that can be used as "byte-oriented sinks" like Files, TcpStreams, and normal Vec<u8>s.
fn main() {
let mut buffer = vec![];
serialize(&mut buffer).unwrap();
println!("{:?}", buffer);
}
[1, 2, 3, 4]
See it running on the playground.
Related
The Read trait is implemented for &[u8]. How can I get a Read trait over several concatenated u8 slices without actually doing any concatenation first?
If I concatenate first, there will be two copies -- multiple arrays into a single array followed by copying from single array to destination via the Read trait. I would like to avoid the first copying.
I want a Read trait over &[&[u8]] that treats multiple slices as a single continuous slice.
fn foo<R: std::io::Read + Send>(data: R) {
// ...
}
let a: &[u8] = &[1, 2, 3, 4, 5];
let b: &[u8] = &[1, 2];
let c: &[&[u8]] = &[a, b];
foo(c); // <- this won't compile because `c` is not a slice of bytes.
You could use the multi_reader crate, which can concatenate any number of values that implement Read:
let a: &[u8] = &[1, 2, 3, 4, 5];
let b: &[u8] = &[1, 2];
let c: &[&[u8]] = &[a, b];
foo(multi_reader::MultiReader::new(c.iter().copied()));
If you don't want to depend on an external crate, you can wrap the slices in a struct of your own and implement Read for it:
struct MultiRead<'a> {
sources: &'a [&'a [u8]],
pos_in_current: usize,
}
impl<'a> MultiRead<'a> {
fn new(sources: &'a [&'a [u8]]) -> MultiRead<'a> {
MultiRead {
sources,
pos_in_current: 0,
}
}
}
impl Read for MultiRead<'_> {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
let current = loop {
if self.sources.is_empty() {
return Ok(0); // EOF
}
let current = self.sources[0];
if self.pos_in_current < current.len() {
break current;
}
self.pos_in_current = 0;
self.sources = &self.sources[1..];
};
let read_size = buf.len().min(current.len() - self.pos_in_current);
buf[..read_size].copy_from_slice(¤t[self.pos_in_current..][..read_size]);
self.pos_in_current += read_size;
Ok(read_size)
}
}
Playground
Create a wrapper type around the slices and implement Read for it. Compared to user4815162342's answer, I delegate down to the implementation of Read for slices:
use std::{io::Read, mem};
struct Wrapper<'a, 'b>(&'a mut [&'b [u8]]);
impl<'a, 'b> Read for Wrapper<'a, 'b> {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
let slices = mem::take(&mut self.0);
match slices {
[head, ..] => {
let n_bytes = head.read(buf)?;
if head.is_empty() {
// Advance the child slice
self.0 = &mut slices[1..];
} else {
// More to read, put back all the child slices
self.0 = slices;
}
Ok(n_bytes)
}
_ => Ok(0),
}
}
}
fn main() {
let parts: &mut [&[u8]] = &mut [b"hello ", b"world"];
let mut w = Wrapper(parts);
let mut buf = Vec::new();
w.read_to_end(&mut buf).unwrap();
assert_eq!(b"hello world", &*buf);
}
A more efficient implementation would implement further methods from Read, such as read_to_end or read_vectored.
See also:
How do I implement a trait I don't own for a type I don't own?
I needed to implement a custom Stream that yields items in a sliding window (ie. [1, 2, 3] => [(1, 2), (2, 3)]). So I implemented and gave it an adapter called .tuple_windows(). Allowing the following code
let iter = stream::iter(0..=3);
assert_eq!(
iter.tuple_windows().collect::<Vec<_>>().await,
vec![(0, 1), (1, 2), (2, 3)]
)
I ran into a weird situation when chaining other adapters with it where the final type doesn't implement the Stream trait.
code (playground):
use anyhow; // 1.0.52
use futures; // 0.3.19
use futures::{stream, Stream, StreamExt};
use pin_project_lite;
use pin_project_lite::pin_project;
use std::{
pin::Pin,
task::{Context, Poll},
};
use tokio; // 1.15.0 // 0.2.8
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let mut stream = stream::iter(0..20)
.map(|_| stream::iter(2..10)) // this works with the custom Stream
// .map(|_| stream::iter(2..10).enumerate()) // but this doesn't
.enumerate(); // this works regardless what happens in `map`
// .tuple_windows(); // this only works with the first map
while let Some(_) = stream.next().await {}
Ok(())
}
impl<T: Stream> TupleWindowsExt for T {}
pub trait TupleWindowsExt: Stream + Sized {
fn tuple_windows(self) -> TupleWindows<Self> {
TupleWindows::new(self)
}
}
pin_project! {
#[derive(Debug)]
pub struct TupleWindows<S: Stream> {
#[pin]
stream: S,
previous: Option<S::Item>,
}
}
impl<S: Stream> TupleWindows<S> {
pub fn new(stream: S) -> Self {
Self {
stream,
previous: None,
}
}
}
impl<S: Stream> Stream for TupleWindows<S>
where
S::Item: Clone,
{
type Item = (S::Item, S::Item);
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let mut this = self.project();
let current = match futures::ready!(this.stream.as_mut().poll_next(cx)) {
Some(next) => next,
None => return Poll::Ready(None),
};
if let Some(previous) = this.previous {
let res = (previous.clone(), current.clone());
*this.previous = Some(current);
Poll::Ready(Some(res))
} else {
let next = match this.stream.poll_next(cx) {
Poll::Ready(next) => next,
Poll::Pending => {
*this.previous = Some(current);
return Poll::Pending;
}
};
*this.previous = next.clone();
Poll::Ready(next.map(|next| (current, next)))
}
}
fn size_hint(&self) -> (usize, Option<usize>) {
let (lower, upper) = self.stream.size_hint();
(
lower.saturating_mul(2),
upper.and_then(|upper| upper.checked_mul(2)),
)
}
}
The compiler error is not helpful either as it only tells me that Stream is not implemented for the newly created type:
error[E0599]: the method `next` exists for struct `TupleWindows<futures::stream::Map<futures::stream::Iter<std::ops::Range<{integer}>>, [closure#src/main.rs:16:11: 16:46]>>`, but its trait bounds were not satisfied
What am I missing here ?
Your Stream implementation requires that items from the inner stream be cloneable:
impl<S: Stream> Stream for TupleWindows<S>
where
S::Item: Clone,
In the working case, stream::iter(0..20).map(|_| stream::iter(2..10)).tuple_windows(), you're passing an stream of futures::stream::Iter<std::ops::Range<i32>> items to tuple_windows(). Iter implements Clone when the inner iterator type implements Clone. The inner iterator type here is std::ops::Range<i32>, which does implement Clone.
When you change the code to add a call to enumerate() within the map(), you're now passing an stream of futures::stream::Enumerate<futures::stream::Iter<std::ops::Range<i32>>> items (i.e. a stream of streams) to tuple_windows(). Enumerate doesn't implement Clone at all (as of futures 0.3.19).
I can't see any reason why Enumerate couldn't implement Clone (with the appropriate trait bounds); I suppose it wasn't implemented because nobody asked for it.
Got it, the Stream impl doesn't work because the items don't satisfy Clone which is required by my custom Stream
I should have put the Clone bound on the adapter as well as the impl block
impl<T: ?Sized> TupleWindowsExt for T where T: Stream {}
pub trait TupleWindowsExt: Stream {
fn tuple_windows(self) -> TupleWindows<Self>
where
Self: Sized,
Self::Item: Clone,
{
TupleWindows::new(self)
}
}
I'm trying to use a struct with multiple implementations of one method:
trait Trait { fn apply(&self) -> vec<usize>; }
struct Bar<X> { vec: Vec<usize> }
impl<X> Bar<X> {
pub fn new(vec: Vec<usize>) -> Self { Self{vec} }
pub fn test(&self) {
// Things here
println!("Method: {:?}", self.apply());
// Things there
}
}
impl Trait for Bar<ThisWay> {
fn apply(&self) -> Vec<usize> { self.vec.iter().map(|x| x.pow(2)).collect() }
}
impl Trait for Bar<ThatWay> {
fn apply(&self) -> Vec<usize> { self.vec.iter().map(|x| x + 2).collect() }
}
fn main() {
Bar<ThisWay>::new(vec![1,2,3]).test();
Bar<ThatWay>::new(vec![1,2,3]).test();
}
Which would return:
>>> [1,4,9];
>>> [3,4,5];
I know I could create 2 structs implementing these methods differently, but that feels wrong as it's potentially a lot of redundant code.
I also know I could have a reference to that implementation method:
trait Trait { fn apply(vec: &Vec<usize>) -> Vec<usize>; }
impl Struct{
// fn new
test(&self, t: &impl Trait) {
// Things here
println!("{:?}", t::apply(&self.vec));
// Things there
}
}
struct ThisWay;
struct ThatWay;
impl Trait for ThisWay {fn apply(vec: &Vec<usize>) -> Vec<usize> {///} };
impl Trait for ThatWay {fn apply(vec: &Vec<usize>) -> Vec<usize> {///} };
fn main() {
let this_way = ThisWay{};
let that_way = ThatWay{};
let problem = Bar::new(vec![1,2,3]);
problem.test(&this_way);
problem.test(&that_way);
}
This approach seems needlessly complicated when I would want to use many arguments inside given struct:
fn hill_climber(&self, nullary_op: &impl NullaryOperator, unary_op: &impl UnaryOperator, ...) {
self.vec = nullary_op();
self.vec = unary_op(&self.vec, self.n, self.m, self.jobs, self.stuff, ...);
}
This seems to be a cursed way of writing code. What happens when a method implementation doesn't use a parameter e.g m, and other uses that?
Traits are used to define shared behaviour. In your example, you want to implement the same trait in different ways. This goes against the purpose of a trait. Instead of having two structs as you tried, you should probably have two traits:
trait ThisWay {
fn apply(&self) -> Vec<usize>;
}
trait ThatWay {
fn apply(&self) -> Vec<usize>;
}
Now you can implement both traits for your struct:
struct Bar {
vec: Vec<usize>,
}
impl ThisWay for Bar {
fn apply(&self) -> Vec<usize> {
self.vec.iter().map(|x| x.pow(2)).collect()
}
}
impl ThatWay for Bar {
fn apply(&self) -> Vec<usize> {
self.vec.iter().map(|x| x + 2).collect()
}
}
Because Bar implements ThisWay and ThatWay, it now has two definitions for the apply method. To disambiguate between them, we have to use Fully Qualified Syntax:
let this_bar = Bar::new(vec![1, 2, 3]);
println!("Method: {:?}", <Bar as ThisWay>::apply(&this_bar));
let that_bar = Bar::new(vec![1, 2, 3]);
println!("Method: {:?}", <Bar as ThatWay>::apply(&that_bar));
And, as expected, you get two different outputs:
Method: [1, 4, 9]
Method: [3, 4, 5]
As an alternative to the other answer, you can also use something more similar to your original example, using generics and zero-sized struct types as "markers" for which method you want to use. Here's a complete example:
// PhantomData allows us to "use" a generic without having an actual field
use std::marker::PhantomData;
// These structs will be used to indicate which implementation we want
struct ThisWay;
struct ThatWay;
trait Trait { fn apply(&self) -> Vec<usize>; }
struct Bar<X> {
vec: Vec<usize>,
// This extra field is here to stop the compiler complaining about not using X
_marker: PhantomData<X>,
}
impl<X> Bar<X> {
pub fn new(vec: Vec<usize>) -> Self { Self { vec, _marker: PhantomData } }
// Note the new "where" clause here - we can only implement this function if Bar<X> implements Trait
pub fn test(&self) where Self: Trait {
// Things here
println!("Method: {:?}", self.apply());
// Things there
}
}
impl Trait for Bar<ThisWay> {
fn apply(&self) -> Vec<usize> { self.vec.iter().map(|x| x.pow(2)).collect() }
}
impl Trait for Bar<ThatWay> {
fn apply(&self) -> Vec<usize> { self.vec.iter().map(|x| x + 2).collect() }
}
fn main() {
Bar::<ThisWay>::new(vec![1,2,3]).test();
Bar::<ThatWay>::new(vec![1,2,3]).test();
}
Running this, the output correctly reflects the different functions being used:
Method: [1, 4, 9]
Method: [3, 4, 5]
This approach has different semantics from the other answer: whereas the other answer allows you to construct a Bar that's capable of being used with both functions, this approach locks you into one implementation at the type level, since Bar<ThisWay> and Bar<ThatWay> are two separate types that each only provide only one apply function. This can be desirable for type safety in some scenarios, but may not be what you need for this particular case.
// since x is declared as `Bar<ThatWay>`, we can only ever use the `ThatWay` implementation of `apply`/`test`
let x: Bar<ThatWay> = Bar::new(vec![1, 2, 3]);
x.test(); // -> Method: [3, 4, 5]
I can't figure out how to provide a Stream where I await async functions to get the data needed for the values of the stream.
I've tried to implement the the Stream trait directly, but I run into issues because I'd like to use async things like awaiting, the compiler does not want me to call async functions.
I assume that I'm missing some background on what the goal of Stream is and I'm just attacking this incorrectly and perhaps I shouldn't be looking at Stream at all, but I don't know where else to turn. I've seen the other functions in the stream module that could be useful, but I'm unsure how I could store any state and use these functions.
As a slightly simplified version of my actual goal, I want to provide a stream of 64-byte Vecs from an AsyncRead object (i.e. TCP stream), but also store a little state inside whatever logic ends up producing values for the stream, in this example, a counter.
pub struct Receiver<T>
where
T: AsyncRead + Unpin,
{
readme: T,
num: u64,
}
// ..code for a simple `new() -> Self` function..
impl<T> Stream for Receiver<T>
where
T: AsyncRead + Unpin,
{
type Item = Result<Vec<u8>, io::Error>;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let mut buf: [u8; 64] = [0; 64];
match self.readme.read_exact(&mut buf).await {
Ok(()) => {
self.num += 1;
Poll::Ready(Some(Ok(buf.to_vec())))
}
Err(e) => Poll::Ready(Some(Err(e))),
}
}
}
This fails to build, saying
error[E0728]: `await` is only allowed inside `async` functions and blocks
I'm using rustc 1.36.0-nightly (d35181ad8 2019-05-20) and my Cargo.toml looks like this:
[dependencies]
futures-preview = { version = "0.3.0-alpha.16", features = ["compat", "io-compat"] }
pin-utils = "0.1.0-alpha.4"
Answer copy/pasted from the reddit post by user Matthias247:
It's unfortunately not possible at the moment - Streams have to be implemented by hand and can not utilize async fn. Whether it's possible to change this in the future is unclear.
You can work around it by defining a different Stream trait which makes use of Futures like:
trait Stream<T> {
type NextFuture: Future<Output=T>;
fn next(&mut self) -> Self::NextFuture;
}
This article and this futures-rs issue have more information around it.
You can do it with gen-stream crate:
#![feature(generators, generator_trait, gen_future)]
use {
futures::prelude::*,
gen_stream::{gen_await, GenTryStream},
pin_utils::unsafe_pinned,
std::{
io,
marker::PhantomData,
pin::Pin,
sync::{
atomic::{AtomicU64, Ordering},
Arc,
},
task::{Context, Poll},
},
};
pub type Inner = Pin<Box<dyn Stream<Item = Result<Vec<u8>, io::Error>> + Send>>;
pub struct Receiver<T> {
inner: Inner,
pub num: Arc<AtomicU64>,
_marker: PhantomData<T>,
}
impl<T> Receiver<T> {
unsafe_pinned!(inner: Inner);
}
impl<T> From<T> for Receiver<T>
where
T: AsyncRead + Unpin + Send + 'static,
{
fn from(mut readme: T) -> Self {
let num = Arc::new(AtomicU64::new(0));
Self {
inner: Box::pin(GenTryStream::from({
let num = num.clone();
static move || loop {
let mut buf: [u8; 64] = [0; 64];
match gen_await!(readme.read_exact(&mut buf)) {
Ok(()) => {
num.fetch_add(1, Ordering::Relaxed);
yield Poll::Ready(buf.to_vec())
}
Err(e) => return Err(e),
}
}
})),
num,
_marker: PhantomData,
}
}
}
impl<T> Stream for Receiver<T>
where
T: AsyncRead + Unpin,
{
type Item = Result<Vec<u8>, io::Error>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
self.inner().poll_next(cx)
}
}
This question already has an answer here:
How can I add new methods to Iterator?
(1 answer)
Closed 4 years ago.
Let's say I want to double each value in an iterator of numbers. I could do this:
vec![1, 2, 3]
.into_iter()
.map(|x| x * 2)
.for_each(|x| println!("{}", x)); //Prints 2, 4, 6.
To get cleaner code, I would prefer to do this:
vec![1, 2, 3]
.into_iter()
.double() //I need to implement this.
.for_each(|x| println!("{}", x));
How do I write my own chainable iterator function, like double in this example? I guess I will have to create an interface and implement it for Iterator? There are a lot of types to get right, so a working solution for this silly example would be helpful.
First you have to create a new-type to implement your iterator. The simplest way is to wrap a iterator of the type you want to target:
struct Double<T: ?Sized> {
inner: T,
}
Now you just implement the Iterator trait for this type if the innert type T is of the correct type. Something like:
impl<T> Iterator for Double<T>
where T: Iterator<Item = i32> {
type Item = i32;
fn next(&mut self) -> Option<i32> {
self.inner.next().map(|x| 2*x)
}
}
And that's it! You just need to add a Double constructor. The most ergonomic solution is to extend Iterator. Something like this:
trait Doubler : Iterator<Item = i32> {
fn into_double(self) -> Double<Self>;
}
impl<T> Doubler for T
where T: Iterator<Item = i32> {
fn into_double(self) -> Double<Self> {
Double { inner: self }
}
}
An example of usage (playground):
fn main() {
vec![1, 2, 3]
.into_iter()
.into_double()
.for_each(|x| println!("{}", x));
}