Goal:
The server should be able to receive a stream of binary data and save it to a file.
I'm getting this error:
mismatched types
expected `&[u8]`, found type parameter `B`
How can I get a &[u8] from generic type B?
use warp::Filter;
use warp::{body};
use futures::stream::Stream;
async fn handle_upload<S, B>(stream: S) -> Result<impl warp::Reply, warp::Rejection>
where
S: Stream<Item = Result<B, warp::Error>>,
S: StreamExt,
B: warp::Buf
{
let mut file = File::create("some_binary_file").unwrap();
let pinnedStream = Box::pin(stream);
while let Some(item) = pinnedStream.next().await {
let data = item.unwrap();
file.write_all(data);
}
Ok(warp::reply())
}
#[tokio::main]
async fn main() {
pretty_env_logger::init();
let upload = warp::put()
.and(warp::path("stream"))
.and(body::stream())
.and_then(handle_upload);
warp::serve(upload).run(([127, 0, 0, 1], 3030)).await;
}
B implements warp::Buf which is re-exported from the bytes crate. It has a .bytes() method that returns a &[u8] which may work.
However, the documentation says that .bytes() may return a shorter slice than what it actually contains. So you can call .bytes() and .advance() the stream while it .has_remaining() OR convert it to Bytes and send that to the file:
let mut data = item.unwrap();
file.write_all(data.to_bytes().as_ref());
Related
Assuming you have a binary file example.bin and you want to read that file in units of f64, i.e. the first 8 bytes give a float, the next 8 bytes give a number, etc. (assuming you know endianess) How can this be done in Rust?
I know that one can use std::fs::read("example.bin") to get a Vec<u8> of the data, but then you have to do quite a bit of "gymnastics" to convert always 8 of the bytes to a f64, i.e.
fn eight_bytes_to_array(barry: &[u8]) -> &[u8; 8] {
barry.try_into().expect("slice with incorrect length")
}
let mut file_content = std::fs::read("example.bin").expect("Could not read file!");
let nr = eight_bytes_to_array(&file_content[0..8]);
let nr = f64::from_be_bytes(*nr_dp_per_spectrum);
I saw this post, but its from 2015 and a lot of changes have happend in Rust since then, so I was wondering if there is a better/faster way these days?
Example without proper error handling and checking for cases when file contains not divisible amount of bytes.
use std::fs::File;
use std::io::{BufReader, Read};
fn main() {
// Using BufReader because files in std is unbuffered by default
// And reading by 8 bytes is really bad idea.
let mut input = BufReader::new(
File::open("floats.bin")
.expect("Failed to open file")
);
let mut floats = Vec::new();
loop {
use std::io::ErrorKind;
// You may use 8 instead of `size_of` but size_of is less error-prone.
let mut buffer = [0u8; std::mem::size_of::<f64>()];
// Using read_exact because `read` may return less
// than 8 bytes even if there are bytes in the file.
// This, however, prevents us from handling cases
// when file size cannot be divided by 8.
let res = input.read_exact(&mut buffer);
match res {
// We detect if we read until the end.
// If there were some excess bytes after last read, they are lost.
Err(error) if error.kind() == ErrorKind::UnexpectedEof => break,
// Add more cases of errors you want to handle.
_ => {}
}
// You should do better error-handling probably.
// This simply panics.
res.expect("Unexpected error during read");
// Use `from_be_bytes` if numbers in file is big-endian
let f = f64::from_le_bytes(buffer);
floats.push(f);
}
}
I would create a generic iterator that returns f64 for flexibility and reusability.
struct F64Reader<R: io::BufRead> {
inner: R,
}
impl<R: io::BufRead> F64Reader<R> {
pub fn new(inner: R) -> Self {
Self{
inner
}
}
}
impl<R: io::BufRead> Iterator for F64Reader<R> {
type Item = f64;
fn next(&mut self) -> Option<Self::Item> {
let mut buff: [u8; 8] = [0;8];
self.inner.read_exact(&mut buff).ok()?;
Some(f64::from_be_bytes(buff))
}
}
This means if the file is large, you can loop through the values without storing it all in memory
let input = fs::File::open("example.bin")?;
for f in F64Reader::new(io::BufReader::new(input)) {
println!("{}", f)
}
Or if you want all the values you can collect them
let input = fs::File::open("example.bin")?;
let values : Vec<f64> = F64Reader::new(io::BufReader::new(input)).collect();
This question already has answers here:
How does the Iterator::collect function work?
(2 answers)
Closed 1 year ago.
i want to read a textfile and convert all lines into int values.
I use this code.
But what i really miss here is a "good" way of error handling.
use std::{
fs::File,
io::{prelude::*, BufReader},
path::Path
};
fn lines_from_file(filename: impl AsRef<Path>) -> Vec<i32> {
let file = File::open(filename).expect("no such file");
let buf = BufReader::new(file);
buf.lines()
.map(|l| l.expect("Could not parse line"))
.map(|l:String| l.parse::<i32>().expect("could not parse int"))
.collect()
}
Question: How to do proper error handling ?
Is this above example "good rust code" ?
or should i use something like this :
fn lines_from_file(filename: impl AsRef<Path>) -> Vec<i32> {
let file = File::open(filename).expect("no such file");
let buf = BufReader::new(file);
buf.lines()
.map(|l| l.expect("Could not parse line"))
.map(|l:String| match l.parse::<i32>() {
Ok(num) => num,
Err(e) => -1 //Do something here
}).collect()
}
You can actually collect into a Result<T, E>.
See docs
So you could collect into a Result<Vec<i32>, MyCustomErrorType>.
This works when you transform your iterator in an iterator which returns a Result<i32, MyCustomErrorType>. The iteration stops at the first Err you map.
Here's your working code example.
I used the thiserror crate for error handling
use std::{
fs::File,
io::{prelude::*, BufReader},
num::ParseIntError,
path::Path,
};
use thiserror::Error;
#[derive(Error, Debug)]
pub enum LineParseError {
#[error("Failed to read line")]
IoError(#[from] std::io::Error),
#[error("Failed to parse int")]
FailedToParseInt(#[from] ParseIntError),
}
fn lines_from_file(filename: impl AsRef<Path>) -> Result<Vec<i32>, LineParseError> {
let file = File::open(filename).expect("no such file");
let buf = BufReader::new(file);
buf.lines().map(|l| Ok(l?.parse()?)).collect()
}
Some small explanation of how the code works by breaking down this line of code:
buf.lines().map(|l| Ok(l?.parse()?)).collect()
Rust infers that we need to collect to a Result<Vec<i32>, LineParseError> because the return type of the function is Result<Vec<i32>, LineParseError>
In the mapping method we write l? this makes the map method return an Err if the l result contains an Err, the #[from] attribute on LineParseError::IoError takes care of the conversion
The .parse()? works the same way: #[from] on LineParseError::FailedToParseInt takes care of the conversion
Last but not least our method must return Ok(...) when the mapping does succeed, this makes the collect into a Result<Vec<i32>, LineParseError> possible.
I need to put some futures in a Vec for later joining. However if I try to collect it using an iterator, the compiler doesn't seem to be able to determine the type for the vector.
I'm trying to create a command line utility that accepts an arbitrary number of IP addresses, communicates with those remotes and collects the results for printing. The communication function works well, I've cut down the program to show the failure I need to understand.
use futures::future::join_all;
use itertools::Itertools;
use std::net::SocketAddr;
use std::str::from_utf8;
use std::fmt;
#[tokio::main(flavor = "current_thread")]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
let socket: Vec<SocketAddr> = vec![
"192.168.20.33:502".parse().unwrap(),
"192.168.20.34:502".parse().unwrap(),];
let async_vec = vec![
MyStruct::get(socket[0]),
MyStruct::get(socket[1]),];
// The above 3 lines happen to work to build a Vec because there are
// 2 sockets. But I need to build a Vec to join_all from an arbitary
// number of addresses. Why doesn't the line below work instead?
//let async_vec = socket.iter().map(|x| MyStruct::get(*x)).collect();
let rt = join_all(async_vec).await;
let results = rt.iter().map(|x| x.as_ref().unwrap().to_string()).join("\n");
let mut rvec: Vec<String> = results.split("\n").map(|x| x.to_string()).collect();
rvec.sort_by(|a, b| a[15..20].cmp(&b[15..20]));
println!("{}", rvec.join("\n"));
Ok(())
}
struct MyStruct {
serial: [u8; 12],
placeholder: String,
}
impl fmt::Display for MyStruct {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
let serial = match from_utf8(&self.serial) {
Ok(v) => v,
Err(_) => "(invalid)",
};
let lines = (1..4).map(|x| format!("{}, line{}, {}", serial, x, self.placeholder)).join("\n");
write!(f, "{}", lines)
}
}
impl MyStruct {
pub async fn get(sockaddr: SocketAddr) -> Result<MyStruct, Box<dyn std::error::Error>> {
let char = sockaddr.ip().to_string().chars().last().unwrap();
let rv = MyStruct{serial: [char as u8;12], placeholder: sockaddr.to_string(), };
Ok(rv)
}
}
This line:
let async_vec = socket.iter().map(|x| MyStruct::get(*x)).collect();
doesn't work because the compiler can't know that you want to collect everything into a Vec. You might want to collect into some other container (e.g. a linked list or a set). Therefore you need to tell the compiler the kind of container you want with:
let async_vec = socket.iter().map(|x| MyStruct::get(*x)).collect::<Vec::<_>>();
or:
let async_vec: Vec::<_> = socket.iter().map(|x| MyStruct::get(*x)).collect();
I'm trying to figure out build a feature which requires reading the contents of a file into a futures::stream::BoxStream but I'm having a tough time figuring out what I need to do.
I have figured out how to read a file byte by byte via Bytes which implements an iterator.
use std::fs::File;
use std::io::prelude::*;
use std::io::{BufReader, Bytes};
// TODO: Convert this to a async Stream
fn async_read() -> Box<dyn Iterator<Item = Result<u8, std::io::Error>>> {
let f = File::open("/dev/random").expect("Could not open file");
let reader = BufReader::new(f);
let iter = reader.bytes().into_iter();
Box::new(iter)
}
fn main() {
ctrlc::set_handler(move || {
println!("received Ctrl+C!");
std::process::exit(0);
})
.expect("Error setting Ctrl-C handler");
for b in async_read().into_iter() {
println!("{:?}", b);
}
}
However, I've been struggling a bunch trying to figure out how I can turn this Box<dyn Iterator<Item = Result<u8, std::io::Error>>> into an Stream.
I would have thought something like this would work:
use futures::stream;
use std::fs::File;
use std::io::prelude::*;
use std::io::{BufReader, Bytes};
// TODO: Convert this to a async Stream
fn async_read() -> stream::BoxStream<'static, dyn Iterator<Item = Result<u8, std::io::Error>>> {
let f = File::open("/dev/random").expect("Could not open file");
let reader = BufReader::new(f);
let iter = reader.bytes().into_iter();
std::pin::Pin::new(Box::new(stream::iter(iter)))
}
fn main() {
ctrlc::set_handler(move || {
println!("received Ctrl+C!");
std::process::exit(0);
})
.expect("Error setting Ctrl-C handler");
while let Some(b) = async_read().poll() {
println!("{:?}", b);
}
}
But I keep getting a ton of compiler errors, I've tried other permutations but generally getting no where.
One of the compiler errors:
std::pin::Pin::new
``` --> src/main.rs:14:24
|
14 | std::pin::Pin::new(Box::new(stream::iter(iter)))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected trait object `dyn std::iter::Iterator`, found enum `std::result::Result`
Anyone have any advice?
I'm pretty new to Rust, and specifically Streams/lower level stuff so I apologize if I got anything wrong, feel free to correct me.
For some additional background, I'm trying to do this so you can CTRL-C out of a command in nushell
I think you are overcomplicating it a bit, you can just return impl Stream from async_read, there is no need to box or pin (same goes for the original Iterator-based version). Then you need to set up an async runtime in order to poll the stream (in this example I just use the runtime provided by futures::executor::block_on). Then you can call futures::stream::StreamExt::next() on the stream to get a future representing the next item.
Here is one way to do this:
use futures::prelude::*;
use std::{
fs::File,
io::{prelude::*, BufReader},
};
fn async_read() -> impl Stream<Item = Result<u8, std::io::Error>> {
let f = File::open("/dev/random").expect("Could not open file");
let reader = BufReader::new(f);
stream::iter(reader.bytes())
}
async fn async_main() {
while let Some(b) = async_read().next().await {
println!("{:?}", b);
}
}
fn main() {
ctrlc::set_handler(move || {
println!("received Ctrl+C!");
std::process::exit(0);
})
.expect("Error setting Ctrl-C handler");
futures::executor::block_on(async_main());
}
I have this lib.rs file.
use std::io::{ Result, Read };
pub trait ReadExt: Read {
/// Read all bytes until EOF in this source, returning them as a new `Vec`.
///
/// See `read_to_end` for other semantics.
fn read_into_vec(&mut self) -> Result<Vec<u8>> {
let mut buf = Vec::new();
let res = self.read_to_end(&mut buf);
res.map(|_| buf)
}
/// Read all bytes until EOF in this source, returning them as a new buffer.
///
/// See `read_to_string` for other semantics.
fn read_into_string(&mut self) -> Result<String> {
let mut buf = String::new();
let res = self.read_to_string(&mut buf);
res.map(|_| buf)
}
}
impl<T> ReadExt for T where T: Read {}
And now I want to write tests for it in a separate test/lib.rs
extern crate readext;
use std::io::{Read,Cursor};
use readext::ReadExt;
#[test]
fn test () {
let bytes = b"hello";
let mut input = Cursor::new(bytes);
let s = input.read_into_string();
assert_eq!(s, "hello");
}
But Rust keeps telling me
type std::io::cursor::Cursor<&[u8; 5]> does not implement any method in scope named read_into_string
I don't know why. Obviously I'm useing it already. Confused.
The answer is already in the error:
type std::io::cursor::Cursor<&[u8; 5]> does not implement any method
in scope named read_into_string
The problem is, Cursor<&[u8; 5]> does not implement Read because the wrapped type is pointer to a fixed-size array instead of a slice, and so it does not implement your trait either. I guess something along these lines should work:
#[test]
fn test () {
let bytes = b"hello";
let mut input = Cursor::new(bytes as &[u8]);
let s = input.read_into_string();
assert_eq!(s, "hello");
}
This way input is of type Cursor<&[u8]> which implements Read and so should implement your trait too.