This question already has answers here:
How to idiomatically / efficiently pipe data from Read+Seek to Write?
(1 answer)
Why does the usage of by_ref().take() differ between the Iterator and Read traits?
(2 answers)
Closed 4 years ago.
The community reviewed whether to reopen this question last year and left it closed:
Original close reason(s) were not resolved
I was trying to write a function that copies a user-specified number of bytes from a reader to a writer, and I came up with this:
fn io_copy(
reader: &mut std::io::Read,
writer: &mut std::io::Write,
byte_count: usize,
) -> std::io::Result<()> {
let mut buffer: [u8; 16384] = unsafe { std::mem::uninitialized() };
let mut remaining = byte_count;
while remaining > 0 {
let to_read = if remaining > 16384 { 16384 } else { remaining };
reader.read_exact(&mut buffer[0..to_read])?;
remaining -= to_read;
writer.write(&buffer[0..to_read])?;
}
Ok(())
}
It works fine, but I wanted to do it without an arbitrarily sized intermediate buffer, and I wondered if such a function already existed. I found std::io::copy, but that copies the whole stream, and I only want to copy a limited amount. I figured I could use take on the reader, but I'm having trouble getting rid of errors. This is what I have so far:
fn io_copy<R>(reader: &mut R, writer: &mut std::io::Write, byte_count: usize) -> std::io::Result<()>
where
R: std::io::Read + Sized,
{
let mut r = reader.by_ref().take(byte_count as u64);
std::io::copy(&mut r, writer)?;
Ok(())
}
This gives me an error:
error[E0507]: cannot move out of borrowed content
--> src/lib.rs:6:21
|
6 | let mut r = reader.by_ref().take(byte_count as u64);
| ^^^^^^^^^^^^^^^ cannot move out of borrowed content
I don't understand how to get around this.
I don't think you're going to get much better than that with just the generic Read/Write interfaces (except you probably shouldn't use read_exact in the case where you're ok with filling the entire buffer, that might lead to blocking unnecessarily).
You might be able to specialise it for specific Reads and Writes though, for example you might be able to use Kernel functions (like sendfile when reading from and writing to file descriptors in Linux) which might allow you to avoid having to copy things through userspace unnecessarily.
My implementation of copy_n would look like this (playground)
pub fn copy_n<R: ?Sized, W: ?Sized>(reader: &mut R, writer: &mut W, count: usize) -> io::Result<()>
where R: Read, W: Write
{
let mut buf = vec![];
unsafe {
buf.reserve(count);
buf.set_len(count);
}
reader.read_exact(&mut buf)?;
writer.write_all(&buf)?;
Ok(())
}
This solution just uses read_exact and write_all, which guarantee, that the complete buffer is read/written, or an error occurs, so it should be fine.
Related
Background (Skippable)
On linux, the file /var/run/utmp contains several utmp structures, each in raw binary format, following each other in a file. utmp itself is a relatively large (384 bytes on my machine). I am trying to read this file to it's raw data, and them implement checks after the fact that the data makes sense. I'm not new to rust, but this is my first real experience with the unsafe side of things.
Problem Statement
I have a file that contains several c sturct utmps (docs). In rust, I would like to read the entire file into an array of Vec<libc::utmpx>. More specifically, given a reader open to this file, how could I read one struct utmp?
What I have so far
Below are three different implementations of read_raw, which accepts a reader and returns a RawEntry(my alias for struct utmp). Which method is most correct? I am trying to write as performant code as possible, and I am worried that read_raw0 might be slower than the others if it involves memcpys. What is the best/fastest way to accomplish this behavior?
use std::io::Read;
use libc::utmpx as RawEntry;
const RawEntrySize = std::mem::size_of::<RawEntry>();
type RawEntryBuffer = [u8; RawEntrySize];
/// Read a raw utmpx struct
// After testing, this method doesn't work
pub fn read_raw0<R: Read>(reader: &mut R) -> RawEntry {
let mut entry: RawEntry = unsafe { std::mem::zeroed() };
unsafe {
let mut entry_buf = std::mem::transmute::<RawEntry, RawEntryBuffer>(entry);
reader.read_exact(&mut entry_buf[..]);
}
return entry;
}
/// Read a raw utmpx struct
pub fn read_raw1<R: Read>(reader: &mut R) -> RawEntry {
// Worried this could cause alignment issues, or maybe it's okay
// because transmute copies
let mut buffer: RawEntryBuffer = [0; RawEntrySize];
reader.read_exact(&mut buffer[..]);
let entry = unsafe {
std::mem::transmute::<RawEntryBuffer, RawEntry>(buffer)
};
return entry;
}
/// Read a raw utmpx struct
pub fn read_raw2<R: Read>(reader: &mut R) -> RawEntry {
let mut entry: RawEntry = unsafe { std::mem::zeroed() };
unsafe {
let entry_ptr = std::mem::transmute::<&mut RawEntry, *mut u8>(&mut entry);
let entry_slice = std::slice::from_raw_parts_mut(entry_ptr, RawEntrySize);
reader.read_exact(entry_slice);
}
return entry;
}
Note: After more testing, it appears read_raw0 doesn't work. I believe this is because transmute creates a new buffer instead of referencing the struct.
This is what I came up with, which I imagine should be about as fast as it gets to read a single entry. It follows the spirit of your last entry, but avoids the transmute (Transmuting &mut T to *mut u8 can be done with two casts: t as *mut T as *mut u8). Also it uses MaybeUninit instead of zeroed to be a bit more explicit (The assembly is likely the same once optimized). Lastly, the function will be unsafe either way, so we may as well mark it as such and do away with the unsafe blocks.
use std::io::{self, Read};
use std::slice::from_raw_parts_mut;
use std::mem::{MaybeUninit, size_of};
pub unsafe fn read_raw_struct<R: Read, T: Sized>(src: &mut R) -> io::Result<T> {
let mut buffer = MaybeUninit::uninit();
let buffer_slice = from_raw_parts_mut(buffer.as_mut_ptr() as *mut u8, size_of::<T>());
src.read_exact(buffer_slice)?;
Ok(buffer.assume_init())
}
To read the bytes of a PNG file, I want to create a function called read_8_bytes which will read the next 8 bytes in the file each time it's called.
fn main(){
let png = File::open("test.png").expect("1");
let mut png_reader = BufReader::new(png);
let mut byteBuffer: Vec<u8> = vec![0;8];
png_reader.read_exact(&mut byteBuffer).expect("2");
}
This works fine and if I keep calling read_exact from main I can read the next 8 bytes. I tried to create a function to do this and the solution just seems needlessly complicated. I'm wondering if there is a better way.
I thought I have to pass the BufReader to the function, but due to how Rust works this makes things complicated and I end up working out I need to do something like:
fn read_eight_bytes<R: BufRead>(fd: &mut R)
This compiles but I'm not happy because I don't understand why this needed to be done and seems complex. Is there a simple way of having a function I can pass a file descriptor type thing to and have it store the position like in C without having to do this?
Looking at your question, I think you are trying to say that you are confused as to why the <R: BufRead> is necessary or furthermore why this even works.
In your example, this generic is not strictly necessary. One could implement the function you describe like so:
use std::{fs, io};
fn main() -> io::Result<()> {
let mut file = fs::File::open("./path/to/file")?;
let bytes = read_eight_bytes(&mut file)?;
println!("{:?}", bytes);
Ok(())
}
fn read_eight_bytes(file: &mut fs::File) -> io::Result<[u8; 8]> {
use io::Read;
let mut bytes = [0; 8];
file.read_exact(&mut bytes)?;
Ok(bytes)
}
Playground
This is perfectly valid and hopefully should make sense.
But then, why does fn read_eight_bytes<R: BufRead>(file: &mut R) -> [u8; 8] work? First of all, I assume you understand the following concepts:
Generics
Traits
Given an understanding of the above concepts, you should know that this syntax means that the function read_eight_bytes is a generic function with a generic type named R. You should then also understand that the generic has a trait bound, requiring the type R to implement BufRead. And that this function takes a parameter which is a mutable reference to the variable file, which is of the type R.
Now taking a look at the definition of BufRead: we see that it contains several functions. But surprisingly there is no read_exact function! Why does a function like this compile?
use std::{fs, io};
use io::BufRead;
fn main() -> io::Result<()> {
let file = fs::File::open("./path/to/file")?;
let mut reader = io::BufReader::new(file);
let bytes = read_eight_bytes(&mut reader)?;
println!("{:?}", bytes);
Ok(())
}
fn read_eight_bytes<R: BufRead>(reader: &mut R) -> io::Result<[u8; 8]> {
let mut bytes = [0; 8];
reader.read_exact(&mut bytes)?;
Ok(bytes)
}
Playground
Note: I have altered the return type to io::Result<...>. This is considered to be a better practice compared to unwraping every Result.
I have also changed the function call to use a BufReader because BufReader implements BufRead whilst File does not. I will cover the difference a little further below.
The reason this works is because BufRead is a Super Trait. This means that any type that implements BufRead must also implement Read too. And thus it must have the read_exact function!
Given our function never requires the functions on BufRead we could change the trait bound to only require Read:
use std::{fs, io};
use io::Read;
fn main() -> io::Result<()> {
let file = fs::File::open("./path/to/file")?;
let mut reader = io::BufReader::new(file);
let bytes = read_eight_bytes(&mut reader)?;
println!("{:?}", bytes);
Ok(())
}
fn read_eight_bytes<R: Read>(reader: &mut R) -> io::Result<[u8; 8]> {
let mut bytes = [0; 8];
reader.read_exact(&mut bytes)?;
Ok(bytes)
}
Playground
Now here is something interesting about this change. The read_eight_bytes function can now be called in (at least) two different ways:
use std::{fs, io};
use io::Read;
fn main() -> io::Result<()> {
let mut file = fs::File::open("./path/to/file")?;
let bytes = read_eight_bytes(&mut file)?;
println!("{:?}", bytes);
let file = fs::File::open("./path/to/file")?;
let mut reader = io::BufReader::new(file);
let bytes = read_eight_bytes(&mut reader)?;
println!("{:?}", bytes);
Ok(())
}
fn read_eight_bytes<R: Read>(reader: &mut R) -> io::Result<[u8; 8]> {
let mut bytes = [0; 8];
reader.read_exact(&mut bytes)?;
Ok(bytes)
}
Playground
Why is this? This is because both File and BufReader implement the Read trait. And thus can both be used with the read_eight_bytes function!
So then why would someone want to use either File or BufReader over the other?
Well the BufReader documentation explains this:
The BufReader struct adds buffering to any reader.
It can be excessively inefficient to work directly with a Read
instance. For example, every call to read on TcpStream results in a
system call. A BufReader performs large, infrequent reads on the
underlying Read and maintains an in-memory buffer of the results.
BufReader can improve the speed of programs that make small and
repeated read calls to the same file or network socket. It does not
help when reading very large amounts at once, or reading just one or a
few times. It also provides no advantage when reading from a source
that is already in memory, like a Vec.
Now, remember how before we wrote this function just for the File type? The primary reason why one would want to write it with generics would be such that a caller can make the choice presented above. This is common practice in libraries where such a choice really does matter. However, generics come at the cost of increased compile times (when used excessively) and increased code complexity.
I met this problem while implementing AsyncRead over a synchronized read to adjust to the async world in Rust.
The sync read implementation I'm handling is a wrapper over a raw C sync implementation, much like the std::fs::File::read; therefore I would use std::io::Read for simplicity hereafter.
Here's the code:
use futures::{AsyncRead, Future};
use std::task::{Context, Poll};
use std::pin::Pin;
use tokio::task;
use std::fs::File;
use std::io::Read;
use std::io::Result;
struct FileAsyncRead {
path: String
}
impl AsyncRead for FileAsyncRead {
fn poll_read(self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut [u8]) -> Poll<Result<usize>> {
let path = self.path.to_owned();
let buf_len = buf.len();
let mut handle = task::spawn_blocking(move || {
let mut vec = vec![0u8; buf_len];
let mut file = File::open(path).unwrap();
let len = file.read(vec.as_mut_slice());
(vec, len)
});
match Pin::new(&mut handle).poll(cx) {
Poll::Ready(l) => {
let v_l = l.unwrap();
let _c_l = v_l.0.as_slice().read(buf);
Poll::Ready(v_l.1)
}
Poll::Pending => Poll::Pending
}
}
}
The current implementation is creating a new vector of the same size with the outer buf: &mut [u8] each time because of :
`buf` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement
buf: &mut [u8],
| --------- this data with an anonymous lifetime `'_`...
My question is:
Is that possible to avoid the vector creation in spwan_blocking and mutate the buf in poll_read? To avoid vector allocation as well as copying?
Is there a better way to express this "wrapper" logic instead of spawn_blocking as well as Pin::new(&mut handle).poll(cx)? What's the more idiomatic way to do this in Rust?
Something is odd about this code:
If this code is called once, it will likely return Poll::Pending, because spawn_blocking takes time to even start a task.
If this is called multiple times, then it creates multiple unrelated tasks reading the same part of the file and potentially ignoring the result due to (1), which is probably not what you want.
What you could do to fix this is to remember the task inside the FileAsyncRead struct first time you create it, and then on the next call only start a new task if needed, and poll the existing task.
With this API you have it doesn't seem possible to avoid double buffering, because since your API is blocking, and the ReadBuf buffer is not shared, you need to do a blocking read into some other buffer, and then copy the data over when a new non-blocking call poll_read() arrives.
Vecs support std::io::Write, so code can be written that takes a File or Vec, for example. From the API reference, it looks like neither Vec nor slices support std::io::Read.
Is there a convenient way to achieve this? Does it require writing a wrapper struct?
Here is an example of working code, that reads and writes a file, with a single line commented that should read a vector.
use ::std::io;
// Generic IO
fn write_4_bytes<W>(mut file: W) -> Result<usize, io::Error>
where W: io::Write,
{
let len = file.write(b"1234")?;
Ok(len)
}
fn read_4_bytes<R>(mut file: R) -> Result<[u8; 4], io::Error>
where R: io::Read,
{
let mut buf: [u8; 4] = [0; 4];
file.read(&mut buf)?;
Ok(buf)
}
// Type specific
fn write_read_vec() {
let mut vec_as_file: Vec<u8> = Vec::new();
{ // Write
println!("Writing Vec... {}", write_4_bytes(&mut vec_as_file).unwrap());
}
{ // Read
// println!("Reading File... {:?}", read_4_bytes(&vec_as_file).unwrap());
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
// Comment this line above to avoid an error!
}
}
fn write_read_file() {
let filepath = "temp.txt";
{ // Write
let mut file_as_file = ::std::fs::File::create(filepath).expect("open failed");
println!("Writing File... {}", write_4_bytes(&mut file_as_file).unwrap());
}
{ // Read
let mut file_as_file = ::std::fs::File::open(filepath).expect("open failed");
println!("Reading File... {:?}", read_4_bytes(&mut file_as_file).unwrap());
}
}
fn main() {
write_read_vec();
write_read_file();
}
This fails with the error:
error[E0277]: the trait bound `std::vec::Vec<u8>: std::io::Read` is not satisfied
--> src/main.rs:29:42
|
29 | println!("Reading File... {:?}", read_4_bytes(&vec_as_file).unwrap());
| ^^^^^^^^^^^^ the trait `std::io::Read` is not implemented for `std::vec::Vec<u8>`
|
= note: required by `read_4_bytes`
I'd like to write tests for a file format encoder/decoder, without having to write to the file-system.
While vectors don't support std::io::Read, slices do.
There is some confusion here caused by Rust being able to coerce a Vec into a slice in some situations but not others.
In this case, an explicit coercion to a slice is needed because at the stage coercions are applied, the compiler doesn't know that Vec<u8> doesn't implement Read.
The code in the question will work when the vector is coerced into a slice using one of the following methods:
read_4_bytes(&*vec_as_file)
read_4_bytes(&vec_as_file[..])
read_4_bytes(vec_as_file.as_slice()).
Note:
When asking the question initially, I was taking &Read instead of Read. This made passing a reference to a slice fail, unless I'd passed in &&*vec_as_file which I didn't think to do.
Recent versions of rust you can also use as_slice() to convert a Vec to a slice.
Thanks to #arete on #rust for finding the solution!
std::io::Cursor
std::io::Cursor is a simple and useful wrapper that implements Read for Vec<u8>, so it allows to use vector as a readable entity.
let mut file = Cursor::new(vector);
read_something(&mut file);
And documentation shows how to use Cursor instead of File to write unit-tests!
Working example:
use std::io::Cursor;
use std::io::Read;
fn read_something(file: &mut impl Read) {
let _ = file.read(&mut [0; 8]);
}
fn main() {
let vector = vec![1, 2, 3, 4];
let mut file = Cursor::new(vector);
read_something(&mut file);
}
From the documentation about std::io::Cursor:
Cursors are typically used with in-memory buffers to allow them to implement Read and/or Write...
The standard library implements some I/O traits on various types which are commonly used as a buffer, like Cursor<Vec<u8>> and Cursor<&[u8]>.
Slice
The example above works for slices as well. In that case it would look like the following:
read_something(&mut &vector[..]);
Working example:
use std::io::Read;
fn read_something(file: &mut impl Read) {
let _ = file.read(&mut [0; 8]);
}
fn main() {
let vector = vec![1, 2, 3, 4];
read_something(&mut &vector[..]);
}
&mut &vector[..] is a "mutable reference to a slice" (a reference to a reference to a part of vector), so I just find the explicit option with Cursor to be more clear and elegant.
Cursor <-> Slice
Even more: if you have a Cursor that owns a buffer, and you need to emulate, for instance, a part of a "file", you can get a slice from the Cursor and pass to the function.
read_something(&mut &file.get_ref()[1..3]);
The following code reads space-delimited records from stdin, and writes comma-delimited records to stdout. Even with optimized builds it's rather slow (about twice as slow as using, say, awk).
use std::io::BufRead;
fn main() {
let stdin = std::io::stdin();
for line in stdin.lock().lines().map(|x| x.unwrap()) {
let fields: Vec<_> = line.split(' ').collect();
println!("{}", fields.join(","));
}
}
One obvious improvement would be to use itertools to join without allocating a vector (the collect call causes an allocation). However, I tried a different approach:
fn main() {
let stdin = std::io::stdin();
let mut cache = Vec::<&str>::new();
for line in stdin.lock().lines().map(|x| x.unwrap()) {
cache.extend(line.split(' '));
println!("{}", cache.join(","));
cache.clear();
}
}
This version tries to reuse the same vector over and over. Unfortunately, the compiler complains:
error: `line` does not live long enough
--> src/main.rs:7:22
|
7 | cache.extend(line.split(' '));
| ^^^^
|
note: reference must be valid for the block suffix following statement 1 at 5:39...
--> src/main.rs:5:40
|
5 | let mut cache = Vec::<&str>::new();
| ^
note: ...but borrowed value is only valid for the for at 6:4
--> src/main.rs:6:5
|
6 | for line in stdin.lock().lines().map(|x| x.unwrap()) {
| ^
error: aborting due to previous error
Which of course makes sense: the line variable is only alive in the body of the for loop, whereas cache keeps a pointer into it across iterations. But that error still looks spurious to me: since the cache is cleared after each iteration, no reference to line can be kept, right?
How can I tell the borrow checker about this?
The only way to do this is to use transmute to change the Vec<&'a str> into a Vec<&'b str>. transmute is unsafe and Rust will not raise an error if you forget the call to clear here. You might want to extend the unsafe block up to after the call to clear to make it clear (no pun intended) where the code returns to "safe land".
use std::io::BufRead;
use std::mem;
fn main() {
let stdin = std::io::stdin();
let mut cache = Vec::<&str>::new();
for line in stdin.lock().lines().map(|x| x.unwrap()) {
let cache: &mut Vec<&str> = unsafe { mem::transmute(&mut cache) };
cache.extend(line.split(' '));
println!("{}", cache.join(","));
cache.clear();
}
}
In this case Rust doesn't know what you're trying to do. Unfortunately, .clear() does not affect how .extend() is checked.
The cache is a "vector of strings that live as long as the main function", but in extend() calls you're appending "strings that live only as long as one loop iteration", so that's a type mismatch. The call to .clear() doesn't change the types.
Usually such limited-time uses are expressed by making a long-lived opaque object that enables access to its memory by borrowing a temporary object with the right lifetime, like RefCell.borrow() gives a temporary Ref object. Implementation of that would be a bit involved and would require unsafe methods for recycling Vec's internal memory.
In this case an alternative solution could be to avoid any allocations at all (.join() allocates too) and stream the printing thanks to Peekable iterator wrapper:
for line in stdin.lock().lines().map(|x| x.unwrap()) {
let mut fields = line.split(' ').peekable();
while let Some(field) = fields.next() {
print!("{}", field);
if fields.peek().is_some() {
print!(",");
}
}
print!("\n");
}
BTW: Francis' answer with transmute is good too. You can use unsafe to say you know what you're doing and override the lifetime check.
Itertools has .format() for the purpose of lazy formatting, which skips allocating a string too.
use std::io::BufRead;
use itertools::Itertools;
fn main() {
let stdin = std::io::stdin();
for line in stdin.lock().lines().map(|x| x.unwrap()) {
println!("{}", line.split(' ').format(","));
}
}
A digression, something like this is a “safe abstraction” in the littlest sense of the solution in another answer here:
fn repurpose<'a, T: ?Sized>(mut v: Vec<&T>) -> Vec<&'a T> {
v.clear();
unsafe {
transmute(v)
}
}
Another approach is to refrain from storing references altogether, and to store indices instead. This trick can also be useful in other data structure contexts, so this might be a nice opportunity to try it out.
use std::io::BufRead;
fn main() {
let stdin = std::io::stdin();
let mut cache = Vec::new();
for line in stdin.lock().lines().map(|x| x.unwrap()) {
cache.push(0);
cache.extend(line.match_indices(' ').map(|x| x.0 + 1));
// cache now contains the indices where new words start
// do something with this information
for i in 0..(cache.len() - 1) {
print!("{},", &line[cache[i]..(cache[i + 1] - 1)]);
}
println!("{}", &line[*cache.last().unwrap()..]);
cache.clear();
}
}
Though you made the remark yourself in the question, I feel the need to point out that there are more elegant methods to do this using iterators, that might avoid the allocation of a vector altogether.
The approach above was inspired by a similar question here, and becomes more useful if you need to do something more complicated than printing.
Elaborating on Francis's answer about using transmute(), this could be safely abstracted, I think, with this simple function:
pub fn zombie_vec<'a, 'b, T: ?Sized>(mut data: Vec<&'a T>) -> Vec<&'b T> {
data.clear();
unsafe {
std::mem::transmute(data)
}
}
Using this, the original code would be:
fn main() {
let stdin = std::io::stdin();
let mut cache0 = Vec::<&str>::new();
for line in stdin.lock().lines().map(|x| x.unwrap()) {
let mut cache = cache0; // into the loop
cache.extend(line.split(' '));
println!("{}", cache.join(","));
cache0 = zombie_vec(cache); // out of the loop
}
}
You need to move the outer vector into every loop iteration, and restore it back to before you finish, while safely erasing the local lifetime.
The safe solution is to use .drain(..) instead of .clear() where .. is a "full range". It returns an iterator, so drained elements can be processed in a loop. It is also available for other collections (String, HashMap, etc.)
fn main() {
let mut cache = Vec::<&str>::new();
for line in ["first line allocates for", "second"].iter() {
println!("Size and capacity: {}/{}", cache.len(), cache.capacity());
cache.extend(line.split(' '));
println!(" {}", cache.join(","));
cache.drain(..);
}
}