I have this code to read a file into a BytesMut:
use std::fs::File;
use std::io;
use std::io::prelude::*;
use bytes::{BufMut, BytesMut, Buf};
fn main() -> io::Result<()> {
let mut f = File::open("foo.txt")?;
let mut b = BytesMut::with_capacity(10);
f.read(b.as_mut())?;
println!("The bytes: {:?}", b.len());
Ok(())
}
But b.len() is always zero. The content of "foo.txt" is 0x00000010 (decimal 16). I can read into a [0;10] then convert it to a BytesMut, but is there a easy way do this?
The f.read() call will write into an existing byte slice &[u8]. You have allocated space for 10 bytes, but the BytesMut doesn't yet hold 10 bytes. When it is deref'd into a byte slice it will return an empty slice, since it holds no data.
You'll want something like BytesMut::zeroed and then it will work:
use std::fs::File;
use std::io::{Read, Result};
use bytes::BytesMut;
fn main() -> Result<()> {
let mut f = File::open("foo.txt")?;
let mut b = BytesMut::zeroed(10);
f.read(b.as_mut())?;
println!("The bytes: {:?}", b.len());
Ok(())
}
The bytes: 10
If you want to read the whole file without allocating prior, you can use std::io::copy with a Writer adapter via .writer():
use bytes::{BufMut, BytesMut};
use std::fs::File;
use std::io::Result;
fn main() -> Result<()> {
let mut f = File::open("foo.txt")?;
let mut b = BytesMut::new().writer();
std::io::copy(&mut f, &mut b)?;
println!("The bytes: {:?}", b.into_inner().len());
Ok(())
}
Related
I'm trying to read some lines from a file, skipping the first few and printing the rest, but I keep getting errors about used value after move:
use std::fs::File;
use std::io::{self, BufRead, BufReader, Read};
use std::path::Path;
fn skip_and_print_file(skip: &usize, path: &Path) {
let mut skip: usize = *skip;
if let Ok(file) = File::open(path) {
let mut buffer = BufReader::new(file);
for (index, line) in buffer.lines().enumerate() {
if index >= skip {
break;
}
}
print_to_stdout(&mut buffer);
}
}
fn print_to_stdout(mut input: &mut Read) {
let mut stdout = io::stdout();
io::copy(&mut input, &mut stdout);
}
fn main() {}
This is the error I'm getting:
error[E0382]: use of moved value: `buffer`
--> src/main.rs:15:30
|
10 | for (index, line) in buffer.lines().enumerate() {
| ------ value moved here
...
15 | print_to_stdout(&mut buffer);
| ^^^^^^ value used here after move
|
= note: move occurs because `buffer` has type `std::io::BufReader<std::fs::File>`, which does not implement the `Copy` trait
In order to avoid the move, use the Read::by_ref() method. That way, you only borrow the BufReader:
for (index, line) in buffer.by_ref().lines().enumerate() { ... }
// ^^^^^^^^^
// you can still use `buffer` here
As Lukas Kalbertodt says, use Read::by_ref.
This prevents lines from consuming the BufReader and instead it consumes a &mut BufReader. The same logic applies to iterators.
Instead of implementing skip yourself, you can use Iterator::take. This has to be driven to completion with a for loop though:
use std::{
fs::File,
io::{self, BufRead, BufReader, Read},
path::Path,
};
fn skip_and_print_file(skip: usize, path: impl AsRef<Path>) {
if let Ok(file) = File::open(path) {
let mut buffer = BufReader::new(file);
for _ in buffer.by_ref().lines().take(skip) {}
// Or: buffer.by_ref().lines().take(skip).for_each(drop);
print_to_stdout(buffer);
}
}
fn print_to_stdout(mut input: impl Read) {
let mut stdout = io::stdout();
io::copy(&mut input, &mut stdout).expect("Unable to copy");
}
fn main() {
skip_and_print_file(2, "/etc/hosts");
}
Note that there's no reason to make the skip variable mutable or even to pass in a reference. You can also take in AsRef<Path> and then callers of skip_and_print_file can just pass in a string literal.
use std::env;
use std::fs::File;
use std::io::{BufReader, BufWriter, Read, Write};
fn main() {
let args = env::args().collect::<Vec<String>>();
let file = File::open(&args[1]).expect("file not found");
let reader = BufReader::new(file);
let mut writer = BufWriter::new(std::io::stdout());
for it in reader.bytes() {
writer.write(&[*it]);
}
}
Why does this give an error?
type `std::result::Result<u8, std::io::Error>` cannot be dereferenced
From the documentation,
fn bytes(self) -> Bytes<Self> where
Self: Sized,
Transforms this Read instance to an Iterator over its bytes.
The returned type implements Iterator where the Item is Result<u8,
io::Error>. The yielded item is Ok if a byte was successfully read and
Err otherwise. EOF is mapped to returning None from this iterator.
Only types implementing std::ops::Deref can be dereferenced.
use std::env;
use std::fs::File;
use std::io::{BufReader, BufWriter, Read, Write};
fn main() {
let args = env::args().collect::<Vec<String>>();
let file = File::open(&args[1]).expect("file not found");
let reader = BufReader::new(file);
let mut writer = BufWriter::new(std::io::stdout());
for it in reader.bytes() {
writer.write(&[it.unwrap()]);
}
}
I am trying to use the read and write of a tcp stream in different threads. This is what I currently have:
use tokio::prelude::*;
use tokio::net::TcpStream;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut stream = TcpStream::connect("localhost:8080").await?;
let (mut read, mut write) = stream.split();
tokio::spawn(async move {
loop {
let mut buf = [0u8; 32];
read.read(&mut buf).await.unwrap();
println!("{}", std::str::from_utf8(&buf));
}
});
Ok(())
}
Im going to use another thread for the write. My problem is that I get the error that 'stream' is dropped while still borrowed.
That happens due to the method signature of Tokio::split, as you can see it takes &mut self, so its part cannot be used in a tokio::spawn future argument due to the 'static bound. So, this is exactly what the error says.
What you are searching is tokio::io::split. Playground
use tokio::prelude::*;
use tokio::net::TcpStream;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut stream = TcpStream::connect("localhost:8080").await?;
let (mut read, mut write) = tokio::io::split(stream);
tokio::spawn(async move {
loop {
let mut buf = [0u8; 32];
read.read(&mut buf).await.unwrap();
println!("{:?}", std::str::from_utf8(&buf));
}
});
Ok(())
}
Reading https://users.rust-lang.org/t/why-i-have-to-use-tokio-tcpstream-split-for-concurrent-read-writes/47755/3
And it suggests to use into_split it is more efficient.
tokio::io::split uses a Mutex which into_split apparently does not have (specific to TcpStream)
I'm trying to read some lines from a file, skipping the first few and printing the rest, but I keep getting errors about used value after move:
use std::fs::File;
use std::io::{self, BufRead, BufReader, Read};
use std::path::Path;
fn skip_and_print_file(skip: &usize, path: &Path) {
let mut skip: usize = *skip;
if let Ok(file) = File::open(path) {
let mut buffer = BufReader::new(file);
for (index, line) in buffer.lines().enumerate() {
if index >= skip {
break;
}
}
print_to_stdout(&mut buffer);
}
}
fn print_to_stdout(mut input: &mut Read) {
let mut stdout = io::stdout();
io::copy(&mut input, &mut stdout);
}
fn main() {}
This is the error I'm getting:
error[E0382]: use of moved value: `buffer`
--> src/main.rs:15:30
|
10 | for (index, line) in buffer.lines().enumerate() {
| ------ value moved here
...
15 | print_to_stdout(&mut buffer);
| ^^^^^^ value used here after move
|
= note: move occurs because `buffer` has type `std::io::BufReader<std::fs::File>`, which does not implement the `Copy` trait
In order to avoid the move, use the Read::by_ref() method. That way, you only borrow the BufReader:
for (index, line) in buffer.by_ref().lines().enumerate() { ... }
// ^^^^^^^^^
// you can still use `buffer` here
As Lukas Kalbertodt says, use Read::by_ref.
This prevents lines from consuming the BufReader and instead it consumes a &mut BufReader. The same logic applies to iterators.
Instead of implementing skip yourself, you can use Iterator::take. This has to be driven to completion with a for loop though:
use std::{
fs::File,
io::{self, BufRead, BufReader, Read},
path::Path,
};
fn skip_and_print_file(skip: usize, path: impl AsRef<Path>) {
if let Ok(file) = File::open(path) {
let mut buffer = BufReader::new(file);
for _ in buffer.by_ref().lines().take(skip) {}
// Or: buffer.by_ref().lines().take(skip).for_each(drop);
print_to_stdout(buffer);
}
}
fn print_to_stdout(mut input: impl Read) {
let mut stdout = io::stdout();
io::copy(&mut input, &mut stdout).expect("Unable to copy");
}
fn main() {
skip_and_print_file(2, "/etc/hosts");
}
Note that there's no reason to make the skip variable mutable or even to pass in a reference. You can also take in AsRef<Path> and then callers of skip_and_print_file can just pass in a string literal.
I don't know how to convert a Vec<i32> into a &[u8] slice.
fn main() {
let v: Vec<i32> = vec![1; 100_000_000];
let v_bytes: &[u8] = /* ... */;
}
I want to write a large Vec<i32> to a file so I can read it back at a future time.
You can use std::slice::from_raw_parts:
let v_bytes: &[u8] = unsafe {
std::slice::from_raw_parts(
v.as_ptr() as *const u8,
v.len() * std::mem::size_of::<i32>(),
)
};
Following the comments on this answer, you should wrap this code in a function and have the return value borrow the input, so that you use the borrow checker as far as possible:
fn as_u8_slice(v: &[i32]) -> &[u8] {
unsafe {
std::slice::from_raw_parts(
v.as_ptr() as *const u8,
v.len() * std::mem::size_of::<i32>(),
)
}
}
Since Rust 1.30, the best solution is to use slice::align_to:
fn main() {
let v: Vec<i32> = vec![1; 8];
let (head, body, tail) = unsafe { v.align_to::<u8>() };
assert!(head.is_empty());
assert!(tail.is_empty());
println!("{:#x?}", body);
}
This properly handles the cases where the alignment of the first type and the second type do not match. In this example, I ensure that the alignment of the i32 is greater than that of the u8 via the assert! statements.
I took #swizards answer and ran with it a bit to get the other side of the coin - reading the vector back in:
use std::fs::File;
use std::io::{Read, Write};
use std::{mem, slice};
fn as_u8_slice(v: &[i32]) -> &[u8] {
let element_size = mem::size_of::<i32>();
unsafe { slice::from_raw_parts(v.as_ptr() as *const u8, v.len() * element_size) }
}
fn from_u8(v: Vec<u8>) -> Vec<i32> {
let data = v.as_ptr();
let len = v.len();
let capacity = v.capacity();
let element_size = mem::size_of::<i32>();
// Make sure we have a proper amount of capacity (may be overkill)
assert_eq!(capacity % element_size, 0);
// Make sure we are going to read a full chunk of stuff
assert_eq!(len % element_size, 0);
unsafe {
// Don't allow the current vector to be dropped
// (which would invalidate the memory)
mem::forget(v);
Vec::from_raw_parts(
data as *mut i32,
len / element_size,
capacity / element_size,
)
}
}
fn do_write(filename: &str, v: &[i32]) {
let mut f = File::create(filename).unwrap();
f.write_all(as_u8_slice(v)).unwrap();
}
fn do_read(filename: &str) -> Vec<i32> {
let mut f = File::open(filename).unwrap();
let mut bytes = Vec::new();
f.read_to_end(&mut bytes).unwrap();
from_u8(bytes)
}
fn main() {
let v = vec![42; 10];
do_write("vector.dump", &v);
let v2 = do_read("vector.dump");
assert_eq!(v, v2);
println!("{:?}", v2)
}