read file(not utf-8) line by line? - rust

Question

Is it possible to read file line by line, if it is not in utf-8 encoding with std::io::File and std::io::BufReader?
I look at std::io::Lines and it return Result<String>, so
I worry, have I implement my own BufReader that do the same, but return Vec<u8> instead, or I can reuse std::io::BufReader in some way?

Answer 1

You do not have to re-implement BufReader itself, it provides exactly the method you need for your usecase read_until:
fn read_until(&mut self, byte: u8, buf: &mut Vec<u8>) -> Result<usize>
You supply your own Vec<u8> and the content of the file will be appended until byte is encountered (0x0A being LF).
There are several potential gotchas:
the buffer may end not only with a LF byte, but with a CR LF sequence,
it is up to you to clear buf between subsequent calls.
A simple while let Ok(_) = reader.read_until(0x0A as u8, buffer) should let you read your file easily enough.
You may consider implement a std::io::Lines equivalent which converts from the encoding to UTF-8 to provide a nice API, though it will have a performance cost.

Answer 2

To read files (utf-8 or not utf-8) line by line, I use:
/*
// Cargo.toml:
[dependencies]
encoding_rs = "0.8"
encoding_rs_io = "0.1.7"
...
*/
use encoding_rs::WINDOWS_1252;
use encoding_rs_io::DecodeReaderBytesBuilder;
use std::{
fs,
io::{Read, BufRead, BufReader, Error},
};
fn main() {
let file_path: &str = "/tmp/file.txt";
let buffer: Box<dyn BufRead> = read_file(file_path);
for (index, result_vec_bytes) in buffer.split(b'\n').enumerate() {
let line_number: usize = index + 1;
let line_utf8: String = get_string_utf8(result_vec_bytes, line_number, file_path);
println!("{line_utf8}");
}
}
Such that:
fn read_file(file_path: &str) -> Box<dyn BufRead> {
let file = match fs::File::open(file_path) {
Ok(f) => f,
Err(why) => panic!("Problem opening the file: \"{file_path}\"\n{why:?}"),
};
Box::new(BufReader::new(file))
}
And
fn get_string_utf8(result_vec_bytes: Result<Vec<u8>, std::io::Error>, line_number: usize) -> String {
let vec_bytes: Vec<u8> = match result_vec_bytes {
Ok(values) => values,
Err(why) => panic!("Failed to read line nº {line_number}: {why}"),
};
// from_utf8() checks to ensure that the bytes are valid UTF-8
let line_utf8: String = match std::str::from_utf8(&vec_bytes) {
Ok(str) => str.to_string(),
Err(_) => {
let mut data = DecodeReaderBytesBuilder::new()
.encoding(Some(WINDOWS_1252))
.build(vec_bytes.as_slice());
let mut buffer = String::new();
let _number_of_bytes = match data.read_to_string(&mut buffer) {
Ok(num) => num,
Err(why) => {
eprintln!("Problem reading data from file in buffer!");
eprintln!("Line nº {line_number}");
eprintln!("Used encoding type: WINDOWS_1252.");
eprintln!("Try another encoding type!");
panic!("Failed to convert data to UTF-8!: {why}")
},
};
buffer
}
};
// remove Window new line: "\r\n"
line_utf8.trim_end_matches('\r').to_string()
}