Delegating the creation of data structures - rust

Question

I’m very new to Rust. While trying out small things, I have written the following code. It simply scans files (given as arguments) for a specific string (“Started “) and prints out the matching lines:
use std::os;
use std::io::BufferedReader;
use std::io::File;
fn main() {
for target in os::args().iter() {
scan_file(target);
}
}
fn scan_file(path_str: &String) {
let path = Path::new(path_str.as_bytes());
let file = File::open(&path);
let mut reader = BufferedReader::new(file);
for line in reader.lines() {
match line {
Ok(s) => {
if s.as_slice().contains("Started ") {
print!("{}", s);
}
}
Err(_) => return,
}
}
}
My question is: how can I refactor the function scan_file so that it looks something like this (or similar enough)?:
fn scan_file(path_str: &String) {
for line in each_line_in_file_with_path(path_str) {
match line {
Ok(s) => {
if s.as_slice().contains("Started ") {
print!("{}", s);
}
}
Err(_) => return,
}
}
}
In this new version of the function, the three variable declarations are gone. Instead, the function each_line_in_file_with_path is expected to handle all the “turn a path into lines”, returning an iterator.
I’ve tried a number of things unsuccessfully, always due to variables going out of scope too early for my needs. I understand the problems I have (I think), but can’t find anywhere a good explanation of how this should be handled.

Answer 1

It is not possible to implement a working each_line_in_file_with_path function — at least, not without adding some overhead and unsafe code.
Let's look at the values involved and their types. First is path, of type Path (either posix::Path or windows::Path). The constructors for these types receive a BytesContainer by value, therefore they take ownership of it. No issues here.
Next is file, of type IoResult<File>. File::open() clones the path it receives, so again, no issues here.
Next is reader, of type BufferedReader<IoResult<File>>. Just like Path, the constructor for BufferedReader takes its argument by value and takes ownership of it.
The problem is with reader.lines(). This value is of type Lines<'r, T: 'r>. As the type signature suggests, this struct contains a borrowed reference. The signature of lines shows the relationship between the loaner and the borrower:
fn lines<'r>(&'r mut self) -> Lines<'r, Self>
How do we define each_line_in_file_with_path now? each_line_in_file_with_path cannot return a Lines directly. You probably tried writing the function like this:
fn each_line_in_file_with_path<'a, T>(path: &T) -> Lines<'a, BufferedReader<IoResult<File>>>
where T: BytesContainer {
let path = Path::new(path);
let file = File::open(&path);
let reader = BufferedReader::new(file);
reader.lines()
}
This gives a compilation error:
main.rs:46:5: 46:11 error: `reader` does not live long enough
main.rs:46 reader.lines()
^~~~~~
main.rs:42:33: 47:2 note: reference must be valid for the lifetime 'a as defined on the block at 42:32...
main.rs:42 where T: BytesContainer {
main.rs:43 let path = Path::new(path);
main.rs:44 let file = File::open(&path);
main.rs:45 let reader = BufferedReader::new(file);
main.rs:46 reader.lines()
main.rs:47 }
main.rs:42:33: 47:2 note: ...but borrowed value is only valid for the block at 42:32
main.rs:42 where T: BytesContainer {
main.rs:43 let path = Path::new(path);
main.rs:44 let file = File::open(&path);
main.rs:45 let reader = BufferedReader::new(file);
main.rs:46 reader.lines()
main.rs:47 }
error: aborting due to previous error
That's because we're trying to return a Lines that refers to a BufferedReader that ceases to exist when the function returns (the Lines would contain a dangling pointer).
Now, one might think, “I'll just return the BufferedReader along with the Lines”.
struct LinesInFileIterator<'a> {
reader: BufferedReader<IoResult<File>>,
lines: Lines<'a, BufferedReader<IoResult<File>>>
}
impl<'a> Iterator<IoResult<String>> for LinesInFileIterator<'a> {
fn next(&mut self) -> Option<IoResult<String>> {
self.lines.next()
}
}
fn each_line_in_file_with_path<'a, T>(path: &T) -> LinesInFileIterator<'a>
where T: BytesContainer {
let path = Path::new(path);
let file = File::open(&path);
let reader = BufferedReader::new(file);
LinesInFileIterator {
reader: reader,
lines: reader.lines()
}
}
This doesn't work either:
main.rs:46:16: 46:22 error: `reader` does not live long enough
main.rs:46 lines: reader.lines()
^~~~~~
main.rs:40:33: 48:2 note: reference must be valid for the lifetime 'a as defined on the block at 40:32...
main.rs:40 where T: BytesContainer {
main.rs:41 let path = Path::new(path);
main.rs:42 let file = File::open(&path);
main.rs:43 let reader = BufferedReader::new(file);
main.rs:44 LinesInFileIterator {
main.rs:45 reader: reader,
...
main.rs:40:33: 48:2 note: ...but borrowed value is only valid for the block at 40:32
main.rs:40 where T: BytesContainer {
main.rs:41 let path = Path::new(path);
main.rs:42 let file = File::open(&path);
main.rs:43 let reader = BufferedReader::new(file);
main.rs:44 LinesInFileIterator {
main.rs:45 reader: reader,
...
main.rs:46:16: 46:22 error: use of moved value: `reader`
main.rs:46 lines: reader.lines()
^~~~~~
main.rs:45:17: 45:23 note: `reader` moved here because it has type `std::io::buffered::BufferedReader<core::result::Result<std::io::fs::File, std::io::IoError>>`, which is non-copyable
main.rs:45 reader: reader,
^~~~~~
error: aborting due to 2 previous errors
Basically, we can't have a struct that contains a borrowed reference that points to another member of the struct, because when the struct is moved, the reference would become invalid.
There are 2 solutions:
Make a function that returns a BufferedReader from a file path, and call .lines() on it in your for loop.
Make a function that accepts a closure that receives each line.
fn main() {
for target in os::args().iter() {
scan_file(target.as_slice());
}
}
fn for_each_line_in_file_with_path_do(path: &str, action: |IoResult<String>|) {
let path = Path::new(path.as_bytes());
let file = File::open(&path);
let mut reader = BufferedReader::new(file);
for line in reader.lines() {
action(line);
}
}
fn scan_file(path_str: &str) {
for_each_line_in_file_with_path_do(path_str, |line| {
match line {
Ok(s) => {
if s.as_slice().contains("Started ") {
print!("{}", s);
}
}
Err(_) => return,
}
});
}

Answer 2

You won't be able to do it without some boilerplate. You need to have some source of data, and because iterators return their data in chunks, they either have to contain the data or to have a reference into some other source of this data (this also includes iterators which return data from external source, e.g. lines in a file).
However, because you want to "encapsulate" your iterator into a function call, this iterator cannot be of the second kind, i.e. it cannot contain references, because all references it could contain would point to this function call stack. Consequently, the iterator's source can only be contained in this iterator.
And this is the boilerplate problem - in general there is no such iterator in the standard library. You will need to create it yourself. In this particular case, though, you can get away without implementing Iterator trait manually. You only need to create some simple structural wrapper:
use std::os;
use std::io::{BufferedReader, File, Lines};
fn main() {
for target in os::args().iter() {
scan_file(target.as_slice());
}
}
struct FileLines {
source: BufferedReader<File>
}
impl FileLines {
fn new(path_str: &str) -> FileLines {
let path = Path::new(path_str.as_bytes());
let file = File::open(&path).unwrap();
let reader = BufferedReader::new(file);
FileLines { source: reader }
}
fn lines(&mut self) -> Lines<BufferedReader<File>> {
self.source.lines()
}
}
fn scan_file(path_str: &str) {
for line in FileLines::new(path_str).lines() {
match line {
Ok(s) => {
if s.as_slice().contains("Started ") {
print!("{}", s);
}
}
Err(_) => return,
}
}
}
(I also changed &String to &str because it is more idiomatic and general)
The FileLines structure owns the data and encapsulates all of the complex logic in its constructor. Then its lines() method just returns an iterator into its internals. This is rather common pattern in Rust, and usually you will be able to find the main owner of your data and build your program around it with methods which return iterators/references into this owner.
This is not exactly what you wanted (there are two function calls in for loop initializer - new() and lines()), but I believe that for all practical purposes they have the same expressiveness and usability.