Pulling a vec out of a vec of structs - rust

I'm working on a compiler for a toy language and I want to be able to check for errors on each file. I have a MooFile struct that has a Vec<anyhow::Error> where I put errors when they are encountered. Now I want to go through the files, get the errors, and flatten that into a single vec to return from the compiler but I'm running up against the borrow checker. I'm going to post my various attempt but they keep running into similar issues.
fn compile(&mut self) -> Result<(), Vec<Error>> {
...
// Check for errors
let has_errors = self.files
.iter()
.map(|file| file.has_errors())
.reduce(|acc, value| acc | value)
.unwrap_or(false);
if has_errors {
let mut errors = vec![];
self.files
.iter()
.for_each(|&mut file| errors.append(&mut file.errors));
Err(errors)
} else {
Ok(())
}
}
error[E0596]: cannot borrow `file.errors` as mutable, as it is behind a `&` reference
--> src\lib.rs:62:48
|
62 | .for_each(|file| errors.append(&mut file.errors));
| ---- ^^^^^^^^^^^^^^^^ `file` is a `&` reference, so the data it refers to cannot be borrowed as mutable
| |
| help: consider changing this to be a mutable reference: `&mut MooFile`
fn compile(&mut self) -> Result<(), Vec<Error>> {
...
// Check for errors
let has_errors = self.files
.iter()
.map(|file| file.has_errors())
.reduce(|acc, value| acc | value)
.unwrap_or(false);
if has_errors {
let mut errors = vec![];
self.files
.into_iter()
.for_each(|mut file| errors.append(&mut file.errors));
Err(errors)
} else {
Ok(())
}
}
error[E0507]: cannot move out of `self.files` which is behind a mutable reference
--> src\lib.rs:63:13
|
63 | self.files
| ^^^^^^^^^^ move occurs because `self.files` has type `Vec<MooFile>`, which does not implement the `Copy` trait
64 | .into_iter()
| ----------- `self.files` moved due to this method call
fn compile(&mut self) -> Result<(), Vec<Error>> {
...
// Check for errors
let errors: Vec<Error> = self.files
.iter()
.map(|file| file.errors)
.flatten()
.collect();
if errors.len() > 0 {
Err(errors)
} else {
Ok(())
}
}
error[E0507]: cannot move out of `file.errors` which is behind a shared reference
--> src\lib.rs:54:25
|
54 | .map(|file| file.errors)
| ^^^^^^^^^^^ move occurs because `file.errors` has type `Vec<anyhow::Error>`, which does not implement the `Copy` trait
For more information about this error, try `rustc --explain E0507`.
error: could not compile `moo2` due to previous error
I feel like there has to be some kind of idiomatic way of doing this kind of thing that I'm missing.

Most of these errors come from how you create your iterators. If you assume x is a Vec<Y>:
x.iter(): Consumes &x and creates an iterator of &Y.
x.iter_mut(): Consumes &mut x and creates an iterator of &mut Y.
x.into_iter(): Consumes ownership of the entire Vec<Y> and creates an iterator of owned Y.
The first version uses .iter() when you want a mutable reference (.iter_mut()). The second uses .into_iter() when you want a mutable reference (.iter_mut()). And the third uses .iter() when you attempt to claim ownership of the errors by disposing of the files (.into_iter(), but you can't drop self.files without issues).
Here is one way you could Clone the errors without moving them out of each file.
let mut errors = Vec::new();
self.files.iter()
.filter(|file| file.has_errors())
.foreach(|file| errors.extend_from_slice(&file.errors));
if has_errors {
return Err(errors)
}
Ok(())
Or you could drain them into a new Vec using extend. This version also uses any to look a bit cleaner, but is not as efficient.
if self.files.iter().any(|f| f.has_errors()) {
let mut errors = Vec::new();
for file in &mut self.files {
errors.extend(file.errors.drain(..));
}
return Err(errors)
}
Ok(())

Related

Difficulty creating a mock 'sink' for testing purposes in Rust

Context
I have a function that will use rayon to paralellize some work. Each thread should write its output to a separate consumer. In practice, these consumers will usually write to (separate) files on disc. For testing, I want them to append to vectors, which I can then make assertions on.
Problem
In fact I hit an issue even with a serial version of this code.
The function I want to test is func_under_test. I want get_sink to create and hand out a new 'sink' (a Vec<i32>) each time it is called, which will be inside func_under_test. But I also want to keep a reference to these vectors in my main function, so I can make assertions on them. I've tried writing this:
fn func_under_test<GetSink, F>(
mut get_sink: GetSink
)
where
GetSink: FnMut(usize) -> F,
F: FnMut(i32) -> (),
{
let idxs : Vec<usize> = (0..2).collect();
idxs.iter().for_each(|&i| {
let mut sink = get_sink(i);
sink(0i32);
sink(1i32);
sink(2i32);
});
}
fn main() {
let mut sinks : Vec<&Vec<i32>> = vec![];
let get_sink = |i: usize| {
let mut sink : Vec<i32> = vec![];
sinks.push(&sink);
move |x : i32| sink.push(x)
};
func_under_test(get_sink);
assert_eq!(**sinks.get(0).unwrap(), vec![0i32, 1i32, 2i32]);
}
But I get this compilation error:
error[E0597]: `sink` does not live long enough
--> src/main.rs:23:20
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
25 | };
| - `sink` dropped here while still borrowed
error[E0505]: cannot move out of `sink` because it is borrowed
--> src/main.rs:24:9
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------------
| | |
| | borrow of `sink` occurs here
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
| ^^^^^^^^^^^^^^ ---- move occurs due to use in closure
| |
| move out of `sink` occurs here
Am I correct in thinking that I need to use something like an Rc in this context, since I need to have two references to each Vec<i32>?
Since only one of those references needs to be mutable, do I need to use Rc<Cell<Vec<i32>>> in one case, and just Rc<Vec<i32>> in the other?
You could use an Arc<Mutex<i32>> to share mutable access to the vectors. (Rc is not thread-safe.)
A tidier solution would be typed_arena::Arena, which would allow you to write essentially the code you tried, but with the roles swapped: the Vec<i32>s are always owned by the arena (thus it outlives func_under_tests). Then, after all of the references to the arena are gone, you can convert it into a vector of its elements.
use typed_arena::Arena;
fn main() {
let sinks: Arena<Vec<i32>> = Arena::new();
let get_sink = |_i: usize| {
let sink_ref = sinks.alloc(Vec::new());
move |x| sink_ref.push(x)
};
func_under_test(get_sink);
assert_eq!(sinks.into_vec()[0], vec![0i32, 1i32, 2i32]);
}
I think I've figured out the answer to this. The thread doing the testing needs to have access to the inner vectors, and so does the thread that is populating this. So they need to be wrapped in an Arc and a Mutex. But the outer vector does not, since it is only borrowed by the get_sink closure.
So sinks can have type Vec<Arc<Mutex<Vec<i32>>>.
Anyway, I can write this and it does compile and behave and expected:
use rayon::prelude::*;
use std::sync::{Arc, Mutex};
fn func_under_test<G, S>(
get_sink: G)
where
G: Fn(&usize) -> S + Sync,
S: FnMut(i32) -> (),
{
let idxs : Vec<usize> = vec![0, 1, 2];
idxs.par_iter().for_each(|i| {
let mut sink : S = get_sink(i);
for j in 0..3 {
sink(j + 3*(*i) as i32);
}
});
}
fn main() {
let sinks : Vec<Arc<Mutex<Vec<i32>>>> = vec![
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
];
let get_sink = |i: &usize| {
let m : Arc<Mutex<Vec<i32>>> = sinks.get(*i).unwrap().clone();
move |x: i32| {
m.lock().unwrap().push(x);
}
};
func_under_test(get_sink);
assert_eq!(*sinks.get(0).unwrap().lock().unwrap(), vec![0, 1, 2]);
assert_eq!(*sinks.get(1).unwrap().lock().unwrap(), vec![3, 4, 5]);
assert_eq!(*sinks.get(2).unwrap().lock().unwrap(), vec![6, 7, 8]);
}
It also seems like G implements the correct traits here: it is Sync, since it needs to be shared between threads. But is is not Send, since it does not need to send information between threads.

Rust borrow checker multiple mutable reference in modifying element in linked list in a loop

I'm trying to build some sort of linked list, each node has a keyword and vectors of the same node. In that for loop, i will traverse the linked list to get the vector, either already exist if keyword is the same, or create a new vector. I find it impossible to *not * violate the double mut reference error since I'll need to do iter_mut(), push() on the same path. Anyone has general idea on how to approach this?
pub struct Knowledge {
pub title: String,
}
pub struct Entry {
pub keyword: String,
pub next: Vec<Box<KnowledgeEntry>>
}
pub enum KnowledgeEntry {
Normal(Knowledge),
Entry(Entry)
}
impl KnowledgeEntry {
pub fn get_entry_mut (&mut self) -> Option<&mut Vec<Box<KnowledgeEntry>>> {
match self {
KnowledgeEntry::Entry(e) => {
Some(&mut e.next)
}
KnowledgeEntry::Normal(k) => {None}
}
}
}
fn main () {
let mut entry = Entry{keyword: "".to_owned(), next: vec![]};
let mut current_container = &mut entry.next;
let new_knowledge = Knowledge{title: "title1".to_owned()};
// new knowledge to be insert in these hierarchy.
let remaining_comp: Vec<_> = vec!["hier1", "hier2", "hier3"];
// each loop will set current_container, either created from new, or found in the vector.
for comp in remaining_comp.iter() {
// true will be replace with body checking with keyword.
let mut _result = current_container.iter_mut().find(|en| {true});
if let Some(mut _res) = _result {
// found the container. Set the current container
if let KnowledgeEntry::Entry(ref mut entry) = &mut **_res {
current_container = &mut entry.next;
} else {
break;
}
} else {
// can't find the resulting container. Create a new vector, and move the container.
let new_container: Vec<Box<KnowledgeEntry>> = vec![];
let mut _temp = KnowledgeEntry::Entry(Entry {
keyword: (*comp).to_string(),
next: new_container
});
current_container.push(Box::new(_temp));
let last_var = current_container.last_mut().unwrap();
// get_entry_mut() to retrieve `_temp` that we have just moved.
current_container = (**last_var).get_entry_mut().unwrap();
}
}
// found the container correspond to "hier1" -> "hier2" -> "hier3", add the entry.
current_container.push(Box::new(KnowledgeEntry::Normal(new_knowledge)));
}
Playground
Produces the following errors:
error[E0499]: cannot borrow `*current_container` as mutable more than once at a time
--> src/main.rs:50:13
|
35 | let mut _result = current_container.iter_mut().find(|en| {true});
| ----------------- first mutable borrow occurs here
...
50 | current_container.push(Box::new(_temp));
| ^^^^^^^^^^^^^^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
error[E0499]: cannot borrow `*current_container` as mutable more than once at a time
--> src/main.rs:51:28
|
35 | let mut _result = current_container.iter_mut().find(|en| {true});
| ----------------- first mutable borrow occurs here
...
51 | let last_var = current_container.last_mut().unwrap();
| ^^^^^^^^^^^^^^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
error[E0499]: cannot borrow `*current_container` as mutable more than once at a time
--> src/main.rs:57:1
|
35 | let mut _result = current_container.iter_mut().find(|en| {true});
| ----------------- first mutable borrow occurs here
...
57 | current_container.push(Box::new(KnowledgeEntry::Normal(new_knowledge)));
| ^^^^^^^^^^^^^^^^^
| |
| second mutable borrow occurs here
| first borrow later used here

Mutably borrowing in match statement and result

I'm trying to determine if a container has an object and return the found object if it does, or add it if it doesn't.
I've found Rust borrow mutable self inside match expression
which has an answer which says what I am trying to do can't (couldn't?) be done.
In my situation, I've got some objects that have vectors of children. I don't want to expose the internals of my object, because I may want to change the representation underneath.
How can you resolve the need to mutably borrow in different match arms in Rust? seems to suggest I may be able to do what I want if I get the lifetimes correct, but I haven't been able to figure out how.
Here's a representation of the issue I'm having:
fn find_val<'a>(container: &'a mut Vec<i32>, to_find: i32) -> Option<&'a mut i32> {
for item in container.iter_mut() {
if *item == to_find {
return Some(item);
}
}
None
}
fn main() {
let mut container = Vec::<i32>::new();
container.push(1);
container.push(2);
container.push(3);
let to_find = 4;
match find_val(&mut container, to_find) {
Some(x) => {
println!("Found {}", x);
}
_ => {
container.push(to_find);
println!("Added {}", to_find);
}
}
}
playground
The error I get is:
error[E0499]: cannot borrow `container` as mutable more than once at a time
--> src/main.rs:24:13
|
19 | match find_val(&mut container, to_find) {
| --------- first mutable borrow occurs here
...
24 | container.push(to_find);
| ^^^^^^^^^ second mutable borrow occurs here
...
27 | }
| - first borrow ends here
Put the change in a function, and use early return instead of an else branch:
fn find_val_or_insert(container: &mut Vec<i32>, to_find: i32) {
if let Some(x) = find_val(&container, to_find) {
println!("Found {}", x);
return; // <- return here instead of an else branch
}
container.push(to_find);
println!("Added {}", to_find);
}
See also Mutable borrow more than once and How to update-or-insert on a Vec?

Cannot borrow as mutable because it is also borrowed as immutable

I am learning Rust and I don't quite get why this is not working.
#[derive(Debug)]
struct Node {
value: String,
}
#[derive(Debug)]
pub struct Graph {
nodes: Vec<Box<Node>>,
}
fn mk_node(value: String) -> Node {
Node { value }
}
pub fn mk_graph() -> Graph {
Graph { nodes: vec![] }
}
impl Graph {
fn add_node(&mut self, value: String) {
if let None = self.nodes.iter().position(|node| node.value == value) {
let node = Box::new(mk_node(value));
self.nodes.push(node);
};
}
fn get_node_by_value(&self, value: &str) -> Option<&Node> {
match self.nodes.iter().position(|node| node.value == *value) {
None => None,
Some(idx) => self.nodes.get(idx).map(|n| &**n),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn some_test() {
let mut graph = mk_graph();
graph.add_node("source".to_string());
graph.add_node("destination".to_string());
let source = graph.get_node_by_value("source").unwrap();
let dest = graph.get_node_by_value("destination").unwrap();
graph.add_node("destination".to_string());
}
}
(playground)
This has the error
error[E0502]: cannot borrow `graph` as mutable because it is also borrowed as immutable
--> src/main.rs:50:9
|
47 | let source = graph.get_node_by_value("source").unwrap();
| ----- immutable borrow occurs here
...
50 | graph.add_node("destination".to_string());
| ^^^^^ mutable borrow occurs here
51 | }
| - immutable borrow ends here
This example from Programming Rust is quite similar to what I have but it works:
pub struct Queue {
older: Vec<char>, // older elements, eldest last.
younger: Vec<char>, // younger elements, youngest last.
}
impl Queue {
/// Push a character onto the back of a queue.
pub fn push(&mut self, c: char) {
self.younger.push(c);
}
/// Pop a character off the front of a queue. Return `Some(c)` if there /// was a character to pop, or `None` if the queue was empty.
pub fn pop(&mut self) -> Option<char> {
if self.older.is_empty() {
if self.younger.is_empty() {
return None;
}
// Bring the elements in younger over to older, and put them in // the promised order.
use std::mem::swap;
swap(&mut self.older, &mut self.younger);
self.older.reverse();
}
// Now older is guaranteed to have something. Vec's pop method // already returns an Option, so we're set.
self.older.pop()
}
pub fn split(self) -> (Vec<char>, Vec<char>) {
(self.older, self.younger)
}
}
pub fn main() {
let mut q = Queue {
older: Vec::new(),
younger: Vec::new(),
};
q.push('P');
q.push('D');
assert_eq!(q.pop(), Some('P'));
q.push('X');
let (older, younger) = q.split(); // q is now uninitialized.
assert_eq!(older, vec!['D']);
assert_eq!(younger, vec!['X']);
}
A MRE of your problem can be reduced to this:
// This applies to the version of Rust this question
// was asked about; see below for updated examples.
fn main() {
let mut items = vec![1];
let item = items.last();
items.push(2);
}
error[E0502]: cannot borrow `items` as mutable because it is also borrowed as immutable
--> src/main.rs:4:5
|
3 | let item = items.last();
| ----- immutable borrow occurs here
4 | items.push(2);
| ^^^^^ mutable borrow occurs here
5 | }
| - immutable borrow ends here
You are encountering the exact problem that Rust was designed to prevent. You have a reference pointing into the vector and are attempting to insert into the vector. Doing so might require that the memory of the vector be reallocated, invalidating any existing references. If that happened and you used the value in item, you'd be accessing uninitialized memory, potentially causing a crash.
In this particular case, you aren't actually using item (or source, in the original) so you could just... not call that line. I assume you did that for some reason, so you could wrap the references in a block so that they go away before you try to mutate the value again:
fn main() {
let mut items = vec![1];
{
let item = items.last();
}
items.push(2);
}
This trick is no longer needed in modern Rust because non-lexical lifetimes have been implemented, but the underlying restriction still remains — you cannot have a mutable reference while there are other references to the same thing. This is one of the rules of references covered in The Rust Programming Language. A modified example that still does not work with NLL:
let mut items = vec![1];
let item = items.last();
items.push(2);
println!("{:?}", item);
In other cases, you can copy or clone the value in the vector. The item will no longer be a reference and you can modify the vector as you see fit:
fn main() {
let mut items = vec![1];
let item = items.last().cloned();
items.push(2);
}
If your type isn't cloneable, you can transform it into a reference-counted value (such as Rc or Arc) which can then be cloned. You may or may not also need to use interior mutability:
struct NonClone;
use std::rc::Rc;
fn main() {
let mut items = vec![Rc::new(NonClone)];
let item = items.last().cloned();
items.push(Rc::new(NonClone));
}
this example from Programming Rust is quite similar
No, it's not, seeing as how it doesn't use references at all.
See also
Cannot borrow `*x` as mutable because it is also borrowed as immutable
Pushing something into a vector depending on its last element
Why doesn't the lifetime of a mutable borrow end when the function call is complete?
How should I restructure my graph code to avoid an "Cannot borrow variable as mutable more than once at a time" error?
Why do I get the error "cannot borrow x as mutable more than once"?
Why does Rust want to borrow a variable as mutable more than once at a time?
Try to put your immutable borrow inside a block {...}.
This ends the borrow after the block.
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn some_test() {
let mut graph = mk_graph();
graph.add_node("source".to_string());
graph.add_node("destination".to_string());
{
let source = graph.get_node_by_value("source").unwrap();
let dest = graph.get_node_by_value("destination").unwrap();
}
graph.add_node("destination".to_string());
}
}
So for anyone else banging their head against this problem and wanting a quick way out - use clones instead of references. Eg I'm iterating this list of cells and want to change an attribute so I first copy the list:
let created = self.cells
.into_iter()
.map(|c| {
BoardCell {
x: c.x,
y: c.y,
owner: c.owner,
adjacency: c.adjacency.clone(),
}
})
.collect::<Vec<BoardCell>>();
And then modify the values in the original by looping the copy:
for c in created {
self.cells[(c.x + c.y * self.size) as usize].adjacency[dir] = count;
}
Using Vec<&BoardCell> would just yield this error. Not sure how Rusty this is but hey, it works.

Swapping values between two hashmaps

Edit Note: This code now compile see What are non-lexical lifetimes?.
I have two HashMaps and want to swap a value between them under certain conditions. If the key does not exist in the second HashMap, it should be inserted. I do not want to clone the value, since that is too expensive.
The (simplified) critical code that is not working is as follows:
use std::collections::HashMap;
use std::collections::hash_map::Entry;
use std::mem;
#[derive(Debug)]
struct ExpensiveStruct {
replace: bool,
// imagine a lot of heap data here
}
fn main() {
let mut hm : HashMap<usize, ExpensiveStruct> = HashMap::new();
let mut hm1 : HashMap<usize, ExpensiveStruct> = HashMap::new();
let dummy = ExpensiveStruct { replace: false };
hm.insert(1, ExpensiveStruct { replace: true});
hm1.insert(1, ExpensiveStruct { replace: true});
match hm1.get_mut(&1) {
Some(ref mut x) =>
match hm.entry(1) {
Entry::Occupied(mut y) => { if y.get().replace {
mem::swap(x, &mut y.get_mut());
}
},
Entry::Vacant(y) => { y.insert(mem::replace(x, dummy)); }
},
None => {}
}
println!("{:?}", hm);
}
(On the Rust Playground)
I get the error:
error[E0597]: `y` does not live long enough
--> src/main.rs:28:9
|
23 | mem::swap(x, &mut y.get_mut());
| - borrow occurs here
...
28 | },
| ^ `y` dropped here while still borrowed
29 | None => {}
30 | }
| - borrowed value needs to live until here
I am really confused about this borrow problem and I do not see a way to fix it. If I replace the Entry by a match hm.get_mut(1), I cannot insert in the None case, since the matching mutably borrows the HashMap.
You're giving references to references where you should be giving references.
&mut y.get_mut()
for instance is
&mut &mut ExpensiveStruct
and you're having a simular issue with
match hm1.get_mut(&1) {
Some(ref mut x) =>
Your code works as expected when the types are trimmed down to &mut ExpensiveStruct:
match hm1.get_mut(&1) {
Some(x) => match hm.entry(1) {
Entry::Occupied(mut y) => if y.get().replace {
mem::swap(x, y.get_mut());
(On the Playground)
Note that the ref mut is removed, because hm1.get_mut(&1) already returns an Option for a mutable reference, and the &mut is removed because y.get_mut() already returns a reference.

Resources