How do I move a vector reference into threads? [duplicate] - multithreading

This question already has an answer here:
How can I pass a reference to a stack variable to a thread?
(1 answer)
Closed last month.
How do I move a vector reference into threads? The closest I get is the (minimized) code below. (I realize that the costly calculation still isn't parallel, as it is locked by the mutex, but one problem at a time.)
Base problem: I'm calculating values based on information saved in a vector. Then I'm storing the results as nodes per vector element. So vector in vector (but only one vector in the example code below). The calculation takes time so I would like to divide it into threads. The structure is big, so I don't want to copy it.
use std::sync::{Arc, Mutex};
use std::thread;
fn main() {
let n = Nodes::init();
n.calc();
println!("Result: nodes {:?}", n);
}
#[derive(Debug)]
struct Nodes {
nodes: Vec<Info>,
}
impl Nodes {
fn init() -> Self {
let mut n = Nodes { nodes: Vec::new() };
n.nodes.push(Info::init(1));
n.nodes.push(Info::init(2));
n
}
fn calc(&self) {
Nodes::calc_associative(&self.nodes);
}
fn calc_associative(nodes: &Vec<Info>) {
let mut handles = vec![];
let arc_nodes = Arc::new(nodes);
let counter = Arc::new(Mutex::new(0));
for _ in 0..2 {
let arc_nodes = Arc::clone(&arc_nodes);
let counter = Arc::clone(&counter);
let handle = thread::spawn(move || {
let mut idx = counter.lock().unwrap();
// costly calculation
arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
*idx += 1;
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
}
}
#[derive(Debug)]
struct Info {
length: u32,
}
impl Info {
fn init(length: u32) -> Self {
Info { length }
}
fn get_length(&self) -> u32 {
self.length
}
fn set_length(&mut self, x: u32) {
self.length = x;
}
}
The compiler complains that life time of the reference isn't fulfilled, but isn't that what Arc::clone() should do? Then Arc require a deref, but maybe there are better solutions before starting to dig into that...?
Compiling threads v0.1.0 (/home/freefox/proj/threads)
error[E0596]: cannot borrow data in an `Arc` as mutable
--> src/main.rs:37:17
|
37 | arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
| ^^^^^^^^^ cannot borrow as mutable
|
= help: trait `DerefMut` is required to modify through a dereference, but it is not implemented for `Arc<&Vec<Info>>`
error[E0521]: borrowed data escapes outside of associated function
--> src/main.rs:34:26
|
25 | fn calc_associative(nodes: &Vec<Info>) {
| ----- - let's call the lifetime of this reference `'1`
| |
| `nodes` is a reference that is only valid in the associated function body
...
34 | let handle = thread::spawn(move || {
| __________________________^
35 | | let mut idx = counter.lock().unwrap();
36 | | // costly calculation
37 | | arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
38 | | *idx += 1;
39 | | });
| | ^
| | |
| |______________`nodes` escapes the associated function body here
| argument requires that `'1` must outlive `'static`
Some errors have detailed explanations: E0521, E0596.
For more information about an error, try `rustc --explain E0521`.
error: could not compile `threads` due to 2 previous errors

You wrap a reference with Arc. Now the type is Arc<&Vec<Info>>. There is still a reference here, so the variable could still be destroyed before the thread return and we have a dangling reference.
Instead, you should take a &Arc<Vec<Info>>, and on the construction of the Vec wrap it in Arc, or take &[Info] and clone it (let arc_nodes = Arc::new(nodes.to_vec());). You also need a mutex along the way (either Arc<Mutex<Vec<Info>>> or Arc<Vec<Mutex<Info>>>), since you want to change the items.
Or better, since you immediately join() the threads, use scoped threads:
fn calc_associative(nodes: &[Mutex<Info>]) {
let counter = std::sync::atomic::AtomicUsize::new(0); // Changed to atomic, prefer it to mutex wherever possible
std::thread::scope(|s| {
for _ in 0..2 {
s.spawn(|| {
let idx = counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
let node = &mut *nodes[idx].lock().unwrap();
// costly calculation
node.set_length(node.get_length() * 2);
});
}
});
}

Related

Cryptic message when dealing with Rust Option type

I have a function like the following:
pub fn process_options<T>(in_refs:& mut Vec<T>,ivals:Vec<Option<T>>) -> i32 {
let mut nones = 0;
for int in 0..ivals.len() {
if let None = ivals[int] {
nones += 1;
} else {
in_refs.push(ivals[int].unwrap());
}
}
return nones;
}
What this function does is that it:
It pushes all the some(T) from one vector to another as T. And it also records the number of none values in the first vector. As written above the code fails to compile and gives me:
error[E0507]: cannot move out of index of `Vec<Option<T>>`
--> src/functions.rs:40:26
|
40 | in_refs.push(ivals[int].unwrap());
| ^^^^^^^^^^ -------- value moved due to this method call
| |
| help: consider calling `.as_ref()` or `.as_mut()` to borrow the type's contents
| move occurs because value has type `Option<T>`, which does not implement the `Copy` trait
|
note: this function takes ownership of the receiver `self`, which moves value
--> /Users/yunfeichen/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:772:25
|
772 | pub const fn unwrap(self) -> T {
Which I frankly do not understand can you explain this to me?
Edit:
This question has been solved:
It turns out I just need to use a different style of for loop, Not sure why this for loop works but it does:
pub fn process_options<T>(in_refs:& mut Vec<T>,ivals:Vec<Option<T>>) -> i32 {
let mut nones = 0;
for element in ivals{
if let None = element {
nones += 1;
} else {
in_refs.push(element.unwrap());
}
}
return nones;
}

How to make multiple mutable references to the same element in an array using unsafe code?

I am trying to make a prime sieve in Rust.
I need several mutable references to the same element of the array, not mutable references to different parts of the array.
For the way this works, I know data races are not a relevant problem, so multiple mutable references are acceptable, but the Rust compiler does not accept my unsafe code.
I am using crossbeam 0.8.0.
fn extend_helper(
primes: &Vec<usize>,
true_block: &mut Vec<bool>,
segment_min: usize,
segment_len: usize,
) {
crossbeam::scope(|scope| {
for prime in primes {
let prime = prime.clone();
let segment_min = &segment_min;
let segment_len = &segment_len;
let shared = unsafe { &mut *true_block };
scope.spawn(move |_| {
let tmp = smallest_multiple_of_n_geq_m(prime, *segment_min) - *segment_min;
for j in (tmp..*segment_len).step_by(prime) {
shared[j] = false;
}
});
}
})
.unwrap();
}
fn smallest_multiple_of_n_geq_m(n: usize, m: usize) -> usize {
m + ((n - (m % n)) % n)
}
error[E0499]: cannot borrow `*true_block` as mutable more than once at a time
--> src/lib.rs:12:35
|
7 | crossbeam::scope(|scope| {
| ----- has type `&Scope<'1>`
...
12 | let shared = unsafe { &mut *true_block };
| ^^^^^^^^^^^^^^^^ `*true_block` was mutably borrowed here in the previous iteration of the loop
13 |
14 | / scope.spawn(move |_| {
15 | | let tmp = smallest_multiple_of_n_geq_m(prime, *segment_min) - *segment_min;
16 | | for j in (tmp..*segment_len).step_by(prime) {
17 | | shared[j] = false;
18 | | }
19 | | });
| |______________- argument requires that `*true_block` is borrowed for `'1`
warning: unnecessary `unsafe` block
--> src/lib.rs:12:26
|
12 | let shared = unsafe { &mut *true_block };
| ^^^^^^ unnecessary `unsafe` block
|
= note: `#[warn(unused_unsafe)]` on by default
How I should write the unsafe in a way that Rust accepts?
Rust does not allow that in its model. You want AtomicBool with relaxed ordering.
This is a somewhat common edge case in most concurrency models - if you have two non-atomic writes of the same value to one location, is that well-defined? In Rust (and C++) it is not, and you need to explicitly use atomic.
On any platform you care about, the atomic bool store with relaxed ordering will have no impact.
For a reason why this matters, consider:
pub fn do_the_thing(x: &mut bool) {
if !*x { return };
// Do some stuff.
*x = true;
}
At setting x to true, the compiler is free to assume (due to no shared mutable references) that x is still false. It may implement this assignment as, e.g., inc x in assembly, moving its value from 0 to 1.
If two threads came through this and both reached *x = true, its value may become something other than 0 or 1, which could violate other assumed invariants.

Swapping 2 Vec properties from mutably referenced object in Rust

In my implementation of Conway's Game of Life in Rust, I've walked into a problem. I want to remember the previous state of the world. For this I've created an extra property on my Universe struct, which now looks like this:
pub struct Universe {
width: usize,
height: usize,
cells: Vec<Cell>,
previous_cells: Option<Vec<Cell>>,
}
The Option is there because at initialization, there is no previous state.
My idea would be that on every tick, we write the new state to the vector with the previous state. At the end, I swap the vectors. This resulted in the following code:
pub fn tick(&mut self) {
// get the previous state or create new one if it doesn't exist (i.e. this is the first tick/update)
let mut next = match self.previous_cells {
Some(cells) => cells,
None => {
let mut new_vec = Vec::with_capacity(self.width() * self.height());
new_vec.resize(new_vec.capacity(), Cell::Dead);
new_vec
}
};
// Calculate next state, not relevant for question
for row in 0..self.height as isize {
for column in 0..self.width as isize {
let index = self.get_index(row , column);
let target_cell = self.cells[index];
let live_neighbors = self.live_neighbor_count(row, column);
next[index] = match target_cell {
Cell::Alive => if live_neighbors == 2 || live_neighbors == 3 { Cell::Alive } else { Cell::Dead }
Cell::Dead => if live_neighbors == 3 { Cell::Alive } else { Cell::Dead }
}
}
}
// swap values
self.previous_cells = Some(self.cells);
self.cells = next;
}
Now I get the following errors from the compiler:
error[E0507]: cannot move out of `self.previous_cells.0` which is behind a mutable reference
--> src/lib.rs:61:30
|
61 | let mut next = match self.previous_cells {
| ^^^^^^^^^^^^^^^^^^^ help: consider borrowing here: `&self.previous_cells`
62 | Some(cells) => cells,
| -----
| |
| data moved here
| move occurs because `cells` has type `std::vec::Vec<Cell>`, which does not implement the `Copy` trait
error[E0507]: cannot move out of `self.cells` which is behind a mutable reference
--> src/lib.rs:83:36
|
83 | self.previous_cells = Some(self.cells);
| ^^^^^^^^^^ move occurs because `self.cells` has type `std::vec::Vec<Cell>`, which does not implement the `Copy` trait
I get why this happens, but I don't see a solution that does not include copying the entire Vec. That would be a solution, but is there maybe a better one?

Iterate through a whole file one character at a time

I'm new to Rust and I'm struggle with the concept of lifetimes. I want to make a struct that iterates through a file a character at a time, but I'm running into issues where I need lifetimes. I've tried to add them where I thought they should be but the compiler isn't happy. Here's my code:
struct Advancer<'a> {
line_iter: Lines<BufReader<File>>,
char_iter: Chars<'a>,
current: Option<char>,
peek: Option<char>,
}
impl<'a> Advancer<'a> {
pub fn new(file: BufReader<File>) -> Result<Self, Error> {
let mut line_iter = file.lines();
if let Some(Ok(line)) = line_iter.next() {
let char_iter = line.chars();
let mut advancer = Advancer {
line_iter,
char_iter,
current: None,
peek: None,
};
// Prime the pump. Populate peek so the next call to advance returns the first char
let _ = advancer.next();
Ok(advancer)
} else {
Err(anyhow!("Failed reading an empty file."))
}
}
pub fn next(&mut self) -> Option<char> {
self.current = self.peek;
if let Some(char) = self.char_iter.next() {
self.peek = Some(char);
} else {
if let Some(Ok(line)) = self.line_iter.next() {
self.char_iter = line.chars();
self.peek = Some('\n');
} else {
self.peek = None;
}
}
self.current
}
pub fn current(&self) -> Option<char> {
self.current
}
pub fn peek(&self) -> Option<char> {
self.peek
}
}
fn main() -> Result<(), Error> {
let file = File::open("input_file.txt")?;
let file_buf = BufReader::new(file);
let mut advancer = Advancer::new(file_buf)?;
while let Some(char) = advancer.next() {
print!("{}", char);
}
Ok(())
}
And here's what the compiler is telling me:
error[E0515]: cannot return value referencing local variable `line`
--> src/main.rs:37:13
|
25 | let char_iter = line.chars();
| ---- `line` is borrowed here
...
37 | Ok(advancer)
| ^^^^^^^^^^^^ returns a value referencing data owned by the current function
error[E0597]: `line` does not live long enough
--> src/main.rs:49:34
|
21 | impl<'a> Advancer<'a> {
| -- lifetime `'a` defined here
...
49 | self.char_iter = line.chars();
| -----------------^^^^--------
| | |
| | borrowed value does not live long enough
| assignment requires that `line` is borrowed for `'a`
50 | self.peek = Some('\n');
51 | } else {
| - `line` dropped here while still borrowed
error: aborting due to 2 previous errors
Some errors have detailed explanations: E0515, E0597.
For more information about an error, try `rustc --explain E0515`.
error: could not compile `advancer`.
Some notes:
The Chars iterator borrows from the String it was created from. So you can't drop the String while the iterator is alive. But that's what happens in your new() method, the line variable owning the String disappears while the iterator referencing it is stored in the struct.
You could also try storing the current line in the struct, then it would live long enough, but that's not an option – a struct cannot hold a reference to itself.
Can you make a char iterator on a String that doesn't store a reference into the String? Yes, probably, for instance by storing the current position in the string as an integer – it shouldn't be the index of the char, because chars can be more than one byte long, so you'd need to deal with the underlying bytes yourself (using e.g. is_char_boundary() to take the next bunch of bytes starting from your current index that form a char).
Is there an easier way? Yes, if performance is not of highest importance, one solution is to make use of Vec's IntoIterator instance (which uses unsafe magic to create an object that hands out parts of itself) :
let char_iter = file_buf.lines().flat_map(|line_res| {
let line = line_res.unwrap_or(String::new());
line.chars().collect::<Vec<_>>()
});
Note that just returning line.chars() would have the same problem as the first point.
You might think that String should have a similar IntoIterator instance, and I wouldn't disagree.

Lifetime issues in rust: borrowed value does not live long enough

I have reproduced my problem in the short code below.
Problem: The inner thread uses reference of variable v from the outer thread. The rust compiler throws an error because "technically" the outer thread could terminate before the inner thread and hence inner thread could loose access to variable v. However in the code below that clearly cannot happen.
Question: How shall I change this code so that it complies while maintaining the same functionality?
fn main() { //outer thread
let v = vec![0, 1];
let test = Test { v: &v }; //inner_thread
std::thread::spawn(move || test.print());
loop {
// this thread will never die because it will never leave this loop
}
}
pub struct Test<'a> {
v: &'a Vec<u32>,
}
impl<'a> Test<'a> {
fn print(&self) {
println!("{:?}", self.v);
}
}
error[E0597]: `v` does not live long enough
--> src/main.rs:3:26
|
3 | let test = Test { v: &v }; //inner_thread
| ^^ borrowed value does not live long enough
4 | std::thread::spawn(move || test.print());
| ---------------------------------------- argument requires that `v` is borrowed for `'static`
...
8 | }
| - `v` dropped here while still borrowed
The obvious solution would be to have Test own the vector instead of just have a reference.
But if you really need to borrow the value in the thread (probably because you want to use it after end of execution), then you may use crossbeam's scope:
let v = vec![0, 1];
let test = Test { v: &v }; //inner_thread
crossbeam::thread::scope(|scope| {
scope.spawn(|_| test.print());
}).unwrap();

Resources