I am trying to make a prime sieve in Rust.
I need several mutable references to the same element of the array, not mutable references to different parts of the array.
For the way this works, I know data races are not a relevant problem, so multiple mutable references are acceptable, but the Rust compiler does not accept my unsafe code.
I am using crossbeam 0.8.0.
fn extend_helper(
primes: &Vec<usize>,
true_block: &mut Vec<bool>,
segment_min: usize,
segment_len: usize,
) {
crossbeam::scope(|scope| {
for prime in primes {
let prime = prime.clone();
let segment_min = &segment_min;
let segment_len = &segment_len;
let shared = unsafe { &mut *true_block };
scope.spawn(move |_| {
let tmp = smallest_multiple_of_n_geq_m(prime, *segment_min) - *segment_min;
for j in (tmp..*segment_len).step_by(prime) {
shared[j] = false;
}
});
}
})
.unwrap();
}
fn smallest_multiple_of_n_geq_m(n: usize, m: usize) -> usize {
m + ((n - (m % n)) % n)
}
error[E0499]: cannot borrow `*true_block` as mutable more than once at a time
--> src/lib.rs:12:35
|
7 | crossbeam::scope(|scope| {
| ----- has type `&Scope<'1>`
...
12 | let shared = unsafe { &mut *true_block };
| ^^^^^^^^^^^^^^^^ `*true_block` was mutably borrowed here in the previous iteration of the loop
13 |
14 | / scope.spawn(move |_| {
15 | | let tmp = smallest_multiple_of_n_geq_m(prime, *segment_min) - *segment_min;
16 | | for j in (tmp..*segment_len).step_by(prime) {
17 | | shared[j] = false;
18 | | }
19 | | });
| |______________- argument requires that `*true_block` is borrowed for `'1`
warning: unnecessary `unsafe` block
--> src/lib.rs:12:26
|
12 | let shared = unsafe { &mut *true_block };
| ^^^^^^ unnecessary `unsafe` block
|
= note: `#[warn(unused_unsafe)]` on by default
How I should write the unsafe in a way that Rust accepts?
Rust does not allow that in its model. You want AtomicBool with relaxed ordering.
This is a somewhat common edge case in most concurrency models - if you have two non-atomic writes of the same value to one location, is that well-defined? In Rust (and C++) it is not, and you need to explicitly use atomic.
On any platform you care about, the atomic bool store with relaxed ordering will have no impact.
For a reason why this matters, consider:
pub fn do_the_thing(x: &mut bool) {
if !*x { return };
// Do some stuff.
*x = true;
}
At setting x to true, the compiler is free to assume (due to no shared mutable references) that x is still false. It may implement this assignment as, e.g., inc x in assembly, moving its value from 0 to 1.
If two threads came through this and both reached *x = true, its value may become something other than 0 or 1, which could violate other assumed invariants.
Related
This question already has an answer here:
How can I pass a reference to a stack variable to a thread?
(1 answer)
Closed last month.
How do I move a vector reference into threads? The closest I get is the (minimized) code below. (I realize that the costly calculation still isn't parallel, as it is locked by the mutex, but one problem at a time.)
Base problem: I'm calculating values based on information saved in a vector. Then I'm storing the results as nodes per vector element. So vector in vector (but only one vector in the example code below). The calculation takes time so I would like to divide it into threads. The structure is big, so I don't want to copy it.
use std::sync::{Arc, Mutex};
use std::thread;
fn main() {
let n = Nodes::init();
n.calc();
println!("Result: nodes {:?}", n);
}
#[derive(Debug)]
struct Nodes {
nodes: Vec<Info>,
}
impl Nodes {
fn init() -> Self {
let mut n = Nodes { nodes: Vec::new() };
n.nodes.push(Info::init(1));
n.nodes.push(Info::init(2));
n
}
fn calc(&self) {
Nodes::calc_associative(&self.nodes);
}
fn calc_associative(nodes: &Vec<Info>) {
let mut handles = vec![];
let arc_nodes = Arc::new(nodes);
let counter = Arc::new(Mutex::new(0));
for _ in 0..2 {
let arc_nodes = Arc::clone(&arc_nodes);
let counter = Arc::clone(&counter);
let handle = thread::spawn(move || {
let mut idx = counter.lock().unwrap();
// costly calculation
arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
*idx += 1;
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
}
}
#[derive(Debug)]
struct Info {
length: u32,
}
impl Info {
fn init(length: u32) -> Self {
Info { length }
}
fn get_length(&self) -> u32 {
self.length
}
fn set_length(&mut self, x: u32) {
self.length = x;
}
}
The compiler complains that life time of the reference isn't fulfilled, but isn't that what Arc::clone() should do? Then Arc require a deref, but maybe there are better solutions before starting to dig into that...?
Compiling threads v0.1.0 (/home/freefox/proj/threads)
error[E0596]: cannot borrow data in an `Arc` as mutable
--> src/main.rs:37:17
|
37 | arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
| ^^^^^^^^^ cannot borrow as mutable
|
= help: trait `DerefMut` is required to modify through a dereference, but it is not implemented for `Arc<&Vec<Info>>`
error[E0521]: borrowed data escapes outside of associated function
--> src/main.rs:34:26
|
25 | fn calc_associative(nodes: &Vec<Info>) {
| ----- - let's call the lifetime of this reference `'1`
| |
| `nodes` is a reference that is only valid in the associated function body
...
34 | let handle = thread::spawn(move || {
| __________________________^
35 | | let mut idx = counter.lock().unwrap();
36 | | // costly calculation
37 | | arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
38 | | *idx += 1;
39 | | });
| | ^
| | |
| |______________`nodes` escapes the associated function body here
| argument requires that `'1` must outlive `'static`
Some errors have detailed explanations: E0521, E0596.
For more information about an error, try `rustc --explain E0521`.
error: could not compile `threads` due to 2 previous errors
You wrap a reference with Arc. Now the type is Arc<&Vec<Info>>. There is still a reference here, so the variable could still be destroyed before the thread return and we have a dangling reference.
Instead, you should take a &Arc<Vec<Info>>, and on the construction of the Vec wrap it in Arc, or take &[Info] and clone it (let arc_nodes = Arc::new(nodes.to_vec());). You also need a mutex along the way (either Arc<Mutex<Vec<Info>>> or Arc<Vec<Mutex<Info>>>), since you want to change the items.
Or better, since you immediately join() the threads, use scoped threads:
fn calc_associative(nodes: &[Mutex<Info>]) {
let counter = std::sync::atomic::AtomicUsize::new(0); // Changed to atomic, prefer it to mutex wherever possible
std::thread::scope(|s| {
for _ in 0..2 {
s.spawn(|| {
let idx = counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
let node = &mut *nodes[idx].lock().unwrap();
// costly calculation
node.set_length(node.get_length() * 2);
});
}
});
}
Context
I have a function that will use rayon to paralellize some work. Each thread should write its output to a separate consumer. In practice, these consumers will usually write to (separate) files on disc. For testing, I want them to append to vectors, which I can then make assertions on.
Problem
In fact I hit an issue even with a serial version of this code.
The function I want to test is func_under_test. I want get_sink to create and hand out a new 'sink' (a Vec<i32>) each time it is called, which will be inside func_under_test. But I also want to keep a reference to these vectors in my main function, so I can make assertions on them. I've tried writing this:
fn func_under_test<GetSink, F>(
mut get_sink: GetSink
)
where
GetSink: FnMut(usize) -> F,
F: FnMut(i32) -> (),
{
let idxs : Vec<usize> = (0..2).collect();
idxs.iter().for_each(|&i| {
let mut sink = get_sink(i);
sink(0i32);
sink(1i32);
sink(2i32);
});
}
fn main() {
let mut sinks : Vec<&Vec<i32>> = vec![];
let get_sink = |i: usize| {
let mut sink : Vec<i32> = vec![];
sinks.push(&sink);
move |x : i32| sink.push(x)
};
func_under_test(get_sink);
assert_eq!(**sinks.get(0).unwrap(), vec![0i32, 1i32, 2i32]);
}
But I get this compilation error:
error[E0597]: `sink` does not live long enough
--> src/main.rs:23:20
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
25 | };
| - `sink` dropped here while still borrowed
error[E0505]: cannot move out of `sink` because it is borrowed
--> src/main.rs:24:9
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------------
| | |
| | borrow of `sink` occurs here
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
| ^^^^^^^^^^^^^^ ---- move occurs due to use in closure
| |
| move out of `sink` occurs here
Am I correct in thinking that I need to use something like an Rc in this context, since I need to have two references to each Vec<i32>?
Since only one of those references needs to be mutable, do I need to use Rc<Cell<Vec<i32>>> in one case, and just Rc<Vec<i32>> in the other?
You could use an Arc<Mutex<i32>> to share mutable access to the vectors. (Rc is not thread-safe.)
A tidier solution would be typed_arena::Arena, which would allow you to write essentially the code you tried, but with the roles swapped: the Vec<i32>s are always owned by the arena (thus it outlives func_under_tests). Then, after all of the references to the arena are gone, you can convert it into a vector of its elements.
use typed_arena::Arena;
fn main() {
let sinks: Arena<Vec<i32>> = Arena::new();
let get_sink = |_i: usize| {
let sink_ref = sinks.alloc(Vec::new());
move |x| sink_ref.push(x)
};
func_under_test(get_sink);
assert_eq!(sinks.into_vec()[0], vec![0i32, 1i32, 2i32]);
}
I think I've figured out the answer to this. The thread doing the testing needs to have access to the inner vectors, and so does the thread that is populating this. So they need to be wrapped in an Arc and a Mutex. But the outer vector does not, since it is only borrowed by the get_sink closure.
So sinks can have type Vec<Arc<Mutex<Vec<i32>>>.
Anyway, I can write this and it does compile and behave and expected:
use rayon::prelude::*;
use std::sync::{Arc, Mutex};
fn func_under_test<G, S>(
get_sink: G)
where
G: Fn(&usize) -> S + Sync,
S: FnMut(i32) -> (),
{
let idxs : Vec<usize> = vec![0, 1, 2];
idxs.par_iter().for_each(|i| {
let mut sink : S = get_sink(i);
for j in 0..3 {
sink(j + 3*(*i) as i32);
}
});
}
fn main() {
let sinks : Vec<Arc<Mutex<Vec<i32>>>> = vec![
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
];
let get_sink = |i: &usize| {
let m : Arc<Mutex<Vec<i32>>> = sinks.get(*i).unwrap().clone();
move |x: i32| {
m.lock().unwrap().push(x);
}
};
func_under_test(get_sink);
assert_eq!(*sinks.get(0).unwrap().lock().unwrap(), vec![0, 1, 2]);
assert_eq!(*sinks.get(1).unwrap().lock().unwrap(), vec![3, 4, 5]);
assert_eq!(*sinks.get(2).unwrap().lock().unwrap(), vec![6, 7, 8]);
}
It also seems like G implements the correct traits here: it is Sync, since it needs to be shared between threads. But is is not Send, since it does not need to send information between threads.
I want to use a double For loop to compare each element with each and if these properties are the same, make a change to the objects. Below is a little demo code of what I want to do.
But the Rust compiler tells me here that I can't have 2 mutable references from the same object. How can I implement the whole thing differently?
fn main() {
struct S {
a: i32
}
let mut v = vec![S{a: 1}, S{a: 1}, S{a: 1}];
let size = v.len();
for i in 0..size {
for j in 0..size {
if i == j {
continue;
}
let a = &mut v[i];
let b = &mut v[j];
if a.a == b.a {
a.a = 5;
}
}
}
}
Console:
error[E0499]: cannot borrow `v` as mutable more than once at a time
--> src\main.rs:35:26
|
34 | let a = &mut v[i];
| - first mutable borrow occurs here
35 | let b = &mut v[j];
| ^ second mutable borrow occurs here
36 |
37 | if a.a == b.a {
| --- first borrow later used here
First thing to note is that you don't need two mutable references since you are only mutating v[i].
Writing it like this will create two immutable borrows and inside the if a mutable borrow:
if v[i].a == v[j].a {
v[i].a = 5;
}
Being very explicit this is what is happening (but its nicer the other way):
let a = &v[i];
let b = &v[j];
if a.a == b.a {
let mut_a = &mut v[i];
mut_a.a = 5;
}
One of the rules of the borrow checker is that no two mutable borrows can exist to the same thing at the same time.
Here is the code:
fn test(){
let mut numbers = vec![2];
let f = || {
for _ in numbers.iter(){
}
false
};
while false {
let res = f();
if res {
numbers.push(10);
}
}
}
The error is:
|
15 | let f = || {
| -- immutable borrow occurs here
16 | for _ in numbers.iter(){
| ------- first borrow occurs due to use of `numbers` in closure
...
22 | let res = f();
| - immutable borrow later used here
23 | if res {
24 | numbers.push(10);
| ^^^^^^^^^^^^^^^^ mutable borrow occurs here
But if I change the while keyword to if, it can be compiled. How to fix this? I want to call the anonymous function in a loop.
We can simplify your example even more by replacing the closure by a simple immutable reference
let mut numbers = vec![2];
let r = &numbers;
while false {
println!("{:?}", r);
numbers.push(10);
}
Here we get this error:
error[E0502]: cannot borrow `numbers` as mutable because it is also borrowed as immutable
--> src/lib.rs:7:5
|
3 | let r = &numbers;
| -------- immutable borrow occurs here
...
6 | println!("{:?}", r); // use reference
| - immutable borrow later used here
7 | numbers.push(10);
| ^^^^^^^^^^^^^^^^ mutable borrow occurs here
And like in your example, replacing the while with if makes the error go away. Why?
You probably know about the important Rust rule: Aliasing nand mutability. It states that, at any given time, a value can either be borrowed immutably arbitrarily many times or mutably exactly once.
The statement numbers.push(10) borrows numbers mutably temporarily (just for the statement). But we also have r which is an immutable reference. In order for numbers.push(10) to work, the compiler has to make sure that no other borrow exists at that time. But there exists the reference r! This reference cannot exist at the same time as numbers.push(10) exists.
Let's see for the if case first:
let mut numbers = vec![2];
let r = &numbers; // <------+ (creation of r)
// |
if false { // |
println!("{:?}", r); // <------+ (last use of r)
numbers.push(10);
}
While the lexical scope means the variable r is only dropped at the end of the function, due to non-lexical lifetimes, the compiler can see that the last use of r is in the println line. Then the compiler can mark r as "dead" after this line. And this in turn means, that there is no other borrow in the line numbers.push(10) and everything works out fine.
And for the loop case? Let's imagine the loop iterating three times:
let mut numbers = vec![2];
let r = &numbers; // <------+ (creation of r)
// |
// First iteration // |
println!("{:?}", r); // |
numbers.push(10); // | <== oh oh!
// |
// Second iteration // |
println!("{:?}", r); // |
numbers.push(10); // |
// |
// Third iteration // |
println!("{:?}", r); // <------+ (last use of r)
numbers.push(10);
As can be seen here, the time in which r is active overlaps numbers.push(10) (except the last one). And as a result, the compiler will produce an error because this code violates the central Rust rule.
And the explanation is the same for your closure case: the closure borrows numbers immutably and f(); uses that closure. In the loop case, the compiler is not able to shrink the "alive time" of the closure enough to make sure it doesn't overlap the mutable borrow for push.
How to fix?
Well, you could pass numbers into the closure each time:
let mut numbers = vec![2];
let f = |numbers: &[i32]| {
for _ in numbers.iter(){
}
false
};
while false {
let res = f(&numbers);
if res {
numbers.push(10);
}
}
This works because now, numbers is borrowed immutably also just temporarily for the f(&numbers); statement.
You can also use a RefCell as the other answer suggested, but that should be a last resort.
It's not exactly sure what you're trying to accomplish, but one way to solve this, without changing your code too drastically, would be to use std::cell::RefCell (described in the std and in the book):
use std::cell::RefCell;
fn test(){
let numbers = RefCell::new(vec![2]);
let f = || {
for _ in numbers.borrow().iter(){
}
false
};
while false {
let res = f();
if res {
numbers.borrow_mut().push(10);
}
}
}
Here's a little bit tweaked demo, which actually does something:
use std::cell::RefCell;
fn main() {
test();
}
fn test() {
let numbers = RefCell::new(vec![0]);
let f = || {
for n in numbers.borrow().iter() {
println!("In closure: {}", n);
}
println!();
true
};
let mut i = 1;
while i <= 3 {
let res = f();
if res {
numbers.borrow_mut().push(i);
}
i += 1;
}
println!("End of test(): {:?}", numbers.borrow());
}
Output:
In closure: 0
In closure: 0
In closure: 1
In closure: 0
In closure: 1
In closure: 2
End of test(): [0, 1, 2, 3]
Rust Playground demo
I have two vectors of itertools::MinMaxResult. I need to iterate through the first vector, and for each element, iterate through the second vector, checking if the minimum of the first vector is equal to the maximum of any elements of the second vector, and vice versa. Here is an MCVE of what I tried:
use itertools::MinMaxResult; // itertools = "0.8.0"
use itertools::MinMaxResult::*;
pub fn mcve() -> Vec<(usize, usize)> {
// dummy variables to make the MCVE compile
let num_rows = 0;
let num_cols = 0;
let row_minmax: Vec<MinMaxResult<&u64>> = vec![];
let col_minmax: Vec<MinMaxResult<&u64>> = vec![];
// Problematic code:
(0..num_rows)
.flat_map(|row_index| {
(0_usize..num_cols).filter_map(|col_index| {
match (row_minmax[row_index], col_minmax[col_index]) {
(MinMax(a, _b), MinMax(_c, d)) if a == d =>
Some((row_index, col_index)),
(MinMax(_a, b), MinMax(c, _d)) if b == c =>
Some((row_index, col_index)),
_ => None,
}
})
})
.collect::<Vec<(usize, usize)>>()
}
Link to the Playground with full code
I'm getting the following error:
error[E0373]: closure may outlive the current function, but it borrows `row_index`,
which is owned by the current function
--> src/main.rs:15:48
|
15 | (0_usize..num_cols).filter_map(|col_index| {
| ^^^^^^^^^^^ may outlive
borrowed value `row_index`
16 | match (row_minmax[row_index], col_minmax[col_index]) {
| --------- `row_index` is borrowed here
|
note: closure is returned here
--> src/main.rs:15:17
|
15 | / (0_usize..num_cols).filter_map(|col_index| {
16 | | match (row_minmax[row_index], col_minmax[col_index]) {
17 | | (MinMax(a, _b), MinMax(_c, d)) if a == d =>
Some((row_index, col_index)),
18 | | (MinMax(_a, b), MinMax(c, _d)) if b == c =>
Some((row_index, col_index)),
19 | | _ => None,
20 | | }
21 | | })
| |__________________^
help: to force the closure to take ownership of `row_index`
(and any other referenced variables), use the `move` keyword
|
15 | (0_usize..num_cols).filter_map(move |col_index| {
| ^^^^^^^^^^^^^^^^
If I add a move, as the compiler suggests, I get twice as many errors, so that doesn't help. How do I get rid of this error?
Your linked gist has a whole bunch of the typical "wrestling with the borrow-checker" errors. Here is how to solve them:
use move as the compiler proposes, but make sure that all moved values implement Copy.
Iteration indices of type usize already implement Copy, so there is nothing to do here.
The vectors are problematic, so just take immutable references of them, and pass the immutable references into the closures instead.
Your into_par_iter causes an unrelated ownership problem. Just use par_iter instead, then you will iterate over references, not the actual values, and you will not destroy your matrix while you are iterating over it.
This here compiles:
use itertools::MinMaxResult; // itertools = "0.8.0"
use itertools::MinMaxResult::*;
use itertools::Itertools;
use rayon::prelude::*; // rayon = "1.0.3"
pub fn find_saddle_points(input: &[Vec<u64>]) -> Vec<(usize, usize)> {
let flattened_matrix: Vec<(&u64)> = input.into_par_iter().flatten().collect();
if flattened_matrix.is_empty() {
vec![]
} else {
let num_rows = input.len();
let num_cols = input[0].len();
let row_minmax: Vec<MinMaxResult<&u64>> = input
.iter()
.map(|row| row.iter().minmax())
.collect::<Vec<MinMaxResult<&u64>>>();
let input_tranpose: Vec<Vec<u64>> = (0_usize..num_cols)
.into_par_iter()
.map(|col_index| {
(0_usize..num_rows)
.map(|row_index| input[row_index][col_index])
.collect::<Vec<u64>>()
})
.collect();
// don't take ownership!
// Use `par_iter` instead, then `col` will be a reference,
// and the borrow-checker is happy.
let col_minmax: Vec<MinMaxResult<&u64>> = input_tranpose
.par_iter()
.map(|col| col.iter().minmax())
.collect();
let rmm_ref = &row_minmax;
let cmm_ref = &col_minmax;
(0_usize..num_rows)
.flat_map(|row_index| {
(0_usize..num_cols).filter_map(move |col_index| {
match (rmm_ref[row_index], cmm_ref[col_index]) {
(MinMax(a, _b), MinMax(_c, d)) if a == d =>
Some((row_index, col_index)),
(MinMax(_a, b), MinMax(c, _d)) if b == c =>
Some((row_index, col_index)),
_ => None,
}
})
})
.collect::<Vec<(usize, usize)>>()
}
}
fn main(){}