I'm trying to write a closure that uses an Arc by cloning it. Ideally I'd like to have the clone inside the closure, but I'm kinda forced to pass the original Arc, which might be the reason I'm getting the error:
use std::sync::Arc;
use std::sync::Condvar;
use std::sync::Mutex;
use std::collections::VecDeque;
type Fifo<T> = Arc<(Mutex<VecDeque<T>>, Condvar)>;
fn executor(f: Box<dyn Fn()>) {
f();
}
fn main() {
let a = Fifo::<u8>::new(
(Mutex::new(VecDeque::new()), Condvar::new())
);
let r = Box::new(||{
let f = a.clone();
f.0.lock().unwrap().push_back(0);
});
executor(r);
}
Error:
error[E0597]: `a` does not live long enough
--> src/main.rs:19:17
|
18 | let r = Box::new(||{
| -- value captured here
19 | let f = a.clone();
| ^ borrowed value does not live long enough
...
22 | executor(r);
| - cast requires that `a` is borrowed for `'static`
23 | }
| - `a` dropped here while still borrowed
error: aborting due to previous error
I thought changing to
let r = Box::new(||{
//let f = a.clone();
a.0.lock().unwrap().push_back(0);
});
would force the closure to decide to clone a, therefore fixing the problem, but I get the same error.
How can I pass an Arc to a closure?
Clone the Arc outside the closure and then move the clone into the closure. Example:
use std::collections::VecDeque;
use std::sync::Arc;
use std::sync::Condvar;
use std::sync::Mutex;
type Fifo<T> = Arc<(Mutex<VecDeque<T>>, Condvar)>;
fn executor(f: Box<dyn Fn()>) {
f();
}
fn main() {
let a = Fifo::<u8>::new((Mutex::new(VecDeque::new()), Condvar::new()));
let f = a.clone();
let r = Box::new(move || {
f.0.lock().unwrap().push_back(0);
});
executor(r);
}
playground
Related
This question already has an answer here:
How can I pass a reference to a stack variable to a thread?
(1 answer)
Closed last month.
How do I move a vector reference into threads? The closest I get is the (minimized) code below. (I realize that the costly calculation still isn't parallel, as it is locked by the mutex, but one problem at a time.)
Base problem: I'm calculating values based on information saved in a vector. Then I'm storing the results as nodes per vector element. So vector in vector (but only one vector in the example code below). The calculation takes time so I would like to divide it into threads. The structure is big, so I don't want to copy it.
use std::sync::{Arc, Mutex};
use std::thread;
fn main() {
let n = Nodes::init();
n.calc();
println!("Result: nodes {:?}", n);
}
#[derive(Debug)]
struct Nodes {
nodes: Vec<Info>,
}
impl Nodes {
fn init() -> Self {
let mut n = Nodes { nodes: Vec::new() };
n.nodes.push(Info::init(1));
n.nodes.push(Info::init(2));
n
}
fn calc(&self) {
Nodes::calc_associative(&self.nodes);
}
fn calc_associative(nodes: &Vec<Info>) {
let mut handles = vec![];
let arc_nodes = Arc::new(nodes);
let counter = Arc::new(Mutex::new(0));
for _ in 0..2 {
let arc_nodes = Arc::clone(&arc_nodes);
let counter = Arc::clone(&counter);
let handle = thread::spawn(move || {
let mut idx = counter.lock().unwrap();
// costly calculation
arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
*idx += 1;
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
}
}
#[derive(Debug)]
struct Info {
length: u32,
}
impl Info {
fn init(length: u32) -> Self {
Info { length }
}
fn get_length(&self) -> u32 {
self.length
}
fn set_length(&mut self, x: u32) {
self.length = x;
}
}
The compiler complains that life time of the reference isn't fulfilled, but isn't that what Arc::clone() should do? Then Arc require a deref, but maybe there are better solutions before starting to dig into that...?
Compiling threads v0.1.0 (/home/freefox/proj/threads)
error[E0596]: cannot borrow data in an `Arc` as mutable
--> src/main.rs:37:17
|
37 | arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
| ^^^^^^^^^ cannot borrow as mutable
|
= help: trait `DerefMut` is required to modify through a dereference, but it is not implemented for `Arc<&Vec<Info>>`
error[E0521]: borrowed data escapes outside of associated function
--> src/main.rs:34:26
|
25 | fn calc_associative(nodes: &Vec<Info>) {
| ----- - let's call the lifetime of this reference `'1`
| |
| `nodes` is a reference that is only valid in the associated function body
...
34 | let handle = thread::spawn(move || {
| __________________________^
35 | | let mut idx = counter.lock().unwrap();
36 | | // costly calculation
37 | | arc_nodes[*idx].set_length(arc_nodes[*idx].get_length() * 2);
38 | | *idx += 1;
39 | | });
| | ^
| | |
| |______________`nodes` escapes the associated function body here
| argument requires that `'1` must outlive `'static`
Some errors have detailed explanations: E0521, E0596.
For more information about an error, try `rustc --explain E0521`.
error: could not compile `threads` due to 2 previous errors
You wrap a reference with Arc. Now the type is Arc<&Vec<Info>>. There is still a reference here, so the variable could still be destroyed before the thread return and we have a dangling reference.
Instead, you should take a &Arc<Vec<Info>>, and on the construction of the Vec wrap it in Arc, or take &[Info] and clone it (let arc_nodes = Arc::new(nodes.to_vec());). You also need a mutex along the way (either Arc<Mutex<Vec<Info>>> or Arc<Vec<Mutex<Info>>>), since you want to change the items.
Or better, since you immediately join() the threads, use scoped threads:
fn calc_associative(nodes: &[Mutex<Info>]) {
let counter = std::sync::atomic::AtomicUsize::new(0); // Changed to atomic, prefer it to mutex wherever possible
std::thread::scope(|s| {
for _ in 0..2 {
s.spawn(|| {
let idx = counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
let node = &mut *nodes[idx].lock().unwrap();
// costly calculation
node.set_length(node.get_length() * 2);
});
}
});
}
I'm new to Rust and still reading the Rust book. Below is my program.
use clap::{App, Arg};
type GenericError = Box<dyn std::error::Error + Send + Sync + 'static>;
type GenericResult<T> = Result<T, GenericError>;
fn main() -> GenericResult<()> {
let matches = App::new("test")
.arg(Arg::new("latency")
.takes_value(true))
.get_matches();
let latency_present = matches.is_present("latency");
let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
let latency_pairs: Vec<&str> = latency.split(",").collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
checker.join().unwrap()?;
Ok(())
}
When I run it, it tells me this:
error[E0597]: `matches` does not live long enough
--> src\main.rs:14:19
|
14 | let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
| ^^^^^^^--------------------
| |
| borrowed value does not live long enough
| argument requires that `matches` is borrowed for `'static`
...
30 | }
| - `matches` dropped here while still borrowed
I don't quite understand the error messages here. But I guess it's because I use latency_pairs in the checker thread and latency_pairs could get dropped while checker is still executing. Is my understanding correct? How to fix the error? I tried for (i, latency_pair) in latency_pairs.clone().iter().enumerate() { in order to pass a cloned value for the thread, but it doesn't help.
latency_pairs holds references into latency which in turn references matches. Thus cloning latency_pairs just clones the references into latency and matches.
Your code would require that latency's type is &'static str but it's actually &'a str where 'a is bound to matches' lifetime.
You can call to_owned() on latency to get an owned value and split the string inside the closure or you can call to_owned() on each of the splits collected in latency_pairs and move that Vec<String> into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
If you need to use latency_pairs outside of the closure, you can clone it before moving it into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let latency_pairs_ = latency_pairs.clone();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs_.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
println!("{:?}", latency_pairs);
Context
I have a function that will use rayon to paralellize some work. Each thread should write its output to a separate consumer. In practice, these consumers will usually write to (separate) files on disc. For testing, I want them to append to vectors, which I can then make assertions on.
Problem
In fact I hit an issue even with a serial version of this code.
The function I want to test is func_under_test. I want get_sink to create and hand out a new 'sink' (a Vec<i32>) each time it is called, which will be inside func_under_test. But I also want to keep a reference to these vectors in my main function, so I can make assertions on them. I've tried writing this:
fn func_under_test<GetSink, F>(
mut get_sink: GetSink
)
where
GetSink: FnMut(usize) -> F,
F: FnMut(i32) -> (),
{
let idxs : Vec<usize> = (0..2).collect();
idxs.iter().for_each(|&i| {
let mut sink = get_sink(i);
sink(0i32);
sink(1i32);
sink(2i32);
});
}
fn main() {
let mut sinks : Vec<&Vec<i32>> = vec![];
let get_sink = |i: usize| {
let mut sink : Vec<i32> = vec![];
sinks.push(&sink);
move |x : i32| sink.push(x)
};
func_under_test(get_sink);
assert_eq!(**sinks.get(0).unwrap(), vec![0i32, 1i32, 2i32]);
}
But I get this compilation error:
error[E0597]: `sink` does not live long enough
--> src/main.rs:23:20
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
25 | };
| - `sink` dropped here while still borrowed
error[E0505]: cannot move out of `sink` because it is borrowed
--> src/main.rs:24:9
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------------
| | |
| | borrow of `sink` occurs here
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
| ^^^^^^^^^^^^^^ ---- move occurs due to use in closure
| |
| move out of `sink` occurs here
Am I correct in thinking that I need to use something like an Rc in this context, since I need to have two references to each Vec<i32>?
Since only one of those references needs to be mutable, do I need to use Rc<Cell<Vec<i32>>> in one case, and just Rc<Vec<i32>> in the other?
You could use an Arc<Mutex<i32>> to share mutable access to the vectors. (Rc is not thread-safe.)
A tidier solution would be typed_arena::Arena, which would allow you to write essentially the code you tried, but with the roles swapped: the Vec<i32>s are always owned by the arena (thus it outlives func_under_tests). Then, after all of the references to the arena are gone, you can convert it into a vector of its elements.
use typed_arena::Arena;
fn main() {
let sinks: Arena<Vec<i32>> = Arena::new();
let get_sink = |_i: usize| {
let sink_ref = sinks.alloc(Vec::new());
move |x| sink_ref.push(x)
};
func_under_test(get_sink);
assert_eq!(sinks.into_vec()[0], vec![0i32, 1i32, 2i32]);
}
I think I've figured out the answer to this. The thread doing the testing needs to have access to the inner vectors, and so does the thread that is populating this. So they need to be wrapped in an Arc and a Mutex. But the outer vector does not, since it is only borrowed by the get_sink closure.
So sinks can have type Vec<Arc<Mutex<Vec<i32>>>.
Anyway, I can write this and it does compile and behave and expected:
use rayon::prelude::*;
use std::sync::{Arc, Mutex};
fn func_under_test<G, S>(
get_sink: G)
where
G: Fn(&usize) -> S + Sync,
S: FnMut(i32) -> (),
{
let idxs : Vec<usize> = vec![0, 1, 2];
idxs.par_iter().for_each(|i| {
let mut sink : S = get_sink(i);
for j in 0..3 {
sink(j + 3*(*i) as i32);
}
});
}
fn main() {
let sinks : Vec<Arc<Mutex<Vec<i32>>>> = vec![
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
];
let get_sink = |i: &usize| {
let m : Arc<Mutex<Vec<i32>>> = sinks.get(*i).unwrap().clone();
move |x: i32| {
m.lock().unwrap().push(x);
}
};
func_under_test(get_sink);
assert_eq!(*sinks.get(0).unwrap().lock().unwrap(), vec![0, 1, 2]);
assert_eq!(*sinks.get(1).unwrap().lock().unwrap(), vec![3, 4, 5]);
assert_eq!(*sinks.get(2).unwrap().lock().unwrap(), vec![6, 7, 8]);
}
It also seems like G implements the correct traits here: it is Sync, since it needs to be shared between threads. But is is not Send, since it does not need to send information between threads.
Consider the following (contrived) way to increment x by 9.
fn main() {
let mut x = 0;
let mut f = || {
x += 4;
};
let _g = || {
f();
x += 5;
};
}
error[E0499]: cannot borrow `x` as mutable more than once at a time
--> x.rs:6:12
|
3 | let mut f = || {
| -- first mutable borrow occurs here
4 | x += 4;
| - first borrow occurs due to use of `x` in closure
5 | };
6 | let _g = || {
| ^^ second mutable borrow occurs here
7 | f();
| - first borrow later captured here by closure
8 | x += 5;
| - second borrow occurs due to use of `x` in closure
error: aborting due to previous error
For more information about this error, try `rustc --explain E0499`.
So, it does not work. How to make an algorithm like the above that would modify a variable in a closure and call another closure from it that also modifies the variable?
In traditional languages it's easy. What to do in Rust?
The accepted answer is the most idiomatic approach, but there is also an alternative that is useful in situations when the additional argument doesn't work, for example when you need to pass the closure to third-party code that will call it without arguments. In that case you can use a Cell, a form of interior mutability:
use std::cell::Cell;
fn main() {
let x = Cell::new(0);
let f = || {
x.set(x.get() + 4);
};
let g = || {
f();
x.set(x.get() + 5);
};
f();
g();
assert_eq!(x.get(), 13);
}
By design, closures have to enclose external objects that are used in it upon creation, in our case closures have to borrow the external x object. So as compiler explained to you, upon creation of the closure f it mutably borrows x, and you can't borrow it once again when you create closure g.
In order to compile it you can't enclose any external object that you want to change. Instead you can directly pass the objects as a closure's argument (arguably it's even more readable). This way you describe that a closure accepts some object of some type, but you don't yet use/pass any actual object. This object will be borrowed only when you call the closure.
fn main() {
let mut x = 0;
let f = |local_x: &mut i32| { // we don't enclose `x`, so no borrow yet
*local_x += 4;
};
let _g = |local_x: &mut i32| { // we don't enclose `x`, so no borrow yet
f(local_x);
*local_x += 5;
};
_g(&mut x); // finally we borrow `x`, and this borrow will later move to `f`,
// so no simultaneous borrowing.
println!("{}", x); // 9
}
To expand on Alex Larionov's answer: you should think of a closure as a callable structure, anything you capture is set as a field of the structure, then implicitly dereferenced inside the function body. The way those fields are used is also what determines whether the closure is Fn, FnMut or FnOnce, basically whether the method would take &self, &mut self or self if it were written longhand.
Here
fn main() {
let mut x = 0;
let mut f = || {
x += 4;
};
// ...
}
essentially translates to:
struct F<'a> { x: &'a mut u32 }
impl F<'_> {
fn call(&mut self) {
*self.x += 4
}
}
fn main() {
let mut x = 0;
let mut f = F { x: &mut x };
// ...
}
from this, you can see that as soon as f is created, x is mutably borrowed, with all that implies.
And with this partial desugaring, we can see essentially the same error: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=e129f19f25dc61de8a6f42cdca1f67b5
If we use the relevant unstable features on nightly we can get just a bit closer. That is pretty much what rustc does for us under the hood.
I'm trying to use the hyper library to make some requests. The Headers::get() method returns Option<&H>, where H is a tuple struct with one field. I can use if let Some() to destructure the Option. But how do we destructure the &H? Sure I could always access the field with .0, but I'm curious if Rust has a syntax to do this.
struct s(String);
fn f(input: &s) -> &s {
input
}
fn main() {
let my_struct1 = s("a".to_owned());
let s(foo) = my_struct1;
let my_struct2 = s("b".to_owned());
let &s(bar) = f(&my_struct2); // this does not work
let baz = &my_struct2.0; // this works
}
When you try to compile this, the Rust compiler will tell you how to fix the error with a nice message:
error[E0507]: cannot move out of borrowed content
--> <anon>:11:9
|
11 | let &s(bar) = f(&my_struct2); // this does not work
| ^^^---^
| | |
| | hint: to prevent move, use `ref bar` or `ref mut bar`
| cannot move out of borrowed content
This is needed to tell the compiler that you only want a reference to the field in the struct; the default matching will perform a move and the original struct value will no longer be valid.
Let's fix the example:
struct s(String);
fn f(input: &s) -> &s {
input
}
fn main() {
let my_struct1 = s("a".to_owned());
let s(foo) = my_struct1;
let my_struct2 = s("b".to_owned());
let &s(ref bar) = f(&my_struct2);
}
Another way is to dereference first and drop the &. I think this is preferred in Rust:
struct s(String);
fn f(input: &s) -> &s {
input
}
fn main() {
let my_struct1 = s("a".to_owned());
let s(foo) = my_struct1;
let my_struct2 = s("b".to_owned());
let s(ref bar) = *f(&my_struct2);
}