Copying in Vectors to a Thread - multithreading

Considering the following code, I wish to have access to both client and requests within my thread, currently I do not:
for _x in 0..100 {
let handle = thread::spawn(move || {
let start = time::precise_time_s();
let res = client.get("http://jacob.uk.com")
.header(Connection::close())
.send().unwrap();
let end = time::precise_time_s();
requests.push(Request::new(end-start));
});
handle.join().unwrap()
}
I get the following compiler error:
Compiling Herd v0.1.0 (file:///Users/jacobclark/Desktop/LearningRust/Herd)
src/main.rs:41:23: 41:29 error: capture of moved value: `client`
src/main.rs:41 let res = client.get("http://jacob.uk.com")
^~~~~~
src/main.rs:38:41: 48:10 note: `client` moved into closure environment here because it has type `[closure(())]`, which is non-copyable
src/main.rs:38 let handle = thread::spawn(move || {
src/main.rs:39 let start = time::precise_time_s();
src/main.rs:40
src/main.rs:41 let res = client.get("http://jacob.uk.com")
src/main.rs:42 .header(Connection::close())
src/main.rs:43 .send().unwrap();
...
src/main.rs:38:41: 48:10 help: perhaps you meant to use `clone()`?
src/main.rs:47:13: 47:21 error: capture of moved value: `requests`
src/main.rs:47 requests.push(Request::new(end-start));
^~~~~~~~
src/main.rs:38:41: 48:10 note: `requests` moved into closure environment here because it has type `[closure(())]`, which is non-copyable
src/main.rs:38 let handle = thread::spawn(move || {
src/main.rs:39 let start = time::precise_time_s();
src/main.rs:40
src/main.rs:41 let res = client.get("http://jacob.uk.com")
src/main.rs:42 .header(Connection::close())
src/main.rs:43 .send().unwrap();
...
src/main.rs:38:41: 48:10 help: perhaps you meant to use `clone()`?
src/main.rs:53:24: 53:32 error: use of moved value: `requests`
src/main.rs:53 Request::mean_time(requests);
^~~~~~~~
src/main.rs:38:41: 48:10 note: `requests` moved into closure environment here because it has type `[closure(())]`, which is non-copyable
src/main.rs:38 let handle = thread::spawn(move || {
src/main.rs:39 let start = time::precise_time_s();
src/main.rs:40
src/main.rs:41 let res = client.get("http://jacob.uk.com")
src/main.rs:42 .header(Connection::close())
src/main.rs:43 .send().unwrap();
...
src/main.rs:38:41: 48:10 help: perhaps you meant to use `clone()`?
error: aborting due to 3 previous errors
Could not compile `Herd`
.

Firstly, minimal examples are really useful:
use std::thread;
fn main() {
let mut items = Vec::new();
for _ in 0..100 {
thread::spawn(move || {
items.push(());
});
}
}
So what's the problem here? Well, you're moving items into a closure 100 times - but you can only move it once!
To share data across multiple threads, you need
To remove data races - put it in a Mutex (or don't mutate it).
To keep it alive - put it in an Arc (Atomically Reference-Counted pointer).
use std::sync::{Arc, Mutex};
use std::thread;
fn main() {
let items = Arc::new(Mutex::new(Vec::new()));
for _ in 0..10 {
let thread_items = items.clone();
let handle = thread::spawn(move || {
thread_items.lock().unwrap().push(());
});
handle.join().unwrap();
}
println!("{:?}", items);
}

Related

Force 'static lifetime for an Arc<Mutex<i32>>

I would like to make a variant of this program where my goal is to spawn a new thread calling a callback inside another spawned thread and the cb is declared outside the handle and keeping full parallelisation :
let counter = Arc::new(Mutex::new(0));
let cb = || {
let counter = Arc::clone(&counter);
for _ in 0..9 {
let mut num = counter.lock().unwrap();
*num += 1;
}
};
let handle = thread::spawn(move || {
for _ in 0..9 {
thread::spawn(cb);
}
})
.join();
println!("Result: {}", *counter.lock().unwrap());
Unfortunately, I got this error :
`counter` does not live long enough
borrowed value does not live long enoughrustcE0597
tests_spawn.rs(28, 5): `counter` dropped here while still borrowed
tests_spawn.rs(12, 18): value captured here
tests_spawn.rs(22, 17): argument requires that `counter` is borrowed for `'static`
How can change the lifetime of the counter variable to make it static ?
A cleaner option is probably to use the scoped threads API, which automatically joins the child threads. Plus, you don't even need Arc.
use std::sync::Mutex;
use std::thread;
fn main() {
let counter = Mutex::new(0);
let cb = || {
for _ in 0..9 {
let mut num = counter.lock().unwrap();
*num += 1;
}
};
thread::scope(|s| {
for _ in 0..9 {
s.spawn(cb);
}
});
println!("Result: {}", *counter.lock().unwrap());
}
playground

Sharing arrays between threads in Rust

I'm new to Rust and I'm struggling with some ownership semantics.
The goal is to do some nonsense measurements on multiplying 2 f64 arrays and writing the result in a third array.
In the single-threaded version, a single thread takes care of the whole range. In the multi-threaded version, each thread takes care of a segment of the range.
The single-threaded version is easy, but my problem is with the multithreaded version where I'm struggling with the ownership rules.
I was thinking to use raw pointers, to bypass the borrow checker. But I'm still not able to make it pass.
#![feature(box_syntax)]
use std::time::SystemTime;
use rand::Rng;
use std::thread;
fn main() {
let nCells = 1_000_000;
let concurrency = 1;
let mut one = box [0f64; 1_000_000];
let mut two = box [0f64; 1_000_000];
let mut res = box [0f64; 1_000_000];
println!("Creating data");
let mut rng = rand::thread_rng();
for i in 0..nCells {
one[i] = rng.gen::<f64>();
two[i] = rng.gen::<f64>();
res[i] = 0 as f64;
}
println!("Finished creating data");
let rounds = 100000;
let start = SystemTime::now();
let one_raw = Box::into_raw(one);
let two_raw = Box::into_raw(two);
let res_raw = Box::into_raw(res);
let mut handlers = Vec::new();
for _ in 0..rounds {
let sizePerJob = nCells / concurrency;
for j in 0..concurrency {
let from = j * sizePerJob;
let to = (j + 1) * sizePerJob;
handlers.push(thread::spawn(|| {
unsafe {
unsafe {
processData(one_raw, two_raw, res_raw, from, to);
}
}
}));
}
for j in 0..concurrency {
handlers.get_mut(j).unwrap().join();
}
handlers.clear();
}
let durationUs = SystemTime::now().duration_since(start).unwrap().as_micros();
let durationPerRound = durationUs / rounds;
println!("duration per round {} us", durationPerRound);
}
// Make sure we can find the function in the generated Assembly
#[inline(never)]
pub fn processData(one: *const [f64;1000000],
two: *const [f64;1000000],
res: *mut [f64;1000000],
from: usize,
to: usize) {
unsafe {
for i in from..to {
(*res)[i] = (*one)[i] * (*two)[i];
}
}
}
This is the error I'm getting
error[E0277]: `*mut [f64; 1000000]` cannot be shared between threads safely
--> src/main.rs:38:27
|
38 | handlers.push(thread::spawn(|| {
| ^^^^^^^^^^^^^ `*mut [f64; 1000000]` cannot be shared between threads safely
|
= help: the trait `Sync` is not implemented for `*mut [f64; 1000000]`
= note: required because of the requirements on the impl of `Send` for `&*mut [f64; 1000000]`
note: required because it's used within this closure
--> src/main.rs:38:41
|
38 | handlers.push(thread::spawn(|| {
| ^^
note: required by a bound in `spawn`
--> /home/pveentjer/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:653:8
|
653 | F: Send + 'static,
| ^^^^ required by this bound in `spawn`
[edit] I know that spawning threads is very expensive. I'll convert this to a pool of worker threads that can be recycled once this code is up and running.
You can use chunks_mut or split_at_mut to get non-overlapping slices of one two and res. You can then access different slices from different threads safely. See: documentation for chunks_mut and documentation for split_at_mut
I was able to compile it using scoped threads and chunks_mut. I have removed all the unsafe stuff because there is no need. See the code:
#![feature(box_syntax)]
#![feature(scoped_threads)]
use rand::Rng;
use std::thread;
use std::time::SystemTime;
fn main() {
let nCells = 1_000_000;
let concurrency = 2;
let mut one = box [0f64; 1_000_000];
let mut two = box [0f64; 1_000_000];
let mut res = box [0f64; 1_000_000];
println!("Creating data");
let mut rng = rand::thread_rng();
for i in 0..nCells {
one[i] = rng.gen::<f64>();
two[i] = rng.gen::<f64>();
res[i] = 0 as f64;
}
println!("Finished creating data");
let rounds = 1000;
let start = SystemTime::now();
for _ in 0..rounds {
let size_per_job = nCells / concurrency;
thread::scope(|s| {
for it in one
.chunks_mut(size_per_job)
.zip(two.chunks_mut(size_per_job))
.zip(res.chunks_mut(size_per_job))
{
let ((one, two), res) = it;
s.spawn(|| {
processData(one, two, res);
});
}
});
}
let durationUs = SystemTime::now().duration_since(start).unwrap().as_micros();
let durationPerRound = durationUs / rounds;
println!("duration per round {} us", durationPerRound);
}
// Make sure we can find the function in the generated Assembly
#[inline(never)]
pub fn processData(one: &[f64], two: &[f64], res: &mut [f64]) {
for i in 0..one.len() {
res[i] = one[i] * two[i];
}
}

Sharing a reference in multiple threads inside a function

I want to build a function that takes a HashMap reference as an argument. This HashMap should be shared between threads for read only access. The code example is very simple:
I insert some value into the HashMap, pass it to the function and want antoher thread to read that value. I get an Error that the borrowed value does not live long enough at line let exit_code = test(&m);. Why is this not working?
use std::thread;
use std::collections::HashMap;
use std::sync::{Arc, RwLock };
fn main(){
let mut m: HashMap<u32, f64> = HashMap::new();
m.insert(0, 0.1);
let exit_code = test(&m);
std::process::exit(exit_code);
}
fn test(m: &'static HashMap<u32, f64>) -> i32{
let map_lock = Arc::new(RwLock::new(m));
let read_thread = thread::spawn(move || {
if let Ok(r_guard) = map_lock.read(){
println!("{:?}", r_guard.get(&0).unwrap());
}
});
read_thread.join().unwrap();
return 0;
}
if I don't put the 'static in the function signature for the HashMap argument, Arc::new(RwLock::new(m)); doesn't work. How can I sovlve this problem?
A reference is not safe to share unless is 'static meaning that something will live for the extent of the program. Otherwise the compiler is not able to track the liveliness of the shared element.
You should wrap it outside of the function, and take ownership of an Arc:
use std::thread;
use std::collections::HashMap;
use std::sync::{Arc, RwLock };
fn main(){
let mut map = HashMap::new();
map.insert(0, 0.1);
let m = Arc::new(RwLock::new(map));
let exit_code = test(m);
std::process::exit(exit_code);
}
fn test(map_lock: Arc<RwLock<HashMap<u32, f64>>>) -> i32 {
let read_thread = thread::spawn(move || {
if let Ok(r_guard) = map_lock.read(){
println!("{:?}", r_guard.get(&0).unwrap());
}
});
read_thread.join().unwrap();
return 0;
}
Playground

Creating threads from a list of tasks

I try to set up my first-ever thread machine and ran into the following issue.
We have this state:
use std::{thread, sync::mpsc};
fn main() {
let tasks = vec![String::from("task1"), String::from("task2")];
let (tx, rx) = mpsc::channel();
let tx1 = tx.clone();
let handle = thread::spawn(move || {
for i in 0..(tasks.len()/2 - 1){
let send = format!("{}, {}", tasks[i], String::from("done"));
tx.send(send).unwrap();
}
});
for i in tasks.len()/2..tasks.len() - 1 {
let send = format!("{}, {}", tasks[i], String::from("done"));
tx1.send(send).unwrap();
}
handle.join().unwrap();
for string in rx {
println!("{}", string);
}
}
I try to set up parallel threads, which do something with the strings in the vector. They should work and send until the whole vector is covered. The vector has reliable 100 entries.
The corresponding error message:
error[E0382]: borrow of moved value: `tasks`
--> src/main.rs:19:14
|
6 | let tasks = vec![String::from("task1"), String::from("task2")];
| ----- move occurs because `tasks` has type `Vec<String>`, which does not implement the `Copy` trait
...
11 | let handle = thread::spawn(move || {
| ------- value moved into closure here
12 | for i in 0..(tasks.len()/2 - 1){
| ----- variable moved due to use in closure
...
19 | for i in tasks.len()/2..tasks.len() - 1 {
| ^^^^^ value borrowed here after move
error: aborting due to previous error
For more information about this error, try `rustc --explain E0382`.
It seems like I really do not understand how to set up parallel threads and how to treat channels together with threads and a list of tasks.
As the compiler indicates, the problem is that you moved the vector of tasks when spawning a thread, and then you try to borrow the vector from the main thread while it was already moved.
The problem you need to solve here is how to share the vectors' ownership between the main thread and the spawned thread. One way to do it is to use Arc.
Based on your code, I have wrote an example,
use std::{thread, sync::mpsc};
use std::sync::Arc;
fn main() {
let tasks = Arc::new(vec![String::from("task1"), String::from("task2")]);
let tasks_arc_clone = Arc::clone(&tasks);
let (tx, rx) = mpsc::channel();
let tx1 = tx.clone();
let mid_idx = tasks.len() / 2 - 1;
let handle = thread::spawn(move || {
for i in 0..mid_idx {
let send = format!("{}, {}", tasks_arc_clone[i], String::from("done"));
tx.send(send).unwrap();
}
});
for i in mid_idx..tasks.len() {
let send = format!("{}, {}", tasks[i], String::from("done"));
tx1.send(send).unwrap();
}
drop(tx1);
for string in rx {
println!("{}", string);
}
handle.join().unwrap()
}
Rust playground link, in case you want to play with it.

What's the proper way to use variables defined in the main thread in a child thread in Rust?

I'm new to Rust and still reading the Rust book. Below is my program.
use clap::{App, Arg};
type GenericError = Box<dyn std::error::Error + Send + Sync + 'static>;
type GenericResult<T> = Result<T, GenericError>;
fn main() -> GenericResult<()> {
let matches = App::new("test")
.arg(Arg::new("latency")
.takes_value(true))
.get_matches();
let latency_present = matches.is_present("latency");
let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
let latency_pairs: Vec<&str> = latency.split(",").collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
checker.join().unwrap()?;
Ok(())
}
When I run it, it tells me this:
error[E0597]: `matches` does not live long enough
--> src\main.rs:14:19
|
14 | let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
| ^^^^^^^--------------------
| |
| borrowed value does not live long enough
| argument requires that `matches` is borrowed for `'static`
...
30 | }
| - `matches` dropped here while still borrowed
I don't quite understand the error messages here. But I guess it's because I use latency_pairs in the checker thread and latency_pairs could get dropped while checker is still executing. Is my understanding correct? How to fix the error? I tried for (i, latency_pair) in latency_pairs.clone().iter().enumerate() { in order to pass a cloned value for the thread, but it doesn't help.
latency_pairs holds references into latency which in turn references matches. Thus cloning latency_pairs just clones the references into latency and matches.
Your code would require that latency's type is &'static str but it's actually &'a str where 'a is bound to matches' lifetime.
You can call to_owned() on latency to get an owned value and split the string inside the closure or you can call to_owned() on each of the splits collected in latency_pairs and move that Vec<String> into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
If you need to use latency_pairs outside of the closure, you can clone it before moving it into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let latency_pairs_ = latency_pairs.clone();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs_.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
println!("{:?}", latency_pairs);

Resources