Split/gather pattern for jobs - multithreading

Question

I have a set of jobs that I am trying to run in parallel. I want to run each task on its own thread and gather the responses on the calling thread.
Some jobs may take much longer than others, so I'd like to start using each result as it comes in, and not have to wait for all jobs to complete.
Here is an attempt:
struct Container<T> {
items : Vec<T>
}
#[derive(Debug)]
struct Item {
x: i32
}
impl Item {
fn foo (&mut self) {
self.x += 1; //consider an expensive mutating computation
}
}
fn main() {
use std;
use std::sync::{Mutex, Arc};
use std::collections::RingBuf;
//set up a container with 2 items
let mut item1 = Item { x: 0};
let mut item2 = Item { x: 1};
let container = Container { items: vec![item1, item2]};
//set a gather system for our results
let ringBuf = Arc::new(Mutex::new(RingBuf::<Item>::new()));
//farm out each job to its own thread...
for item in container.items {
std::thread::Thread::spawn(|| {
item.foo(); //job
ringBuf.lock().unwrap().push_back(item); //push item back to caller
});
}
loop {
let rb = ringBuf.lock().unwrap();
if rb.len() > 0 { //gather results as soon as they are available
println!("{:?}",rb[0]);
rb.pop_front();
}
}
}
For starters, this does not compile due to the impenetrable cannot infer an appropriate lifetime due to conflicting requirements error.
What am I doing wrong and how do I do it right?

Answer 1

You've got a couple compounding issues, but the first one is a misuse / misunderstanding of Arc. You need to give each thread it's own copy of the Arc. Arc itself will make sure that changes are synchronized. The main changes were the addition of .clone() and the move keyword:
for item in container.items {
let mrb = ringBuf.clone();
std::thread::Thread::spawn(move || {
item.foo(); //job
mrb.lock().unwrap().push_back(item); //push item back to caller
});
}
After changing this, you'll run into some simpler errors about forgotten mut qualifiers, and then you hit another problem - you are trying to send mutable references across threads. Your for loop will need to return &mut Item to call foo, but this doesn't match your Vec. Changing it, we can get to something that compiles:
for mut item in container.items.into_iter() {
let mrb = ringBuf.clone();
std::thread::Thread::spawn(move || {
item.foo(); //job
mrb.lock().unwrap().push_back(item); //push item back to caller
});
}
Here, we consume the input vector, moving each of the Items to the worker thread. Unfortunately, this hits the Playpen timeout, so there's probably some deeper issue.
All that being said, I'd highly recommend using channels:
#![feature(std_misc)]
use std::sync::mpsc::channel;
#[derive(Debug)]
struct Item {
x: i32
}
impl Item {
fn foo(&mut self) { self.x += 1; }
}
fn main() {
let items = vec![Item { x: 0 }, Item { x: 1 }];
let rx = {
let (tx, rx) = channel();
for item in items.into_iter() {
let my_tx = tx.clone();
std::thread::Thread::spawn(move || {
let mut item = item;
item.foo();
my_tx.send(item).unwrap();
});
}
rx
};
for item in rx.iter() {
println!("{:?}", item);
}
}
This also times-out in the playpen, but works fine when compiled and run locally.