I am modifiying a crate (rust-argon2) to some custom needs. That crate uses multithreading with scoped threads using crossbeam. I want to have a mutable state between threads, that may depend on each other.
The following snippet showcases the situation: fill_memory_blocks_mt gets some mutable state reference state, that will be mutated in fill_segment across multiple threads using crossbeam_utils::thread::scope. They are synced via common::SYNC_POINTS.
fn fill_memory_blocks_mt(context: &Context, memory: &mut Memory, state: &mut Argon2Result) {
for p in 0..context.config.time_cost {
for s in 0..common::SYNC_POINTS {
let _ = scope(|scoped| {
for (l, mem) in (0..context.config.lanes).zip(memory.as_lanes_mut()) {
let position = Position {
pass: p,
lane: l,
slice: s,
index: 0,
};
scoped.spawn(move |_| {
fill_segment(context, &position, mem, state);
});
}
});
}
}
}
The compiler outputs following message:
error[E0382]: use of moved value: `state`
--> src/core.rs:223:34
|
223 | scoped.spawn(move |_| {
| ^^^^^^^^ value moved into closure here, in previous iteration of loop
224 | fill_segment(context, &position, mem, state);
| ----- use occurs due to use in closure
|
= note: move occurs because `state` has type `&mut Argon2Result`, which does not implement the `Copy` trait
help: consider creating a fresh reborrow of `state` here
|
223 | scoped.spawn(&mut *move |_| {
| ++++++
error[E0382]: use of moved value: `state`
--> src/core.rs:215:27
|
212 | fn fill_memory_blocks_mt(context: &Context, memory: &mut Memory, state: &mut Argon2Result) {
| ----- move occurs because `state` has type `&mut Argon2Result`, which does not implement the `Copy` trait
...
215 | let _ = scope(|scoped| {
| ^^^^^^^^ value moved into closure here, in previous iteration of loop
...
224 | fill_segment(context, &position, mem, state);
| ----- use occurs due to use in closure
|
help: consider creating a fresh reborrow of `state` here
|
215 | let _ = scope(&mut *|scoped| {
| ++++++
For more information about this error, try `rustc --explain E0382`.
error: could not compile `rust-argon2-wasm` due to 2 previous errors
Of course I don't want to copy state but use one instance for all threads. They don't overwrite fields between each other, but may depend on each other.
Related
I have a multithreaded program that wants to access resources across threads. Some want to write to them, some want to read from them.
I'm not sure if this counts as a global mutable singleton, because my setup isn't global, but the solution might be similar?
A very much simplified version would be the code below.
What it tries to do, is have one thread write to a struct, and another read from this same struct. So when thread A mutates the data, thread B would read the mutated data then.
use std::{thread, time::Duration};
struct TagList {
list: Vec<String>
}
impl TagList {
fn add(self: &mut TagList, tag: String) {
self.list.push(tag);
}
fn read(&self) -> Vec<String> {
self.list.clone()
}
}
fn main() {
let mut list = TagList { list: vec![] };
thread::spawn(move || {
["fee", "foo", "faa", "fuu"].into_iter().for_each(|tag| {
list.add(tag.to_string());
thread::sleep(Duration::from_millis(100));
});
});
thread::spawn(move || {
loop {
dbg!(list.read());
thread::sleep(Duration::from_millis(20));
}
});
}
This, rather obviously, fails with a borrow error:
error[E0382]: use of moved value: `list`
--> src/main.rs:79:19
|
70 | let mut list = TagList { list: vec![] };
| -------- move occurs because `list` has type `TagList`, which does not implement the `Copy` trait
71 |
72 | thread::spawn(move || {
| ------- value moved into closure here
73 | ["fee", "foo", "faa", "fuu"].into_iter().for_each(|tag| {
74 | list.add(tag.to_string());
| ---- variable moved due to use in closure
...
79 | thread::spawn(move || {
| ^^^^^^^ value used here after move
80 | dbg!(list.read());
| ---- use occurs due to use in closure
I've tried to solve this by wrapping the list in an Arc:
use std::sync::Arc;
// ...
let list = Arc::new(TagList { list: vec![] });
let write_list = Arc::get_mut(&mut list).unwrap();
let read_list = Arc::clone(&list);
thread::spawn(move || {
["fee", "foo", "faa", "fuu"].into_iter().for_each(|tag| {
write_list.add(tag.to_string());
thread::sleep(Duration::from_millis(100));
});
});
thread::spawn(move || {
loop {
dbg!(read_list.read());
thread::sleep(Duration::from_millis(20));
}
});
This fails because I probably don't understand how Arc is supposed to work or how it relates to lifetimes:
error[E0597]: `list` does not live long enough
--> src/main.rs:71:35
|
71 | let write_list = Arc::get_mut(&mut list).unwrap();
| -------------^^^^^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `list` is borrowed for `'static`
...
85 | }
| - `list` dropped here while still borrowed
error[E0502]: cannot borrow `list` as immutable because it is also borrowed as mutable
--> src/main.rs:72:32
|
71 | let write_list = Arc::get_mut(&mut list).unwrap();
| -----------------------
| | |
| | mutable borrow occurs here
| argument requires that `list` is borrowed for `'static`
72 | let read_list = Arc::clone(&list);
| ^^^^^ immutable borrow occurs here
Is what I want possible at all? (I'm pretty sure i've seen this used with e.g. std::sync::mpsc where somehow messages are pushed and read over threads).
What should I use? Is Arc the proper structure for this, or am I looking at the wrong solution there? What should I read up on to understand how such problems are typically solved in Rust?
Arc does not allow mutation, and Arc::get_mut() is not a solution to that. It allows mutation when the Arc has only one instance (thus the second error) and return a reference which is not 'static, therefore you cannot move it into a thread (the first error).
If you need to mutate the content of an Arc, use Mutex or RwLock.
I basically have the following code that needs to modify a vector field (by removing some items through the retains function) and in the predicate of that retains function, I need to call a mutable function on self :
struct Task {}
struct Processor {
counter: u32,
tasks: Vec<Task>
}
impl Processor {
fn execute(&mut self, task: &Task) -> bool {
self.counter += 1;
// Do some stuff with task and return wether it was sucessfully executed
true
}
fn run(&mut self) {
self.tasks.retain(|task| !self.execute(task))
}
}
The compiler complains with two errors:
error[E0501]: cannot borrow `self.tasks` as mutable because previous closure requires unique access
--> src/main.rs:90:9
|
90 | self.tasks.retain(|task| !self.execute(task))
| ^^^^^^^^^^^------^------^^----^^^^^^^^^^^^^^^
| | | | |
| | | | first borrow occurs due to use of `self` in closure
| | | closure construction occurs here
| | first borrow later used by call
| second borrow occurs here
error[E0500]: closure requires unique access to `self` but it is already borrowed
--> src/main.rs:90:27
|
90 | self.tasks.retain(|task| !self.execute(task))
| ---------- ------ ^^^^^^ ---- second borrow occurs due to use of `self` in closure
| | | |
| | | closure construction occurs here
| | first borrow later used by call
| borrow occurs here
I do understand the issue, but how do I make this work ?
Rust doesn't yet have a way of only borrowing certain members of self. Because of this, it can't infer that you aren't mutating both self.tasks and self.counter at the same time. All it knows is that you're trying to mutate self twice at the same time.
To work around this, you'll need to move tasks out of self, then perform the operation you want, then move it back into self.
Rust has a handy operation that does exactly this in the standard library. Since Vec implements Default (yielding an empty vector, no allocation necessary), we can use std::mem::take:
fn run(&mut self) {
// swaps `self.tasks` with an empty `Vec`, and yields `tasks` to us
// this won't allocate, because `Vec::new` and therefore `Vec::default` do not
let mut tasks = std::mem::take(&mut self.tasks);
// do what we want with it
tasks.retain(|task| !self.execute(task));
// move it back into `self`
self.tasks = tasks;
}
Playground
Option 1: Split your struct into the right parts so they can be borrowed separately. The key part here is that execute doesn't take &mut Processor, only &mut Engine.
struct Task {}
struct Engine {
counter: u32,
}
struct Processor {
engine: Engine,
tasks: Vec<Task>
}
impl Engine {
fn execute(&mut self, task: &Task) -> bool {
self.counter += 1;
true
}
}
impl Processor {
fn run(&mut self) {
let engine = &mut self.engine;
self.tasks.retain(|task| !engine.execute(task))
}
}
Option 2: Move the tasks vector out with std::mem::take, so they're temporarily owned by the function instead of the Processor, then put them back:
impl Processor {
fn run(&mut self) {
let mut tasks = std::mem::take(&mut self.tasks);
tasks.retain(|task| !self.execute(task));
self.tasks = tasks;
}
}
(This does not copy the memory for all the tasks, because the Vec's heap allocated storage is undisturbed by moving the Vec value itself.)
Option 3: If in a more complex situation you can't find a split between parts or a temporary move that works for all your operations, then you could use RefCell to do run-time instead of compile-time borrow checking — you'd need one cell for tasks and another cell or cells for the parts execute needs.
My code:
fn add(&mut self, key: &str, path: std::path::PathBuf) {
self.key.push(key.to_string());
self.path.push(path);
if key == "img" {
self.img.push(self.create_img(path))
}
}
Error:
error[E0499]: cannot borrow `*self` as mutable more than once at a time
--> src/software/init/image.rs:44:27
|
44 | self.img.push(self.create_img(path))
| -------- ---- ^^^^ second mutable borrow occurs here
| | |
| | first borrow later used by call
| first mutable borrow occurs here
error[E0382]: use of moved value: `path`
--> src/software/init/image.rs:44:43
|
40 | fn add(&mut self, key: &str, path: std::path::PathBuf) {
| ---- move occurs because `path` has type `std::path::PathBuf`, which does not implement the `Copy` trait
41 | self.key.push(key.to_string());
42 | self.path.push(path);
| ---- value moved here
43 | if key == "img" {
44 | self.img.push(self.create_img(path))
| ^^^^ value used here after move
It is fairly hard to reason about this code without other methods, but the following code should work (or at least be closer to working solution):
fn add(&mut self, key: &str, path: std::path::PathBuf) {
self.key.push(key.to_string());
// Here, path must be explicitly cloned because of move semantics.
self.path.push(path.clone());
if key == "img" {
// Sometimes compiler cannot determine dependencies of compound
// statement, so with `img` bounding `img` we un-borrow `self`
// (if `create_img` returns owned type). Otherwise you need some
// reference counter like std::rc::Rc.
let img = self.create_img(path);
self.img.push(img);
}
}
A rust newbie here.
I would like to launch an external long-running process and talk with it over pipes from multiple threads in Rust.
I am getting lifetime errors and can't figure the proper way to please the lifetimes checker. What are the ways to restructure this?
Consider the following example:
use std::process::{Command, Stdio, ChildStdin};
use std::sync::Mutex;
use std::io::{Write};
use std::thread;
struct Element {
sink: Mutex<Option<ChildStdin>>
}
impl Element {
fn launch_process(&self) {
let child =
Command::new("sed").args(&["s/foo/bar/g"])
.stdin(Stdio::piped())
.spawn()
.unwrap();
let mut sink = self.sink.lock().unwrap();
*sink = child.stdin;
}
fn tx(&self, content: &[u8]) {
let mut sink = self.sink.lock().unwrap();
sink.as_mut().unwrap().write(content);
}
fn start_tx(&self) {
thread::spawn( || {
self.tx(b"foo fighters");
});
}
}
fn main() {
let e = Element {
sink: Mutex::new(None)
};
e.launch_process();
e.start_tx();
}
If I remove the thread::spawn bit then everything works as expected. With thread::spawn in place, I get the error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:28:24
|
28 | thread::spawn( || {
| ________________________^
29 | | self.tx(b"foo fighters");
30 | | });
| |_________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 27:5...
--> src/main.rs:27:5
|
27 | / fn start_tx(&self) {
28 | | thread::spawn( || {
29 | | self.tx(b"foo fighters");
30 | | });
31 | | }
| |_____^
= note: ...so that the types are compatible:
expected &&Element
found &&Element
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `[closure#src/main.rs:28:24: 30:10 self:&&Element]` will meet its required lifetime bounds
--> src/main.rs:28:9
|
28 | thread::spawn( || {
| ^^^^^^^^^^^^^
error: aborting due to previous error
You can't pass &self (a temporary borrow) to a thread, because the thread may keep running after the reference is no longer valid.
For using data from threads you have only two options:
Give ownership (which is exclusive) of the object to the thread, i.e. use move || closure, and don't try to use that object afterwards from the main thread, or any other thread.
Wrap the object in Arc to get shared thread-safe ownership, and send a clone to the thread (with Arc::clone it's cheap and the underlying data is shared).
When the compiler says that you need a "static lifetime", ignore that. For all practical purposes, it means "references are not allowed".
I have a data stream that I want to process in the background, but I want to create a struct or some functions to manage this stream.
In C++ land, I would create a class that abstracts all of this away. It would have a start method which would initialize the data stream and start a thread for processing. It would have a stop method that stops the processing and joins the thread.
However, this isn't really Rusty, and it doesn't even work in Rust.
Example (Playground)
use std::thread;
use std::time::Duration;
struct Handler {
worker_handle: Option<thread::JoinHandle<()>>,
stop_flag: bool, // Should be atomic, but lazy for example
}
impl Handler {
pub fn new() -> Handler {
let worker_handle = None;
let stop_flag = true;
return Handler { worker_handle, stop_flag };
}
pub fn start(&mut self) {
self.stop_flag = false;
self.worker_handle = Some(std::thread::spawn(move || {
println!("Spawned");
self.worker_fn();
}));
}
pub fn stop(&mut self) {
let handle = match self.worker_handle {
None => return,
Some(x) => x,
};
self.stop_flag = true;
handle.join();
}
fn worker_fn(&mut self) {
while !self.stop_flag {
println!("Working!");
}
}
}
fn main() {
let mut handler = Handler::new();
handler.start();
thread::sleep(Duration::from_millis(10000));
handler.stop();
}
Output:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:20:54
|
20 | self.worker_handle = Some(std::thread::spawn(move || {
| ______________________________________________________^
21 | | println!("Spawned");
22 | | self.worker_fn();
23 | | }));
| |_________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 18:5...
--> src/main.rs:18:5
|
18 | / pub fn start(&mut self) {
19 | | self.stop_flag = false;
20 | | self.worker_handle = Some(std::thread::spawn(move || {
21 | | println!("Spawned");
22 | | self.worker_fn();
23 | | }));
24 | | }
| |_____^
= note: ...so that the types are compatible:
expected &mut Handler
found &mut Handler
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `[closure#src/main.rs:20:54: 23:10 self:&mut Handler]` will meet its required lifetime bounds
--> src/main.rs:20:35
|
20 | self.worker_handle = Some(std::thread::spawn(move || {
| ^^^^^^^^^^^^^^^^^^
error: aborting due to previous error
Even if I cheat and remove the call the worker_fn, I still can't really work with JoinHandles like I might expect from C++ land.
error[E0507]: cannot move out of `self.worker_handle.0` which is behind a mutable reference
--> src/main.rs:27:28
|
27 | let handle = match self.worker_handle {
| ^^^^^^^^^^^^^^^^^^ help: consider borrowing here: `&self.worker_handle`
28 | None => return,
29 | Some(x) => x,
| -
| |
| data moved here
| move occurs because `x` has type `std::thread::JoinHandle<()>`, which does not implement the `Copy` trait
error: aborting due to previous error
So it's clear that I'm going outside of the usual Rust model and probably shouldn't be using this strategy.
But I still want to build some kind of interface that lets me simply spin up the data stream without worrying about managing threads, and I'm not really sure the best way to do this.
So it seems I have two core problems.
1) How can I create a function to be run in a thread which accepts data from an outside source, and can be signaled to quit safely? If I had an atomic bool for killing it, how would I share that between the threads?
2) How do I handle joining the thread when I'm done? The stop method needs to clean up the thread, but I don't know how to track a reference to it.