Rust: Using Mutexes to Allow Access to Data from multiple threads - multithreading

I'm trying to create a reference to the current TCP connections. In this code, I'm trying to just print the current TCP connection's peer addresses in a loop and sleep for 10 seconds and the error I'm having is accessing the data between multiple threads.
I want to be able to manipulate a TCPStream from a different thread at any given point in time, to do things like shut down the TCPStream or get the peer address.
Can you please let me know what I'm doing wrong in a way that I can get a better understanding of how Arc and Mutex work?
use std::io::Read;
use std::net::{TcpListener, TcpStream};
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
fn main() {
let server = Server {
connected_clients: Arc::new(Mutex::new(Vec::new()))
};
thread::spawn(move || {
let listener = TcpListener::bind("127.0.0.1:25565").unwrap();
// For each new connection start a new thread
for stream in listener.incoming() {
let stream = Arc::new(Mutex::new(stream.unwrap()));
let client = Client {
stream: stream.clone()
};
let cc = server.connected_clients.clone();
cc.lock().unwrap().push(client);
thread::spawn(move || {
// TODO: Add client to the connected_clients Vec
let mut buffer = [0; 1024];
loop {
stream.lock().unwrap().read(&mut buffer).unwrap();
println!("{}", String::from_utf8(Vec::from(&buffer[..])).unwrap().trim_end_matches(char::from(0)));
}
});
}
});
loop {
thread::sleep(Duration::from_secs(10));
// let vec = server.lock().unwrap().connected_clients.lock().unwrap().iter();
for client in server.connected_clients.lock().unwrap().iter() {
println!("{:?}", client.stream.lock().unwrap().peer_addr().unwrap())
}
}
}
#[derive(Debug)]
struct Server {
connected_clients: Arc<Mutex<Vec<Client>>>,
}
#[derive(Debug)]
struct Client {
stream: Arc<Mutex<TcpStream>>
}
ERROR:
error[E0382]: borrow of moved value: `server.connected_clients`
--> src\main.rs:40:23
|
12 | thread::spawn(move || {
| ------- value moved into closure here
...
22 | let cc = server.connected_clients.clone();
| ------------------------ variable moved due to use in closure
...
40 | for client in server.connected_clients.lock().unwrap().iter() {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ value borrowed here after move
|
= note: move occurs because `server.connected_clients` has type `Arc<Mutex<Vec<Client>>>`, which does not implement the `Copy` trait
= note: borrow occurs due to deref coercion to `Mutex<Vec<Client>>`

Just move the line let cc = server.connected_clients.clone(); before the first line thread::spawn(move || {.
The move keyword of the closure will now take ownership of cc, then the original server.connected_clients will stay available for the loop at the end of the program.
The idea behind Rc::clone() or Arc::clone() is exactly for the purpose of move closures: instead of moving the original ref-counted pointer to the resource into the closure, we move its clone and the original ref-counted pointer to the resource is still available in its original context.

Related

Proper way to share references to Vec between threads

I am new to rust and I am attempting to create a Vec that will live on the main thread, and pass a reference to another thread, which then pushes members onto the vector, for the main thread to use.
use std::{thread};
fn main() {
let mut v: Vec<u8> = Vec::new();
let _ = thread::spawn(move || {
vec_push(&mut v, 0)
});
for i in v.iter_mut() {
println!("poo {}", i);
}
}
fn vec_push(v: &mut Vec<u8>, n: u8) {
v.push(n);
}
This is a simplified version of what I am trying to do. In my main code I am want it to be a Vec of TcpStreams.
I think this post would also apply to maintaining a struct (that doesn't implement Copy) between threads.
I get this error
error[E0382]: borrow of moved value: `v`
--> src/main.rs:8:11
|
4 | let mut v: Vec<u8> = Vec::new();
| ----- move occurs because `v` has type `Vec<u8>`, which does not implement the `Copy` trait
5 | let _ = thread::spawn(move || {
| ------- value moved into closure here
6 | vec_push(&mut v, 0)
| - variable moved due to use in closure
7 | });
8 | for i in v.iter_mut() {
| ^^^^^^^^^^^^ value borrowed here after move
Is there a better way to do this? Am I missing some basic concept?
Any help would be useful, I am used to C where I can just throw around references willy-nilly
What you are doing is wildly unsound. You are trying to have two mutable references to a object, which is strictly forbidden in rust. Rust forbids this to prevent you from having data races that would result in memory unsafety.
If you want to mutate an object from different threads you have to synchronize it somehow. The easiest way to do it is by using Mutex. This probably won't be very efficient in a high-congestion scenario (as locking a mutex can become your bottle neck), but it will be safe.
To share this Mutex between threads you can wrap it in an Arc (an atomic counted shared reference smart pointer). So your code can be transformed to something like this:
use std::thread;
use std::sync::{Arc, Mutex};
fn main() {
let v = Arc::new(Mutex::new(Vec::new()));
let v_clone = Arc::clone(&v);
let t = thread::spawn(move || {
vec_push(v_clone, 0)
});
t.join().unwrap();
for i in v.lock().unwrap().iter_mut() {
println!("poo {}", i);
}
}
fn vec_push(v: Arc<Mutex<Vec<u8>>>, n: u8) {
v.lock().unwrap().push(n);
}
You probably will also want to join your spawned thread, so you should name it.

How to fix "data is required to be 'static" using tokio::spawn?

I'm trying to listen to multiple TCP streams via tokio but I get an error:
use rybw_config::ListenerConfig;
use tokio::prelude::*;
use tokio::task::JoinHandle;
pub mod tcp;
use tcp::TCPListener;
pub struct Listener {
config: ListenerConfig,
tcp: Vec<tcp::TCPListener>,
}
impl Listener {
pub fn new(config: ListenerConfig) -> Listener {
let tcp: Vec<TCPListener> = config
.tcp
.clone()
.unwrap()
.iter()
.map(|x| TCPListener::new((*x).clone()))
.collect();
Listener {
tcp: tcp,
config: config.clone(),
}
}
pub async fn run(&self) {
let handles: Vec<JoinHandle<()>> = self.tcp.iter().map(|i| {
tokio::spawn(async {
i.run().await
})
}).collect();
futures::future::join_all(handles);
}
error: cannot infer an appropriate lifetime
--> rybw-listener/src/lib.rs:28:22
|
28 | pub async fn run(&self) {
| ^^^^^
| |
| data with this lifetime...
| ...is captured here...
29 | let handles: Vec<JoinHandle<()>> = self.tcp.iter().map(|i| {
30 | tokio::spawn(async {
| ------------ ...and required to be `'static` by this
Solved by myself.
rust closure captures the struct, not by individual fields. So we wrapper the TCPListener with std::sync::Arc and cloning it then used in spawn async {}
The type of i in your example is &tcp::TCPListener where the reference is still tied to self. However, tokio::spawn requires the spawned task to be 'static meaning it can't keep references to local variables.
The solution is to move owned values into the task. Common ways to do this are:
Use Clone to create a copy of the data needed by the task.
let handles: Vec<_> = self.tcp.iter().map(|listener| {
let listener = listener.clone();
// ^^^^^^^^ create a copy
tokio::spawn(async move {
// ^^^^ and move it into the async block
listener.run().await
})
}).collect();
Use shared ownership via Arc and Clone the handle needed by the task.
use std::sync::Arc;
struct Listener {
config: ListenerConfig,
tcp: Vec<Arc<tcp::TCPListener>>,
// ^^^ allow shared ownership
}
let handles: Vec<_> = self.tcp.iter().map(|listener| {
let listener = Arc::clone(listener);
// ^^^^^^^^^^ create a new handle to the same data
tokio::spawn(async move {
// ^^^^ and move it into the async block
listener.run().await
})
}).collect();
Provide ownership directly to the task. This won't really be an option in your example since &self is an immutable reference. But if you had an owned self or mutable reference and didn't need to keep your listeners after spawning tasks for them, you could use something like .into_iter() or .drain() on Vec to consume the listeners so their type is tcp::TCPListener (not a reference) and can easily be move-d into the task.

How can I send non-static data to a thread in Rust and is it needed in this example?

I am trying to fire up a new thread using some heap data in Rust and I am getting a bunch of errors that stem from the need of the data to have 'static lifetime. I've worked my way backwards up my program but hit a problem.
use std::sync::Arc;
use std::thread;
struct ThreadData {
vector_of_strings: Vec<String>,
terms: Vec<&'static str>,
quotient: usize,
}
fn perform_search(slice: &[String], terms: &[&str]) {
/* ... */
}
fn threaded_search(td_arc: &Arc<ThreadData>) {
let no_of_lines = td_arc.vector_of_strings.len();
let new_tda1 = td_arc.clone();
let strings_as_slice1 = new_tda1.vector_of_strings.as_slice();
thread::spawn(move || {
perform_search(&strings_as_slice1[0..td_arc.quotient], &new_tda1.terms);
});
}
fn main() {
let td = ThreadData {
vector_of_strings: Vec::new(),
terms: Vec::new(),
quotient: 0,
};
let td_arc = Arc::new(td);
threaded_search(&td_arc);
}
Error:
error[E0621]: explicit lifetime required in the type of `td_arc`
--> src/main.rs:20:5
|
14 | fn threaded_search(td_arc: &Arc<ThreadData>) {
| ---------------- help: add explicit lifetime `'static` to the type of `td_arc`: `&'static std::sync::Arc<ThreadData>`
...
20 | thread::spawn(move || {
| ^^^^^^^^^^^^^ lifetime `'static` required
The error about 'static is because the new thread created within thread::spawn may outlive the invocation of threaded_search during which the thread is initially created, which means the thread must not be permitted to use any local variables from threaded_search with a lifetime shorter than 'static.
In your code the new thread is referring to strings_as_slice1 and td_arc.
Generally with thread::spawn and Arc you will want to move ownership of one reference count into the thread and have the thread access whatever it needs through that reference counted pointer rather than from the enclosing short-lived scope directly.
fn threaded_search(td_arc: &Arc<ThreadData>) {
// Increment reference count that we can move into the new thread.
let td_arc = td_arc.clone();
thread::spawn(move || {
perform_search(&td_arc.vector_of_strings[0..td_arc.quotient], &td_arc.terms);
});
}

Nested closure borrow failed

I want to parse a YAML file and use the value inside a service for a HTTP request. Line 35 is the end of main function.
extern crate hyper;
extern crate libc;
extern crate yaml_rust;
use hyper::rt::Future;
use hyper::service::service_fn_ok;
use hyper::{Body, Response, Server};
use std::sync::Arc;
use yaml_rust::YamlLoader;
fn main() {
let content: String = String::from("response: Hello world");
let cfg = Arc::new(YamlLoader::load_from_str(content.as_str()).unwrap());
let cfg0 = (&cfg[0]).clone();
let cfg_response = (&cfg0)["response"].as_str().unwrap();
// A `Service` is needed for every connection, so this
// creates on of our `hello_world` function.
let handle = move || {
let cfg_response = cfg_response.clone();
service_fn_ok(move |_| Response::new(Body::from(String::from(cfg_response.clone()))))
};
// Serve HTTP protocol
// This is our socket address...
let addr: std::net::SocketAddr = ([127, 0, 0, 1], 3000).into();
let server = Server::bind(&addr)
.serve(handle)
.map_err(|e| eprintln!("server error: {}", e));
// Run this server for... forever!
hyper::rt::run(server);
}
Unfortunately, I encountered a nested closure leading to a strange borrow error:
error[E0597]: `cfg0` does not live long enough
--> src/main.rs:15:26
|
15 | let cfg_response = (&cfg0)["response"].as_str().unwrap();
| ^^^^ borrowed value does not live long enough
...
35 | }
| - borrowed value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
I tried to
clone it before borrow
use Arc to make it counter-based,
modify assignment
all to no avail
Why does this happen? How do I solve this?
The functions you pass the closure into - hyper::server::Builder::serve and hyper::rt::run() - require their arguments to be 'static, rather than being bounded by any function. main isn't considered special in this regard.
The value bounding it, cfg_response, is captured by the outer closure, so the nested closure isn't necessary to get the error.
Here's a very small program that has the same problem:
fn main() {
let cfg0 = String::from("hello world");
let cfg_response: &str = &cfg0;
let handle = move || {
// this closure takes ownership of cfg_response, a reference to cfg0. Since cfg0 will not
// outlive the function, neither can handle. If it instead took ownership of cfg0 or a
// clone of it, it would have no outside references and could live forever.
return cfg_response.to_owned();
};
serve(handle);
}
fn serve<F: Fn() -> String + 'static>(handle: F) {
loop {
println!("{}", handle());
}
}
As #Stargateur pointed out, this can be solved by making cfg_response owned.
Alternatively, you could initialize cfg0 in a lazy_static like so:
#[macro_use]
extern crate lazy_static;
lazy_static! {
static ref cfg0: String = String::from("hello world");
}
This way, you can still use a borrowed value because it meets the lifetime requirements.

Why do I need to implement `Copy` and `Clone` if I'm sharing an immutable reference with multiple threads?

Still struggling with the Rust mind shift, now I have this use case - one configuration for a multi-threaded TcpListener:
use std::net::{TcpListener, TcpStream, ToSocketAddrs};
use std::thread;
fn main() {
serve("127.0.0.1:3333", Configuration { do_something: true });
}
//#[derive(Copy, Clone)]
pub struct Configuration {
pub do_something: bool,
}
pub fn serve<A: ToSocketAddrs>(addr: A, conf: Configuration) {
let listener = TcpListener::bind(addr).expect("bind failed");
for stream in listener.incoming() {
match stream {
Ok(stream) => {
thread::spawn(move || {
handle_client(stream, &conf);
});
}
Err(e) => {
println!("Connection failed: {}", e);
}
}
}
}
fn handle_client(stream: TcpStream, conf: &Configuration) {
if conf.do_something {
//stream....
}
}
I'm happy that the TcpStream is consumed by handle_client, that's it's purpose, but why does Configuration have to be copied for each thread? I'd like to have just one copy and share an immutable reference with all threads. Is that possible? Or perhaps I'm missing the point.
Why do I need the Copy and Clone traits if I'm passing Configuration by reference? This confused me a good deal:
error[E0382]: capture of moved value: `conf`
--> src/main.rs:19:64
|
19 | thread::spawn(move || { handle_client(stream, &conf); });
| ------- ^^^^ value captured here after move
| |
| value moved (into closure) here
|
= note: move occurs because `conf` has type `Configuration`, which does not implement the `Copy` trait
Implementing Copy (Clone is incidental here) only fixes the problem because implementing it allows the compiler to implicitly duplicate the Configuration struct, passing a copied value into the thread.
It needs to pass the variable by value because you told the compiler to move all used values into the closure:
thread::spawn(move || {
// ^^^^ HERE
handle_client(stream, &conf);
});
The entire purpose of the move keyword is to tell the compiler "no, don't try to infer how variables are used inside this closure, just move everything in".
When you move &conf, the compiler says "OK, I'll move conf into the closure then take a reference to it".
In your case, you can just remove the move keyword:
thread::spawn(|| {
handle_client(stream, &conf);
});
If you really need to be able to use the move keyword and pass in a reference, you need to move in a reference:
let conf = &conf;
thread::spawn(move || {
handle_client(stream, conf);
});
This still doesn't allow your code to compile because there's no guarantee that the reference outlives the thread. That's discussed thoroughly in Passing a reference to a stack variable to a scoped thread.

Resources