Avoid deadlock in rust when multiple spawns execute code in a loop - multithreading

Question

I am trying to run 2 threads in parallel and share some data between them. When either one of the threads contain a loop statement, the shared data in the other thread goes into a deadlock.
But if I were to add a line to code to break out of the loop statement after a certain number of iterations, the deadlock gets released and the operation in the next thread starts.
Rust Playground
Code:
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
#[derive(Clone, Copy)]
struct SomeNetwork {
is_connected: bool,
}
impl SomeNetwork {
fn connection_manager(&mut self) {
loop {
// if I exit the loop after a few iterations then the deadlock is removed
// eg: when I use `for i in 0..10 {` instead of `loop`
println!("connection_manager thread...");
thread::sleep(Duration::from_millis(2000));
}
}
fn api_calls(&self) {
loop {
if self.is_connected {
//make_an_api_call()
}
println!("api_calls thread...");
thread::sleep(Duration::from_millis(5000));
}
}
pub fn start() {
let self_arc = SomeNetwork {
is_connected: false,
};
let self_arc = Arc::new(Mutex::new(self_arc));
let self_cloned1 = Arc::clone(&self_arc);
let self_cloned2 = Arc::clone(&self_arc);
thread::Builder::new()
.spawn(move || {
let mut n = self_cloned1.lock().unwrap();
n.connection_manager();
})
.unwrap();
thread::Builder::new()
.spawn(move || {
let n = self_cloned2.lock().unwrap(); // <---- deadlock here
n.api_calls();
})
.unwrap();
loop {
thread::sleep(Duration::from_millis(5000))
}
}
}
fn main() {
SomeNetwork::start();
}
Output:
connection_manager thread...
connection_manager thread...
connection_manager thread...
connection_manager thread...
connection_manager thread...
....
Wouldn't the underlying OS take care of the scheduling once a thread goes into sleep?
What could be done here, so that I can run both threads in parallel?

Answer 1

The issue is the mutex you created stays locked during connection_manager.
The way you use a mutex in Rust is that it wraps the data it locks. When you lock the mutex, it blocks the current thread until it can obtain the mutex. Once it has, it gives you a MutexGuard which you can think of as a wrapper for a reference to the mutex. The MutexGuard gives you mutable access to the data inside the mutex. Then once the MutexGuard is no longer needed Rust invokes MutexGuard's implementation of Drop which unlocks the mutex and allows other threads to obtain it.
// Block until mutex is locked for this thread and return MutexGuard
let mut n = self_cloned1.lock().unwrap();
// Do stuff with the locked mutex
n.connection_manager();
// MutexGuard is no longer needed so it gets dropped and the mutex is released
As you can see, if connection_manager never exits the mutex will remain locked for the first thread to obtain the mutex.
What you want is probably to use a mutex with a condvar so the mutex can be released while the thread is sleeping.
Edit:
Here is a rough idea of what that using condvars to handle connecting and channels to pass jobs to workers would look like. Playground Link
use std::sync::{Arc, Mutex, Condvar};
use std::thread::{self, current};
use std::time::Duration;
use crossbeam_channel::{unbounded, Receiver};
#[derive(Clone, Copy)]
struct SomeNetwork {
is_connected: bool,
}
const TIMEOUT: Duration = Duration::from_secs(5);
impl SomeNetwork {
fn connect(&mut self) {
println!("connection_manager thread...");
self.is_connected = true;
}
fn api_calls(&self, job: i32) {
//println!("api_calls thread...");
println!("[Worker {:?}] Handling job {}", current().id(), job);
thread::sleep(Duration::from_millis(50))
}
pub fn start_connection_thread(
self_data: Arc<Mutex<Self>>,
connect_condvar: Arc<Condvar>,
worker_condvar: Arc<Condvar>,
) {
thread::Builder::new()
.spawn(move || {
let mut guard = self_data.lock().unwrap();
loop {
// Do something with the data
if !guard.is_connected {
guard.connect();
// Notify all workers that the connection is ready
worker_condvar.notify_all();
}
// Use condvar to release mutex and wait until signaled to start again
let (new_guard, _) = connect_condvar.wait_timeout(guard, TIMEOUT).unwrap();
guard = new_guard;
}
})
.unwrap();
}
pub fn start_worker_thread(
self_data: Arc<Mutex<Self>>,
connect_condvar: Arc<Condvar>,
worker_condvar: Arc<Condvar>,
requests: Receiver<i32>,
) {
thread::Builder::new()
.spawn(move || {
loop {
// Wait until a request is received
let request = requests.recv().unwrap();
// Lock mutex once we have a request
let mut guard = self_data.lock().unwrap();
// Make sure we are connected before starting tasks
while !guard.is_connected {
// Wake up 1 connection thread if the connection breaks
connect_condvar.notify_one();
// Sleep until signaled that the connection has been fixed
let (new_guard, _) = worker_condvar.wait_timeout(guard, TIMEOUT).unwrap();
guard = new_guard;
}
// Now that we have verified we are connected, handle the request
guard.api_calls(request);
}
})
.unwrap();
}
pub fn start() {
let self_arc = SomeNetwork {
is_connected: false,
};
let self_arc = Arc::new(Mutex::new(self_arc));
let connect_condvar = Arc::new(Condvar::new());
let worker_condvar = Arc::new(Condvar::new());
// Create a channel to send jobs to workers
let (send, recv) = unbounded();
Self::start_connection_thread(self_arc.clone(), connect_condvar.clone(), worker_condvar.clone());
// Start some workers
for _ in 0..5 {
Self::start_worker_thread(self_arc.clone(), connect_condvar.clone(), worker_condvar.clone(), recv.clone());
}
// Send messages to workers
for message in 1..100 {
send.send(message);
}
loop {
thread::sleep(Duration::from_millis(5000))
}
}
}
fn main() {
SomeNetwork::start();
}