This question already has an answer here:
What does “&*” do in Rust? [duplicate]
(1 answer)
Closed last year.
https://doc.rust-lang.org/std/sync/struct.Condvar.html#examples
use std::sync::{Arc, Mutex, Condvar};
use std::thread;
let pair = Arc::new((Mutex::new(false), Condvar::new()));
let pair2 = Arc::clone(&pair);
// Inside of our lock, spawn a new thread, and then wait for it to start.
thread::spawn(move|| {
let (lock, cvar) = &*pair2;
let mut started = lock.lock().unwrap();
*started = true;
// We notify the condvar that the value has changed.
cvar.notify_one();
});
// Wait for the thread to start up.
let (lock, cvar) = &*pair;
let mut started = lock.lock().unwrap();
while !*started {
started = cvar.wait(started).unwrap();
}
Why isn't 17th line written like this:
let (lock, cvar) = pair;
Or like this:
let (lock, cvar) = *pair;
Or somehow else
The asterisk makes Arc<(Mutex<bool>, Condvar)> become (Mutex<bool>, Condvar). Then the ampersand makes it become a &(Mutex<bool>, Condvar).
So as a result, lock's type is &Mutex<bool> and cvar's type is &Condvar.
Now, let's see why we need the & by trying to remove it:
let (lock, cvar) = *pair;
The compiler explains the problem:
error[E0507]: cannot move out of an `Arc`
--> src/main.rs:26:24
|
26 | let (lock, cvar) = *pair;
| ---- ---- ^^^^^ help: consider borrowing here: `&*pair`
| | |
| | ...and here
| data moved here
|
= note: move occurs because these variables have types that don't implement the `Copy` trait
Related
I have a requirement to send and receive normal data on the same TcpStream, while sending heartbeat data at regular intervals. In the current implementation, I used Arc<Mutex<TcpStream>>, but it compiled with errors:
use anyhow::Result;
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
#[tokio::main]
async fn main() -> Result<()> {
let stream = TcpStream::connect("127.0.0.1:8888").await.unwrap();
let stream = Arc::new(Mutex::new(stream));
let common_stream = stream.clone();
let handler1 = tokio::spawn(async {
loop {
let mut stream = common_stream.lock().unwrap();
let mut buf = [0u8; 10];
stream.read_exact(&mut buf).await.unwrap();
buf.reverse();
stream.write(&buf).await.unwrap();
}
});
let heartbeat_stream = stream.clone();
let handler2 = tokio::spawn(async {
loop {
let mut stream = heartbeat_stream.lock().unwrap();
stream.write_u8(1).await.unwrap();
thread::sleep(Duration::from_millis(200));
}
});
handler1.await?;
handler2.await?;
Ok(())
}
error: future cannot be sent between threads safely
--> src\main.rs:14:20
|
14 | let handler1 = tokio::spawn(async {
| ^^^^^^^^^^^^ future created by async block is not `Send`
|
= help: within `impl Future<Output = [async output]>`, the trait `Send` is not implemented for `std::sync::MutexGuard<'_, tokio::net::TcpStream>`
note: future is not `Send` as this value is used across an await
--> src\main.rs:20:31
|
16 | let mut stream = common_stream.lock().unwrap();
| ---------- has type `std::sync::MutexGuard<'_, tokio::net::TcpStream>` which is not `Send`
...
20 | stream.write(&buf).await.unwrap();
| ^^^^^^ await occurs here, with `mut stream` maybe used later
21 | }
| - `mut stream` is later dropped here
note: required by a bound in `tokio::spawn`
--> .cargo\registry\src\mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd\tokio-1.17.0\src\task\spawn.rs:127:21
|
127 | T: Future + Send + 'static,
| ^^^^ required by this bound in `tokio::spawn`
error: future cannot be sent between threads safely
--> src\main.rs:25:20
|
25 | let handler2 = tokio::spawn(async {
| ^^^^^^^^^^^^ future created by async block is not `Send`
|
= help: within `impl Future<Output = [async output]>`, the trait `Send` is not implemented for `std::sync::MutexGuard<'_, tokio::net::TcpStream>`
note: future is not `Send` as this value is used across an await
--> src\main.rs:28:31
|
27 | let mut stream = heartbeat_stream.lock().unwrap();
| ---------- has type `std::sync::MutexGuard<'_, tokio::net::TcpStream>` which is not `Send`
28 | stream.write_u8(1).await.unwrap();
| ^^^^^^ await occurs here, with `mut stream` maybe used later
...
31 | }
| - `mut stream` is later dropped here
note: required by a bound in `tokio::spawn`
--> .cargo\registry\src\mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd\tokio-1.17.0\src\task\spawn.rs:127:21
|
127 | T: Future + Send + 'static,
| ^^^^ required by this bound in `tokio::spawn`
How can these errors be fixed or is there another way to achieve the same goal?
Here is a solution that splits the stream to two parts for reading and writing plus does in a loop:
waiting for heartbeat events and sends a byte to write half of stream when this happens
waits data from read half (10 bytes), reverses it and writes again to write half
Also this does not spawn threads and does everything nicely in current one without locks.
use anyhow::Result;
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
#[tokio::main]
async fn main() -> Result<()> {
let mut stream = TcpStream::connect("127.0.0.1:8888").await?;
let (mut read, mut write) = stream.split();
let mut heartbeat_interval = tokio::time::interval(Duration::from_millis(200));
let mut buf = [0u8; 10];
loop {
tokio::select! {
_ = heartbeat_interval.tick() => {
write.write(&[1]).await?;
}
result = read.read_exact(&mut buf) => {
let _bytes_read = result?;
buf.reverse();
write.write(&buf).await?;
}
}
}
}
Several errors in your code, although the idea behind it is almost good. You should use any available tool in async as possible. Some of the needed/desired changes:
Use tokio::time::sleep because it is async, otherwise the call is blocking
Use an async version of mutex (the one from futures crate for example)
Use some kind of generic error handling (anyhow would help)
use futures::lock::Mutex;
use anyhow::Error;
use tokio::time::sleep;
use std::sync::Arc;
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
#[tokio::main]
async fn main() -> Result<(), Error> {
let stream = TcpStream::connect("127.0.0.1:8888").await.unwrap();
let stream = Arc::new(Mutex::new(stream));
let common_stream = stream.clone();
let handler1 = tokio::spawn(async move {
loop {
let mut stream = common_stream.lock().await;
let mut buf = [0u8; 10];
stream.read_exact(&mut buf).await.unwrap();
buf.reverse();
stream.write(&buf).await.unwrap();
}
});
let heartbeat_stream = stream.clone();
let handler2 = tokio::spawn(async move {
loop {
let mut stream = heartbeat_stream.lock().await;
stream.write_u8(1).await.unwrap();
sleep(Duration::from_millis(200)).await;
}
});
handler1.await?;
handler2.await?;
Ok(())
}
Playground
I try to set up my first-ever thread machine and ran into the following issue.
We have this state:
use std::{thread, sync::mpsc};
fn main() {
let tasks = vec![String::from("task1"), String::from("task2")];
let (tx, rx) = mpsc::channel();
let tx1 = tx.clone();
let handle = thread::spawn(move || {
for i in 0..(tasks.len()/2 - 1){
let send = format!("{}, {}", tasks[i], String::from("done"));
tx.send(send).unwrap();
}
});
for i in tasks.len()/2..tasks.len() - 1 {
let send = format!("{}, {}", tasks[i], String::from("done"));
tx1.send(send).unwrap();
}
handle.join().unwrap();
for string in rx {
println!("{}", string);
}
}
I try to set up parallel threads, which do something with the strings in the vector. They should work and send until the whole vector is covered. The vector has reliable 100 entries.
The corresponding error message:
error[E0382]: borrow of moved value: `tasks`
--> src/main.rs:19:14
|
6 | let tasks = vec![String::from("task1"), String::from("task2")];
| ----- move occurs because `tasks` has type `Vec<String>`, which does not implement the `Copy` trait
...
11 | let handle = thread::spawn(move || {
| ------- value moved into closure here
12 | for i in 0..(tasks.len()/2 - 1){
| ----- variable moved due to use in closure
...
19 | for i in tasks.len()/2..tasks.len() - 1 {
| ^^^^^ value borrowed here after move
error: aborting due to previous error
For more information about this error, try `rustc --explain E0382`.
It seems like I really do not understand how to set up parallel threads and how to treat channels together with threads and a list of tasks.
As the compiler indicates, the problem is that you moved the vector of tasks when spawning a thread, and then you try to borrow the vector from the main thread while it was already moved.
The problem you need to solve here is how to share the vectors' ownership between the main thread and the spawned thread. One way to do it is to use Arc.
Based on your code, I have wrote an example,
use std::{thread, sync::mpsc};
use std::sync::Arc;
fn main() {
let tasks = Arc::new(vec![String::from("task1"), String::from("task2")]);
let tasks_arc_clone = Arc::clone(&tasks);
let (tx, rx) = mpsc::channel();
let tx1 = tx.clone();
let mid_idx = tasks.len() / 2 - 1;
let handle = thread::spawn(move || {
for i in 0..mid_idx {
let send = format!("{}, {}", tasks_arc_clone[i], String::from("done"));
tx.send(send).unwrap();
}
});
for i in mid_idx..tasks.len() {
let send = format!("{}, {}", tasks[i], String::from("done"));
tx1.send(send).unwrap();
}
drop(tx1);
for string in rx {
println!("{}", string);
}
handle.join().unwrap()
}
Rust playground link, in case you want to play with it.
I'm new to Rust and still reading the Rust book. Below is my program.
use clap::{App, Arg};
type GenericError = Box<dyn std::error::Error + Send + Sync + 'static>;
type GenericResult<T> = Result<T, GenericError>;
fn main() -> GenericResult<()> {
let matches = App::new("test")
.arg(Arg::new("latency")
.takes_value(true))
.get_matches();
let latency_present = matches.is_present("latency");
let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
let latency_pairs: Vec<&str> = latency.split(",").collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
checker.join().unwrap()?;
Ok(())
}
When I run it, it tells me this:
error[E0597]: `matches` does not live long enough
--> src\main.rs:14:19
|
14 | let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
| ^^^^^^^--------------------
| |
| borrowed value does not live long enough
| argument requires that `matches` is borrowed for `'static`
...
30 | }
| - `matches` dropped here while still borrowed
I don't quite understand the error messages here. But I guess it's because I use latency_pairs in the checker thread and latency_pairs could get dropped while checker is still executing. Is my understanding correct? How to fix the error? I tried for (i, latency_pair) in latency_pairs.clone().iter().enumerate() { in order to pass a cloned value for the thread, but it doesn't help.
latency_pairs holds references into latency which in turn references matches. Thus cloning latency_pairs just clones the references into latency and matches.
Your code would require that latency's type is &'static str but it's actually &'a str where 'a is bound to matches' lifetime.
You can call to_owned() on latency to get an owned value and split the string inside the closure or you can call to_owned() on each of the splits collected in latency_pairs and move that Vec<String> into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
If you need to use latency_pairs outside of the closure, you can clone it before moving it into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let latency_pairs_ = latency_pairs.clone();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs_.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
println!("{:?}", latency_pairs);
Context
I have a function that will use rayon to paralellize some work. Each thread should write its output to a separate consumer. In practice, these consumers will usually write to (separate) files on disc. For testing, I want them to append to vectors, which I can then make assertions on.
Problem
In fact I hit an issue even with a serial version of this code.
The function I want to test is func_under_test. I want get_sink to create and hand out a new 'sink' (a Vec<i32>) each time it is called, which will be inside func_under_test. But I also want to keep a reference to these vectors in my main function, so I can make assertions on them. I've tried writing this:
fn func_under_test<GetSink, F>(
mut get_sink: GetSink
)
where
GetSink: FnMut(usize) -> F,
F: FnMut(i32) -> (),
{
let idxs : Vec<usize> = (0..2).collect();
idxs.iter().for_each(|&i| {
let mut sink = get_sink(i);
sink(0i32);
sink(1i32);
sink(2i32);
});
}
fn main() {
let mut sinks : Vec<&Vec<i32>> = vec![];
let get_sink = |i: usize| {
let mut sink : Vec<i32> = vec![];
sinks.push(&sink);
move |x : i32| sink.push(x)
};
func_under_test(get_sink);
assert_eq!(**sinks.get(0).unwrap(), vec![0i32, 1i32, 2i32]);
}
But I get this compilation error:
error[E0597]: `sink` does not live long enough
--> src/main.rs:23:20
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
25 | };
| - `sink` dropped here while still borrowed
error[E0505]: cannot move out of `sink` because it is borrowed
--> src/main.rs:24:9
|
20 | let mut sinks : Vec<&Vec<i32>> = vec![];
| --------- lifetime `'1` appears in the type of `sinks`
...
23 | sinks.push(&sink);
| -----------------
| | |
| | borrow of `sink` occurs here
| argument requires that `sink` is borrowed for `'1`
24 | move |x : i32| sink.push(x)
| ^^^^^^^^^^^^^^ ---- move occurs due to use in closure
| |
| move out of `sink` occurs here
Am I correct in thinking that I need to use something like an Rc in this context, since I need to have two references to each Vec<i32>?
Since only one of those references needs to be mutable, do I need to use Rc<Cell<Vec<i32>>> in one case, and just Rc<Vec<i32>> in the other?
You could use an Arc<Mutex<i32>> to share mutable access to the vectors. (Rc is not thread-safe.)
A tidier solution would be typed_arena::Arena, which would allow you to write essentially the code you tried, but with the roles swapped: the Vec<i32>s are always owned by the arena (thus it outlives func_under_tests). Then, after all of the references to the arena are gone, you can convert it into a vector of its elements.
use typed_arena::Arena;
fn main() {
let sinks: Arena<Vec<i32>> = Arena::new();
let get_sink = |_i: usize| {
let sink_ref = sinks.alloc(Vec::new());
move |x| sink_ref.push(x)
};
func_under_test(get_sink);
assert_eq!(sinks.into_vec()[0], vec![0i32, 1i32, 2i32]);
}
I think I've figured out the answer to this. The thread doing the testing needs to have access to the inner vectors, and so does the thread that is populating this. So they need to be wrapped in an Arc and a Mutex. But the outer vector does not, since it is only borrowed by the get_sink closure.
So sinks can have type Vec<Arc<Mutex<Vec<i32>>>.
Anyway, I can write this and it does compile and behave and expected:
use rayon::prelude::*;
use std::sync::{Arc, Mutex};
fn func_under_test<G, S>(
get_sink: G)
where
G: Fn(&usize) -> S + Sync,
S: FnMut(i32) -> (),
{
let idxs : Vec<usize> = vec![0, 1, 2];
idxs.par_iter().for_each(|i| {
let mut sink : S = get_sink(i);
for j in 0..3 {
sink(j + 3*(*i) as i32);
}
});
}
fn main() {
let sinks : Vec<Arc<Mutex<Vec<i32>>>> = vec![
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
Arc::new(Mutex::new(vec![])),
];
let get_sink = |i: &usize| {
let m : Arc<Mutex<Vec<i32>>> = sinks.get(*i).unwrap().clone();
move |x: i32| {
m.lock().unwrap().push(x);
}
};
func_under_test(get_sink);
assert_eq!(*sinks.get(0).unwrap().lock().unwrap(), vec![0, 1, 2]);
assert_eq!(*sinks.get(1).unwrap().lock().unwrap(), vec![3, 4, 5]);
assert_eq!(*sinks.get(2).unwrap().lock().unwrap(), vec![6, 7, 8]);
}
It also seems like G implements the correct traits here: it is Sync, since it needs to be shared between threads. But is is not Send, since it does not need to send information between threads.
I'm using an example provided by the Tokio library and attempting to have a vector of all the currently active TCP connections. Ultimately, I would like to be able to broadcast a message to each of the active connections, by looping through them and writing a message to the socket.
To start with, I am trying to print out the current number of connections in one thread whilst accepting connections in another.
To do this, I'm trying to use a shared vector. I've not yet implemented the removal of connections from the vector as and when they disconnect.
// A tiny async echo server with tokio-core
extern crate futures;
extern crate tokio_core;
extern crate tokio_io;
use futures::{Future, Stream};
use tokio_io::{io, AsyncRead};
use tokio_core::net::TcpListener;
use tokio_core::reactor::Core;
use std::thread;
use std::sync::{Arc, Mutex};
use std::io::stdout;
use std::io::Write;
fn main() {
// Create the event loop that will drive this server
let mut core = Core::new().unwrap();
let handle = core.handle();
// Bind the server's socket
let addr = "127.0.0.1:12345".parse().unwrap();
let tcp = TcpListener::bind(&addr, &handle).unwrap();
let mut connections = Arc::new((Mutex::new(Vec::new())));
thread::spawn(move || {
//Every 10 seconds print out the current number of connections
let mut i;
loop {
i = connections.lock().unwrap().len();
println!("There are {} connections", i);
stdout().flush();
thread::sleep_ms(10000);
}
});
// Iterate incoming connections
let server = tcp.incoming().for_each(|(tcp, _)| {
connections.lock().unwrap().push(tcp);
// Split up the read and write halves
let (reader, writer) = tcp.split();
// Future of the copy
let bytes_copied = io::copy(reader, writer);
// ... after which we'll print what happened
let handle_conn = bytes_copied.map(|(n, _, _)| {
println!("wrote {} bytes", n)
}).map_err(|err| {
println!("IO error {:?}", err)
});
// Spawn the future as a concurrent task
handle.spawn(handle_conn);
Ok(())
});
// Spin up the server on the event loop
core.run(server).unwrap();
}
At the moment this is failing to build with the following errors:
error[E0382]: capture of moved value: `connections`
--> src/main.rs:36:42
|
26 | thread::spawn(move || {
| ------- value moved (into closure) here
...
36 | let server = tcp.incoming().for_each(|(tcp, _)| {
| ^^^^^^^^^^ value captured here after move
|
= note: move occurs because `connections` has type `std::sync::Arc<std::sync::Mutex<std::vec::Vec<tokio_core::net::TcpStream>>>`, which does not implement the `Copy` trait
error[E0382]: use of moved value: `tcp`
--> src/main.rs:40:32
|
38 | connections.lock().unwrap().push(tcp);
| --- value moved here
39 | // Split up the read and write halves
40 | let (reader, writer) = tcp.split();
| ^^^ value used here after move
|
= note: move occurs because `tcp` has type `tokio_core::net::TcpStream`, which does not implement the `Copy` trait
Is it possible to achieve this without writing any unsafe code?
You get the first error because of the move closure:
let mut connections = Arc::new((Mutex::new(Vec::new())));
thread::spawn(move || {
let mut i = connections.lock().unwrap().len();
....
}
This actually moves the whole Arc, while you only want to move "a part" of it (that is, move it in such a way that the reference count is incremented, and that both threads can use it).
To do this, we can use Arc::clone:
let mut connections = Arc::new((Mutex::new(Vec::new())));
let conn = connections.clone();
thread::spawn(move || {
let mut i = conn.lock().unwrap().len();
....
}
This way, the cloned Arc, conn, is moved into the closure, and the original Arc, connections, is not, and hence still usable.
I'm not sure exactly what you are doing with your second error, but for the sake of simply counting the connections you do not need to push the entire thing.