Changing a member while in an async stream - rust

I have a struct with a member. The implementation returns an async stream which changes this member on a specific event (in the example on each).
This fails with a lifetime error, which is understandable, since the struct itself is not necessarily in the same lifetime as the map in the stream. it sounds similar to lifetime around async and stream, but I am not sure if it is related.
use async_std::pin::Pin;
use futures::{Stream, StreamExt};
use std::time::Duration;
struct StreamProvider {
value: u16,
}
impl StreamProvider {
fn returnastream(self: &mut Self) -> Pin<Box<dyn Stream<Item = i32>>> {
return async_std::stream::interval(Duration::from_millis(1000))
.map(|_| {
// change a value of Self within the stream
self.value = self.value + 1;
1
})
.boxed();
}
}
#[async_std::main]
async fn main() {
let mut object = StreamProvider { value: 1 };
let mut worx = object.returnastream();
// subscribing to the items
while let item = worx.next().await {
match item {
Some(value) => println!("{}", value),
_ => {}
}
}
}
[dependencies]
futures = "0.3.6"
async-std = { version = "1.6.5", features = ["attributes", "unstable"] }
The Error message:
/Users/andre/.cargo/bin/cargo run --color=always --package traittest --bin traittest
Compiling traittest v0.1.0 (/Users/andre/repos/traittest)
error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'a` due to conflicting requirements
--> src/main.rs:20:12
|
20 | }).boxed();
| ^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 14:3...
--> src/main.rs:14:3
|
14 | / fn returnastream(self: &mut Self) -> Pin<Box<dyn Stream<Item=i32>>> {
15 | | return async_std::stream::interval(Duration::from_millis(1000))
16 | | .map(|_| {
17 | | // change a value of Self within the stream
... |
20 | | }).boxed();
21 | | }
| |___^
note: ...so that the type `futures_util::stream::stream::map::Map<async_std::stream::interval::Interval, [closure#src/main.rs:16:14: 20:10 self:&mut &mut StreamProvider]>` will meet its required lifetime bounds
--> src/main.rs:20:12
|
20 | }).boxed();
| ^^^^^
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the expression is assignable
--> src/main.rs:15:12
|
15 | return async_std::stream::interval(Duration::from_millis(1000))
| ____________^
16 | | .map(|_| {
17 | | // change a value of Self within the stream
18 | | self.value = self.value + 1;
19 | | 1
20 | | }).boxed();
| |__________________^
= note: expected `std::pin::Pin<std::boxed::Box<(dyn futures_core::stream::Stream<Item = i32> + 'static)>>`
found `std::pin::Pin<std::boxed::Box<dyn futures_core::stream::Stream<Item = i32>>>`
error: aborting due to previous error

Thanks to Peter Hall:
You need to use synchonisation primitives, for example Arc and Mutex, in order to access and mutate values that are owned by another thread.
This was the link I missed. Here's the result:
use std::time::Duration;
use async_std::pin::Pin;
use futures::{Stream, StreamExt};
use std::sync::{Arc, Mutex};
struct StreamProvider {
value: Arc<Mutex<u16>>,
}
impl StreamProvider {
fn returnastream(self: &mut Self) -> Pin<Box<dyn Stream<Item = i32>>> {
let v = self.value.clone();
return async_std::stream::interval(Duration::from_millis(1000))
.map(move |_| {
let mut a = v.lock().unwrap();
// change a value of Self within the stream
*a += 1;
1
})
.boxed();
}
}
#[async_std::main]
async fn main() {
let mut object = StreamProvider {
value: Arc::new(Mutex::new(1)),
};
let mut worx = object.returnastream();
// subscribing to the items
while let item = worx.next().await {
match item {
Some(_) => println!("{}", object.value.lock().unwrap()),
_ => {}
}
}
}
[dependencies]
futures = "0.3.6"
async-std = { version = "1.6.5", features = ["attributes", "unstable"] }

Related

future created by async block is not `Send`

I do a server update in rust. it create patches between 2 binaries files, and serves static files
I try to do
let mut update_state;
if let Some(state) = update_stream.next().await {
if let Ok(state) = state {
update_state = state
} else if let Err(err) = state {
reply = BuildOutput { error: "Update failed: ".to_string() + &err.to_string() }
}
} else {
reply = BuildOutput { error: "Unreacheable".to_string() }
}
let state = update_state.borrow();
let progress = state.histogram.progress();
let res = update_stream.try_for_each(|_state| future::ready(Ok(()))).await;
but get
note: future is not `Send` as this value is used across an await
--> server\grpc\src\rpc.rs:260:50
|
259 | let mut update_state;
| ---------------- has type `SharedUpdateProgress` which is not `Send`
260 | if let Some(state) = update_stream.next().await {
| ^^^^^^ await occurs here, with `mut update_state` maybe used later
...
305 | }
| - `mut update_state` is later dropped here
= note: required for the cast from `impl futures::Future<Output = Result<tonic::Response<BuildOutput>, Status>>` to the object type `dyn futures::Future<Output = Result<tonic::Response<BuildOutput>, Status>> + std::marker::Send
SharedUpdateProgress:
#[derive(Clone)]
pub struct SharedUpdateProgress {
state: Rc<RefCell<UpdateProgress>>,
}
impl SharedUpdateProgress {
pub fn new(target_revision: CleanName) -> Self {
Self { state: Rc::new(RefCell::new(UpdateProgress::new(target_revision))) }
}
pub fn borrow(&self) -> Ref<'_, UpdateProgress> {
self.state.borrow()
}
pub(crate) fn borrow_mut(&self) -> RefMut<'_, UpdateProgress> {
self.state.borrow_mut()
}
}
I don't know why and don't know how to fix it
I assume a minimal reproducible example of your problem is as follows:
use std::{cell::RefCell, rc::Rc};
use tokio::time::{sleep, Duration};
#[derive(Clone)]
pub struct SharedString {
state: Rc<RefCell<String>>,
}
impl SharedString {
pub fn new(initial: &str) -> Self {
Self {
state: Rc::new(RefCell::new(initial.into())),
}
}
}
async fn run() {
let shared_string = SharedString::new("Hello,");
sleep(Duration::from_millis(1)).await;
*shared_string.state.borrow_mut() += " world!";
sleep(Duration::from_millis(1)).await;
println!("{:?}", shared_string.state.borrow());
}
#[tokio::main]
async fn main() {
tokio::task::spawn(run()).await.unwrap();
}
error: future cannot be sent between threads safely
--> src/main.rs:27:24
|
27 | tokio::task::spawn(run()).await.unwrap();
| ^^^^^ future returned by `run` is not `Send`
|
= help: within `impl Future<Output = ()>`, the trait `Send` is not implemented for `Rc<RefCell<String>>`
note: future is not `Send` as this value is used across an await
--> src/main.rs:19:36
|
18 | let shared_string = SharedString::new("Hello,");
| ------------- has type `SharedString` which is not `Send`
19 | sleep(Duration::from_millis(1)).await;
| ^^^^^^ await occurs here, with `shared_string` maybe used later
...
23 | }
| - `shared_string` is later dropped here
note: required by a bound in `tokio::spawn`
--> /home/martin/.cargo/git/checkouts/tokio-dd4afa005f1f4b79/686577b/tokio/src/task/spawn.rs:163:21
|
163 | T: Future + Send + 'static,
| ^^^^ required by this bound in `tokio::spawn`
The tokio Runtime is usually multi-threaded, meaning that at any .await point your task could get moved from one thread to another. That's why everything that is held across an .await point must be Send. Which Rc<RefCell<>> is explicitely not, because it's a single-threaded reference counter.
Solution: Replace Rc<RefCell<>> with Arc<Mutex<>>, which is the thread-safe equivalent.
use std::sync::{Arc, Mutex};
use tokio::time::{sleep, Duration};
#[derive(Clone)]
pub struct SharedString {
state: Arc<Mutex<String>>,
}
impl SharedString {
pub fn new(initial: &str) -> Self {
Self {
state: Arc::new(Mutex::new(initial.into())),
}
}
}
async fn run() {
let shared_string = SharedString::new("Hello,");
sleep(Duration::from_millis(1)).await;
*shared_string.state.lock().unwrap() += " world!";
sleep(Duration::from_millis(1)).await;
println!("{:?}", shared_string.state.lock().unwrap());
}
#[tokio::main]
async fn main() {
tokio::task::spawn(run()).await.unwrap();
}
"Hello, world!"

How do I process a vector in a struct method using multiple threads in Rust? [duplicate]

This question already has answers here:
How can I pass a reference to a stack variable to a thread?
(1 answer)
How do I pass disjoint slices from a vector to different threads?
(1 answer)
Concurrent access to vector from multiple threads using a mutex lock
(1 answer)
Processing vec in parallel: how to do safely, or without using unstable features?
(2 answers)
Closed 2 years ago.
I have a struct in Rust representing a container for a heavy data array. I need a method to perform some calculations over this data:
pub struct MultithreadTest {
data: Vec<usize>,
}
impl MultithreadTest {
const WORKERS_COUNT: usize = 2;
pub fn new(data: Vec<usize>) -> Self {
Self { data }
}
pub fn calculate(&self) {
let data = std::sync::Arc::new(&self.data);
let (tx, rx) = std::sync::mpsc::channel();
(0..Self::WORKERS_COUNT).for_each(|thread_index| {
let data = data.clone();
let tx = tx.clone();
std::thread::spawn(move || {
// Do some stuff with data
let sum: usize = data.iter().skip(thread_index).sum();
tx.send(sum).unwrap();
});
});
let result: usize = rx.iter().take(Self::WORKERS_COUNT).sum();
println!("{:}", result);
}
}
If I try to provide self by reference, Rust stops me:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> src/lib.rs:13:40
|
13 | let data = std::sync::Arc::new(&self.data);
| ^^^^^^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 12:5...
--> src/lib.rs:12:5
|
12 | / pub fn calculate(&self) {
13 | | let data = std::sync::Arc::new(&self.data);
14 | |
15 | | let (tx, rx) = std::sync::mpsc::channel();
... |
31 | | println!("{:}", result);
32 | | }
| |_____^
note: ...so that reference does not outlive borrowed content
--> src/lib.rs:13:40
|
13 | let data = std::sync::Arc::new(&self.data);
| ^^^^^^^^^^
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `[closure#src/lib.rs:21:32: 26:14 data:std::sync::Arc<&std::vec::Vec<usize>>, thread_index:usize, tx:std::sync::mpsc::Sender<usize>]` will meet its required lifetime bounds
--> src/lib.rs:21:13
|
21 | std::thread::spawn(move || {
| ^^^^^^^^^^^^^^^^^^```
I totally understand where it's coming from, but I can't understand how to solve it.
I can't clone `data` because it might be huge and I don't want to consume `self` by value.
Thanks to Masklinn's replies, I found a working example using the crossbeam crate:
pub struct MultithreadTest {
data: Vec<usize>,
}
impl MultithreadTest {
const WORKERS_COUNT: usize = 2;
pub fn new(data: Vec<usize>) -> Self {
Self { data }
}
pub fn calculate(&self) {
crossbeam::scope(|scope| {
let workers: Vec<_> = (0..Self::WORKERS_COUNT)
.map(|thread_index| {
scope.spawn(move || {
// Do some stuff with data
let sum: usize = self.data.iter().skip(thread_index).sum();
sum
})
})
.collect();
let result: usize = workers.into_iter().map(|w| w.join()).sum();
println!("{:}", result);
});
}
}

How to solve "lifetime must be valid for the static lifetime" in calling thread methods

I am a Rust beginner and I can’t get the following code to compile. I know this problem might be not new, tried searching all over the place but couldn't find a proper answer to the below problem.
Basically i m trying to call methods from thread, and also using different structs for sending and receiving objects between the threads.
use std::{thread, time};
struct SenderStruct;
impl SenderStruct {
fn send(&self, sndr: Sender<(Option<String>)>) {
let count = 0;
loop {
sndr.send(Some(String::from("Hello"))).unwrap();
thread::sleep(time::Duration::from_millis(1000));
count = count + 1;
if count == 50 {
break;
}
}
sndr.send(None);
}
}
struct ReceiveStruct;
impl ReceiveStruct {
fn receive(&self, rec: Receiver<Option<String>>) {
loop {
let recv_out = rec.recv().unwrap();
match recv_out {
Some(some_str) => println!("{}", some_str),
None => break,
}
}
}
}
struct SendReceiveStruct {
m_ss: SenderStruct,
m_sos: ReceiveStruct,
m_recv_hndlr: Option<thread::JoinHandle<()>>,
}
impl SendReceiveStruct {
fn new() -> Self {
SendReceiveStruct {
m_ss: SenderStruct {},
m_sos: ReceiveStruct {},
m_recv_hndlr: None,
}
}
fn start(&mut self) {
let (tx, rx): (Sender<(Option<String>)>, Receiver<Option<String>>) = channel();
thread::spawn(move || self.m_ss.send(tx));
self.m_recv_hndlr = Some(thread::spawn(move || self.m_sos.receive(rx)));
}
fn wait_for_recevier(&mut self) {
self.m_recv_hndlr.unwrap().join();
}
}
fn main() {
println!("Hello, world!");
let mut ubs = SendReceiveStruct::new();
ubs.start();
ubs.wait_for_recevier();
}
But i m getting lifetime issues all over the place
$ cargo build
Compiling threads v0.1.0 (/root/learn-rust/threads)
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:55:23
|
55 | thread::spawn(move || self.m_ss.send(tx));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 52:5...
--> src/main.rs:52:5
|
52 | / fn start(&mut self) {
53 | | let (tx, rx): (Sender<(Option<String>)>, Receiver<Option<String>>) = channel();
54 | |
55 | | thread::spawn(move || self.m_ss.send(tx));
56 | | self.m_recv_hndlr = Some(thread::spawn(move || self.m_sos.receive(rx)));
57 | | }
| |_____^
= note: ...so that the types are compatible:
expected &mut SendReceiveStruct
found &mut SendReceiveStruct
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `[closure#src/main.rs:55:23: 55:49 self:&mut SendReceiveStruct, tx:std::sync::mpsc::Sender<std::option::Option<std::string::String>>]` will meet its required lifetime bounds
--> src/main.rs:55:9
|
55 | thread::spawn(move || self.m_ss.send(tx));
Any pointers (or other references) would really help, and also any other possible approaches for the above problem ?
If you examine the signature of std::thread::spawn:
pub fn spawn<F, T>(f: F) -> JoinHandle<T>
where
F: FnOnce() -> T,
F: Send + 'static,
T: Send + 'static,
and its documentation closely:
The 'static constraint means that the closure and its return value must have a lifetime of the whole program execution. The reason for this is that threads can detach and outlive the lifetime they have been created in.
However, the &mut self value closure closes over may not live long enough. One way to overcome that is to clone the value the closure actually uses:
#[derive(Clone)]
struct SenderStruct;
#[derive(Clone)]
struct ReceiveStruct;
impl SendReceiveStruct {
fn start(&mut self) {
let (tx, rx): (Sender<Option<String>>, Receiver<Option<String>>) = channel();
thread::spawn({
let ss = self.m_ss.clone();
move || ss.send(tx)
});
self.m_recv_hndlr = Some(thread::spawn({
let sos = self.m_sos.clone();
move || sos.receive(rx)
}));
}
fn wait_for_recevier(&mut self) {
self.m_recv_hndlr.take().unwrap().join();
}
}
Except for several other minor issues, your code now compiles.

How can I write a boxed closure which mutates a reference to a struct?

I have some code that looks like this:
type Callback<T> = Box<Fn(&T) -> ()>;
struct Foo {
name: String,
}
impl Foo {
fn name_updater(&mut self) -> Callback<String> {
Box::new(|new_name| {
self.name = *new_name;
})
}
}
This code doesn't compile because the closure requires the static lifetime. However I'm unsure if this will resolve the question and because there aren't explicit lifetimes, it's not immediately clear to me what I'd need to do to resolve that.
Here is an example of the compilation error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/lib.rs:9:9
|
9 | / Box::new(|new_name| {
10 | | self.name = *new_name;
11 | | })
| |__________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 8:5...
--> src/lib.rs:8:5
|
8 | / fn name_updater(&mut self) -> Callback<String> {
9 | | Box::new(|new_name| {
10 | | self.name = *new_name;
11 | | })
12 | | }
| |_____^
note: ...so that the type `[closure#src/lib.rs:9:18: 11:10 self:&mut &mut Foo]` will meet its required lifetime bounds
--> src/lib.rs:9:9
|
9 | / Box::new(|new_name| {
10 | | self.name = *new_name;
11 | | })
| |__________^
= note: but, the lifetime must be valid for the static lifetime...
= note: ...so that the expression is assignable:
expected std::boxed::Box<(dyn for<'r> std::ops::Fn(&'r std::string::String) + 'static)>
found std::boxed::Box<dyn for<'r> std::ops::Fn(&'r std::string::String)>
How can I write a closure that is capable of mutating the struct's name property?
You need to bound a lifetime to your closure because you borrow something aka &mut self:
type Callback<'a, T> = Box<dyn FnMut(&T) -> () + 'a>;
#[derive(Debug)]
struct Foo {
name: String,
}
impl Foo {
fn name_updater(&mut self) -> Callback<str> {
Box::new(move |new_name| {
self.name.replace_range(.., new_name);
})
}
}
fn main() {
let mut foo = Foo {
name: String::from("foo"),
};
foo.name_updater()("bar");
println!("{:?}", foo);
}
Also note that you don't need to use a box:
#[derive(Debug)]
struct Foo {
name: String,
}
impl Foo {
fn name_updater<'a>(&'a mut self) -> impl FnMut(&str) -> () + 'a {
move |new_name| {
self.name.replace_range(.., new_name);
}
}
}
fn main() {
let mut foo = Foo {
name: String::from("foo"),
};
foo.name_updater()("bar");
println!("{:?}", foo);
}

Trouble with closure lifetimes

I am really struggling with closure lifetimes. I'm making a simple hangman game where all connections play one game. I'm attempting to pass a closure to a channel that will update the game and then broadcast JSON, but I am running into lifetime issues.
extern crate names;
extern crate ws;
#[macro_use]
extern crate json;
use names::{Generator, Name};
use std::collections::HashSet;
use std::sync::mpsc;
use std::thread;
use ws::{listen, CloseCode, Handler, Message, Result, Sender};
type Job = Box<FnMut(&mut Game) + Send>;
#[derive(Debug)]
struct Game {
word: Vec<String>,
guesses: HashSet<String>,
misses: u32,
progress: Vec<String>,
}
impl Game {
fn increment_miss(&mut self) {
self.misses += 1;
}
fn update_progress(&mut self, guess: &String) {
for (i, letter) in self.word.iter().enumerate() {
if letter == guess {
self.progress[i] = letter.clone();
}
}
}
fn status(&self) -> &str {
if self.misses > 10 {
"lose"
} else if self.progress == self.word {
"win"
} else {
"active"
}
}
}
struct Server {
out: Sender,
tx: std::sync::mpsc::Sender<Job>,
}
impl Handler for Server {
fn on_message(&mut self, msg: Message) -> Result<()> {
let string_msg = msg.to_string();
self.tx
.send(Box::new(move |mut game: &mut Game| {
if game.guesses.insert(string_msg.clone()) {
check_letter(&mut game, &string_msg);
};
let status = game.status();
let progress = game.progress.clone();
let guesses = game.guesses.clone().into_iter().collect::<Vec<String>>();
println!(
"guesses: {:?}, progress: {:?}, misses: {}, status: {}",
guesses, progress, game.misses, status
);
self.out.broadcast(json::stringify(object!{
"status" => "status",
"progress" => "progress",
"guesses" => "guesses",
"misses" => "misses",
}));
}))
.unwrap();
Ok(())
}
fn on_close(&mut self, code: CloseCode, reason: &str) {
match code {
CloseCode::Normal => println!("The client is done with the connection."),
CloseCode::Away => println!("The client is leaving the site."),
_ => println!("The client encountered an error: {}", reason),
}
}
}
fn check_letter(game: &mut Game, guess: &String) {
if game.word.contains(guess) {
game.update_progress(guess);
} else {
game.increment_miss();
}
}
fn generate_word() -> Vec<String> {
let mut generator = Generator::with_naming(Name::Plain);
generator
.next()
.unwrap()
.split("")
.map(|c| c.to_string())
.filter(|s| s != "")
.collect::<Vec<String>>()
}
fn start_game() -> Game {
let word = generate_word();
Game {
progress: vec!["".to_string(); word.len()],
word: word,
guesses: HashSet::new(),
misses: 0,
}
}
fn main() {
let (tx, rx) = mpsc::channel::<Job>();
thread::spawn(move || {
let mut game = start_game();
for mut received in rx {
received(&mut game)
}
});
listen("127.0.0.1:3000", |out| Server {
out: out,
tx: mpsc::Sender::clone(&tx),
}).unwrap();
}
I am getting the following error:
Compiling hang_man v0.1.0 (file:///Users/smykowski/workspace/rust/hang_man)
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:57:22
|
57 | self.tx.send(Box::new(move |mut game: &mut Game| {
| ______________________^
58 | | if game.guesses.insert(string_msg.clone()) {
59 | | check_letter(&mut game, &string_msg);
60 | | };
... |
75 | | }));
76 | | })).unwrap();
| |__________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 54:5...
--> src/main.rs:54:5
|
54 | / fn on_message(&mut self, msg: Message) -> Result<()> {
55 | | let string_msg = msg.to_string();
56 | |
57 | | self.tx.send(Box::new(move |mut game: &mut Game| {
... |
78 | | Ok(())
79 | | }
| |_____^
note: ...so that the type `[closure#src/main.rs:57:31: 76:10 string_msg:std::string::String, self:&mut Server]` will meet its required lifetime bounds
--> src/main.rs:57:22
|
57 | self.tx.send(Box::new(move |mut game: &mut Game| {
| ______________________^
58 | | if game.guesses.insert(string_msg.clone()) {
59 | | check_letter(&mut game, &string_msg);
60 | | };
... |
75 | | }));
76 | | })).unwrap();
| |__________^
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that expression is assignable (expected std::boxed::Box<for<'r> std::ops::FnMut(&'r mut Game) + std::marker::Send + 'static>, found std::boxed::Box<for<'r> std::ops::FnMut(&'r mut Game) + std::marker::Send>)
--> src/main.rs:57:22
|
57 | self.tx.send(Box::new(move |mut game: &mut Game| {
| ______________________^
58 | | if game.guesses.insert(string_msg.clone()) {
59 | | check_letter(&mut game, &string_msg);
60 | | };
... |
75 | | }));
76 | | })).unwrap();
| |__________^
There is some confusion here because Rust is not very explicit about some features of lifetimes. In this example,
A closure's lifetime is limited by the lifetime of all the params moved into it. In your case, the closure is limited by self because of the line self.out.broadcast. Note that self is actually a reference in this case coming from the &self argument in fn on_message. Essentially you create something like
Box<FnMut(&mut Game) + Send + 'a>
Where 'a is a lifetime of &self.
When you create a boxed trait object, the default lifetime is 'static. That means that
type Job = Box<FnMut(&mut Game) + Send> = Box<FnMut(&mut Game) + Send + 'static>
To avoid this, you can make self cloneable and move self.clone() inside this closure.

Resources