How to solve lifetime inside for_each - rust

I try to get the following code to compile.
Due to the buffer it wanted more explicit lifetimes inserted. So I tried that but still
cannot figure out yet how to make it also work in for_each.
use std::{sync::mpsc::Sender, ffi::OsStr};
use inotify::{Event, EventMask, Inotify, WatchMask};
pub struct Watcher<'a> {
buffer: [u8; 1024],
sender: Sender<Event<&'a OsStr>>,
}
impl <'a> Watcher<'a> {
pub fn new(sender: Sender<Event<&OsStr>>) -> Watcher{
Watcher{
buffer: [0; 1024],
sender: sender,
}
}
pub fn watch_for_led_directories(&mut self, dir: String) {
let sender = self.sender.clone();
let mut inotify = Inotify::init().expect("Error while initializing inotify instance");
// Watch for modify and close events.
inotify
.add_watch(dir, WatchMask::CREATE | WatchMask::DELETE)
.expect("Failed to add file watch");
// Read events that were added with `add_watch` above.
let events = inotify
.read_events_blocking(&mut self.buffer)
.expect("Error while reading events");
events.filter(|e| -> bool { !e.mask.intersects(EventMask::ISDIR) }).for_each(|e| sender.send(e).unwrap());
}
}
EDIT:
I ended up giving up on no-copy of strings:
use std::{ffi::OsStr, sync::mpsc::Sender};
use inotify::{Event, EventMask, Inotify, WatchMask};
pub struct Watcher {
buffer: [u8; 1024],
sender: Sender<Event<String>>,
}
impl Watcher {
pub fn new(sender: Sender<Event<String>>) -> Watcher {
Watcher {
buffer: [0; 1024],
sender: sender,
}
}
pub fn watch_for_led_directories(&mut self, dir: String) {
let sender = self.sender.clone();
let mut inotify = Inotify::init().expect("Error while initializing inotify instance");
// Watch for modify and close events.
inotify
.add_watch(dir, WatchMask::CREATE | WatchMask::DELETE)
.expect("Failed to add file watch");
// Read events that were added with `add_watch` above.
let events = inotify
.read_events_blocking(&mut self.buffer)
.expect("Error while reading events");
let filter = |e: &Event<&OsStr>| -> bool {
!e.mask.intersects(EventMask::ISDIR) || e.name.is_none()
};
events.filter(filter).for_each(|e| {
sender
.send(Event {
cookie: e.cookie,
mask: e.mask,
name: Some(e.name.unwrap().to_str().expect("Expected valid unicode string").into()),
wd: e.wd,
})
.unwrap()
});
}
}

The issue is that while self has the lifetime of 'a, &mut self in watch_for_led_directories() has a different lifetime.
inotify.read_events_blocking() returns events that has the same lifetime as &mut self.buffer, which, as a mutable borrow, has the lifetime of &mut self. But sender is bound to life time 'a. Therefore compiler cannot reconcile these two lifetimes.
One way to solve it is to provide an external buffer, instead of having it owned by Watcher, e.g.:
pub fn watch_for_led_directories(&self, buffer: &'a mut [u8], dir: String) {
...
let events: Events = inotify
.read_events_blocking(buffer)
.expect("Error while reading events");
...
Compiler derives the lifetime of events from buffer and therefore it will be 'a.
To use this method the externally allocated buffer must live at least as long as the Watcher object, which should not be too difficult to arrange.

Related

Change a channel receiver type from Receiver<T> to Receiver<U>

I've got an interface, which defines a method that returns a receiver:
pub fn subscribe(to: &str) -> crossbeam_channel::Receiver<Message>;
I am using a library method that returns a Receiver, but of a different message type:
pub fn subscribe(to: &str) -> crossbeam_channel::Receiver<lib::Message>;
It is easy enough to convert lib::Message to Message but how could I implement the interface, which would act as a wrapper for this library, such that the type returned is correct?
I've tried to create a new channel, but this doesn't work, (I think) since the method will return and then no longer pass messages to the new channel, therefore the receiver will always be empty.
let sub_recv = subscription.receiver();
let (send, receiver) = crossbeam_channel::unbounded::<Message>();
for m in sub_recv.try_recv() {
send.send(m.into()).map_err(|_| MQError::ConversionError)?;
}
Thanks
To convert between your types you should run background task to do so, because otwerwise you would block the thread where you are trying to do so.
Playground link
use crossbeam_channel::{Receiver, unbounded}; // 0.5.0
trait ReceiverCompatExt<T>
{
fn convert(self) -> Receiver<T>;
}
impl<T, U> ReceiverCompatExt<U> for Receiver<T>
where U: From<T>,
T: Send + 'static,
U: Send + 'static,
{
fn convert(self) -> Receiver<U> {
let (sender, receiver) = unbounded();
std::thread::spawn(move || {
while let Ok(value) = self.recv() {
if sender.send(value.into()).is_err() {
break;
}
}
});
receiver
}
}

How to idiomatically share data between closures with wasm-bindgen?

In my browser application, two closures access data stored in a Rc<RefCell<T>>. One closure mutably borrows the data, while the other immutably borrows it. The two closures are invoked independently of one another, and this will occasionally result in a BorrowError or BorrowMutError.
Here is my attempt at an MWE, though it uses a future to artificially inflate the likelihood of the error occurring:
use std::cell::RefCell;
use std::future::Future;
use std::pin::Pin;
use std::rc::Rc;
use std::task::{Context, Poll, Waker};
use wasm_bindgen::prelude::*;
use wasm_bindgen::JsValue;
#[wasm_bindgen]
extern "C" {
#[wasm_bindgen(js_namespace = console)]
pub fn log(s: &str);
#[wasm_bindgen(js_name = setTimeout)]
fn set_timeout(closure: &Closure<dyn FnMut()>, millis: u32) -> i32;
#[wasm_bindgen(js_name = setInterval)]
fn set_interval(closure: &Closure<dyn FnMut()>, millis: u32) -> i32;
}
pub struct Counter(u32);
#[wasm_bindgen(start)]
pub async fn main() -> Result<(), JsValue> {
console_error_panic_hook::set_once();
let counter = Rc::new(RefCell::new(Counter(0)));
let counter_clone = counter.clone();
let log_closure = Closure::wrap(Box::new(move || {
let c = counter_clone.borrow();
log(&c.0.to_string());
}) as Box<dyn FnMut()>);
set_interval(&log_closure, 1000);
log_closure.forget();
let counter_clone = counter.clone();
let increment_closure = Closure::wrap(Box::new(move || {
let counter_clone = counter_clone.clone();
wasm_bindgen_futures::spawn_local(async move {
let mut c = counter_clone.borrow_mut();
// In reality this future would be replaced by some other
// time-consuming operation manipulating the borrowed data
SleepFuture::new(5000).await;
c.0 += 1;
});
}) as Box<dyn FnMut()>);
set_timeout(&increment_closure, 3000);
increment_closure.forget();
Ok(())
}
struct SleepSharedState {
waker: Option<Waker>,
completed: bool,
closure: Option<Closure<dyn FnMut()>>,
}
struct SleepFuture {
shared_state: Rc<RefCell<SleepSharedState>>,
}
impl Future for SleepFuture {
type Output = ();
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
let mut shared_state = self.shared_state.borrow_mut();
if shared_state.completed {
Poll::Ready(())
} else {
shared_state.waker = Some(cx.waker().clone());
Poll::Pending
}
}
}
impl SleepFuture {
fn new(duration: u32) -> Self {
let shared_state = Rc::new(RefCell::new(SleepSharedState {
waker: None,
completed: false,
closure: None,
}));
let state_clone = shared_state.clone();
let closure = Closure::wrap(Box::new(move || {
let mut state = state_clone.borrow_mut();
state.completed = true;
if let Some(waker) = state.waker.take() {
waker.wake();
}
}) as Box<dyn FnMut()>);
set_timeout(&closure, duration);
shared_state.borrow_mut().closure = Some(closure);
SleepFuture { shared_state }
}
}
panicked at 'already mutably borrowed: BorrowError'
The error makes sense, but how should I go about resolving it?
My current solution is to have the closures use try_borrow or try_borrow_mut, and if unsuccessful, use setTimeout for an arbitrary amount of time before attempting to borrow again.
Think about this problem independently of Rust's borrow semantics. You have a long-running operation that's updating some shared state.
How would you do it if you were using threads? You would put the shared state behind a lock. RefCell is like a lock except that you can't block on unlocking it — but you can emulate blocking by using some kind of message-passing to wake up the reader.
How would you do it if you were using pure JavaScript? You don't automatically have anything like RefCell, so either:
The state can be safely read while the operation is still ongoing (in a concurrency-not-parallelism sense): in this case, emulate that by not holding a single RefMut (result of borrow_mut()) alive across an await boundary.
The state is not safe to be read: you'd either write something lock-like as described above, or perhaps arrange so that it's only written once when the operation is done, and until then, the long-running operation has its own private state not shared with the rest of the application (so there can be no BorrowError conflicts).
Think about what your application actually needs and pick a suitable solution. Implementing any of these solutions will most likely involve having additional interior-mutable objects used for communication.

How to run futures containing borrowed TcpStream concurrently?

I am trying to make this code snippet run concurrently instead of sequentially since the number of peers can be a large value. I am using async_std 1.4 and rust 1.41
pub struct Peer {
pub peer_id: String,
pub tcp_stream: Arc<TcpStream>,
pub public_key: [u8; 32],
}
async fn send_to_all_peers(message: Protocol, peers: &HashMap<String,Peer>) -> Result<()> {
for peer in peers.values() {
let mut stream = &*peer.tcp_stream;
stream.write_all(&bincode::serialize(&message)?).await?;
}
Ok(())
}
I've tried to use the futures::future::join_all method without any luck since wrapping future I created and used within async_std::task::spawn requires a static lifetime. Here is what I tried:
async fn send_to_all_peers(message: Protocol, peers: &HashMap<String,Peer>) {
let handles = peers.values().into_iter().map(|peer| {
task::spawn(
async {
let mut stream = &*peer.tcp_stream;
if let Err(err) = stream
.write_all(&bincode::serialize(&message).unwrap())
.await
{
error!("Error when writing to tcp_stream: {}", err);
}
}
)
});
futures::future::join_all(handles).await;
}
I'm sure there is some method I am missing, thanks for any help!
Since you are trying to send message concurrently, each task has to have its own copy of the message:
use async_std::{task, net::TcpStream};
use futures::{future, io::AsyncWriteExt};
use serde::Serialize;
use std::{
collections::HashMap,
error::Error,
sync::Arc,
};
pub struct Peer {
pub peer_id: String,
pub tcp_stream: Arc<TcpStream>,
pub public_key: [u8; 32],
}
#[derive(Serialize)]
struct Protocol;
async fn send_to_all_peers(
message: Protocol,
peers: &HashMap<String, Peer>)
-> Result<(), Box<dyn Error>>
{
let msg = bincode::serialize(&message)?;
let handles = peers.values()
.map(|peer| {
let msg = msg.clone();
let socket = peer.tcp_stream.clone();
task::spawn(async move {
let mut socket = &*socket;
socket.write_all(&msg).await
})
});
future::try_join_all(handles).await?;
Ok(())
}
Have you tried something like
let handles = peers.values().into_iter().map(|peer| {
let mut stream = &*peer.tcp_stream;
stream.write_all(&bincode::serialize(&message).unwrap())
}
let results = futures::future::join_all(handles).await
?
Notice how the .map closure doesn’t await, but straight up returns a future, which is then passed to join_all, and then awaited.

Manipulating an object from inside a loop that borrows it

I'm writing some code in Rust that connects to a remote server, and depending on the messages sent by that server, computes some statistics or executes actions based on these statistics. But this is more of a learning project for me and I've run into an issue.
Here is the code that I have reduced to a bare minimum to reproduce the problem :
// Repro code for error[E0502]: cannot borrow `*self` as mutable because `self.server` is also borrowed as immutable
use std::collections::HashMap;
struct ServerReader {
server: Vec<u32>, // A vec for demo purposes, but please imagine this is a server object
counters: HashMap<u32, usize>,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6), // Filling my "server" with some messages
counters: HashMap::new(),
}
}
fn run(&mut self) {
println!("Connecting..."); // ... here there should be some code to connect to the server ...
for message in self.server.iter() { // We wait for the network messages sent by the server, and process them as they come
// ----------- immutable borrow occurs here
println!("Received {}", message);
self.process_message(*message); // HOW
// ^^^^ mutable borrow occurs here
}
// - immutable borrow ends here
println!("Disconnected");
}
fn process_message(&mut self, message: u32) {
// Please imagine that this function contains complex stuff
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
fn main() {
let mut reader = ServerReader::new();
reader.run();
println!("Done");
}
While I think I understand why the compiler is unhappy, I'm struggling to come up with a solution. I cannot manipulate my structure outside of the loop, since I have to work while connected and listening to the server. I also could put everything directly in the loop and not call any method, but I don't want to end up with a 1000 line loop (and I'd prefer to understand what an actual solution would look like).
As you've worked out, you can't call a &mut self method while you're borrowing part of self, so you need to restructure somehow.
The way I would do it is to split the state needed by process_message into a separate type (in your example that's basically the HashMap, but in the real application it's likely to contain more), and move the method to that type. This works because you can separately borrow fields from a struct.
struct SomeState {
counters: HashMap<u32, usize>,
}
impl SomeState {
pub fn new() -> SomeState {
SomeState {
counters: HashMap::new(),
}
}
fn process_message(&mut self, message: u32) {
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
struct ServerReader {
server: Vec<u32>,
state: SomeState,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6),
state: SomeState::new(),
}
}
fn run(&mut self) {
println!("Connecting...");
for message in self.server.iter() {
println!("Received {}", message);
self.state.process_message(*message);
}
println!("Disconnected");
}
}
An alternative (which may or may not be possible in your real example) would be to avoid borrowing in the loop, making it more like:
loop {
// if next_message() returns an owned message, ie not still borrowing
// self
let message = self.next_message();
// now no borrow left
self.process_message(message);
}
Given that you don't need the full ServerReader for processing a message, you could make process_message a free function and just pass &mut self.counters to it. Then you have disjoint borrows of server and counters, which is fine.
Or if your non-server part of ServerReader is larger, extract that into its own struct, and make process_message an impl method of that struct.
In order to allow mutability in an Iterator, you should use iter_mut() and work on mutable references (&mut message). Then, to avoid the additional borrow, you could just perform the addition in the body of the loop:
for &mut message in self.server.iter_mut() {
println!("Received {}", message);
*self.counters.entry(message).or_insert(0) += 1;
}

Multiple self borrows [duplicate]

I'm writing some code in Rust that connects to a remote server, and depending on the messages sent by that server, computes some statistics or executes actions based on these statistics. But this is more of a learning project for me and I've run into an issue.
Here is the code that I have reduced to a bare minimum to reproduce the problem :
// Repro code for error[E0502]: cannot borrow `*self` as mutable because `self.server` is also borrowed as immutable
use std::collections::HashMap;
struct ServerReader {
server: Vec<u32>, // A vec for demo purposes, but please imagine this is a server object
counters: HashMap<u32, usize>,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6), // Filling my "server" with some messages
counters: HashMap::new(),
}
}
fn run(&mut self) {
println!("Connecting..."); // ... here there should be some code to connect to the server ...
for message in self.server.iter() { // We wait for the network messages sent by the server, and process them as they come
// ----------- immutable borrow occurs here
println!("Received {}", message);
self.process_message(*message); // HOW
// ^^^^ mutable borrow occurs here
}
// - immutable borrow ends here
println!("Disconnected");
}
fn process_message(&mut self, message: u32) {
// Please imagine that this function contains complex stuff
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
fn main() {
let mut reader = ServerReader::new();
reader.run();
println!("Done");
}
While I think I understand why the compiler is unhappy, I'm struggling to come up with a solution. I cannot manipulate my structure outside of the loop, since I have to work while connected and listening to the server. I also could put everything directly in the loop and not call any method, but I don't want to end up with a 1000 line loop (and I'd prefer to understand what an actual solution would look like).
As you've worked out, you can't call a &mut self method while you're borrowing part of self, so you need to restructure somehow.
The way I would do it is to split the state needed by process_message into a separate type (in your example that's basically the HashMap, but in the real application it's likely to contain more), and move the method to that type. This works because you can separately borrow fields from a struct.
struct SomeState {
counters: HashMap<u32, usize>,
}
impl SomeState {
pub fn new() -> SomeState {
SomeState {
counters: HashMap::new(),
}
}
fn process_message(&mut self, message: u32) {
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
struct ServerReader {
server: Vec<u32>,
state: SomeState,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6),
state: SomeState::new(),
}
}
fn run(&mut self) {
println!("Connecting...");
for message in self.server.iter() {
println!("Received {}", message);
self.state.process_message(*message);
}
println!("Disconnected");
}
}
An alternative (which may or may not be possible in your real example) would be to avoid borrowing in the loop, making it more like:
loop {
// if next_message() returns an owned message, ie not still borrowing
// self
let message = self.next_message();
// now no borrow left
self.process_message(message);
}
Given that you don't need the full ServerReader for processing a message, you could make process_message a free function and just pass &mut self.counters to it. Then you have disjoint borrows of server and counters, which is fine.
Or if your non-server part of ServerReader is larger, extract that into its own struct, and make process_message an impl method of that struct.
In order to allow mutability in an Iterator, you should use iter_mut() and work on mutable references (&mut message). Then, to avoid the additional borrow, you could just perform the addition in the body of the loop:
for &mut message in self.server.iter_mut() {
println!("Received {}", message);
*self.counters.entry(message).or_insert(0) += 1;
}

Resources