In my browser application, two closures access data stored in a Rc<RefCell<T>>. One closure mutably borrows the data, while the other immutably borrows it. The two closures are invoked independently of one another, and this will occasionally result in a BorrowError or BorrowMutError.
Here is my attempt at an MWE, though it uses a future to artificially inflate the likelihood of the error occurring:
use std::cell::RefCell;
use std::future::Future;
use std::pin::Pin;
use std::rc::Rc;
use std::task::{Context, Poll, Waker};
use wasm_bindgen::prelude::*;
use wasm_bindgen::JsValue;
#[wasm_bindgen]
extern "C" {
#[wasm_bindgen(js_namespace = console)]
pub fn log(s: &str);
#[wasm_bindgen(js_name = setTimeout)]
fn set_timeout(closure: &Closure<dyn FnMut()>, millis: u32) -> i32;
#[wasm_bindgen(js_name = setInterval)]
fn set_interval(closure: &Closure<dyn FnMut()>, millis: u32) -> i32;
}
pub struct Counter(u32);
#[wasm_bindgen(start)]
pub async fn main() -> Result<(), JsValue> {
console_error_panic_hook::set_once();
let counter = Rc::new(RefCell::new(Counter(0)));
let counter_clone = counter.clone();
let log_closure = Closure::wrap(Box::new(move || {
let c = counter_clone.borrow();
log(&c.0.to_string());
}) as Box<dyn FnMut()>);
set_interval(&log_closure, 1000);
log_closure.forget();
let counter_clone = counter.clone();
let increment_closure = Closure::wrap(Box::new(move || {
let counter_clone = counter_clone.clone();
wasm_bindgen_futures::spawn_local(async move {
let mut c = counter_clone.borrow_mut();
// In reality this future would be replaced by some other
// time-consuming operation manipulating the borrowed data
SleepFuture::new(5000).await;
c.0 += 1;
});
}) as Box<dyn FnMut()>);
set_timeout(&increment_closure, 3000);
increment_closure.forget();
Ok(())
}
struct SleepSharedState {
waker: Option<Waker>,
completed: bool,
closure: Option<Closure<dyn FnMut()>>,
}
struct SleepFuture {
shared_state: Rc<RefCell<SleepSharedState>>,
}
impl Future for SleepFuture {
type Output = ();
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
let mut shared_state = self.shared_state.borrow_mut();
if shared_state.completed {
Poll::Ready(())
} else {
shared_state.waker = Some(cx.waker().clone());
Poll::Pending
}
}
}
impl SleepFuture {
fn new(duration: u32) -> Self {
let shared_state = Rc::new(RefCell::new(SleepSharedState {
waker: None,
completed: false,
closure: None,
}));
let state_clone = shared_state.clone();
let closure = Closure::wrap(Box::new(move || {
let mut state = state_clone.borrow_mut();
state.completed = true;
if let Some(waker) = state.waker.take() {
waker.wake();
}
}) as Box<dyn FnMut()>);
set_timeout(&closure, duration);
shared_state.borrow_mut().closure = Some(closure);
SleepFuture { shared_state }
}
}
panicked at 'already mutably borrowed: BorrowError'
The error makes sense, but how should I go about resolving it?
My current solution is to have the closures use try_borrow or try_borrow_mut, and if unsuccessful, use setTimeout for an arbitrary amount of time before attempting to borrow again.
Think about this problem independently of Rust's borrow semantics. You have a long-running operation that's updating some shared state.
How would you do it if you were using threads? You would put the shared state behind a lock. RefCell is like a lock except that you can't block on unlocking it — but you can emulate blocking by using some kind of message-passing to wake up the reader.
How would you do it if you were using pure JavaScript? You don't automatically have anything like RefCell, so either:
The state can be safely read while the operation is still ongoing (in a concurrency-not-parallelism sense): in this case, emulate that by not holding a single RefMut (result of borrow_mut()) alive across an await boundary.
The state is not safe to be read: you'd either write something lock-like as described above, or perhaps arrange so that it's only written once when the operation is done, and until then, the long-running operation has its own private state not shared with the rest of the application (so there can be no BorrowError conflicts).
Think about what your application actually needs and pick a suitable solution. Implementing any of these solutions will most likely involve having additional interior-mutable objects used for communication.
Related
I am working on a Linux-PAM module and I implemented this function to get the user.
pub fn get_user(pamh: PamHandleT) -> PamResult<&'static CStr> {
let mut raw_user: *const c_char = ptr::null();
let r = unsafe { pam_get_user(pamh, &mut raw_user, ptr::null()) };
if raw_user.is_null() {
Err(r)
} else {
let user = unsafe {CStr::from_ptr(raw_user)};
Ok(user)
}
}
pam_get_user is a C function from libpam that returns a * const c_char via its second argument. PAM documentation states that I must not free that pointer to allow interoperability with other modules.
By using the 'static lifetime for the return value, I believe this value will not be deallocated, is that correct? Maybe I could copy the value to use it in a more Rust-idiomatic way, how could I do that?
CStr is responsible for handling the value and in contrast to CString it does not allocate or deallocate memory, just like str and String. You just pass a pointer to it and have to ensure its requirements. Make sure to read std::ffi::CStr carefully and understand what you are doing.
Your code looks fine so far, so you should be ready to go.
You, probably, need to make owned value. This would allocate though.
pub fn get_user(pamh: PamHandleT) -> PamResult<CString> {
let mut raw_user: *const c_char = ptr::null();
let r = unsafe { pam_get_user(pamh, &mut raw_user, ptr::null()) };
if raw_user.is_null() {
Err(r)
} else {
let user = unsafe {CStr::from_ptr(raw_user)};
Ok(user.to_owned())
}
}
If you want to avoid allocation, you should create some context object.
"The pam_end function terminates the PAM transaction and is the last function an application should call in the PAM context. Upon return the handle pamh is no longer valid and all memory associated with it will be invalid. "
struct TransactionContext{
pamh: PamHandleT
}
impl Drop for TransactionContext{
fn drop(&mut self){
unsafe {pam_end(pamh);}
}
}
pub fn get_user(pamh: &TransactionContext) -> PamResult<&CStr> {
let mut raw_user: *const c_char = ptr::null();
let r = unsafe { pam_get_user(pamh.pamh, &mut raw_user, ptr::null()) };
if raw_user.is_null() {
Err(r)
} else {
let user = unsafe {CStr::from_ptr(raw_user)};
Ok(user)
}
}
This would make result CStr to have same lifetime as TransactionContext and borrow checker would ensure that you don't use result CStr after your TransactionContext is dropped.
more on reimplementing c++ timerqueue in rust
This code tries to create a vec of tasks and run them on a different thread. Still cannot compile it. I am at the point of typing words at random into random locations to see if it helps , 'static here, move there dyn sync ,....
I think if I were to see a working version I would be able to work out why I was stuck. But I cannot make it go.
heres the complete code
use std::thread;
use std::time::Instant;
fn main() {
let x = || {
println!("hello");
};
let y = || {
println!("hello2");
};
let mut tq = TimerQueue::new();
tq.set(Box::new(x), String::from("yo"), Instant::now());
tq.set(Box::new(y), String::from("yo"), Instant::now());
tq.thr();
}
pub struct TimerQueueItem {
when: Instant,
name: String,
what: QIFunc,
}
type QIFunc = Box<dyn Fn() -> () + Send>;
struct TimerQueue {
queue: Vec<TimerQueueItem>,
}
impl TimerQueue {
fn thr(&mut self) -> () {
thread::spawn(move || {
self.runq();
});
}
fn set(&mut self, f: QIFunc, n: String, when: Instant) {
let qi = TimerQueueItem {
what: f,
name: n,
when: when,
};
self.queue.push(qi);
}
fn new() -> TimerQueue {
TimerQueue { queue: Vec::new() }
}
fn runq(&mut self) -> () {
for qi in self.queue.iter() {
(qi.what)();
}
}
}
all sorts of errors all pointing at the same place
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src\main.rs:31:23
|
31 | thread::spawn(move || {
| _______________________^
32 | | self.runq();
33 | | });
| |_________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 30:5...
--> src\main.rs:30:5
this one seems particularly designed to confuse
= note: expected `&mut TimerQueue`
found `&mut TimerQueue`
Your TimerQueue object is stored on the stack of the main thread. Because Rust cannot guarantee that the new thread will finish before the main one, it has imposed some restrictions on how you can pull off recreating your timerqueue.
I actually found a simple fix. Basically, I removed the thr method and moved its body into main.
use std::thread;
use std::time::Instant;
fn main() {
let x = || {
println!("hello");
};
let y = || {
println!("hello2");
};
let mut tq = TimerQueue::new();
tq.set(Box::new(x), String::from("yo"), Instant::now());
tq.set(Box::new(y), String::from("yo"), Instant::now());
thread::spawn(move || {
tq.runq();
});
thread::sleep_ms(1000);
}
pub struct TimerQueueItem {
when: Instant,
name: String,
what: QIFunc,
}
type QIFunc = Box<dyn Fn() -> () + Send>;
struct TimerQueue {
queue: Vec<TimerQueueItem>,
}
impl TimerQueue {
fn set(&mut self, f: QIFunc, n: String, when: Instant) {
let qi = TimerQueueItem {
what: f,
name: n,
when: when,
};
self.queue.push(qi);
}
fn new() -> TimerQueue {
TimerQueue { queue: Vec::new() }
}
fn runq(&mut self) -> () {
for qi in self.queue.iter() {
(qi.what)();
}
}
}
Note: I added the
thread::sleep_ms(1000);
which ensures that the main thread will last as long as the spawned thread - at least in this example, other scenarios might have the new thread require more time. If you want to pause the main thread until the spawned one completes (regardless of how long the spawned one takes), you can replace
thread::spawn(move || {
tq.runq();
});
thread::sleep_ms(1000);
with
let thread_handle = thread::spawn(move || {
tq.runq();
});
thread_handle.join();
For reference, my cargo version is 1.41.0
Instead of thr(&mut self), write thr(mut self).
(mut because runq wants a mutable reference. It might not need to though.)
You can't transfer ownership of the TimerQueue to the new thread with just a reference. When you "move" something into a closure, you transfer ownership there. On the other hand, if you didn't move (i.e. left the move keyword out), you couldn't ensure that self would live long enough.
A C++ analogy to this would be
// BAD: v may go out of scope in the main
// thread before the new thread completes.
// Also facilitates data races.
std::vector<...> v;
std::thread([&v] () {
// use/modify v
});
// BETTER: v is moved to be "owned" by the
// closure, so it will certainly live as long as the
// thread is alive.
std::vector<...> v;
std::thread([v=std::move(v)] () mutable {
// use/modify v
})
There's really no such thing as a member function that takes this as moved in C++, but there is in Rust. You can think of it as a static class method that looks like this instead:
static void thr(TimerQueue &&tq);
// called as
TimerQueue::thr(std::move(tq));
Of course, that's not identical to the Rust version in many ways, but hopefully this helps you understand what's going on.
That error message is certainly unhelpful. Not sure what's going on there.
I am a beginner in Rust.
I have a long running IO-bound process that I want to spawn and monitor via a REST API. I chose Iron for that, following this tutorial . Monitoring means getting its progress and its final result.
When I spawn it, I give it an id and map that id to a resource that I can GET to get the progress. I don't have to be exact with the progress; I can report the progress from 5 seconds ago.
My first attempt was to have a channel via which I send request for progress and receive the status. I got stuck where to store the receiver, as in my understanding it belongs to one thread only. I wanted to put it in the context of the request, but that won't work as there are different threads handling subsequent requests.
What would be the idiomatic way to do this in Rust?
I have a sample project.
Later edit:
Here is a self contained example which follows the sample principle as the answer, namely a map where each thread updates its progress:
extern crate iron;
extern crate router;
extern crate rustc_serialize;
use iron::prelude::*;
use iron::status;
use router::Router;
use rustc_serialize::json;
use std::io::Read;
use std::sync::{Mutex, Arc};
use std::thread;
use std::time::Duration;
use std::collections::HashMap;
#[derive(Debug, Clone, RustcEncodable, RustcDecodable)]
pub struct Status {
pub progress: u64,
pub context: String
}
#[derive(RustcEncodable, RustcDecodable)]
struct StartTask {
id: u64
}
fn start_process(status: Arc<Mutex<HashMap<u64, Status>>>, task_id: u64) {
let c = status.clone();
thread::spawn(move || {
for i in 1..100 {
{
let m = &mut c.lock().unwrap();
m.insert(task_id, Status{ progress: i, context: "in progress".to_string()});
}
thread::sleep(Duration::from_secs(1));
}
let m = &mut c.lock().unwrap();
m.insert(task_id, Status{ progress: 100, context: "done".to_string()});
});
}
fn main() {
let status: Arc<Mutex<HashMap<u64, Status>>> = Arc::new(Mutex::new(HashMap::new()));
let status_clone: Arc<Mutex<HashMap<u64, Status>>> = status.clone();
let mut router = Router::new();
router.get("/:taskId", move |r: &mut Request| task_status(r, &status.lock().unwrap()));
router.post("/start", move |r: &mut Request|
start_task(r, status_clone.clone()));
fn task_status(req: &mut Request, statuses: & HashMap<u64,Status>) -> IronResult<Response> {
let ref task_id = req.extensions.get::<Router>().unwrap().find("taskId").unwrap_or("/").parse::<u64>().unwrap();
let payload = json::encode(&statuses.get(&task_id)).unwrap();
Ok(Response::with((status::Ok, payload)))
}
// Receive a message by POST and play it back.
fn start_task(request: &mut Request, statuses: Arc<Mutex<HashMap<u64, Status>>>) -> IronResult<Response> {
let mut payload = String::new();
request.body.read_to_string(&mut payload).unwrap();
let task_start_request: StartTask = json::decode(&payload).unwrap();
start_process(statuses, task_start_request.id);
Ok(Response::with((status::Ok, json::encode(&task_start_request).unwrap())))
}
Iron::new(router).http("localhost:3000").unwrap();
}
One possibility is to use a global HashMap that associate each worker id with the progress (and result). Here is simple example (without the rest stuff)
#[macro_use]
extern crate lazy_static;
use std::sync::Mutex;
use std::collections::HashMap;
use std::thread;
use std::time::Duration;
lazy_static! {
static ref PROGRESS: Mutex<HashMap<usize, usize>> = Mutex::new(HashMap::new());
}
fn set_progress(id: usize, progress: usize) {
// insert replaces the old value if there was one.
PROGRESS.lock().unwrap().insert(id, progress);
}
fn get_progress(id: usize) -> Option<usize> {
PROGRESS.lock().unwrap().get(&id).cloned()
}
fn work(id: usize) {
println!("Creating {}", id);
set_progress(id, 0);
for i in 0..100 {
set_progress(id, i + 1);
// simulates work
thread::sleep(Duration::new(0, 50_000_000));
}
}
fn monitor(id: usize) {
loop {
if let Some(p) = get_progress(id) {
if p == 100 {
println!("Done {}", id);
// to avoid leaks, remove id from PROGRESS.
// maybe save that the task ends in a data base.
return
} else {
println!("Progress {}: {}", id, p);
}
}
thread::sleep(Duration::new(1, 0));
}
}
fn main() {
let w = thread::spawn(|| work(1));
let m = thread::spawn(|| monitor(1));
w.join().unwrap();
m.join().unwrap();
}
You need to register one channel per request thread, because if cloning Receivers were possible the responses might/will end up with the wrong thread if two request are running at the same time.
Instead of having your thread create a channel for answering requests, use a future. A future allows you to have a handle to an object, where the object doesn't exist yet. You can change the input channel to receive a Promise, which you then fulfill, no output channel necessary.
This question already has an answer here:
Why is it possible to implement Read on an immutable reference to File?
(1 answer)
Closed 6 years ago.
I am writing a simple TCP-based echo server. When I tried to use BufReader and BufWriter to read from and write to a TcpStream, I found that passing a TcpStream to BufReader::new() by value moves its ownership so that I couldn't pass it to a BufWriter. Then, I found an answer in this thread that solves the problem:
fn handle_client(stream: TcpStream) {
let mut reader = BufReader::new(&stream);
let mut writer = BufWriter::new(&stream);
// Receive a message
let mut message = String::new();
reader.read_line(&mut message).unwrap();
// ingored
}
This is simple and it works. However, I can not quite understand why this code works. Why can I just pass an immutable reference to BufReader::new(), instead of a mutable reference ?
The whole program can be found here.
More Details
In the above code, I used reader.read_line(&mut message). So I opened the source code of BufRead in Rust standard library and saw this:
fn read_line(&mut self, buf: &mut String) -> Result<usize> {
// ignored
append_to_string(buf, |b| read_until(self, b'\n', b))
}
Here we can see that it passes the self (which may be a &mut BufReader in my case) to read_until(). Next, I found the following code in the same file:
fn read_until<R: BufRead + ?Sized>(r: &mut R, delim: u8, buf: &mut Vec<u8>)
-> Result<usize> {
let mut read = 0;
loop {
let (done, used) = {
let available = match r.fill_buf() {
Ok(n) => n,
Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
Err(e) => return Err(e)
};
match memchr::memchr(delim, available) {
Some(i) => {
buf.extend_from_slice(&available[..i + 1]);
(true, i + 1)
}
None => {
buf.extend_from_slice(available);
(false, available.len())
}
}
};
r.consume(used);
read += used;
if done || used == 0 {
return Ok(read);
}
}
}
In this part, there are two places using the BufReader: r.fill_buf() and r.consume(used). I thought r.fill_buf() is what I want to see. Therefore, I went to the code of BufReader in Rust standard library and found this:
fn fill_buf(&mut self) -> io::Result<&[u8]> {
// ignored
if self.pos == self.cap {
self.cap = try!(self.inner.read(&mut self.buf));
self.pos = 0;
}
Ok(&self.buf[self.pos..self.cap])
}
It seems like it uses self.inner.read(&mut self.buf) to read the data from self.inner. Then, we take a look at the structure of BufReader and the BufReader::new():
pub struct BufReader<R> {
inner: R,
buf: Vec<u8>,
pos: usize,
cap: usize,
}
// ignored
impl<R: Read> BufReader<R> {
// ignored
#[stable(feature = "rust1", since = "1.0.0")]
pub fn new(inner: R) -> BufReader<R> {
BufReader::with_capacity(DEFAULT_BUF_SIZE, inner)
}
// ignored
#[stable(feature = "rust1", since = "1.0.0")]
pub fn with_capacity(cap: usize, inner: R) -> BufReader<R> {
BufReader {
inner: inner,
buf: vec![0; cap],
pos: 0,
cap: 0,
}
}
// ignored
}
From the above code, we can know that inner is a type which implements Read. In my case, the inner may be a &TcpStream.
I knew the signature of Read.read() is:
fn read(&mut self, buf: &mut [u8]) -> Result<usize>
It requires a mutable reference here, but I only lent it an immutable reference. Is this supposed to be a problem when the program reaches self.inner.read() in fill_buf() ?
Quick anser: we pass a &TcpStream as R: Read, not TcpStream. Thus self in Read::read is &mut & TcpStream, not &mut TcpStream. Read is implement for &TcpStream as you can see in the documentation.
Look at this working code:
let stream = TcpStream::connect("...").unwrap();
let mut buf = [0; 100];
Read::read(&mut (&stream), &mut buf);
Note that stream is not even bound as mut, because we use it immutably, just having a mutable reference to the immutable one.
Next, you could ask why Read can be implemented for &TcpStream, because it's necessary to mutate something during the read operation.
This is where the nice Rust-world 🌈 ☮ ends, and the evil C-/operating system-world starts 😈. For example, on Linux you have a simple integer as "file descriptor" for the stream. You can use this for all operations on the stream, including reading and writing. Since you pass the integer by value (it's also a Copy-type), it doesn't matter if you have a mutable or immutable reference to the integer as you can just copy it.
Therefore a minimal amount of synchronization has to be done by the operating system or by the Rust std implementation, because usually it's strange and dangerous to mutate through an immutable reference. This behavior is called "interior mutability" and you can read a little bit more about it...
in the cell documentation
in the book 📖
I have an EventRegistry which people can use to register event listeners. It then calls the appropriate listeners when an event is broadcast. But, when I try to multithread it, it doesn't compile. How would I get this code working?
use std::collections::HashMap;
use std::thread;
struct EventRegistry<'a> {
event_listeners: HashMap<&'a str, Vec<Box<Fn() + Sync>>>
}
impl<'a> EventRegistry<'a> {
fn new() -> EventRegistry<'a> {
EventRegistry {
event_listeners: HashMap::new()
}
}
fn add_event_listener(&mut self, event: &'a str, listener: Box<Fn() + Sync>) {
match self.event_listeners.get_mut(event) {
Some(listeners) => {
listeners.push(listener);
return
},
None => {}
};
let mut listeners = Vec::with_capacity(1);
listeners.push(listener);
self.event_listeners.insert(event, listeners);
}
fn broadcast_event(&mut self, event: &str) {
match self.event_listeners.get(event) {
Some(listeners) => {
for listener in listeners.iter() {
let _ = thread::spawn(|| {
listener();
});
}
}
None => {}
}
}
}
fn main() {
let mut main_registry = EventRegistry::new();
main_registry.add_event_listener("player_move", Box::new(|| {
println!("Hey, look, the player moved!");
}));
main_registry.broadcast_event("player_move");
}
Playpen (not sure if it's minimal, but it produces the error)
If I use thread::scoped, it works too, but that's unstable, and I think it only works because it immediately joins back to the main thread.
Updated question
I meant "call them in their own thread"
The easiest thing to do is avoid the Fn* traits, if possible. If you know that you are only using full functions, then it's straightforward:
use std::thread;
fn a() { println!("a"); }
fn b() { println!("b"); }
fn main() {
let fns = vec![a as fn(), b as fn()];
for &f in &fns {
thread::spawn(move || f());
}
thread::sleep_ms(500);
}
If you can't use that for some reason (like you want to accept closures), then you will need to be a bit more explicit and use Arc:
use std::thread;
use std::sync::Arc;
fn a() { println!("a"); }
fn b() { println!("b"); }
fn main() {
let fns = vec![
Arc::new(Box::new(a) as Box<Fn() + Send + Sync>),
Arc::new(Box::new(b) as Box<Fn() + Send + Sync>),
];
for f in &fns {
let my_f = f.clone();
thread::spawn(move || my_f());
}
thread::sleep_ms(500);
}
Here, we can create a reference-counted trait object. We can clone the trait object (increasing the reference count) each time we spawn a new thread. Each thread gets its own reference to the trait object.
If I use thread::scoped, it works too
thread::scoped is pretty awesome; it's really unfortunate that it needed to be marked unstable due to some complex interactions that weren't the best.
One of the benefits of a scoped thread is that the thread is guaranteed to end by a specific time: when the JoinGuard is dropped. That means that scoped threads are allowed to contain non-'static references, so long as those references last longer than the thread!
A spawned thread has no such guarantees about how long they live; these threads may live "forever". Any references they take must also live "forever", thus the 'static restriction.
This serves to explain your original problem. You have a vector with a non-'static lifetime, but you are handing references that point into that vector to the thread. If the vector were to be deallocated before the thread exited, you could attempt to access undefined memory, which leads to crashes in C or C++ programs. This is Rust helping you out!
Original question
Call functions in vector without consuming them
The answer is that you just call them:
fn a() { println!("a"); }
fn b() { println!("b"); }
fn main() {
let fns = vec![Box::new(a) as Box<Fn()>, Box::new(b) as Box<Fn()>];
fns[0]();
fns[1]();
fns[0]();
fns[1]();
}
Playpen