I am trying to declare and read/write an instance of a custom struct, using lazy_static as I had to use non-const function at its initialization (string).
As I saw here in an other Stackoverflow post, I tried to use RwLock, which works fine when it comes to write, but fails when it comes to read, with the following error:
thread 'main' panicked at 'rwlock read lock would result in deadlock', /Users/adrien/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:47:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
pub struct Authentication {
access_token: String,
refresh_token: String,
expiration: u32
}
lazy_static! {
static ref LATEST_AUTH: RwLock<Authentication> = RwLock::new(Authentication {
access_token: "access".to_string(),
refresh_token: "refresh".to_string(),
expiration: 0
});
}
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap(); //From a reqwest res
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
println!("LATEST_AUTH:{}", LATEST_AUTH.read().unwrap()); //Fails
}
A RwLock is locked for the entire scope of the lock guard, which is obtained from calls to read() or write().
In your case, the write lock guard, au, lives for the entire duration of the auth function. This is what the error is saying: you've already locked it and then trying to lock it again in the same thread will make it block forever.
Write locks can also be used for reading, so you can fix this by just re-using that instead of trying to lock it again:
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap();
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
println!("LATEST_AUTH:{}", au);
}
Alternatively, you can force the lock to be dropped sooner, so that you can lock it for reading separately:
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap();
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
std::mem::drop(au);
println!("LATEST_AUTH:{}", LATEST_AUTH.read().unwrap());
}
Or:
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap();
// a new scope
{
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
// au will be dropped here, when it goes out of scope
}
println!("LATEST_AUTH:{}", LATEST_AUTH.read().unwrap());
}
Related
I'm trying to use a static HashMap<String, Object> to store some data that I want to use and modify globally in the future. I found out that some way of declaring such a global map is by using lazy_static and mutex in order to share data safely. However, I'm getting some ownership troubles when I want to return these objects as reference, as I do in the code above:
use std::error::Error;
use std::collections::HashMap;
use std::sync::Mutex;
use super::domain::{Session, SessionRepository}; // object and trait declaration
lazy_static! {
static ref REPOSITORY: Mutex<HashMap<String, Session>> = {
let mut repo = HashMap::new();
Mutex::new(repo)
};
}
impl SessionRepository for REPOSITORY {
fn find(cookie: &str) -> Result<&mut Session, Box<dyn Error>> {
let mut repo = REPOSITORY.lock()?;
if let Some(sess) = repo.get_mut(cookie) {
return Ok(sess);
}
Err("Not found".into())
}
}
So the question is: Is there any way of doing this correctly? Does it exists any design pattern in Rust I could use in order to reach this behavior?
Thank you very much!
What you're trying to do is correctly rejected by the compiler, because if you could return a reference, then bad things can happen. Since after the function returns, the mutex is no longer locked,
Some other thread could lock the mutex and get its own mutable reference to the session, violating the rule that mutable references provide exclusive access.
Some other thread could lock the mutex and remove the session from the HashMap — now you're accessing deallocated memory.
Solution: each session should be in its own Arc<Mutex<_>>. That is:
lazy_static! {
static ref REPOSITORY: Mutex<HashMap<String, Arc<Mutex<Session>>>> = {
...
impl SessionRepository for REPOSITORY {
fn find(cookie: &str) -> Result<Arc<Mutex<Session>>, Box<dyn Error>> {
let mut repo = REPOSITORY.lock()?;
if let Some(sess) = repo.get(cookie) {
return Ok(Arc::clone(sess));
}
...
}
The Arc allows the session to be kept alive both by the repository and whichever threads have called find() on it and thus gotten their own clone of the Arc reference-counted pointer. The Mutex allows each session to be independently mutated.
I have a tokio core whose main task is running a websocket (client). When I receive some messages from the server, I want to execute a new task that will update some data. Below is a minimal failing example:
use tokio_core::reactor::{Core, Handle};
use futures::future::Future;
use futures::future;
struct Client {
handle: Handle,
data: usize,
}
impl Client {
fn update_data(&mut self) {
// spawn a new task that updates the data
self.handle.spawn(future::ok(()).and_then(|x| {
self.data += 1; // error here
future::ok(())
}));
}
}
fn main() {
let mut runtime = Core::new().unwrap();
let mut client = Client {
handle: runtime.handle(),
data: 0,
};
let task = future::ok::<(), ()>(()).and_then(|_| {
// under some conditions (omitted), we update the data
client.update_data();
future::ok::<(), ()>(())
});
runtime.run(task).unwrap();
}
Which produces this error:
error[E0477]: the type `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:13:51: 16:10 self:&mut &mut Client]>` does not fulfill the required lifetime
--> src/main.rs:13:21
|
13 | self.handle.spawn(future::ok(()).and_then(|x| {
| ^^^^^
|
= note: type must satisfy the static lifetime
The problem is that new tasks that are spawned through a handle need to be static. The same issue is described here. Sadly it is unclear to me how I can fix the issue. Even some attempts with and Arc and a Mutex (which really shouldn't be needed for a single-threaded application), I was unsuccessful.
Since developments occur rather quickly in the tokio landscape, I am wondering what the current best solution is. Do you have any suggestions?
edit
The solution by Peter Hall works for the example above. Sadly when I built the failing example I changed tokio reactor, thinking they would be similar. Using tokio::runtime::current_thread
use futures::future;
use futures::future::Future;
use futures::stream::Stream;
use std::cell::Cell;
use std::rc::Rc;
use tokio::runtime::current_thread::{Builder, Handle};
struct Client {
handle: Handle,
data: Rc<Cell<usize>>,
}
impl Client {
fn update_data(&mut self) {
// spawn a new task that updates the data
let mut data = Rc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
data.set(data.get() + 1);
future::ok(())
}));
}
}
fn main() {
// let mut runtime = Core::new().unwrap();
let mut runtime = Builder::new().build().unwrap();
let mut client = Client {
handle: runtime.handle(),
data: Rc::new(Cell::new(1)),
};
let task = future::ok::<(), ()>(()).and_then(|_| {
// under some conditions (omitted), we update the data
client.update_data();
future::ok::<(), ()>(())
});
runtime.block_on(task).unwrap();
}
I obtain:
error[E0277]: `std::rc::Rc<std::cell::Cell<usize>>` cannot be sent between threads safely
--> src/main.rs:17:21
|
17 | self.handle.spawn(future::ok(()).and_then(move |_x| {
| ^^^^^ `std::rc::Rc<std::cell::Cell<usize>>` cannot be sent between threads safely
|
= help: within `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<std::cell::Cell<usize>>`
= note: required because it appears within the type `[closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]`
= note: required because it appears within the type `futures::future::chain::Chain<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`
= note: required because it appears within the type `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`
So it does seem like in this case I need an Arc and a Mutex even though the entire code is single-threaded?
In a single-threaded program, you don't need to use Arc; Rc is sufficient:
use std::{rc::Rc, cell::Cell};
struct Client {
handle: Handle,
data: Rc<Cell<usize>>,
}
impl Client {
fn update_data(&mut self) {
let data = Rc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
data.set(data.get() + 1);
future::ok(())
}));
}
}
The point is that you no longer have to worry about the lifetime because each clone of the Rc acts as if it owns the data, rather than accessing it via a reference to self. The inner Cell (or RefCell for non-Copy types) is needed because the Rc can't be dereferenced mutably, since it has been cloned.
The spawn method of tokio::runtime::current_thread::Handle requires that the future is Send, which is what is causing the problem in the update to your question. There is an explanation (of sorts) for why this is the case in this Tokio Github issue.
You can use tokio::runtime::current_thread::spawn instead of the method of Handle, which will always run the future in the current thread, and does not require that the future is Send. You can replace self.handle.spawn in the code above and it will work just fine.
If you need to use the method on Handle then you will also need to resort to Arc and Mutex (or RwLock) in order to satisfy the Send requirement:
use std::sync::{Mutex, Arc};
struct Client {
handle: Handle,
data: Arc<Mutex<usize>>,
}
impl Client {
fn update_data(&mut self) {
let data = Arc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
*data.lock().unwrap() += 1;
future::ok(())
}));
}
}
If your data is really a usize, you could also use AtomicUsize instead of Mutex<usize>, but I personally find it just as unwieldy to work with.
I do not know Rust well enough to understand lifetimes and closures yet...
Trying to collect the downloaded data into a vector using tokio-curl:
extern crate curl;
extern crate futures;
extern crate tokio_core;
extern crate tokio_curl;
use std::io::{self, Write};
use std::str;
use curl::easy::Easy;
use tokio_core::reactor::Core;
use tokio_curl::Session;
fn main() {
// Create an event loop that we'll run on, as well as an HTTP `Session`
// which we'll be routing all requests through.
let mut lp = Core::new().unwrap();
let mut out = Vec::new();
let session = Session::new(lp.handle());
// Prepare the HTTP request to be sent.
let mut req = Easy::new();
req.get(true).unwrap();
req.url("https://www.rust-lang.org").unwrap();
req.write_function(|data| {
out.extend_from_slice(data);
io::stdout().write_all(data).unwrap();
Ok(data.len())
})
.unwrap();
// Once we've got our session, issue an HTTP request to download the
// rust-lang home page
let request = session.perform(req);
// Execute the request, and print the response code as well as the error
// that happened (if any).
let mut req = lp.run(request).unwrap();
println!("{:?}", req.response_code());
println!("out: {}", str::from_utf8(&out).unwrap());
}
Produces an error:
error[E0373]: closure may outlive the current function, but it borrows `out`, which is owned by the current function
--> src/main.rs:25:24
|
25 | req.write_function(|data| {
| ^^^^^^ may outlive borrowed value `out`
26 | out.extend_from_slice(data);
| --- `out` is borrowed here
|
help: to force the closure to take ownership of `out` (and any other referenced variables), use the `move` keyword, as shown:
| req.write_function(move |data| {
Investigating further, I see that Easy::write_function requires the 'static lifetime, but the example of how to collect output from the curl-rust docs uses Transfer::write_function instead:
use curl::easy::Easy;
let mut data = Vec::new();
let mut handle = Easy::new();
handle.url("https://www.rust-lang.org/").unwrap();
{
let mut transfer = handle.transfer();
transfer.write_function(|new_data| {
data.extend_from_slice(new_data);
Ok(new_data.len())
}).unwrap();
transfer.perform().unwrap();
}
println!("{:?}", data);
The Transfer::write_function does not require the 'static lifetime:
impl<'easy, 'data> Transfer<'easy, 'data> {
/// Same as `Easy::write_function`, just takes a non `'static` lifetime
/// corresponding to the lifetime of this transfer.
pub fn write_function<F>(&mut self, f: F) -> Result<(), Error>
where F: FnMut(&[u8]) -> Result<usize, WriteError> + 'data
{
...
But I can't use a Transfer instance on tokio-curl's Session::perform because it requires the Easy type:
pub fn perform(&self, handle: Easy) -> Perform {
transfer.easy is a private field that is directly passed to session.perform.
It this an issue with tokio-curl? Maybe it should mark the transfer.easy field as public or implement new function like perform_transfer? Is there another way to collect output using tokio-curl per transfer?
The first thing you have to understand when using the futures library is that you don't have any control over what thread the code is going to run on.
In addition, the documentation for curl's Easy::write_function says:
Note that the lifetime bound on this function is 'static, but that is often too restrictive. To use stack data consider calling the transfer method and then using write_function to configure a callback that can reference stack-local data.
The most straight-forward solution is to use some type of locking primitive to ensure that only one thread at a time may have access to the vector. You also have to share ownership of the vector between the main thread and the closure:
use std::sync::Mutex;
use std::sync::Arc;
let out = Arc::new(Mutex::new(Vec::new()));
let out_closure = out.clone();
// ...
req.write_function(move |data| {
let mut out = out_closure.lock().expect("Unable to lock output");
// ...
}).expect("Cannot set writing function");
// ...
let out = out.lock().expect("Unable to lock output");
println!("out: {}", str::from_utf8(&out).expect("Data was not UTF-8"));
Unfortunately, the tokio-curl library does not currently support using the Transfer type that would allow for stack-based data.
I'm trying to do a simple extension to the comments example by creating a REST API and committing the post to the database. I'm creating the connection outside the scope of the handler itself which I'm assuming is where my problem lies. I'm just not sure how to fix it.
This is the code for the post handler:
server.get("/comments", middleware! {
let mut stmt = conn.prepare("SELECT * FROM comment").unwrap();
let mut iter = stmt.query_map(&[], |row| {
Comment { id: row.get(0), author: row.get(1), text: row.get(2) }
}).unwrap();
let mut out: Vec<Comment> = Vec::new();
for comment in iter {
out.push(comment.unwrap());
}
json::encode(&out).unwrap()
});
This is the error I get:
<nickel macros>:22:50: 22:66 error: the trait `core::marker::Sync` is not implemented for the type `core::cell::UnsafeCell<rusqlite::InnerConnection>` [E0277]
I assume the error is because I have created the instance and then tried to use it in a closure and that variable is probably destroyed once my main function completes.
Here's an MCVE that reproduces the problem (you should provide these when asking questions):
extern crate rusqlite;
#[macro_use]
extern crate nickel;
use nickel::{Nickel, HttpRouter};
use rusqlite::Connection;
fn main() {
let mut server = Nickel::new();
let conn = Connection::open_in_memory().unwrap();
server.get("/comments", middleware! {
let _stmt = conn.prepare("SELECT * FROM comment").unwrap();
""
});
server.listen("127.0.0.1:6767");
}
The Sync trait says:
Types that are not Sync are those that have "interior mutability" in a non-thread-safe way, such as Cell and RefCell
Which matches with the error message you get. Something inside the Connection has interior mutability which means that the compiler cannot automatically guarantee that sharing it across threads is safe. I had a recent question that might be useful to the implementor of Connection, if they can guarantee it's safe to share (perhaps SQLite itself makes guarantees).
The simplest thing you can do is to ensure that only one thread has access to the database object at a time:
use std::sync::Mutex;
fn main() {
let mut server = Nickel::new();
let conn = Mutex::new(Connection::open_in_memory().unwrap());
server.get("/comments", middleware! {
let conn = conn.lock().unwrap();
let _stmt = conn.prepare("SELECT * FROM comment").unwrap();
""
});
server.listen("127.0.0.1:6767");
}
I'm trying to share a mutable object between threads in Rust using Arc, but I get this error:
error[E0596]: cannot borrow data in a `&` reference as mutable
--> src/main.rs:11:13
|
11 | shared_stats_clone.add_stats();
| ^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
This is the sample code:
use std::{sync::Arc, thread};
fn main() {
let total_stats = Stats::new();
let shared_stats = Arc::new(total_stats);
let threads = 5;
for _ in 0..threads {
let mut shared_stats_clone = shared_stats.clone();
thread::spawn(move || {
shared_stats_clone.add_stats();
});
}
}
struct Stats {
hello: u32,
}
impl Stats {
pub fn new() -> Stats {
Stats { hello: 0 }
}
pub fn add_stats(&mut self) {
self.hello += 1;
}
}
What can I do?
Arc's documentation says:
Shared references in Rust disallow mutation by default, and Arc is no exception: you cannot generally obtain a mutable reference to something inside an Arc. If you need to mutate through an Arc, use Mutex, RwLock, or one of the Atomic types.
You will likely want a Mutex combined with an Arc:
use std::{
sync::{Arc, Mutex},
thread,
};
struct Stats;
impl Stats {
fn add_stats(&mut self, _other: &Stats) {}
}
fn main() {
let shared_stats = Arc::new(Mutex::new(Stats));
let threads = 5;
for _ in 0..threads {
let my_stats = shared_stats.clone();
thread::spawn(move || {
let mut shared = my_stats.lock().unwrap();
shared.add_stats(&Stats);
});
// Note: Immediately joining, no multithreading happening!
// THIS WAS A LIE, see below
}
}
This is largely cribbed from the Mutex documentation.
How can I use shared_stats after the for? (I'm talking about the Stats object). It seems that the shared_stats cannot be easily converted to Stats.
As of Rust 1.15, it's possible to get the value back. See my additional answer for another solution as well.
[A comment in the example] says that there is no multithreading. Why?
Because I got confused! :-)
In the example code, the result of thread::spawn (a JoinHandle) is immediately dropped because it's not stored anywhere. When the handle is dropped, the thread is detached and may or may not ever finish. I was confusing it with JoinGuard, a old, removed API that joined when it is dropped. Sorry for the confusion!
For a bit of editorial, I suggest avoiding mutability completely:
use std::{ops::Add, thread};
#[derive(Debug)]
struct Stats(u64);
// Implement addition on our type
impl Add for Stats {
type Output = Stats;
fn add(self, other: Stats) -> Stats {
Stats(self.0 + other.0)
}
}
fn main() {
let threads = 5;
// Start threads to do computation
let threads: Vec<_> = (0..threads).map(|_| thread::spawn(|| Stats(4))).collect();
// Join all the threads, fail if any of them failed
let result: Result<Vec<_>, _> = threads.into_iter().map(|t| t.join()).collect();
let result = result.unwrap();
// Add up all the results
let sum = result.into_iter().fold(Stats(0), |i, sum| sum + i);
println!("{:?}", sum);
}
Here, we keep a reference to the JoinHandle and then wait for all the threads to finish. We then collect the results and add them all up. This is the common map-reduce pattern. Note that no thread needs any mutability, it all happens in the master thread.