I'm trying to do a simple extension to the comments example by creating a REST API and committing the post to the database. I'm creating the connection outside the scope of the handler itself which I'm assuming is where my problem lies. I'm just not sure how to fix it.
This is the code for the post handler:
server.get("/comments", middleware! {
let mut stmt = conn.prepare("SELECT * FROM comment").unwrap();
let mut iter = stmt.query_map(&[], |row| {
Comment { id: row.get(0), author: row.get(1), text: row.get(2) }
}).unwrap();
let mut out: Vec<Comment> = Vec::new();
for comment in iter {
out.push(comment.unwrap());
}
json::encode(&out).unwrap()
});
This is the error I get:
<nickel macros>:22:50: 22:66 error: the trait `core::marker::Sync` is not implemented for the type `core::cell::UnsafeCell<rusqlite::InnerConnection>` [E0277]
I assume the error is because I have created the instance and then tried to use it in a closure and that variable is probably destroyed once my main function completes.
Here's an MCVE that reproduces the problem (you should provide these when asking questions):
extern crate rusqlite;
#[macro_use]
extern crate nickel;
use nickel::{Nickel, HttpRouter};
use rusqlite::Connection;
fn main() {
let mut server = Nickel::new();
let conn = Connection::open_in_memory().unwrap();
server.get("/comments", middleware! {
let _stmt = conn.prepare("SELECT * FROM comment").unwrap();
""
});
server.listen("127.0.0.1:6767");
}
The Sync trait says:
Types that are not Sync are those that have "interior mutability" in a non-thread-safe way, such as Cell and RefCell
Which matches with the error message you get. Something inside the Connection has interior mutability which means that the compiler cannot automatically guarantee that sharing it across threads is safe. I had a recent question that might be useful to the implementor of Connection, if they can guarantee it's safe to share (perhaps SQLite itself makes guarantees).
The simplest thing you can do is to ensure that only one thread has access to the database object at a time:
use std::sync::Mutex;
fn main() {
let mut server = Nickel::new();
let conn = Mutex::new(Connection::open_in_memory().unwrap());
server.get("/comments", middleware! {
let conn = conn.lock().unwrap();
let _stmt = conn.prepare("SELECT * FROM comment").unwrap();
""
});
server.listen("127.0.0.1:6767");
}
Related
I've been trying to move from postgres to tokio_postgres but struggle with some async.
use scraper::Html;
use std::sync::Arc;
use tokio::sync::Mutex;
use tokio::task;
struct Url {}
impl Url {
fn scrapped_home(&self, symbol: String) -> Html {
let url = format!(
"https://finance.yahoo.com/quote/{}?p={}&.tsrc=fin-srch", symbol, symbol
);
let response = reqwest::blocking::get(url).unwrap().text().unwrap();
scraper::Html::parse_document(&response)
}
}
#[derive(Clone)]
struct StockData {
symbol: String,
}
#[tokio::main]
async fn main() {
let stock_data = StockData { symbol: "".to_string() };
let url = Url {};
let mut uri_test: Arc<Mutex<Html>> = Arc::new(Mutex::from(url.scrapped_home(stock_data.clone().symbol)));
let mut uri_test_closure = Arc::clone(&uri_test);
let uri = task::spawn_blocking(|| {
uri_test_closure.lock()
});
}
Without putting a mutex on
url.scrapped_home(stock_data.clone().symbol)),
I would get the error that a runtime cannot drop in a context where blocking is not allowed, so I put in inside spawn_blocking. Then I get the error that Cell cannot be shared between threads safely. This, from what I could gather, is because Cell isn'it Sync. I then wrapped in within a Mutex. This on the other hand throws Cell cannot be shared between threads safely'.
Now, is that because it contains a reference to a Cell and therefore isn't memory-safe? If so, would I need to implement Sync for Html? And how?
Html is from the scraper crate.
UPDATE:
Sorry, here's the error.
error: future cannot be sent between threads safely
--> src/database/queries.rs:141:40
|
141 | let uri = task::spawn_blocking(|| {
| ________________________________________^
142 | | uri_test_closure.lock()
143 | | });
| |_________^ future is not `Send`
|
= help: within `tendril::tendril::NonAtomic`, the trait `Sync` is not implemented for `Cell<usize>`
note: required by a bound in `spawn_blocking`
--> /home/a/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/task/blocking.rs:195:12
|
195 | R: Send + 'static,
| ^^^^ required by this bound in `spawn_blocking`
UPDATE:
Adding Cargo.toml as requested:
[package]
name = "reprod"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
reqwest = { version = "0.11", features = ["json", "blocking"] }
tokio = { version = "1", features = ["full"] }
tokio-postgres = "0"
scraper = "0.12.0"
UPDATE: Added original sync code:
fn main() {
let stock_data = StockData { symbol: "".to_string() };
let url = Url {};
url.scrapped_home(stock_data.clone().symbol);
}
UPDATE: Thanks to Kevin I was able to get it to work. As he pointed out Html was neither Send nor Sync. This part of the Rust lang doc helped me to understand how message passing works.
pub fn scrapped_home(&self, symbol: String) -> Html {
let (tx, rx) = mpsc::channel();
let url = format!(
"https://finance.yahoo.com/quote/{}?p={}&.tsrc=fin-srch", symbol, symbol
);
thread::spawn(move || {
let val = reqwest::blocking::get(url).unwrap().text().unwrap();
tx.send(val).unwrap();
});
scraper::Html::parse_document(&rx.recv().unwrap())
}
Afterwards I had some sort of epiphany and got it to work with tokio, without message passing, as well
pub async fn scrapped_home(&self, symbol: String) -> Html {
let url = format!(
"https://finance.yahoo.com/quote/{}?p={}&.tsrc=fin-srch", symbol, symbol
);
let response = task::spawn_blocking(move || {
reqwest::blocking::get(url).unwrap().text().unwrap()
}).await.unwrap();
scraper::Html::parse_document(&response)
}
I hope that this might help someone.
This illustrates it a bit more clearly now: you're trying to return a tokio::sync::MutexGuard across a thread boundary. When you call this:
let mut uri_test: Arc<Mutex<Html>> = Arc::new(Mutex::from(url.scrapped_home(stock_data.clone().symbol)));
let mut uri_test_closure = Arc::clone(&uri_test);
let uri = task::spawn_blocking(|| {
uri_test_closure.lock()
});
The uri_test_closure.lock() call (tokio::sync::Mutex::lock()) doesn't have a semicolon, which means it's returning the object that's the result of the call. But you can't return a MutexGuard across a thread boundary.
I suggest you read up on the linked lock() call, as well as blocking_lock() and such there.
I'm not certain of the point of your call to task::spawn_blocking here. If you're trying to illustrate a use case for something, that's not coming across.
Edit:
The problem is deeper. Html is both !Send and !Sync which means you can't even wrap it up in an Arc<Mutex<Html>> or Arc<Mutex<Optional<Html>>> or whatever. You need to get the data from another thread in another way, and not as that "whole" object. See this post on the rust user forum for more detailed information. But whatever you're wrapping must be Send and that struct is explicitly not.
So if a type is Send and !Sync, you can wrap in a Mutex and an Arc. But if it's !Send, you're hooped, and need to use message passing, or other synchronization mechanisms.
I am trying to declare and read/write an instance of a custom struct, using lazy_static as I had to use non-const function at its initialization (string).
As I saw here in an other Stackoverflow post, I tried to use RwLock, which works fine when it comes to write, but fails when it comes to read, with the following error:
thread 'main' panicked at 'rwlock read lock would result in deadlock', /Users/adrien/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:47:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
pub struct Authentication {
access_token: String,
refresh_token: String,
expiration: u32
}
lazy_static! {
static ref LATEST_AUTH: RwLock<Authentication> = RwLock::new(Authentication {
access_token: "access".to_string(),
refresh_token: "refresh".to_string(),
expiration: 0
});
}
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap(); //From a reqwest res
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
println!("LATEST_AUTH:{}", LATEST_AUTH.read().unwrap()); //Fails
}
A RwLock is locked for the entire scope of the lock guard, which is obtained from calls to read() or write().
In your case, the write lock guard, au, lives for the entire duration of the auth function. This is what the error is saying: you've already locked it and then trying to lock it again in the same thread will make it block forever.
Write locks can also be used for reading, so you can fix this by just re-using that instead of trying to lock it again:
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap();
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
println!("LATEST_AUTH:{}", au);
}
Alternatively, you can force the lock to be dropped sooner, so that you can lock it for reading separately:
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap();
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
std::mem::drop(au);
println!("LATEST_AUTH:{}", LATEST_AUTH.read().unwrap());
}
Or:
pub fn auth(){
let api_resp: ApiResponse = res.json().unwrap();
// a new scope
{
let mut au = LATEST_AUTH.write().unwrap();
au.access_token = api_resp.access_token.clone();
// au will be dropped here, when it goes out of scope
}
println!("LATEST_AUTH:{}", LATEST_AUTH.read().unwrap());
}
I am trying to get a value from a thread, in this case a HashMap. I reduced the code to the following (I originally tried to share a HashMap containig a Vec):
use std::thread;
use std::sync::mpsc;
use std::sync::Mutex;
use std::sync::Arc;
use std::collections::HashMap;
fn main() {
let(tx, rx) = mpsc::channel();
let n_handle= thread::spawn( || {
tx.send(worker());
});
print!("{:?}", rx.recv().unwrap().into_inner().unwrap());
}
fn worker() -> Arc<Mutex<HashMap<String, i32>>>{
let result: HashMap<String, i32> = HashMap::new();
// some computation
Arc::from(Mutex::from(result))
}
Still Rust says that:
std::sync::mpsc::Sender<std::sync::Arc<std::sync::Mutex<std::collections::HashMap<std::string::String, i32>>>> cannot be shared between threads safely
I read some confusing stuff about putting everything into Arc<Mutex<..>> which I also tried with the value:
let result: HashMap<String, Arc<Mutex<i32>>> = HashMap::new();
Can anyone point me to a document that explains the usage of the mpsc::channel with values such as HashMaps? I understand why it is not working, as the trait Sync is not implemented for the HashMap, which is required to share the stuff. Still I have no idea how to get it to work.
You can pass the values between threads with using mpsc channel.
Until you tag your thread::spawn with the move keyword like following:
thread::spawn(move || {});
Since you did not tag it with move keyword then it is not moving the outer variables into the thread scope but only sharing their references. Thus you need to implement Sync trait that every outer variable you use.
mpsc::Sender does not implement Sync that is why you get the error cannot be shared between threads.
The solution for your case would be ideal to move the sender to inside of the thread scope with move like following:
use std::collections::HashMap;
use std::sync::mpsc;
use std::sync::Arc;
use std::sync::Mutex;
use std::thread;
fn main() {
let (tx, rx) = mpsc::channel();
thread::spawn(move || {
let _ = tx.send(worker());
});
let arc = rx.recv().unwrap();
let hashmap_guard = arc.lock().unwrap();
print!(
"HashMap that retrieved from thread : {:?}",
hashmap_guard.get("Hello").unwrap()
);
}
fn worker() -> Arc<Mutex<HashMap<String, i32>>> {
let mut result: HashMap<String, i32> = HashMap::new();
result.insert("Hello".to_string(), 2);
// some computation
Arc::new(Mutex::new(result))
}
Playground
For further info: I'd recommend reading The Rust Programming Language, specifically the chapter on concurrency. In it, you are introduced to Arc: especially if you want to share your data in between threads.
I have a tokio core whose main task is running a websocket (client). When I receive some messages from the server, I want to execute a new task that will update some data. Below is a minimal failing example:
use tokio_core::reactor::{Core, Handle};
use futures::future::Future;
use futures::future;
struct Client {
handle: Handle,
data: usize,
}
impl Client {
fn update_data(&mut self) {
// spawn a new task that updates the data
self.handle.spawn(future::ok(()).and_then(|x| {
self.data += 1; // error here
future::ok(())
}));
}
}
fn main() {
let mut runtime = Core::new().unwrap();
let mut client = Client {
handle: runtime.handle(),
data: 0,
};
let task = future::ok::<(), ()>(()).and_then(|_| {
// under some conditions (omitted), we update the data
client.update_data();
future::ok::<(), ()>(())
});
runtime.run(task).unwrap();
}
Which produces this error:
error[E0477]: the type `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:13:51: 16:10 self:&mut &mut Client]>` does not fulfill the required lifetime
--> src/main.rs:13:21
|
13 | self.handle.spawn(future::ok(()).and_then(|x| {
| ^^^^^
|
= note: type must satisfy the static lifetime
The problem is that new tasks that are spawned through a handle need to be static. The same issue is described here. Sadly it is unclear to me how I can fix the issue. Even some attempts with and Arc and a Mutex (which really shouldn't be needed for a single-threaded application), I was unsuccessful.
Since developments occur rather quickly in the tokio landscape, I am wondering what the current best solution is. Do you have any suggestions?
edit
The solution by Peter Hall works for the example above. Sadly when I built the failing example I changed tokio reactor, thinking they would be similar. Using tokio::runtime::current_thread
use futures::future;
use futures::future::Future;
use futures::stream::Stream;
use std::cell::Cell;
use std::rc::Rc;
use tokio::runtime::current_thread::{Builder, Handle};
struct Client {
handle: Handle,
data: Rc<Cell<usize>>,
}
impl Client {
fn update_data(&mut self) {
// spawn a new task that updates the data
let mut data = Rc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
data.set(data.get() + 1);
future::ok(())
}));
}
}
fn main() {
// let mut runtime = Core::new().unwrap();
let mut runtime = Builder::new().build().unwrap();
let mut client = Client {
handle: runtime.handle(),
data: Rc::new(Cell::new(1)),
};
let task = future::ok::<(), ()>(()).and_then(|_| {
// under some conditions (omitted), we update the data
client.update_data();
future::ok::<(), ()>(())
});
runtime.block_on(task).unwrap();
}
I obtain:
error[E0277]: `std::rc::Rc<std::cell::Cell<usize>>` cannot be sent between threads safely
--> src/main.rs:17:21
|
17 | self.handle.spawn(future::ok(()).and_then(move |_x| {
| ^^^^^ `std::rc::Rc<std::cell::Cell<usize>>` cannot be sent between threads safely
|
= help: within `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<std::cell::Cell<usize>>`
= note: required because it appears within the type `[closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]`
= note: required because it appears within the type `futures::future::chain::Chain<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`
= note: required because it appears within the type `futures::future::and_then::AndThen<futures::future::result_::FutureResult<(), ()>, futures::future::result_::FutureResult<(), ()>, [closure#src/main.rs:17:51: 20:10 data:std::rc::Rc<std::cell::Cell<usize>>]>`
So it does seem like in this case I need an Arc and a Mutex even though the entire code is single-threaded?
In a single-threaded program, you don't need to use Arc; Rc is sufficient:
use std::{rc::Rc, cell::Cell};
struct Client {
handle: Handle,
data: Rc<Cell<usize>>,
}
impl Client {
fn update_data(&mut self) {
let data = Rc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
data.set(data.get() + 1);
future::ok(())
}));
}
}
The point is that you no longer have to worry about the lifetime because each clone of the Rc acts as if it owns the data, rather than accessing it via a reference to self. The inner Cell (or RefCell for non-Copy types) is needed because the Rc can't be dereferenced mutably, since it has been cloned.
The spawn method of tokio::runtime::current_thread::Handle requires that the future is Send, which is what is causing the problem in the update to your question. There is an explanation (of sorts) for why this is the case in this Tokio Github issue.
You can use tokio::runtime::current_thread::spawn instead of the method of Handle, which will always run the future in the current thread, and does not require that the future is Send. You can replace self.handle.spawn in the code above and it will work just fine.
If you need to use the method on Handle then you will also need to resort to Arc and Mutex (or RwLock) in order to satisfy the Send requirement:
use std::sync::{Mutex, Arc};
struct Client {
handle: Handle,
data: Arc<Mutex<usize>>,
}
impl Client {
fn update_data(&mut self) {
let data = Arc::clone(&self.data);
self.handle.spawn(future::ok(()).and_then(move |_x| {
*data.lock().unwrap() += 1;
future::ok(())
}));
}
}
If your data is really a usize, you could also use AtomicUsize instead of Mutex<usize>, but I personally find it just as unwieldy to work with.
I'm trying to share a mutable object between threads in Rust using Arc, but I get this error:
error[E0596]: cannot borrow data in a `&` reference as mutable
--> src/main.rs:11:13
|
11 | shared_stats_clone.add_stats();
| ^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
This is the sample code:
use std::{sync::Arc, thread};
fn main() {
let total_stats = Stats::new();
let shared_stats = Arc::new(total_stats);
let threads = 5;
for _ in 0..threads {
let mut shared_stats_clone = shared_stats.clone();
thread::spawn(move || {
shared_stats_clone.add_stats();
});
}
}
struct Stats {
hello: u32,
}
impl Stats {
pub fn new() -> Stats {
Stats { hello: 0 }
}
pub fn add_stats(&mut self) {
self.hello += 1;
}
}
What can I do?
Arc's documentation says:
Shared references in Rust disallow mutation by default, and Arc is no exception: you cannot generally obtain a mutable reference to something inside an Arc. If you need to mutate through an Arc, use Mutex, RwLock, or one of the Atomic types.
You will likely want a Mutex combined with an Arc:
use std::{
sync::{Arc, Mutex},
thread,
};
struct Stats;
impl Stats {
fn add_stats(&mut self, _other: &Stats) {}
}
fn main() {
let shared_stats = Arc::new(Mutex::new(Stats));
let threads = 5;
for _ in 0..threads {
let my_stats = shared_stats.clone();
thread::spawn(move || {
let mut shared = my_stats.lock().unwrap();
shared.add_stats(&Stats);
});
// Note: Immediately joining, no multithreading happening!
// THIS WAS A LIE, see below
}
}
This is largely cribbed from the Mutex documentation.
How can I use shared_stats after the for? (I'm talking about the Stats object). It seems that the shared_stats cannot be easily converted to Stats.
As of Rust 1.15, it's possible to get the value back. See my additional answer for another solution as well.
[A comment in the example] says that there is no multithreading. Why?
Because I got confused! :-)
In the example code, the result of thread::spawn (a JoinHandle) is immediately dropped because it's not stored anywhere. When the handle is dropped, the thread is detached and may or may not ever finish. I was confusing it with JoinGuard, a old, removed API that joined when it is dropped. Sorry for the confusion!
For a bit of editorial, I suggest avoiding mutability completely:
use std::{ops::Add, thread};
#[derive(Debug)]
struct Stats(u64);
// Implement addition on our type
impl Add for Stats {
type Output = Stats;
fn add(self, other: Stats) -> Stats {
Stats(self.0 + other.0)
}
}
fn main() {
let threads = 5;
// Start threads to do computation
let threads: Vec<_> = (0..threads).map(|_| thread::spawn(|| Stats(4))).collect();
// Join all the threads, fail if any of them failed
let result: Result<Vec<_>, _> = threads.into_iter().map(|t| t.join()).collect();
let result = result.unwrap();
// Add up all the results
let sum = result.into_iter().fold(Stats(0), |i, sum| sum + i);
println!("{:?}", sum);
}
Here, we keep a reference to the JoinHandle and then wait for all the threads to finish. We then collect the results and add them all up. This is the common map-reduce pattern. Note that no thread needs any mutability, it all happens in the master thread.