I'm trying to use a static HashMap<String, Object> to store some data that I want to use and modify globally in the future. I found out that some way of declaring such a global map is by using lazy_static and mutex in order to share data safely. However, I'm getting some ownership troubles when I want to return these objects as reference, as I do in the code above:
use std::error::Error;
use std::collections::HashMap;
use std::sync::Mutex;
use super::domain::{Session, SessionRepository}; // object and trait declaration
lazy_static! {
static ref REPOSITORY: Mutex<HashMap<String, Session>> = {
let mut repo = HashMap::new();
Mutex::new(repo)
};
}
impl SessionRepository for REPOSITORY {
fn find(cookie: &str) -> Result<&mut Session, Box<dyn Error>> {
let mut repo = REPOSITORY.lock()?;
if let Some(sess) = repo.get_mut(cookie) {
return Ok(sess);
}
Err("Not found".into())
}
}
So the question is: Is there any way of doing this correctly? Does it exists any design pattern in Rust I could use in order to reach this behavior?
Thank you very much!
What you're trying to do is correctly rejected by the compiler, because if you could return a reference, then bad things can happen. Since after the function returns, the mutex is no longer locked,
Some other thread could lock the mutex and get its own mutable reference to the session, violating the rule that mutable references provide exclusive access.
Some other thread could lock the mutex and remove the session from the HashMap — now you're accessing deallocated memory.
Solution: each session should be in its own Arc<Mutex<_>>. That is:
lazy_static! {
static ref REPOSITORY: Mutex<HashMap<String, Arc<Mutex<Session>>>> = {
...
impl SessionRepository for REPOSITORY {
fn find(cookie: &str) -> Result<Arc<Mutex<Session>>, Box<dyn Error>> {
let mut repo = REPOSITORY.lock()?;
if let Some(sess) = repo.get(cookie) {
return Ok(Arc::clone(sess));
}
...
}
The Arc allows the session to be kept alive both by the repository and whichever threads have called find() on it and thus gotten their own clone of the Arc reference-counted pointer. The Mutex allows each session to be independently mutated.
Related
I am looking to use a member function of a struct as a mutable callback. The standard way I would do this is by wrapping my struct instance in a mutex and then in an Arc. In this use case however, my callback is a real-time audio callback which needs to be real-time safe and is triggered on a dedicated thread. This means no locking can happen that might delay the callback from occurring.
My actual member callback method must take a mutable reference to the struct (I think this is where things fall apart). The struct does contain members but these can all be atomic variables.
Here is an example of the kind of thing I'm looking for:
use std::sync::Arc;
use clap::error;
use cpal::{traits::{HostTrait, DeviceTrait}, StreamConfig, OutputCallbackInfo, StreamError};
use std::sync::atomic::AtomicI16;
pub struct AudioProcessor {
var: AtomicI16,
}
impl AudioProcessor {
fn audio_block_f32(&mut self, audio: &mut [f32], _info: &OutputCallbackInfo) {
}
fn audio_error(&mut self, error: StreamError) {
}
}
fn initAudio()
{
let out_dev = cpal::default_host().default_output_device().expect("No available output device found");
let mut supported_configs_range = out_dev.supported_output_configs().expect("Could not obtain device configs");
let config = supported_configs_range.next().expect("No available configs").with_max_sample_rate();
let proc = Arc::new(AudioProcessor{var: AtomicI16::new(5)});
let audio_callback_instance = proc.clone();
let error_callback_instance = proc.clone();
out_dev.build_output_stream(&StreamConfig::from(config), move |audio: &mut [f32], info: &OutputCallbackInfo| audio_callback_instance.audio_block_f32(audio, info), move |stream_error| error_callback_instance.clone().audio_error(stream_error));
}
Currently this code fails in the build_output_stream() function with this error message:
cannot borrow data in an Arc as mutable
trait DerefMut is required to modify through a dereference, but it is not implemented for Arc<AudioProcessor>
Are there any standard ways of dealing with problems like this?
I'm quite new to Rust programming, and I'm trying to convert a code that I had in js to Rust.
A plain concept of it is as below:
fn main() {
let mut ds=DataSource::new();
let mut pp =Processor::new(&mut ds);
}
struct DataSource {
st2r: Option<&Processor>,
}
struct Processor {
st1r: &DataSource,
}
impl DataSource {
pub fn new() -> Self {
DataSource {
st2r: None,
}
}
}
impl Processor {
pub fn new(ds: &mut DataSource) -> Self {
let pp = Processor {
st1r: ds,
};
ds.st2r = Some(&pp);
pp
}
}
As you can see I have two main modules in my system that are inter-connected to each other and I need a reference of each in another.
Well, this code would complain about lifetimes and such stuff, of course 😑. So I started throwing lifetime specifiers around like a madman and even after all that, it still complains that in "Processor::new" I can't return something that has been borrowed. Legit. But I can't find any solution around it! No matter how I try to handle the referencing of each other, it ends with this borrowing error.
So, can anyone point out a solution for this situation? Is my app's structure not valid in Rust and I should do it in another way? or there's a trick to this that my inexperienced mind can't find?
Thanks.
What you're trying to do can't be expressed with references and lifetimes because:
The DataSource must live longer than the Processor so that pp.st1r is guaranteed to be valid,
and the Processor must live longer than the DataSource so that ds.st2r is guaranteed to be valid. You might think that since ds.st2r is an Option and since the None variant doesn't contain a reference this allows a DataSource with a None value in st2r to outlive any Processors, but unfortunately the compiler can't know at compile-time whether st2r contains Some value, and therefore must assume it does.
Your problem is compounded by the fact that you need a mutable reference to the DataSource so that you can set its st2r field at a time when you also have an immutable outstanding reference inside the Processor, which Rust won't allow.
You can make your code work by switching to dynamic lifetime and mutability tracking using Rc (for dynamic lifetime tracking) and RefCell (for dynamic mutability tracking):
use std::cell::RefCell;
use std::rc::{ Rc, Weak };
fn main() {
let ds = Rc::new (RefCell::new (DataSource::new()));
let pp = Processor::new (Rc::clone (&ds));
}
struct DataSource {
st2r: Weak<Processor>,
}
struct Processor {
st1r: Rc<RefCell<DataSource>>,
}
impl DataSource {
pub fn new() -> Self {
DataSource {
st2r: Weak::new(),
}
}
}
impl Processor {
pub fn new(ds: Rc::<RefCell::<DataSource>>) -> Rc<Self> {
let pp = Rc::new (Processor {
st1r: ds,
});
pp.st1r.borrow_mut().st2r = Rc::downgrade (&pp);
pp
}
}
Playground
Note that I've replaced your Option<&Processor> with a Weak<Processor>. It would be possible to use an Option<Rc<Processor>> but this would risk leaking memory if you dropped all references to DataSource without setting st2r to None first. The Weak<Processor> behaves more or less like an Option<Rc<Processor>> that is set to None automatically when all other references are dropped, ensuring that memory will be freed properly.
I am using async-tungstenite to listen to a websocket, and async-std's StreamExt to operate on the resulting stream.
I want to use a HashMap to accumulate the latest Ticker values from the websocket. These Ticker values will be looked up later to be used in calculations. I'm using the symbol (String) value of the Ticker struct as the key for the HashMap. I'm using the .scan StreamExt method to perform the accumulation.
However, I get a compilation error related to lifetimes. Here's some stripped-down code:
let tickers = HashMap::new();
let mut stream = ws.
.scan(tickers, accumulate_tickers);
while let msg = stream.next().await {
println!("{:?}", msg)
}
...and the accumulate_tickers function:
fn accumulate_tickers(tps: &mut HashMap<String, Ticker>, bt: Ticker) -> Option<&HashMap<String, Ticker>> {
tps.insert((*bt.symbol).to_string(), bt);
Some(tps)
}
The compilation error I receive is as follows:
error[E0271]: type mismatch resolving `for<'r> <for<'s> fn(&'s mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker) -> std::option::Option<&'s std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>> {accumulate_tickers} as std::ops::FnOnce<(&'r mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker)>>::Output == std::option::Option<_>`
--> examples/async_std-ws.rs:64:4
|
64 | .scan(tickers, accumulate_tickers);
| ^^^^ expected bound lifetime parameter, found concrete lifetime
I'm unaware of a way to provide a lifetime parameter to the scan method.
I wonder whether the issue may be related to the fact that I modify the HashMap and then try to return it (is it a move issue?). How could I resolve this, or at least narrow down the cause?
I was able to get this working. Working my way through the compiler errors, I ended up with this signature for the accumulate_tickers function:
fn accumulate_tickers<'a>(tps: &'a mut &'static HashMap<String, Ticker>, bt: Ticker) -> Option<&'static HashMap<String, Ticker>>
I do want the accumulator HashMap to have a static lifetime so that makes sense. tps: &'a mut &'static HashMap... does look a bit strange, but it works.
Then, this issue was was that tickers also had to have a static lifetime (it's the initial value for the accumulator. I tried declaring it as static outside the main but it wouldn't let me set it to the result of a function - HashMap::new().
I then turned to lazy_static which allows me to create a static value which does just that:
lazy_static! {
static ref tickers: HashMap<String, Ticker> = HashMap::new();
}
This gave me a HashMap accumulator that had a static lifetime. However, like normal static values declared in the root scope, it was immutable. To fix that, I read some hints from the lazy_static team and then found https://pastebin.com/YES8dsHH. This showed me how to make my static accumulator mutable by wrapping it in Arc<Mutex<_>>.
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
This does mean that I have to retrieve the accumulator from the Mutex (and lock it) before reading or modifying it but, again, it works.
Putting it all together, the stripped-down code now looks like this:
#[macro_use]
extern crate lazy_static;
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
// SNIP
// Inside main()
let mut ticks = ws
.scan(&tickers, accumulate_tickers);
while let Some(msg) = ticks.next().await {
println!("{:?}", msg.lock().unwrap());
}
// SNIP
fn accumulate_tickers<'a>(tps: &'a mut &'static tickers, bt: Ticker) -> Option<&'static tickers> {
tps.lock().unwrap().insert((*bt.symbol).to_string(), bt);
Some(tps)
}
I'd be happy to hear suggestions for ways in which this could be made simpler or more elegant.
I am trying to access a static hashmap for reading and writing but I am always getting error:
use std::collections::HashMap;
use std::sync::Mutex;
pub struct ModuleItem {
pub absolute_path: String,
}
lazy_static! {
static ref MODULE_MAP: Mutex<HashMap<i32, ModuleItem>> = Mutex::new(HashMap::new());
}
pub fn insert(identity_hash: i32, module_item: ModuleItem) {
MODULE_MAP
.lock()
.unwrap()
.insert(identity_hash, module_item);
}
pub fn get(identity_hash: i32) -> Option<&'static ModuleItem> {
MODULE_MAP.lock().unwrap().get(&identity_hash).clone()
}
But I am getting an error on the get function cannot return value referencing temporary value
I tried with .cloned(), .clone() or even nothing but I don't manage to get it to work. Can you help me?
I tried with .cloned(), .clone() or even nothing but I don't manage to get it to work. Can you help me?
All Option::clone does is clone the underlying structure, which in this case is an &ModuleItem so it just clones the reference, and you still have a reference, which you can't return because you only have access to the hashmap's contents while you hold the lock (otherwise it could not work).
Option::cloned actually clones the object being held by reference, but doesn't compile here because ModuleItem can't be cloned.
First you have to return a Option<ModuleItem>, you can not return a reference to the map contents since the lock is going to be released at the end of the function, and you can't keep a handle on hashmap contents across mutex boundaries as they could go away at any moment (e.g. an other thread could move them, or even clear the map entirely).
Then copy the ModuleItem, either by deriving Clone on ModuleItem (then calling Option::cloned) or by creating a new ModuleItem "by hand" e.g.
pub fn get(identity_hash: i32) -> Option<ModuleItem> {
MODULE_MAP.lock().unwrap().get(&identity_hash).map(|m|
ModuleItem { absolute_path: m.absolute_path.clone() }
)
}
If you need to get keys out a lot and are worried about performances, you could always store Arc<ModuleItem>. That has something of a cost (as it's a pointer so your string is now behind two pointers) however cloning an Arc is very cheap.
To avoid the double pointer you could make ModuleItem into an unsized type and have it store an str but... that's pretty difficult to work with so I wouldn't recommend it.
The function get cannot use a static lifetime because the data does not live for the entire life of the program (from the Rust book):
As a reference lifetime 'static indicates that the data pointed to by the reference lives for the entire lifetime of the running program. It can still be coerced to a shorter lifetime.
So you have to return either a none-static reference or a copy of the value of the HashMap. Reference is not possible because MODULE_MAP.lock().unwrap() returns a MutexGuard which is a local and therefore a temporary variable that hold the HashMap. And get() of the HashMap returns a reference.
Due to the fact that the temporary MutexGuard will be destroy at the end of the function, the reference returned by get would point to a temporary value.
To fix this, you could make ModuleItem clonable and return a copy of the value:
use std::collections::HashMap;
use std::sync::Mutex;
#[derive(Clone)]
pub struct ModuleItem {
pub absolute_path: String,
}
lazy_static::lazy_static! {
static ref MODULE_MAP: Mutex<HashMap<i32, ModuleItem>> = Mutex::new(HashMap::new());
}
pub fn insert(identity_hash: i32, module_item: ModuleItem) {
MODULE_MAP
.lock()
.unwrap()
.insert(identity_hash, module_item);
}
pub fn get(identity_hash: i32) -> Option<ModuleItem> {
MODULE_MAP.lock().unwrap().get(&identity_hash).cloned()
}
I want my method of struct to perform in a synchronized way. I wanted to do this by using Mutex (Playground):
use std::sync::Mutex;
use std::collections::BTreeMap;
pub struct A {
map: BTreeMap<String, String>,
mutex: Mutex<()>,
}
impl A {
pub fn new() -> A {
A {
map: BTreeMap::new(),
mutex: Mutex::new(()),
}
}
}
impl A {
fn synchronized_call(&mut self) {
let mutex_guard_res = self.mutex.try_lock();
if mutex_guard_res.is_err() {
return
}
let mut _mutex_guard = mutex_guard_res.unwrap(); // safe because of check above
let mut lambda = |text: String| {
let _ = self.map.insert("hello".to_owned(),
"d".to_owned());
};
lambda("dd".to_owned());
}
}
Error message:
error[E0500]: closure requires unique access to `self` but `self.mutex` is already borrowed
--> <anon>:23:26
|
18 | let mutex_guard_res = self.mutex.try_lock();
| ---------- borrow occurs here
...
23 | let mut lambda = |text: String| {
| ^^^^^^^^^^^^^^ closure construction occurs here
24 | if let Some(m) = self.map.get(&text) {
| ---- borrow occurs due to use of `self` in closure
...
31 | }
| - borrow ends here
As I understand when we borrow anything from the struct we are unable to use other struct's fields till our borrow is finished. But how can I do method synchronization then?
The closure needs a mutable reference to the self.map in order to insert something into it. But closure capturing works with whole bindings only. This means, that if you say self.map, the closure attempts to capture self, not self.map. And self can't be mutably borrowed/captured, because parts of self are already immutably borrowed.
We can solve this closure-capturing problem by introducing a new binding for the map alone such that the closure is able to capture it (Playground):
let mm = &mut self.map;
let mut lambda = |text: String| {
let _ = mm.insert("hello".to_owned(), text);
};
lambda("dd".to_owned());
However, there is something you overlooked: since synchronized_call() accepts &mut self, you don't need the mutex! Why? Mutable references are also called exclusive references, because the compiler can assure at compile time that there is only one such mutable reference at any given time.
Therefore you statically know, that there is at most one instance of synchronized_call() running on one specific object at any given time, if the function is not recursive (calls itself).
If you have mutable access to a mutex, you know that the mutex is unlocked. See the Mutex::get_mut() method for more explanation. Isn't that amazing?
Rust mutexes do not work the way you are trying to use them. In Rust, a mutex protects specific data relying on the borrow-checking mechanism used elsewhere in the language. As a consequence, declaring a field Mutex<()> doesn't make sense, because it is protecting read-write access to the () unit object that has no values to mutate.
As Lukas explained, your call_synchronized as declared doesn't need to do synchronization because its signature already requests an exclusive (mutable) reference to self, which prevents it from being invoked from multiple threads on the same object. In other words, you need to change the signature of call_synchronized because the current one does not match the functionality it is intended to provide.
call_synchronized needs to accept a shared reference to self, which will signal to Rust that it can be called from multiple threads in the first place. Inside call_synchronized a call to Mutex::lock will simultaneously lock the mutex and provide a mutable reference to the underlying data, carefully scoped so that the lock is held for the duration of the reference:
use std::sync::Mutex;
use std::collections::BTreeMap;
pub struct A {
synced_map: Mutex<BTreeMap<String, String>>,
}
impl A {
pub fn new() -> A {
A {
synced_map: Mutex::new(BTreeMap::new()),
}
}
}
impl A {
fn synchronized_call(&self) {
let mut map = self.synced_map.lock().unwrap();
// omitting the lambda for brevity, but it would also work
// (as long as it refers to map rather than self.map)
map.insert("hello".to_owned(), "d".to_owned());
}
}