I use Rayons par_iter()to iterate over different variations of the expensive method I need to run. These runs need to access the same set of checked usizes because they all need to add to it and check it from time to time. I also need them all to shutdown when first thread finishes, this is why I have a kill_switch which will force the iterations to exit when its set to true.
let mut checked: HashSet<usize> = HashSet::new();
let mut kill_switch: bool = false;
permutations.par_iter().for_each(|order| {
let board = Board::new(board_map.clone(), order.clone());
let mut bubbles: Vec<(i8, i8)> = Vec::new();
if let Some(bubbles) = board.solve(&mut bubbles, &mut checked, &kill_switch) {
kill_switch = true;
bubbles.into_iter().for_each(|bubble| {
dbg!(bubble);
});
}
})
This is the code I currently have but I get errors for how I'm using checked and kill_switch. How do I make this work?
Errors:
cannot borrow checked as mutable, as it is a captured variable in a Fn closure
cannot borrow as mutable [E0596]
cannot assign to kill_switch, as it is a captured variable in a Fn closure
cannot assign [E0594]
To fix the errors, you will need to use RefCells to wrap the checked and kill_switch variables and use the borrow_mut method to get a mutable reference to them in the closure.
Here is an example of how you can modify your code:
use std::cell::RefCell;
use std::collections::HashSet;
let checked: RefCell<HashSet<usize>> = RefCell::new(HashSet::new());
let kill_switch: RefCell<bool> = RefCell::new(false);
permutations.par_iter().for_each(|order| {
let board = Board::new(board_map.clone(), order.clone());
let mut bubbles: Vec<(i8, i8)> = Vec::new();
if let Some(bubbles) = board.solve(&mut bubbles, &mut checked.borrow_mut(), &mut kill_switch.borrow_mut()) {
*kill_switch.borrow_mut() = true;
bubbles.into_iter().for_each(|bubble| {
dbg!(bubble);
});
}
})
Note that you will also need to add RefCell as a dependency in your project.
In the following code snippet I am trying to create a tuple that contains a String value and a struct that contains an attribute that is set as the reference to the first value (which would be the String) in the tuple.
My question is how do I make the String (which is the first value in the tuple) live long enough.
Here is the code:
#[derive(Debug, Clone)]
struct RefTestStruct {
key: usize,
ref_value: Option<&'static str>,
}
fn init3<'a>( cache: &Arc<Mutex<HashMap<usize, (String, Option<RefTestStruct>)>>> ) {
let mut handles: Vec<JoinHandle<()>> = vec![];
for idx in 0..10 {
//create reference copy of cache
let cache_clone = Arc::clone(cache);
let handle = thread::spawn(move || {
//lock cache
let mut locked_cache = cache_clone.lock().unwrap();
// add new object to cache
let tuple_value = (format!("value: {}", idx), None );
let mut ts_obj =
RefTestStruct {
key: idx,
ref_value: Some( tuple_value.0.as_str() ),
};
tuple_value.1 = Some(ts_obj);
locked_cache.insert(idx as usize, tuple_value);
println!("IDX: {} - CACHE: {:?}", idx, locked_cache.get( &(idx as usize) ).unwrap() )
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
println!("\n");
}
fn main() {
// init
let cache = HashMap::<usize, (String, Option<RefTestStruct>)>::new();
let am_cache = Arc::new(Mutex::new(cache));
init3(&am_cache);
// change cache contents
let mut handles: Vec<JoinHandle<()>> = vec![];
for idx in 0..10 {
let cache_clone = Arc::clone(&am_cache);
let handle = thread::spawn(move || {
let mut locked_cache = cache_clone.lock().unwrap();
let tuple_value = locked_cache.get_mut( &(idx as usize) ).unwrap();
(*tuple_value).0 = format!("changed value: {}", (*tuple_value).1.unwrap().key + 11);
let ts_obj =
RefTestStruct {
key: (*tuple_value).1.unwrap().key,
ref_value: Some( (*tuple_value).0.as_str() ),
};
tuple_value.1 = Some(ts_obj);
// locked_cache.insert(idx as usize, tuple_value);
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
// display cache contents
let mut handles: Vec<JoinHandle<()>> = vec![];
for idx in 0..10 {
let cache_clone = Arc::clone(&am_cache);
let handle = thread::spawn(move || {
let locked_cache = cache_clone.lock().unwrap();
let ts_obj = locked_cache.get( &(idx as usize) ).unwrap();
println!("IDX: {} - CACHE: {:?}", idx, &ts_obj );
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
}
ERROR:
error[E0597]: `tuple_value.0` does not live long enough
--> src\main.rs:215:42
|
215 | ref_value: Some( tuple_value.0.as_str() ),
| ^^^^^^^^^^^^^^^^^^^^^^
| |
| borrowed value does not live long enough
| argument requires that `tuple_value.0` is borrowed for `'static`
...
221 | });
| - `tuple_value.0` dropped here while still borrowed
How do you pass a struct that contains a String reference between threads?
First I answer the question in the caption and ignore the program you posted (And I'll try to ignore my itch to guess what you really want to do.)
I'll assume, you want to avoid unsafe in your own code. (So dependencies with unsafe code might be ok.)
To pass a reference to a thread, the reference has to live as long as the thread will live.
Your options:
Have a 'static lifetime on your reference.
Use threads which have a non 'static lifetime.
Regarding 1. have a 'static lifetime on the reference.
So now the questions changed into, how to obtain a reference to
a String with 'static lifetime.
We can't use a constant, as we can't initialize it.
This leaves us with the possibility to Box the String and leak it.
But of course, this leaks memory, as the String can't be dropped any more. Also you get a &'mut String out
of it, so only one thread can have it at a time and after being
done with it, pass it on. So to me it looks like it is essentially
the same you could do with the owned value of type String too,
so why bother and risk the memory leak. (Putting a Arc<Mutex<..>> around it, is also "same you coud do with the owned value".).
Other methods might exist, which I am not aware of.
Regarding 2. Use threads which have a non `'static' lifetime.
These are called scoped threads and guarantee, that the thread exits,
before the lifetime ends. These are not in standard rust yet, but there
are crates implementing those (I guess using unsafe).
For example crossbeam offers an implementation of this.
Looking at your program though, I guess you want to have long lived
threads.
The program you posted and what you probably want to achieve.
The program you posted has a different issue besides.
You try to create a self referential value in your cache. As far as I know, this is not possible in rust, without using a shared ownership tool, like Arc.
If you try to use a plain Arc, the access would be read only and users of the cache could only change, which String the cache entry points to, but not the "pointed to" String itself.
So If one user replaced the Arc<String> in the cache, other threads would still have a reference to the "old" String and don't see the changed value. But I guess that is what you want to achieve.
But that is a conceptual problem, you can't have unsynchronized read access to a value, which can then be mutated concurrently (by whatever means).
So, if you want to have the ability to change the value with all prior users being able to see it, this leaves you with Mutex or RWLock depending on the access pattern, you are anticipating.
A solution using Arc<String> with the aforementioned defects:
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::thread::{self, JoinHandle};
#[derive(Debug, Clone)]
struct RefTestStruct {
key: usize,
_ref_value: Arc<String>,
}
type Cache = HashMap<usize, (Arc<String>, RefTestStruct)>;
type AmCache = Arc<Mutex<Cache>>;
fn init3(cache: &AmCache) {
let mut handles: Vec<JoinHandle<()>> = vec![];
for idx in 0..10_usize {
//create reference copy of cache
let cache_clone = Arc::clone(cache);
let handle = thread::spawn(move || {
//lock cache
let mut locked_cache = cache_clone.lock().unwrap();
// add new object to cache
let s = Arc::new(format!("value: {}", idx));
let ref_struct = RefTestStruct {
key: idx,
_ref_value: s.clone(),
};
let tuple_value = (s, ref_struct);
locked_cache.insert(idx, tuple_value);
println!(
"IDX: {} - CACHE: {:?}",
idx,
locked_cache.get(&idx).unwrap()
)
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
println!("\n");
}
fn main() {
// init
let cache = Cache::new();
let am_cache = Arc::new(Mutex::new(cache));
init3(&am_cache);
// change cache contents
let mut handles: Vec<JoinHandle<()>> = vec![];
for idx in 0..10_usize {
let cache_clone = Arc::clone(&am_cache);
let handle = thread::spawn(move || {
let mut locked_cache = cache_clone.lock().unwrap();
let tuple_value = locked_cache.get_mut(&idx).unwrap();
let new_key = tuple_value.1.key + 11;
let new_s = Arc::new(format!("changed value: {}", new_key));
(*tuple_value).1 = RefTestStruct {
key: new_key,
_ref_value: new_s.clone(),
};
(*tuple_value).0 = new_s;
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
// display cache contents
let mut handles: Vec<JoinHandle<()>> = vec![];
for idx in 0..10_usize {
let cache_clone = Arc::clone(&am_cache);
let handle = thread::spawn(move || {
let locked_cache = cache_clone.lock().unwrap();
let ts_obj = locked_cache.get(&idx).unwrap();
println!("IDX: {} - CACHE: {:?}", idx, &ts_obj);
});
handles.push(handle);
}
for handle in handles {
handle.join().unwrap();
}
}
I am sure it is not difficult to derive the Arc<Mutex<String>> version from that, if desired.
That would not be as pointless, as one might think. The Mutex around the cache protects the HashMap for insertion and deletion. And the Mutex around the String values, protects the string values themselves and can be locked indepently of the Mutex around the HashMap. Usual caveats apply, if you ever need both locks at the same time, lock them in the same order always, or risk a deadlock.
I'm trying to process data from a channel, so the whole structure can't be serialized at once. In fact it won't all fit in memory. The trouble I'm running into is that I can't create an Option<SerializeSeq> because that object depends on Serializer (which doesn't live long enough). I'd like to initialize them both together, or not at all:
use serde::ser::{SerializeSeq, Serializer};
fn process_stream(output: bool) -> serde_json::Result<()> {
let rows = /* whatever iterator */ "serde".chars();
let mut seq = if output {
let out = std::io::stdout();
let mut ser = serde_json::Serializer::new(out);
let seq = ser.serialize_seq(None)?
Some(Ok(seq))
} else {
None
}.transpose()?;
for row in rows {
//process_data(row);
if let Some(seq) = &mut seq {
seq.serialize_element(&row)?;
}
}
if let Some(seq) = seq {
seq.end()?;
}
Ok(())
}
(Original code snippet from here.)
The problem is: ser does not live long enough. But I don't want to initialize ser in the outer scope because it may not be enabled (and its writer would create or truncate a file that should not be created). I tried returning ser and seq as a tuple. I tried putting them together in a helper struct, but I couldn't figure out all the template parameters and lifetimes.
How can serde serializer and sequence be initialized based on a condition and stored in Option?
Make sure that ser doesn't get dropped prematurely by declaring it outside the if body:
let mut ser;
let mut seq = if output {
let out = std::io::stdout();
ser = serde_json::Serializer::new(out);
let seq = ser.serialize_seq(None)?;
Some(Ok(seq))
} else {
None
}.transpose()?;
Is there a way to borrow as mutable all the way down the chain and return ownership to config? The memory operations here seem really excessive to update a single variable.
The code is working as expected but coming from high level languages it seems a little "hacky". Am I on the right track? The documentation is a little sparse on this library.
// Read Config File
let mut file = File::open("config.yaml").unwrap();
let mut contents = String::new();
file.read_to_string(&mut contents);
// Get Mutable 'endpoints'
let mut docs = YamlLoader::load_from_str(&contents).unwrap();
let mut config = docs[0].as_hash().unwrap().clone();
let mut endpoints = config[&Yaml::from_str("endpoints")].as_vec().unwrap().clone();
// Find the correct 'endpoint'
for (i, endpoint) in endpoints.clone().iter().enumerate() {
let mut endpoint = endpoint.as_hash().unwrap().clone();
if endpoint[&Yaml::from_str("name")] == Yaml::from_str("Current Name") {
// Found the endpoint, let's update it's name
endpoint.insert(Yaml::from_str("name"), Yaml::from_str("New Name"));
endpoints.insert(i, Yaml::Hash(endpoint));
config.insert(Yaml::from_str("endpoints"), Yaml::Array(endpoints.clone()));
}
}
This is working, and I think solves all the pitfalls of my previous code.
let mut docs = YamlLoader::load_from_str(&contents).unwrap();
let mut config = docs[0].borrow_mut();
if let Yaml::Hash(hash_map) = config {
if let Yaml::Array(endpoints) = hash_map[&Yaml::from_str("endpoints")].borrow_mut() {
for endpoint in endpoints.iter_mut() {
if let Yaml::Hash(endpoint) = endpoint.borrow_mut() {
if endpoint[&Yaml::from_str("name")] == Yaml::from_str("Current Name") {
endpoint.insert(Yaml::from_str("name"), Yaml::from_str("New Name"));
}
}
}
}
}
The function run_av_check receives a vector filled with URLs which need to get called using reqwest. Iterating through the URLs gives me the error that the borrowed value does not live long enough. My understanding is that as long as check_availability is not finished for all URLs which are called in run_av_check using a for loop the run_av_check function is not finished, therefore the variable URLs should still exist during the runtime of all check_availibility calls.
I have read the Tokio and Rust documentation but I am not able to find a clue how and why this behaviour occurs. Why does the error pop up and what needs to be changed to make this work, both program wise and in my way of thinking?
async fn check_availability(host: &str) -> Result<reqwest::StatusCode, reqwest::Error> {
// ... building url
match client.get(&url).send().await {
Ok(r) => Ok(r.status()),
Err(_e) => Ok(reqwest::StatusCode::GATEWAY_TIMEOUT),
}
}
async fn run_av_check(urls: Vec<std::string::String>) -> Result<AvResponse, AvError> {
let mut av_urls: Vec<std::string::String> = vec![];
let mut s: std::string::String = "".to_string();
let mut attributes = Vec::new();
for url in urls.iter() {
attributes.push(("instance", url));
let resp = tokio::task::spawn(check_availability(url));
// ... processing result
let res = AvResponse {
av_urls,
prom_details: s,
};
Ok(res)
}
}
The problem is that you are starting independent tasks in tokio::task::spawn(check_availability(url));
This function takes only 'static references but your url refers to the local vector urls.
You need to make check_availability accept the String type (fn check_availability(host: String)) and call it like let resp = tokio::task::spawn(check_availability(url.to_string()));