Rust lifetime issue with closures within same global scope - rust

Here's the minimal representation of the problem in my code :
use std::collections::HashMap;
fn main() {
let mut map : HashMap<String, String> = HashMap::new();
map.insert(String::from("a"), String::from("first"));
map.insert(String::from("b"), String::from("second"));
let mut funcs: Vec<Box<dyn FnMut(String) -> ()>> = Vec::new();
for (key, val) in map {
funcs.push(Box::new(|v| {
println!("{} : {} : {}",key,val,v);
}))
}
for mut func in funcs {
func(String::from("c"));
}
}
The error : (same for val) :
`key` does not live long enough
values in a scope are dropped in the opposite order they are defined rustcE0597
main.rs(12, 5): `key` dropped here while still borrowed
main.rs(9, 29): value captured here
main.rs(17, 1): borrow might be used here, when `funcs` is dropped and runs the
`Drop` code for type `Vec`
This is a minimalist representation of my real life use case - I'm using a library which requires me to feed it a bunch of closures for an combined execution later on. I first build a HashMap, then loop over it to create a list of FnMut (in this minimal example Fn would also be fine)
I get the general idea of the error that I'm creating key (and val) with temporary lifespan which ends at the end of each loop, and the closures have a "deferred" execution where they will want to access this data which will no longer exist - but I have no idea how to fix it.
I tried map.into_iter(), map.drain() (to not use references), wrapping the data with Rc - without avail. What is the correct solution here ?
Since building the map is a somewhat complex process, I'd like to still have it built before making closures if possible.
EDIT1:
The solution to add move in front of the closure by #Masklinn does work here, but it creates other issues in my real code
The value of the map I use is Vec<MyStruct> which is not copyable, so it cannot be moved. I think I can solve it by implementing Copy trait (my struct is small enough, basically a bunch of Option)
The second issue is that as I create multiple closures, I borrow yet a second HashMap in multiple closures (non mutably), so after the second closure forth I get use of moved value error : how to handle this case ?
EDIT2: Illustration of second problem when using move :
fn main() {
let mut map : HashMap<String, Vec<String>> = HashMap::new();
map.insert(String::from("a"), Vec::from([String::from("first")]));
map.insert(String::from("b"), Vec::from([String::from("first")]));
let map2 : HashMap<String, Vec<String>> = HashMap::new();
let mut funcs: Vec<Box<dyn FnMut(String) -> ()>> = Vec::new();
for (key, val) in map {
funcs.push(Box::new(move |v| {
println!("{} : {:?} : {}, {}",key,val,v, map2.capacity());
}))
}
for mut func in funcs {
func(String::from("c"));
}
}
Error:
use of moved value: `map2`
value moved into closure here, in previous iteration of looprustcE0382
main.rs(12, 54): use occurs due to use in closure
main.rs(7, 9): move occurs because `map2` has type `HashMap<String, Vec<String>>`, which does not implement the `Copy` trait
Should I only be using copyable data (if yes, how ? Wrap the hashmap into a custom copyable struct?), or is there another way?
EDIT : SOLUTION
Thanks to #Masklinn : wrapping non-copyable data into Rc works perfectly !
use std::{collections::HashMap, rc::Rc};
fn main() {
let mut map : HashMap<String, Vec<String>> = HashMap::new();
map.insert(String::from("a"), Vec::from([String::from("first")]));
map.insert(String::from("b"), Vec::from([String::from("first")]));
let map2 : HashMap<String, Vec<String>> = HashMap::new();
let rc_map2: Rc<HashMap<String, Vec<String>>> = Rc::new(map2);
let mut funcs: Vec<Box<dyn FnMut(String) -> ()>> = Vec::new();
for (key, val) in map {
let rc_map2_clone = rc_map2.clone();
funcs.push(Box::new(move |v| {
println!("{} : {:?} : {}, {}",key,val,v, rc_map2_clone.capacity());
}))
}
for mut func in funcs {
func(String::from("c"));
}
}

What is the correct solution here ?
Just add the keyword move before the closure:
funcs.push(Box::new(move |v| {
println!("{} : {} : {}",key,val,v);
}))
By default, lambdas try to infer how the variable closed over should be captured (reference, mutable reference, or value). However this is based exclusively on usage.
Here you're just printing the key and value, so the usage requirements are just shared references, therefore inside the closure you have key: &String and val: &String.
By using move closures, you're telling the lambda to always capture by value (and you'll manage capturing e.g. references by value).
By using a move closure here we're making the closure own the key and value Strings, therefore their lifetimes are not an issue anymore.

Related

Multiple Mutable Borrows from Struct Hashmap

Running into an ownership issue when attempting to reference multiple values from a HashMap in a struct as parameters in a function call. Here is a PoC of the issue.
use std::collections::HashMap;
struct Resource {
map: HashMap<String, String>,
}
impl Resource {
pub fn new() -> Self {
Resource {
map: HashMap::new(),
}
}
pub fn load(&mut self, key: String) -> &mut String {
self.map.get_mut(&key).unwrap()
}
}
fn main() {
// Initialize struct containing a HashMap.
let mut res = Resource {
map: HashMap::new(),
};
res.map.insert("Item1".to_string(), "Value1".to_string());
res.map.insert("Item2".to_string(), "Value2".to_string());
// This compiles and runs.
let mut value1 = res.load("Item1".to_string());
single_parameter(value1);
let mut value2 = res.load("Item2".to_string());
single_parameter(value2);
// This has ownership issues.
// multi_parameter(value1, value2);
}
fn single_parameter(value: &String) {
println!("{}", *value);
}
fn multi_parameter(value1: &mut String, value2: &mut String) {
println!("{}", *value1);
println!("{}", *value2);
}
Uncommenting multi_parameter results in the following error:
28 | let mut value1 = res.load("Item1".to_string());
| --- first mutable borrow occurs here
29 | single_parameter(value1);
30 | let mut value2 = res.load("Item2".to_string());
| ^^^ second mutable borrow occurs here
...
34 | multi_parameter(value1, value2);
| ------ first borrow later used here
It would technically be possible for me to break up the function calls (using the single_parameter function approach), but it would be more convenient to pass the
variables to a single function call.
For additional context, the actual program where I'm encountering this issue is an SDL2 game where I'm attempting to pass multiple textures into a single function call to be drawn, where the texture data may be modified within the function.
This is currently not possible, without resorting to unsafe code or interior mutability at least. There is no way for the compiler to know if two calls to load will yield mutable references to different data as it cannot always infer the value of the key. In theory, mutably borrowing both res.map["Item1"] and res.map["Item2"] would be fine as they would refer to different values in the map, but there is no way for the compiler to know this at compile time.
The easiest way to do this, as already mentioned, is to use a structure that allows interior mutability, like RefCell, which typically enforces the memory safety rules at run-time before returning a borrow of the wrapped value. You can also work around the borrow checker in this case by dealing with mut pointers in unsafe code:
pub fn load_many<'a, const N: usize>(&'a mut self, keys: [&str; N]) -> [&'a mut String; N] {
// TODO: Assert that keys are distinct, so that we don't return
// multiple references to the same value
keys.map(|key| self.load(key) as *mut _)
.map(|ptr| unsafe { &mut *ptr })
}
Rust Playground
The TODO is important, as this assertion is the only way to ensure that the safety invariant of only having one mutable reference to any value at any time is upheld.
It is, however, almost always better (and easier) to use a known safe interior mutation abstraction like RefCell rather than writing your own unsafe code.

"expected bound lifetime parameter, found concrete lifetime" on StreamExt .scan() method

I am using async-tungstenite to listen to a websocket, and async-std's StreamExt to operate on the resulting stream.
I want to use a HashMap to accumulate the latest Ticker values from the websocket. These Ticker values will be looked up later to be used in calculations. I'm using the symbol (String) value of the Ticker struct as the key for the HashMap. I'm using the .scan StreamExt method to perform the accumulation.
However, I get a compilation error related to lifetimes. Here's some stripped-down code:
let tickers = HashMap::new();
let mut stream = ws.
.scan(tickers, accumulate_tickers);
while let msg = stream.next().await {
println!("{:?}", msg)
}
...and the accumulate_tickers function:
fn accumulate_tickers(tps: &mut HashMap<String, Ticker>, bt: Ticker) -> Option<&HashMap<String, Ticker>> {
tps.insert((*bt.symbol).to_string(), bt);
Some(tps)
}
The compilation error I receive is as follows:
error[E0271]: type mismatch resolving `for<'r> <for<'s> fn(&'s mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker) -> std::option::Option<&'s std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>> {accumulate_tickers} as std::ops::FnOnce<(&'r mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker)>>::Output == std::option::Option<_>`
--> examples/async_std-ws.rs:64:4
|
64 | .scan(tickers, accumulate_tickers);
| ^^^^ expected bound lifetime parameter, found concrete lifetime
I'm unaware of a way to provide a lifetime parameter to the scan method.
I wonder whether the issue may be related to the fact that I modify the HashMap and then try to return it (is it a move issue?). How could I resolve this, or at least narrow down the cause?
I was able to get this working. Working my way through the compiler errors, I ended up with this signature for the accumulate_tickers function:
fn accumulate_tickers<'a>(tps: &'a mut &'static HashMap<String, Ticker>, bt: Ticker) -> Option<&'static HashMap<String, Ticker>>
I do want the accumulator HashMap to have a static lifetime so that makes sense. tps: &'a mut &'static HashMap... does look a bit strange, but it works.
Then, this issue was was that tickers also had to have a static lifetime (it's the initial value for the accumulator. I tried declaring it as static outside the main but it wouldn't let me set it to the result of a function - HashMap::new().
I then turned to lazy_static which allows me to create a static value which does just that:
lazy_static! {
static ref tickers: HashMap<String, Ticker> = HashMap::new();
}
This gave me a HashMap accumulator that had a static lifetime. However, like normal static values declared in the root scope, it was immutable. To fix that, I read some hints from the lazy_static team and then found https://pastebin.com/YES8dsHH. This showed me how to make my static accumulator mutable by wrapping it in Arc<Mutex<_>>.
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
This does mean that I have to retrieve the accumulator from the Mutex (and lock it) before reading or modifying it but, again, it works.
Putting it all together, the stripped-down code now looks like this:
#[macro_use]
extern crate lazy_static;
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
// SNIP
// Inside main()
let mut ticks = ws
.scan(&tickers, accumulate_tickers);
while let Some(msg) = ticks.next().await {
println!("{:?}", msg.lock().unwrap());
}
// SNIP
fn accumulate_tickers<'a>(tps: &'a mut &'static tickers, bt: Ticker) -> Option<&'static tickers> {
tps.lock().unwrap().insert((*bt.symbol).to_string(), bt);
Some(tps)
}
I'd be happy to hear suggestions for ways in which this could be made simpler or more elegant.

Vector holding mutable functions

I would like to have a vector with functions. Then I would like to iterate on this vector and execute functions one by one. The functions would mutate an external state. Additionally, I would like to be able to place the same function twice in the vector.
The problems I have are:
I cannot dereference and execute the function from the vector,
Adding the same function to the vector twice fails with, understandable, error that I cannot have two mutable references.
The closest I got is:
fn main() {
let mut c = 0;
{
let mut f = ||{c += 1};
let mut v: Vec<&mut FnMut()> = vec![];
v.push(&mut f);
// How to execute the stored function? The following complains about
// an immutable reference:
// assignment into an immutable reference
// (v[0])();
// How to store the same function twice? The following will fail with:
// cannot borrow `f` as mutable more than once at a time
// v.push(&mut f);
}
println!("c {}", c);
}
For the first problem, I don't really know why no mutable dereference happens here (in my opinion, it should), but there is a simple workaround: just do the dereference and then reference manually:
(&mut *v[0])();
Your second problem is more complex, though. There is no simple solution, because what you're trying to do violates Rust aliasing guarantees, and since you did not describe the purpose of it, I can't suggest alternatives properly. In general, however, you can overcome this error by switching to runtime borrow-checking with Cell/RefCell or Mutex (the latter is when you need concurrent access). With Cell (works nice for primitives):
use std::cell::Cell;
fn main() {
let c = Cell::new(0);
{
let f = || { c.set(c.get() + 1); };
let mut v: Vec<&Fn()> = vec![];
v.push(&f);
v.push(&f);
v[0]();
v[1]();
}
println!("c {}", c.get());
}
With RefCell (works nice for more complex types):
use std::cell::RefCell;
fn main() {
let c = RefCell::new(0);
{
let f = || { *c.borrow_mut() += 1; };
let mut v: Vec<&Fn()> = vec![];
v.push(&f);
v.push(&f);
v[0]();
v[1]();
}
println!("c {}", *c.borrow());
}
As you can see, now you have &Fn() instead of &mut FnMut(), which can be aliased freely, and whose captured environment may also contain aliased references (immutable, of course).

Mutating a variable in the body of a closure

I'm writing a bunch of assertions that all involve popping a value from a list.
Coming from a Scala background, I naturally did this:
let mut list = List::new();
let assert_pop = |expected| assert_eq!(list.pop(), expected);
So that I could just write assert_pop(None) or assert_pop(Some(3)) instead of having to write assert_eq!(list.pop(), None) or assert_eq!(list.pop(), Some(3)) every time.
Of course the borrow checker doesn't like this one bit because the closure essentially needs to borrow the value for an undisclosed amount of time, while the rest of my code goes around mutating, thus violating the rule of "no aliasing if you're mutating".
The question is: is there a way to get around this? Do I have to write a macro, or is there a funky memory-safe way I can get around this?
Note that I know that I can just define the closure like this:
let_assert_pop = |lst: &mut List, expected| assert_eq!(lst.pop(), expected);
But that would be less DRY, as I'd have to pass in a &mut list as the first argument at every call.
They key is to define the closure as mut, since it needs a mutable reference.
This works:
let mut v = vec![1, 2];
let mut assert_pop = |expected| assert_eq!(v.pop(), expected);
assert_pop(Some(2));
assert_pop(Some(1));
assert_pop(None);
Note that the the pop closure borrows mutably, so you if you want to use the list afterwards, you have to scope it:
let mut v = vec![1,2];
{
let mut assert_pop = |expected| assert_eq!(v.pop(), expected);
assert_pop(Some(2));
v.push(33); // ERROR: v is borrowed mutably...
}
v.push(33); // Works now, since pop is out of scope.
Instead of directly answering your question (which is well-enough answered already), I'll instead address your other points:
just write assert_pop(None) or assert_pop(Some(3))
a memory-safe way solution
don't pass in a &mut list
To solve all that, don't use a closure, just make a new type:
type List<T> = Vec<T>;
struct Thing<T>(List<T>);
impl<T> Thing<T> {
fn assert_pop(&mut self, expected: Option<T>)
where T: PartialEq + std::fmt::Debug,
{
assert_eq!(self.0.pop(), expected);
}
}
fn main() {
let list = List::new();
let mut list = Thing(list);
list.0.push(1);
list.assert_pop(Some(1));
list.assert_pop(None);
// Take it back if we need to
let _list = list.0;
}

Why does HashMap have iter_mut() but HashSet doesn't?

What is the design rationale for supplying an iter_mut function for HashMap but not HashSet in Rust?
Would it be a faux pas to roll one's own (assuming that can even be done)?
Having one could alleviate situations that give rise to
previous borrow of X occurs here; the immutable borrow prevents
subsequent moves or mutable borrows of X until the borrow ends
Example
An extremely convoluted example (Gist) that does not show-case why the parameter passing is the way that it is. Has a short comment explaining the pain-point:
use std::collections::HashSet;
fn derp(v: i32, unprocessed: &mut HashSet<i32>) {
if unprocessed.contains(&v) {
// Pretend that v has been processed
unprocessed.remove(&v);
}
}
fn herp(v: i32) {
let mut unprocessed: HashSet<i32> = HashSet::new();
unprocessed.insert(v);
// I need to iterate over the unprocessed values
while let Some(u) = unprocessed.iter().next() {
// And them pass them mutably to another function
// as I will process the values inside derp and
// remove them from the set.
//
// This is an extremely convoluted example but
// I need for derp to be a separate function
// as I will employ recursion there, as it is
// much more succinct than an iterative version.
derp(*u, &mut unprocessed);
}
}
fn main() {
println!("Hello, world!");
herp(10);
}
The statement
while let Some(u) = unprocessed.iter().next() {
is an immutable borrow, hence
derp(*u, &mut unprocessed);
is impossible as unprocessed cannot be borrowed mutably. The immutable borrow does not end until the end of the while-loop.
I have tried to use this as reference and essentially ended up with trying to fool the borrow checker through various permutations of assignments, enclosing braces, but due to the coupling of the intended expressions the problem remains.
You have to think about what HashSet actually is. The IterMut that you get from HashMap::iter_mut() is only mutable on the value part: (&key, &mut val), ((&'a K, &'a mut V))
HashSet is basically a HashMap<T, ()>, so the actual values are the keys, and if you would modify the keys the hash of them would have to be updated or you get an invalid HashMap.
If your HashSet contains a Copy type, such as i32, you can work on a copy of the value to release the borrow on the HashSet early. To do this, you need to eliminate all borrows from the bindings in the while let expression. In your original code, u is of type &i32, and it keeps borrowing from unprocessed until the end of the loop. If we change the pattern to Some(&u), then u is of type i32, which doesn't borrow from anything, so we're free to use unprocessed as we like.
fn herp(v: i32) {
let mut unprocessed: HashSet<i32> = HashSet::new();
unprocessed.insert(v);
while let Some(&u) = unprocessed.iter().next() {
derp(u, &mut unprocessed);
}
}
If the type is not Copy or is too expensive to copy/clone, you can wrap them in Rc or Arc, and clone them as you iterate on them using cloned() (cloning an Rc or Arc doesn't clone the underlying value, it just clones the Rc pointer and increments the reference counter).
use std::collections::HashSet;
use std::rc::Rc;
fn derp(v: &i32, unprocessed: &mut HashSet<Rc<i32>>) {
if unprocessed.contains(v) {
unprocessed.remove(v);
}
}
fn herp(v: Rc<i32>) {
let mut unprocessed: HashSet<Rc<i32>> = HashSet::new();
unprocessed.insert(v);
while let Some(u) = unprocessed.iter().cloned().next() {
// If you don't use u afterwards,
// you could also pass if by value to derp.
derp(&u, &mut unprocessed);
}
}
fn main() {
println!("Hello, world!");
herp(Rc::new(10));
}

Resources