"expected bound lifetime parameter, found concrete lifetime" on StreamExt .scan() method - rust

I am using async-tungstenite to listen to a websocket, and async-std's StreamExt to operate on the resulting stream.
I want to use a HashMap to accumulate the latest Ticker values from the websocket. These Ticker values will be looked up later to be used in calculations. I'm using the symbol (String) value of the Ticker struct as the key for the HashMap. I'm using the .scan StreamExt method to perform the accumulation.
However, I get a compilation error related to lifetimes. Here's some stripped-down code:
let tickers = HashMap::new();
let mut stream = ws.
.scan(tickers, accumulate_tickers);
while let msg = stream.next().await {
println!("{:?}", msg)
}
...and the accumulate_tickers function:
fn accumulate_tickers(tps: &mut HashMap<String, Ticker>, bt: Ticker) -> Option<&HashMap<String, Ticker>> {
tps.insert((*bt.symbol).to_string(), bt);
Some(tps)
}
The compilation error I receive is as follows:
error[E0271]: type mismatch resolving `for<'r> <for<'s> fn(&'s mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker) -> std::option::Option<&'s std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>> {accumulate_tickers} as std::ops::FnOnce<(&'r mut std::collections::HashMap<std::string::String, ws_async::model::websocket::Ticker>, ws_async::model::websocket::Ticker)>>::Output == std::option::Option<_>`
--> examples/async_std-ws.rs:64:4
|
64 | .scan(tickers, accumulate_tickers);
| ^^^^ expected bound lifetime parameter, found concrete lifetime
I'm unaware of a way to provide a lifetime parameter to the scan method.
I wonder whether the issue may be related to the fact that I modify the HashMap and then try to return it (is it a move issue?). How could I resolve this, or at least narrow down the cause?

I was able to get this working. Working my way through the compiler errors, I ended up with this signature for the accumulate_tickers function:
fn accumulate_tickers<'a>(tps: &'a mut &'static HashMap<String, Ticker>, bt: Ticker) -> Option<&'static HashMap<String, Ticker>>
I do want the accumulator HashMap to have a static lifetime so that makes sense. tps: &'a mut &'static HashMap... does look a bit strange, but it works.
Then, this issue was was that tickers also had to have a static lifetime (it's the initial value for the accumulator. I tried declaring it as static outside the main but it wouldn't let me set it to the result of a function - HashMap::new().
I then turned to lazy_static which allows me to create a static value which does just that:
lazy_static! {
static ref tickers: HashMap<String, Ticker> = HashMap::new();
}
This gave me a HashMap accumulator that had a static lifetime. However, like normal static values declared in the root scope, it was immutable. To fix that, I read some hints from the lazy_static team and then found https://pastebin.com/YES8dsHH. This showed me how to make my static accumulator mutable by wrapping it in Arc<Mutex<_>>.
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
This does mean that I have to retrieve the accumulator from the Mutex (and lock it) before reading or modifying it but, again, it works.
Putting it all together, the stripped-down code now looks like this:
#[macro_use]
extern crate lazy_static;
lazy_static! {
// from https://pastebin.com/YES8dsHH
static ref tickers: Arc<Mutex<HashMap<String, Ticker>>> = {
let mut ts = HashMap::new();
Arc::new(Mutex::new(ts))
};
}
// SNIP
// Inside main()
let mut ticks = ws
.scan(&tickers, accumulate_tickers);
while let Some(msg) = ticks.next().await {
println!("{:?}", msg.lock().unwrap());
}
// SNIP
fn accumulate_tickers<'a>(tps: &'a mut &'static tickers, bt: Ticker) -> Option<&'static tickers> {
tps.lock().unwrap().insert((*bt.symbol).to_string(), bt);
Some(tps)
}
I'd be happy to hear suggestions for ways in which this could be made simpler or more elegant.

Related

Rust lifetime issue with closures within same global scope

Here's the minimal representation of the problem in my code :
use std::collections::HashMap;
fn main() {
let mut map : HashMap<String, String> = HashMap::new();
map.insert(String::from("a"), String::from("first"));
map.insert(String::from("b"), String::from("second"));
let mut funcs: Vec<Box<dyn FnMut(String) -> ()>> = Vec::new();
for (key, val) in map {
funcs.push(Box::new(|v| {
println!("{} : {} : {}",key,val,v);
}))
}
for mut func in funcs {
func(String::from("c"));
}
}
The error : (same for val) :
`key` does not live long enough
values in a scope are dropped in the opposite order they are defined rustcE0597
main.rs(12, 5): `key` dropped here while still borrowed
main.rs(9, 29): value captured here
main.rs(17, 1): borrow might be used here, when `funcs` is dropped and runs the
`Drop` code for type `Vec`
This is a minimalist representation of my real life use case - I'm using a library which requires me to feed it a bunch of closures for an combined execution later on. I first build a HashMap, then loop over it to create a list of FnMut (in this minimal example Fn would also be fine)
I get the general idea of the error that I'm creating key (and val) with temporary lifespan which ends at the end of each loop, and the closures have a "deferred" execution where they will want to access this data which will no longer exist - but I have no idea how to fix it.
I tried map.into_iter(), map.drain() (to not use references), wrapping the data with Rc - without avail. What is the correct solution here ?
Since building the map is a somewhat complex process, I'd like to still have it built before making closures if possible.
EDIT1:
The solution to add move in front of the closure by #Masklinn does work here, but it creates other issues in my real code
The value of the map I use is Vec<MyStruct> which is not copyable, so it cannot be moved. I think I can solve it by implementing Copy trait (my struct is small enough, basically a bunch of Option)
The second issue is that as I create multiple closures, I borrow yet a second HashMap in multiple closures (non mutably), so after the second closure forth I get use of moved value error : how to handle this case ?
EDIT2: Illustration of second problem when using move :
fn main() {
let mut map : HashMap<String, Vec<String>> = HashMap::new();
map.insert(String::from("a"), Vec::from([String::from("first")]));
map.insert(String::from("b"), Vec::from([String::from("first")]));
let map2 : HashMap<String, Vec<String>> = HashMap::new();
let mut funcs: Vec<Box<dyn FnMut(String) -> ()>> = Vec::new();
for (key, val) in map {
funcs.push(Box::new(move |v| {
println!("{} : {:?} : {}, {}",key,val,v, map2.capacity());
}))
}
for mut func in funcs {
func(String::from("c"));
}
}
Error:
use of moved value: `map2`
value moved into closure here, in previous iteration of looprustcE0382
main.rs(12, 54): use occurs due to use in closure
main.rs(7, 9): move occurs because `map2` has type `HashMap<String, Vec<String>>`, which does not implement the `Copy` trait
Should I only be using copyable data (if yes, how ? Wrap the hashmap into a custom copyable struct?), or is there another way?
EDIT : SOLUTION
Thanks to #Masklinn : wrapping non-copyable data into Rc works perfectly !
use std::{collections::HashMap, rc::Rc};
fn main() {
let mut map : HashMap<String, Vec<String>> = HashMap::new();
map.insert(String::from("a"), Vec::from([String::from("first")]));
map.insert(String::from("b"), Vec::from([String::from("first")]));
let map2 : HashMap<String, Vec<String>> = HashMap::new();
let rc_map2: Rc<HashMap<String, Vec<String>>> = Rc::new(map2);
let mut funcs: Vec<Box<dyn FnMut(String) -> ()>> = Vec::new();
for (key, val) in map {
let rc_map2_clone = rc_map2.clone();
funcs.push(Box::new(move |v| {
println!("{} : {:?} : {}, {}",key,val,v, rc_map2_clone.capacity());
}))
}
for mut func in funcs {
func(String::from("c"));
}
}
What is the correct solution here ?
Just add the keyword move before the closure:
funcs.push(Box::new(move |v| {
println!("{} : {} : {}",key,val,v);
}))
By default, lambdas try to infer how the variable closed over should be captured (reference, mutable reference, or value). However this is based exclusively on usage.
Here you're just printing the key and value, so the usage requirements are just shared references, therefore inside the closure you have key: &String and val: &String.
By using move closures, you're telling the lambda to always capture by value (and you'll manage capturing e.g. references by value).
By using a move closure here we're making the closure own the key and value Strings, therefore their lifetimes are not an issue anymore.

Trying to pin box a dynamic writer to pass it across FFI with no luck in RUST

I seem to have a basic problem that I can't solve.
I have this struct that expects apinned dynamic writer to pass to "C" and then later gets it back as a part of a callback function:
Here is the struct:
pub struct ExecutionContext<'a> {
pub log: Pin<&'a mut Box<dyn std::io::Write>>,
}
I can't seem to find a way to pass a simple stderr to this struct though.
If I try
let mut stderr2 = Pin::new (&mut Box::<dyn Write>::new(stderr()));
I get this error:
function or associated item cannot be called on `Box<dyn std::io::Write>` due to unsatisfied trait bounds
When I try this:
let mut stderr2 = Pin::new (&mut Box::new(stderr()));
let mut ctx = ExecutionContext{
log: stderr2,
};
I get :
expected trait object `dyn std::io::Write`, found struct `Stderr`
The first error continues with:
the following trait bounds were not satisfied: dyn std::io::Write: Sized
The problem is that somehow calling Box::new with this type bounds requires the inner value of the Box to have a known size. Trait objects cannot provide that.
You can avoid that by creating explicitly a variable with your type annotations.
let mut b: Box<dyn Write> = Box::new(stderr());
let stderr2 = Pin::new(&mut b);
let mut ctx = ExecutionContext { log: stderr2 };
playground
But may I ask for the reason of putting a Box behind a mutable reference?
Isn't a Box enough?
If you are willing to change the type of ExecutionContext::log I would recommend the following:
store the Box directly behind the Pin in ExecutionContext::log (without the reference)
use Box::pin, a constructor which creates a Pin<Box<T>> for you
pub struct ExecutionContext {
pub log: Pin<Box<dyn std::io::Write>>,
}
let stderr2 = Box::pin(stderr());
let mut ctx = ExecutionContext { log: stderr2 };
playground

How to preserve unsized data created in a closure

I'm trying to create unsized data in a callback, and store the reference in a vector. Is this possible in any way?
fn main() {
let mut results = vec![];
let mut callback = |val: &str| results.push(val.clone());
// ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^^ `val` escapes the closure body here
for s in ["a","b","c"] {
callback(s);
};
}
Okay, so in working this I found a solution, but I have questions about it.
use std::clone::Clone;
pub fn main() {
let mut results = vec![];
let mut callback = |val: &str| results.push(Clone::clone(&val.to_owned()));
for s in ["a","b","c"] {
callback(s);
};
}
Why does this work but neither val.clone() or &(val.clone()) work?
Why doesn't the complier complain about not knowing the size of results beforehand?
Does this work for any unsized type?
If I call callback(s) twice it works as expected. Why doesn't to_owned() consume the resource?
Is there anything else important here that I (or someone else at my level who might be reading this) am missing?
You've run into bunch misconceptions.
When you call clone on &str, you get another &str. It is useless, as long as any reference implements Copy.
You're trying to insert reference to str with non-static lifetime into a vector with static lifetime. Ask yourself: what would happen, if the object behind the reference will die before this vector? You'll get a vector of dangling references => undefined behavior.
This code will work:
fn main() {
let mut results = vec![]; // results has type `Vec<&'static str>`
let mut callback = |val: &'static str| results.push(val);
for s in ["a","b","c"] {
callback(s);
};
}
You don't need to use std::clone::Clone as long as it's already in std::prelude and accessible in every scope.
When you call to_owned on &str, you actually create a String which has 'static lifetime. So you can insert any String into a Vec. But it is not a reference to the original str anymore. Also you don't need to call clone here.
So this will work too:
pub fn main() {
let mut results = vec![]; // results has type `Vec<String>`
let mut callback = |val: &str| results.push(val.to_owned());
for s in ["a","b","c"] {
callback(s);
};
}

Multiple Mutable Borrows from Struct Hashmap

Running into an ownership issue when attempting to reference multiple values from a HashMap in a struct as parameters in a function call. Here is a PoC of the issue.
use std::collections::HashMap;
struct Resource {
map: HashMap<String, String>,
}
impl Resource {
pub fn new() -> Self {
Resource {
map: HashMap::new(),
}
}
pub fn load(&mut self, key: String) -> &mut String {
self.map.get_mut(&key).unwrap()
}
}
fn main() {
// Initialize struct containing a HashMap.
let mut res = Resource {
map: HashMap::new(),
};
res.map.insert("Item1".to_string(), "Value1".to_string());
res.map.insert("Item2".to_string(), "Value2".to_string());
// This compiles and runs.
let mut value1 = res.load("Item1".to_string());
single_parameter(value1);
let mut value2 = res.load("Item2".to_string());
single_parameter(value2);
// This has ownership issues.
// multi_parameter(value1, value2);
}
fn single_parameter(value: &String) {
println!("{}", *value);
}
fn multi_parameter(value1: &mut String, value2: &mut String) {
println!("{}", *value1);
println!("{}", *value2);
}
Uncommenting multi_parameter results in the following error:
28 | let mut value1 = res.load("Item1".to_string());
| --- first mutable borrow occurs here
29 | single_parameter(value1);
30 | let mut value2 = res.load("Item2".to_string());
| ^^^ second mutable borrow occurs here
...
34 | multi_parameter(value1, value2);
| ------ first borrow later used here
It would technically be possible for me to break up the function calls (using the single_parameter function approach), but it would be more convenient to pass the
variables to a single function call.
For additional context, the actual program where I'm encountering this issue is an SDL2 game where I'm attempting to pass multiple textures into a single function call to be drawn, where the texture data may be modified within the function.
This is currently not possible, without resorting to unsafe code or interior mutability at least. There is no way for the compiler to know if two calls to load will yield mutable references to different data as it cannot always infer the value of the key. In theory, mutably borrowing both res.map["Item1"] and res.map["Item2"] would be fine as they would refer to different values in the map, but there is no way for the compiler to know this at compile time.
The easiest way to do this, as already mentioned, is to use a structure that allows interior mutability, like RefCell, which typically enforces the memory safety rules at run-time before returning a borrow of the wrapped value. You can also work around the borrow checker in this case by dealing with mut pointers in unsafe code:
pub fn load_many<'a, const N: usize>(&'a mut self, keys: [&str; N]) -> [&'a mut String; N] {
// TODO: Assert that keys are distinct, so that we don't return
// multiple references to the same value
keys.map(|key| self.load(key) as *mut _)
.map(|ptr| unsafe { &mut *ptr })
}
Rust Playground
The TODO is important, as this assertion is the only way to ensure that the safety invariant of only having one mutable reference to any value at any time is upheld.
It is, however, almost always better (and easier) to use a known safe interior mutation abstraction like RefCell rather than writing your own unsafe code.

Discerning lifetimes understanding the move keyword

I've been playing around with AudioUnit via Rust and the Rust library coreaudio-rs. Their example seems to work well:
extern crate coreaudio;
use coreaudio::audio_unit::{AudioUnit, IOType};
use coreaudio::audio_unit::render_callback::{self, data};
use std::f32::consts::PI;
struct Iter {
value: f32,
}
impl Iterator for Iter {
type Item = [f32; 2];
fn next(&mut self) -> Option<[f32; 2]> {
self.value += 440.0 / 44_100.0;
let amp = (self.value * PI * 2.0).sin() as f32 * 0.15;
Some([amp, amp])
}
}
fn main() {
run().unwrap()
}
fn run() -> Result<(), coreaudio::Error> {
// 440hz sine wave generator.
let mut samples = Iter { value: 0.0 };
//let buf: Vec<[f32; 2]> = vec![[0.0, 0.0]];
//let mut samples = buf.iter();
// Construct an Output audio unit that delivers audio to the default output device.
let mut audio_unit = try!(AudioUnit::new(IOType::DefaultOutput));
// Q: What is this type?
let callback = move |args| {
let Args { num_frames, mut data, .. } = args;
for i in 0..num_frames {
let sample = samples.next().unwrap();
for (channel_idx, channel) in data.channels_mut().enumerate() {
channel[i] = sample[channel_idx];
}
}
Ok(())
};
type Args = render_callback::Args<data::NonInterleaved<f32>>;
try!(audio_unit.set_render_callback(callback));
try!(audio_unit.start());
std::thread::sleep(std::time::Duration::from_millis(30000));
Ok(())
}
However, changing it up a little bit to load via a buffer doesn't work as well:
extern crate coreaudio;
use coreaudio::audio_unit::{AudioUnit, IOType};
use coreaudio::audio_unit::render_callback::{self, data};
fn main() {
run().unwrap()
}
fn run() -> Result<(), coreaudio::Error> {
let buf: Vec<[f32; 2]> = vec![[0.0, 0.0]];
let mut samples = buf.iter();
// Construct an Output audio unit that delivers audio to the default output device.
let mut audio_unit = try!(AudioUnit::new(IOType::DefaultOutput));
// Q: What is this type?
let callback = move |args| {
let Args { num_frames, mut data, .. } = args;
for i in 0..num_frames {
let sample = samples.next().unwrap();
for (channel_idx, channel) in data.channels_mut().enumerate() {
channel[i] = sample[channel_idx];
}
}
Ok(())
};
type Args = render_callback::Args<data::NonInterleaved<f32>>;
try!(audio_unit.set_render_callback(callback));
try!(audio_unit.start());
std::thread::sleep(std::time::Duration::from_millis(30000));
Ok(())
}
It says, correctly so, that buf only lives until the end of run and does not live long enough for the audio unit—which makes sense, because "borrowed value must be valid for the static lifetime...".
In any case, that doesn't bother me; I can modify the iterator to load and read from the buffer just fine. However, it does raise some questions:
Why does the Iter { value: 0.0 } have the 'static lifetime?
If it doesn't have the 'static lifetime, why does it say the borrowed value must be valid for the 'static lifetime?
If it does have the 'static lifetime, why? It seems like it would be on the heap and closed on by callback.
I understand that the move keyword allows moving inside the closure, which doesn't help me understand why it interacts with lifetimes. Why can't it move the buffer? Do I have to move both the buffer and the iterator into the closure? How would I do that?
Over all this, how do I figure out the expected lifetime without trying to be a compiler myself? It doesn't seem like guessing and compiling is always a straightforward method to resolving these issues.
Why does the Iter { value: 0.0 } have the 'static lifetime?
It doesn't; only references have lifetimes.
why does it say the borrowed value must be valid for the 'static lifetime
how do I figure out the expected lifetime without trying to be a compiler myself
Read the documentation; it tells you the restriction:
fn set_render_callback<F, D>(&mut self, f: F) -> Result<(), Error>
where
F: FnMut(Args<D>) -> Result<(), ()> + 'static, // <====
D: Data
This restriction means that any references inside of F must live at least as long as the 'static lifetime. Having no references is also acceptable.
All type and lifetime restrictions are expressed at the function boundary — this is a hard rule of Rust.
I understand that the move keyword allows moving inside the closure, which doesn't help me understand why it interacts with lifetimes.
The only thing that the move keyword does is force every variable directly used in the closure to be moved into the closure. Otherwise, the compiler tries to be conservative and move in references/mutable references/values based on the usage inside the closure.
Why can't it move the buffer?
The variable buf is never used inside the closure.
Do I have to move both the buffer and the iterator into the closure? How would I do that?
By creating the iterator inside the closure. Now buf is used inside the closure and will be moved:
let callback = move |args| {
let mut samples = buf.iter();
// ...
}
It doesn't seem like guessing and compiling is always a straightforward method to resolving these issues.
Sometimes it is, and sometimes you have to think about why you believe the code to be correct and why the compiler states it isn't and come to an understanding.

Resources