Storing a boxed closure which references an object in that object - rust

I'm trying to implement a console system for the game I'm writing and have found a fairly simple system: I define a Console object that stores commands as boxed closures (specifically Box<FnMut + 'a> for some 'a). This works for any component of the engine so long as the Console is created before anything else.
Unfortunately, this prevents me from adding commands that modify the Console itself, which means I can't create commands that simply print text or define other variables or commands. I've written a small example that replicates the error:
use std::cell::Cell;
struct Console<'a> {
cmds: Vec<Box<FnMut() + 'a>>,
}
impl<'a> Console<'a> {
pub fn println<S>(&self, msg: S)
where S: AsRef<str>
{
println!("{}", msg.as_ref());
}
pub fn add_cmd(&mut self, cmd: Box<FnMut() + 'a>) {
self.cmds.push(cmd);
}
}
struct Example {
val: Cell<i32>,
}
fn main() {
let ex = Example {
val: Cell::new(0),
};
let mut con = Console {
cmds: Vec::new(),
};
// this works
con.add_cmd(Box::new(|| ex.val.set(5)));
(con.cmds[0])();
// this doesn't
let cmd = Box::new(|| con.println("Hello, world!"));
con.add_cmd(cmd);
(con.cmds[1])();
}
And the error:
error: `con` does not live long enough
--> console.rs:34:31
|
34 | let cmd = Box::new(|| con.println("Hello, world!"));
| -- ^^^ does not live long enough
| |
| capture occurs here
35 | con.add_cmd(cmd);
36 | }
| - borrowed value dropped before borrower
|
= note: values in a scope are dropped in the opposite order they are created
error: aborting due to previous error
Is there a workaround for this, or a better system I should look into? This is on rustc 1.18.0-nightly (53f4bc311 2017-04-07).

This is one of those fairly tricky resource borrowing conundrums that the compiler could not allow. Basically, we have a Console that owns multiple closures, which in turn capture an immutable reference to the same console. This means two constraints:
Since Console owns the closures, they will live for as long as the console itself, and the inner vector will drop them right after Console is dropped.
At the same time, each closure must not outlive Console, because otherwise we would end up with dangling references to the console.
It may seem harmless from the fact that the console and respective closures go out of scope at once. However, the drop method follows a strict order here: first the console, then the closures.
Not to mention of course, that if you wish for closures to freely apply modifications to the console without interior mutability, you would have to mutably borrow it, which cannot be done over multiple closures.
An approach to solving the problem is to separate the two: let the console not own the closures, instead having them in a separate registry, and let the closures only borrow the console when calling the closure.
This can be done by passing the console as an argument to the closures and moving the closure vector to another object (Playground):
use std::cell::Cell;
struct CommandRegistry<'a> {
cmds: Vec<Box<Fn(&mut Console) + 'a>>,
}
impl<'a> CommandRegistry<'a> {
pub fn add_cmd(&mut self, cmd: Box<Fn(&mut Console) + 'a>) {
self.cmds.push(cmd);
}
}
struct Console {
}
impl Console {
pub fn println<S>(&mut self, msg: S)
where S: AsRef<str>
{
println!("{}", msg.as_ref());
}
}
struct Example {
val: Cell<i32>,
}
fn main() {
let ex = Example {
val: Cell::new(0),
};
let mut reg = CommandRegistry{ cmds: Vec::new() };
let mut con = Console {};
// this works
reg.add_cmd(Box::new(|_: &mut Console| ex.val.set(5)));
(reg.cmds[0])(&mut con);
// and so does this now!
let cmd = Box::new(|c: &mut Console| c.println("Hello, world!"));
reg.add_cmd(cmd);
(reg.cmds[1])(&mut con);
}
I also took the liberty of making closures accept a mutable reference. No conflicts emerge here because we are no longer borrowing the console that was already borrowed when fetching the borrowing closure. This way, the closures can also outlive the console.

Related

Callbacks that mutate local state

I have two structs:
Client, which stores a callback and calls it in response to receiving new data. As an example, you can think of this as a websocket client, and we want to provide a hook for incoming messages.
BusinessLogic, which wants to hold a Client initialized with a callback that will update its local value in response to changes that the Client sees.
After following compiler hints, I arrived at the following minimal example:
Rust playground link
use rand::Rng;
struct Client<'cb> {
callback: Box<dyn FnMut(i64) + 'cb>,
}
impl<'cb> Client<'cb> {
fn do_thing(&mut self) {
// does stuff
let value = self._get_new_value();
// does more stuff
(self.callback)(value);
// does even more stuff
}
fn _get_new_value(&self) -> i64 {
let mut rng = rand::thread_rng();
rng.gen()
}
}
struct BusinessLogic<'cb> {
value: Option<i64>,
client: Option<Client<'cb>>,
}
impl<'cb> BusinessLogic<'cb> {
fn new() -> Self {
Self {
value: None,
client: None,
}
}
fn subscribe(&'cb mut self) {
self.client = Some(Client {
callback: Box::new(|value| {
self.value = Some(value);
})
})
}
}
fn main() {
let mut bl = BusinessLogic::new();
bl.subscribe();
println!("Hello, world!");
}
Problem is, I am still getting the following compiler error:
Compiling playground v0.0.1 (/playground)
error[E0597]: `bl` does not live long enough
--> src/main.rs:51:5
|
51 | bl.subscribe();
| ^^^^^^^^^^^^^^ borrowed value does not live long enough
...
54 | }
| -
| |
| `bl` dropped here while still borrowed
| borrow might be used here, when `bl` is dropped and runs the destructor for type `BusinessLogic<'_>`
For more information about this error, try `rustc --explain E0597`.
error: could not compile `playground` due to previous error
I understand why I'm seeing this error: the call to subscribe uses a borrow of bl with a lifetime of 'cb, which is not necessarily contained within the scope of main(). However, I don't see how to resolve this issue. Won't I always need to provide a lifetime for the callback stored in Client, which will end up bleeding through my code in the form of 'cb lifetime annotations?
More generally, I'm interested in understanding what is the canonical way of solving this callback/hook problem in Rust. I'm open to designs different from the one I have proposed, and if there are relevant performance concerns for various options, that would be useful to know also.
What you've created is a self-referential structure, which is problematic and not really expressible with references and lifetime annotations. See: Why can't I store a value and a reference to that value in the same struct? for the potential problems and workarounds. Its an issue here because you want to be able to mutate the BusinessLogic in the callback, but since it holds the Client, you can mutate the callback while its running, which is no good.
I would instead suggest that the callback has full ownership of the BusinessLogic which does not directly reference the Client:
use rand::Rng;
struct Client {
callback: Box<dyn FnMut(i64)>,
}
impl Client {
fn do_thing(&mut self) {
let value = rand::thread_rng().gen();
(self.callback)(value);
}
}
struct BusinessLogic {
value: Option<i64>,
}
fn main() {
let mut bl = BusinessLogic {
value: None
};
let mut client = Client {
callback: Box::new(move |value| {
bl.value = Some(value);
})
};
client.do_thing();
println!("Hello, world!");
}
if you need the subscriber to have backwards communication to the Client, you can pass an additional parameter that the callback can mutate, or simply do it via return value
if you need more complicated communication from the Client to the callback, either send a Message enum as the argument, or make the callback a custom trait instead of just FnMut with additional methods
if you need a single BusinessLogic to operate from multiple Clients use Arc+Mutex to allow shared ownership

Multiple Mutable Borrows from Struct Hashmap

Running into an ownership issue when attempting to reference multiple values from a HashMap in a struct as parameters in a function call. Here is a PoC of the issue.
use std::collections::HashMap;
struct Resource {
map: HashMap<String, String>,
}
impl Resource {
pub fn new() -> Self {
Resource {
map: HashMap::new(),
}
}
pub fn load(&mut self, key: String) -> &mut String {
self.map.get_mut(&key).unwrap()
}
}
fn main() {
// Initialize struct containing a HashMap.
let mut res = Resource {
map: HashMap::new(),
};
res.map.insert("Item1".to_string(), "Value1".to_string());
res.map.insert("Item2".to_string(), "Value2".to_string());
// This compiles and runs.
let mut value1 = res.load("Item1".to_string());
single_parameter(value1);
let mut value2 = res.load("Item2".to_string());
single_parameter(value2);
// This has ownership issues.
// multi_parameter(value1, value2);
}
fn single_parameter(value: &String) {
println!("{}", *value);
}
fn multi_parameter(value1: &mut String, value2: &mut String) {
println!("{}", *value1);
println!("{}", *value2);
}
Uncommenting multi_parameter results in the following error:
28 | let mut value1 = res.load("Item1".to_string());
| --- first mutable borrow occurs here
29 | single_parameter(value1);
30 | let mut value2 = res.load("Item2".to_string());
| ^^^ second mutable borrow occurs here
...
34 | multi_parameter(value1, value2);
| ------ first borrow later used here
It would technically be possible for me to break up the function calls (using the single_parameter function approach), but it would be more convenient to pass the
variables to a single function call.
For additional context, the actual program where I'm encountering this issue is an SDL2 game where I'm attempting to pass multiple textures into a single function call to be drawn, where the texture data may be modified within the function.
This is currently not possible, without resorting to unsafe code or interior mutability at least. There is no way for the compiler to know if two calls to load will yield mutable references to different data as it cannot always infer the value of the key. In theory, mutably borrowing both res.map["Item1"] and res.map["Item2"] would be fine as they would refer to different values in the map, but there is no way for the compiler to know this at compile time.
The easiest way to do this, as already mentioned, is to use a structure that allows interior mutability, like RefCell, which typically enforces the memory safety rules at run-time before returning a borrow of the wrapped value. You can also work around the borrow checker in this case by dealing with mut pointers in unsafe code:
pub fn load_many<'a, const N: usize>(&'a mut self, keys: [&str; N]) -> [&'a mut String; N] {
// TODO: Assert that keys are distinct, so that we don't return
// multiple references to the same value
keys.map(|key| self.load(key) as *mut _)
.map(|ptr| unsafe { &mut *ptr })
}
Rust Playground
The TODO is important, as this assertion is the only way to ensure that the safety invariant of only having one mutable reference to any value at any time is upheld.
It is, however, almost always better (and easier) to use a known safe interior mutation abstraction like RefCell rather than writing your own unsafe code.

Rust calling FnMut within implemented Struct

i have the following code snippet which implements some kind of Emitter Struct:
type Callback<'a> = Option<&'a mut dyn FnMut()>;
struct Emitter<'a> {
cb: Callback<'a>
}
impl<'a> Emitter<'a> {
fn emit(&mut self) {
if self.cb.is_some() {
let f = self.cb.unwrap();
f()
}
}
}
fn main() {
let mut cb = || println!("test");
let mut e = Emitter {
cb : Some(&mut cb)
};
e.emit();
}
The emit() function tries to run the saved callback clojure. But i cannot wrap my head around how to run the callback, since the code produces the following error:
--> src/main.rs:11:15
|
11 | let f = self.cb.unwrap();
| ^^^^^^^
| |
| move occurs because `self.cb` has type `Option<&mut dyn FnMut()>`, which does not implement the `Copy` trait
| help: consider borrowing the `Option`'s content: `self.cb.as_ref()`
Appreciate some help :)
Here is the snippet on replit: https://replit.com/#lutzer/RustEmitterTest
What's going on here is that your line
let f = self.cb.unwrap();
would want to move the closure out of the Option enum. This operation consumes that enum, which isn't allowed for things that belong to a struct.
Here is a simpler example to show what I mean:
fn main() {
let an_option = Some(String::from("Woot!");
let the_value = an_option.unwrap();
println!("The value is {}", the_value);
println!("The option is {:?}", an_option); // error here! Can't use an_option any more!!!
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=4a4a3660b68ebada99113db5165b6e76
So if you take ownership of something stored inside the Some part of an Option, via unwrap, then the whole Option gets moved out. You can see that in the signature of unwrap:
pub const fn unwrap(self) -> T
Note how it says self, and not &self or &mut self. That means, after calling unwrap, that the enum gets consumed and cannot be used any more, unless the value inside the Some part can simply be copied (If you replace the String in my example with, say, an integer, it will compile without issue).
The comment by Omer Erden then explains a way around that: Ask the Option to give you a mutable reference instead via as_mut.
Or skip all that directly and use the map method of option, which you can use to do something if the option is Some and just not do anything if it's None.

How to correctly use trait objects , mutably iterating over mutable trait objects container, while mutating container itself?

I am trying to write a packet parser, where basically one builds up a packet by parsing each Layer in the packet. The packet then holds those 'layers' in a vector.
The ~pseudo code~ code with compilation errors is something like the following -
Also added comments below - for each step. I have experimented with RefCell , but could not get that working. Essentially the challenges are enumerated at the end of the code.
The basic pattern is as follows - Get the object of a Layer type (Every Layer type will return a default next object based upon some field in the current layer as a 'boxed trait object'.)
Edit: I am adding a code that's more than a pseudo code - Also added following compilation errors. May be a way to figure out how to fix these errors could solve the problems.!
#[derive(Debug, Default)]
pub struct Packet<'a> {
data: Option<&'a [u8]>,
meta: PacketMetadata,
layers: Vec<Box<dyn Layer<'a>>>,
}
pub trait Layer<'a>: Debug {
fn from_u8<'b>(&mut self, bytes: &'b [u8]) -> Result<(Option<Box<dyn Layer>>, usize), Error>;
}
#[derive(Debug, Default)]
pub struct PacketMetadata {
timestamp: Timestamp,
inface: i8,
len: u16,
caplen: u16,
}
impl<'a> Packet<'a> {
fn from_u8(bytes: &'a [u8], _encap: EncapType) -> Result<Self, Error> {
let mut p = Packet::default();
let eth = ethernet::Ethernet::default();
let mut layer: RefCell<Box<dyn Layer>> = RefCell::new(Box::new(eth));
let mut res: (Option<Box<dyn Layer>>, usize);
let mut start = 0;
loop {
let mut decode_layer = layer.borrow_mut();
// process it
res = decode_layer.from_u8(&bytes[start..])?;
if res.0.is_none() {
break;
}
// if the layer exists, get it in a layer.
let boxed = layer.replace(res.0.unwrap());
start = res.1;
// append the layer to layers.
p.layers.push(boxed);
}
Ok(p)
}
}
Compilation Errors
error[E0515]: cannot return value referencing local variable `decode_layer`
--> src/lib.rs:81:9
|
68 | res = decode_layer.from_u8(&bytes[start..])?;
| ------------ `decode_layer` is borrowed here
...
81 | Ok(p)
| ^^^^^ returns a value referencing data owned by the current function
error[E0515]: cannot return value referencing local variable `layer`
--> src/lib.rs:81:9
|
65 | let mut decode_layer = layer.borrow_mut();
| ----- `layer` is borrowed here
...
81 | Ok(p)
| ^^^^^ returns a value referencing data owned by the current function
error: aborting due to 2 previous errors; 3 warnings emitted
It's not clear why the above errors come. I am using the values returned by the calls. (The 3: warnings shown above can be ignored, they are unused warnings.)
The challenges -
p.layers.last_mut and p.layers.push are simultaneous mutable borrows - not allowed. I could somehow put it behind a RefCell, but how that's not clear.
This code is similar in pattern to syn::token::Tokens, however one basic difference being, there an Enum is used(TokenTree). In the above example I cannot use Enum because the list of protocols to be supported is potentially unbounded.
I cannot use Layer trait without Trait Objects due to the loop construct.
The pattern can be thought of as - mutably iterating over a container of Trait objects while updating the container itself.
Perhaps I am missing something very basic.
The problem with the above code is due to lifetime annotation on the Layer trait. If that lifetime annotation is removed, the above code indeed compiles with a few modifications as posted below -
// Layer Trait definition
pub trait Layer: Debug {
fn from_u8(&mut self, bytes: &[u8]) -> Result<(Option<Box<dyn Layer>>, usize), Error>;
}
impl<'a> Packet<'a> {
fn from_u8(bytes: &'a [u8], _encap: EncapType) -> Result<Self, Error> {
let mut p = Packet::default();
let eth = ethernet::Ethernet::default();
let layer: RefCell<Box<dyn Layer>> = RefCell::new(Box::new(eth));
let mut res: (Option<Box<dyn Layer>>, usize);
let mut start = 0;
loop {
{
// Do a `borrow_mut` in it's own scope, that gets dropped at the end.
let mut decode_layer = layer.borrow_mut();
res = decode_layer.from_u8(&bytes[start..])?;
}
if res.0.is_none() {
// This is just required to push something to the RefCell, that will get dropped anyways.
let fake_boxed = Box::new(FakeLayer {});
let boxed = layer.replace(fake_boxed);
p.layers.push(boxed);
break;
}
// if the layer exists, get it in a layer.
let boxed = layer.replace(res.0.unwrap());
start = res.1;
// append the layer to layers.
p.layers.push(boxed);
}
Ok(p)
}
}

Cannot split a string into string slices with explicit lifetimes because the string does not live long enough

I'm writing a library that should read from something implementing the BufRead trait; a network data stream, standard input, etc. The first function is supposed to read a data unit from that reader and return a populated struct filled mostly with &'a str values parsed from a frame from the wire.
Here is a minimal version:
mod mymod {
use std::io::prelude::*;
use std::io;
pub fn parse_frame<'a, T>(mut reader: T)
where
T: BufRead,
{
for line in reader.by_ref().lines() {
let line = line.expect("reading header line");
if line.len() == 0 {
// got empty line; done with header
break;
}
// split line
let splitted = line.splitn(2, ':');
let line_parts: Vec<&'a str> = splitted.collect();
println!("{} has value {}", line_parts[0], line_parts[1]);
}
// more reads down here, therefore the reader.by_ref() above
// (otherwise: use of moved value).
}
}
use std::io;
fn main() {
let stdin = io::stdin();
let locked = stdin.lock();
mymod::parse_frame(locked);
}
An error shows up which I cannot fix after trying different solutions:
error: `line` does not live long enough
--> src/main.rs:16:28
|
16 | let splitted = line.splitn(2, ':');
| ^^^^ does not live long enough
...
20 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the lifetime 'a as defined on the body at 8:4...
--> src/main.rs:8:5
|
8 | / {
9 | | for line in reader.by_ref().lines() {
10 | | let line = line.expect("reading header line");
11 | | if line.len() == 0 {
... |
22 | | // (otherwise: use of moved value).
23 | | }
| |_____^
The lifetime 'a is defined on a struct and implementation of a data keeper structure because the &str requires an explicit lifetime. These code parts were removed as part of the minimal example.
BufReader has a lines() method which returns Result<String, Err>. I handle errors using expect or match and thus unpack the Result so that the program now has the bare String. This will then be done multiple times to populate a data structure.
Many answers say that the unwrap result needs to be bound to a variable otherwise it gets lost because it is a temporary value. But I already saved the unpacked Result value in the variable line and I still get the error.
How to fix this error - could not get it working after hours trying.
Does it make sense to do all these lifetime declarations just for &str in a data keeper struct? This will be mostly a readonly data structure, at most replacing whole field values. String could also be used, but have found articles saying that String has lower performance than &str - and this frame parser function will be called many times and is performance-critical.
Similar questions exist on Stack Overflow, but none quite answers the situation here.
For completeness and better understanding, following is an excerpt from complete source code as to why lifetime question came up:
Data structure declaration:
// tuple
pub struct Header<'a>(pub &'a str, pub &'a str);
pub struct Frame<'a> {
pub frameType: String,
pub bodyType: &'a str,
pub port: &'a str,
pub headers: Vec<Header<'a>>,
pub body: Vec<u8>,
}
impl<'a> Frame<'a> {
pub fn marshal(&'a self) {
//TODO
println!("marshal!");
}
}
Complete function definition:
pub fn parse_frame<'a, T>(mut reader: T) -> Result<Frame<'a>, io::Error> where T: BufRead {
Your problem can be reduced to this:
fn foo<'a>() {
let thing = String::from("a b");
let parts: Vec<&'a str> = thing.split(" ").collect();
}
You create a String inside your function, then declare that references to that string are guaranteed to live for the lifetime 'a. Unfortunately, the lifetime 'a isn't under your control — the caller of the function gets to pick what the lifetime is. That's how generic parameters work!
What would happen if the caller of the function specified the 'static lifetime? How would it be possible for your code, which allocates a value at runtime, to guarantee that the value lives longer than even the main function? It's not possible, which is why the compiler has reported an error.
Once you've gained a bit more experience, the function signature fn foo<'a>() will jump out at you like a red alert — there's a generic parameter that isn't used. That's most likely going to mean bad news.
return a populated struct filled mostly with &'a str
You cannot possibly do this with the current organization of your code. References have to point to something. You are not providing anywhere for the pointed-at values to live. You cannot return an allocated String as a string slice.
Before you jump to it, no you cannot store a value and a reference to that value in the same struct.
Instead, you need to split the code that creates the String and that which parses a &str and returns more &str references. That's how all the existing zero-copy parsers work. You could look at those for inspiration.
String has lower performance than &str
No, it really doesn't. Creating lots of extraneous Strings is a bad idea, sure, just like allocating too much is a bad idea in any language.
Maybe the following program gives clues for others who also also having their first problems with lifetimes:
fn main() {
// using String und &str Slice
let my_str: String = "fire".to_owned();
let returned_str: MyStruct = my_func_str(&my_str);
println!("Received return value: {ret}", ret = returned_str.version);
// using Vec<u8> und &[u8] Slice
let my_vec: Vec<u8> = "fire".to_owned().into_bytes();
let returned_u8: MyStruct2 = my_func_vec(&my_vec);
println!("Received return value: {ret:?}", ret = returned_u8.version);
}
// using String -> str
fn my_func_str<'a>(some_str: &'a str) -> MyStruct<'a> {
MyStruct {
version: &some_str[0..2],
}
}
struct MyStruct<'a> {
version: &'a str,
}
// using Vec<u8> -> & [u8]
fn my_func_vec<'a>(some_vec: &'a Vec<u8>) -> MyStruct2<'a> {
MyStruct2 {
version: &some_vec[0..2],
}
}
struct MyStruct2<'a> {
version: &'a [u8],
}

Resources