Multiple self borrows [duplicate] - rust

I'm writing some code in Rust that connects to a remote server, and depending on the messages sent by that server, computes some statistics or executes actions based on these statistics. But this is more of a learning project for me and I've run into an issue.
Here is the code that I have reduced to a bare minimum to reproduce the problem :
// Repro code for error[E0502]: cannot borrow `*self` as mutable because `self.server` is also borrowed as immutable
use std::collections::HashMap;
struct ServerReader {
server: Vec<u32>, // A vec for demo purposes, but please imagine this is a server object
counters: HashMap<u32, usize>,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6), // Filling my "server" with some messages
counters: HashMap::new(),
}
}
fn run(&mut self) {
println!("Connecting..."); // ... here there should be some code to connect to the server ...
for message in self.server.iter() { // We wait for the network messages sent by the server, and process them as they come
// ----------- immutable borrow occurs here
println!("Received {}", message);
self.process_message(*message); // HOW
// ^^^^ mutable borrow occurs here
}
// - immutable borrow ends here
println!("Disconnected");
}
fn process_message(&mut self, message: u32) {
// Please imagine that this function contains complex stuff
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
fn main() {
let mut reader = ServerReader::new();
reader.run();
println!("Done");
}
While I think I understand why the compiler is unhappy, I'm struggling to come up with a solution. I cannot manipulate my structure outside of the loop, since I have to work while connected and listening to the server. I also could put everything directly in the loop and not call any method, but I don't want to end up with a 1000 line loop (and I'd prefer to understand what an actual solution would look like).

As you've worked out, you can't call a &mut self method while you're borrowing part of self, so you need to restructure somehow.
The way I would do it is to split the state needed by process_message into a separate type (in your example that's basically the HashMap, but in the real application it's likely to contain more), and move the method to that type. This works because you can separately borrow fields from a struct.
struct SomeState {
counters: HashMap<u32, usize>,
}
impl SomeState {
pub fn new() -> SomeState {
SomeState {
counters: HashMap::new(),
}
}
fn process_message(&mut self, message: u32) {
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
struct ServerReader {
server: Vec<u32>,
state: SomeState,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6),
state: SomeState::new(),
}
}
fn run(&mut self) {
println!("Connecting...");
for message in self.server.iter() {
println!("Received {}", message);
self.state.process_message(*message);
}
println!("Disconnected");
}
}
An alternative (which may or may not be possible in your real example) would be to avoid borrowing in the loop, making it more like:
loop {
// if next_message() returns an owned message, ie not still borrowing
// self
let message = self.next_message();
// now no borrow left
self.process_message(message);
}

Given that you don't need the full ServerReader for processing a message, you could make process_message a free function and just pass &mut self.counters to it. Then you have disjoint borrows of server and counters, which is fine.
Or if your non-server part of ServerReader is larger, extract that into its own struct, and make process_message an impl method of that struct.

In order to allow mutability in an Iterator, you should use iter_mut() and work on mutable references (&mut message). Then, to avoid the additional borrow, you could just perform the addition in the body of the loop:
for &mut message in self.server.iter_mut() {
println!("Received {}", message);
*self.counters.entry(message).or_insert(0) += 1;
}

Related

How to solve lifetime inside for_each

I try to get the following code to compile.
Due to the buffer it wanted more explicit lifetimes inserted. So I tried that but still
cannot figure out yet how to make it also work in for_each.
use std::{sync::mpsc::Sender, ffi::OsStr};
use inotify::{Event, EventMask, Inotify, WatchMask};
pub struct Watcher<'a> {
buffer: [u8; 1024],
sender: Sender<Event<&'a OsStr>>,
}
impl <'a> Watcher<'a> {
pub fn new(sender: Sender<Event<&OsStr>>) -> Watcher{
Watcher{
buffer: [0; 1024],
sender: sender,
}
}
pub fn watch_for_led_directories(&mut self, dir: String) {
let sender = self.sender.clone();
let mut inotify = Inotify::init().expect("Error while initializing inotify instance");
// Watch for modify and close events.
inotify
.add_watch(dir, WatchMask::CREATE | WatchMask::DELETE)
.expect("Failed to add file watch");
// Read events that were added with `add_watch` above.
let events = inotify
.read_events_blocking(&mut self.buffer)
.expect("Error while reading events");
events.filter(|e| -> bool { !e.mask.intersects(EventMask::ISDIR) }).for_each(|e| sender.send(e).unwrap());
}
}
EDIT:
I ended up giving up on no-copy of strings:
use std::{ffi::OsStr, sync::mpsc::Sender};
use inotify::{Event, EventMask, Inotify, WatchMask};
pub struct Watcher {
buffer: [u8; 1024],
sender: Sender<Event<String>>,
}
impl Watcher {
pub fn new(sender: Sender<Event<String>>) -> Watcher {
Watcher {
buffer: [0; 1024],
sender: sender,
}
}
pub fn watch_for_led_directories(&mut self, dir: String) {
let sender = self.sender.clone();
let mut inotify = Inotify::init().expect("Error while initializing inotify instance");
// Watch for modify and close events.
inotify
.add_watch(dir, WatchMask::CREATE | WatchMask::DELETE)
.expect("Failed to add file watch");
// Read events that were added with `add_watch` above.
let events = inotify
.read_events_blocking(&mut self.buffer)
.expect("Error while reading events");
let filter = |e: &Event<&OsStr>| -> bool {
!e.mask.intersects(EventMask::ISDIR) || e.name.is_none()
};
events.filter(filter).for_each(|e| {
sender
.send(Event {
cookie: e.cookie,
mask: e.mask,
name: Some(e.name.unwrap().to_str().expect("Expected valid unicode string").into()),
wd: e.wd,
})
.unwrap()
});
}
}
The issue is that while self has the lifetime of 'a, &mut self in watch_for_led_directories() has a different lifetime.
inotify.read_events_blocking() returns events that has the same lifetime as &mut self.buffer, which, as a mutable borrow, has the lifetime of &mut self. But sender is bound to life time 'a. Therefore compiler cannot reconcile these two lifetimes.
One way to solve it is to provide an external buffer, instead of having it owned by Watcher, e.g.:
pub fn watch_for_led_directories(&self, buffer: &'a mut [u8], dir: String) {
...
let events: Events = inotify
.read_events_blocking(buffer)
.expect("Error while reading events");
...
Compiler derives the lifetime of events from buffer and therefore it will be 'a.
To use this method the externally allocated buffer must live at least as long as the Watcher object, which should not be too difficult to arrange.

Hashmap multiple mutable borrow issue after reference drop

I am trying to pass around a HashMap which stores values through a set of nested enums/structs. The problem of multiple mutability happens during iteration, even all references should be dropped.
The general idea is to have a vector of values, iterate through them and simplify them, keeping track of them within the HashMap. There are two stages of simplification.
The general flow looks something like
run(Vec<ComplexVal>)
-for each val->
val.fix_complex(holder)
-for each `smp` SimpleVal in val->
basicval = Simplifier::step(smp, holder)
holder.insert("name", basicval)
But the problem is that the holder is borrowed mutably in each stage, and there isn't supposed to be any reference from the ComplexVal to the holder and since the borrowchecker doesn't like multiple borrows, it fails.
Full playground snippet: here
It happens in this snippet:
pub fn run(&mut self, mut vals: Vec<ComplexVal>) {
let mut holder = Holder{hold:HashMap::new()};
// .. setup holder code omitted
let len = vals.len();
for _ in 0..len {
let mut val = vals.remove(0); // remove from vec, should drop after running
println!("Running {:?}", val);
match val {
ComplexVal::Cmplx1(mut c) => {
c.fix_complex(&mut holder)
},
//... more cases of different types of values omitted for simplicity
}
// val *should* be dropped here, and therefore the mutable borrow of holder?
}
println!("Holder: {:?}", holder);
}
}
The only thing I can think of is that it somehow is related to the BasicVal::Ref(&BasicVal) value when created.
I need to return a reference of type &BasicVal so I can't use a regular fn() -> &BasicVal as the reference would be dangling, so I pass a ret value which is to be modified and used as the storage for the return value.
I have also tried just returning the enum BasicVal::Ref(&BasicVal), but run into the same mutability issues.
The example below is a much more simple version which (sort of) demonstrates the same error, just thought I'd include this context in case someone has another idea on how to implement this which wouldn't have these issues
Code (edited)
Updated playground link
Edit: I made a mistake in not needing the lifetimes of both holder and ret to explicitly be the same, so I have made an updated example for it
use std::borrow::BorrowMut;
///////////////////////////////
use std::cell::{RefCell, RefMut};
use std::collections::HashMap;
#[derive(Debug)]
enum BasicVal<'a> {
Ref(&'a BasicVal<'a>),
Val1(BasicStruct),
}
#[derive(Debug)]
struct Holder<'b> {
hold: HashMap<String, RefCell<BasicVal<'b>>>,
}
#[derive(Debug)]
struct BasicStruct {
val: i32,
}
impl<'a> BasicVal<'a> {
pub fn empty() -> Self { BasicVal::Val1(BasicStruct { val: 0 }) }
}
// must match sig of modify_val_ref
fn modify_val<'f>(holder: &'f mut Holder<'f>, mut ret: RefMut<BasicVal<'f>>) {
*ret = BasicVal::Val1(BasicStruct { val: 5 });
}
// must match sig of modify_val
fn modify_val_ref<'f>(holder: &'f mut Holder<'f>, mut ret: RefMut<BasicVal<'f>>) {
ret = holder.hold.get("reference_val").unwrap().borrow_mut();
}
fn do_modify<'f>(holder: &'f mut Holder<'f>) {
let mut v = RefCell::new(BasicVal::empty());
println!("Original {:?}", v);
modify_val(holder, v.borrow_mut());
holder.hold.insert("Data".to_string(), v);
println!("Modified {:?}", holder.hold.get("Data"));
}
pub fn test_dropborrow() {
let mut holder = Holder { hold: HashMap::new() };
holder.hold.insert(
"reference_val".to_string(),
RefCell::new(BasicVal::Val1(BasicStruct { val: 8 })),
);
do_modify(&mut holder);
}
pub fn main() {
test_dropborrow();
}
Edit: Using just the holder for a temp return value gives me a multiple mutable borrow issue, so that workaround doesn't work. I have also tried it with a RefCell with the same issue.
fn modify_val<'f>(holder: &'f mut Holder<'f>) {
holder.hold.insert("$return".to_string(), BasicVal::Val1(BasicStruct{val: 5}));
}
fn do_modify<'f>(holder: &'f mut Holder<'f>) {
modify_val(holder);
let mut v = holder.hold.remove("$return").unwrap();
holder.hold.insert("Data".to_string(), v);
println!("Modified {:?}", v);
}
Error:
935 | fn do_modify<'f>(holder: &'f mut Holder<'f>) {
| -- lifetime `'f` defined here
936 |
937 | modify_val(holder);
| ------------------
| | |
| | first mutable borrow occurs here
| argument requires that `*holder` is borrowed for `'f`
938 | let mut v = holder.hold.remove("$return").unwrap();
| ^^^^^^^^^^^ second mutable borrow occurs here
Any help is greatly appreciated!!!
Figured it out, essentially the BasicVal<'a> was causing Holder to mutably borrow itself in successive iterations of the loop, so removing the lifetime was pretty much the only solution

How to return a reference to a global vector or an internal Option?

I'm trying to create a method that can return a reference to Data that is either in a constant global array or inside an Option in an item. The lifetimes are certainly different, but it's safe to assume that the lifetime of the data is at least as long as the lifetime of the item. While doing this, I expected the compiler to warn if I did anything wrong, but it's instead generating wrong instructions and the program is crashing with SIGILL.
Concretely speaking, I have the following code failing in Rust 1.27.2:
#[derive(Debug)]
pub enum Type {
TYPE1,
TYPE2,
}
#[derive(Debug)]
pub struct Data {
pub ctype: Type,
pub int: i32,
}
#[derive(Debug)]
pub struct Entity {
pub idata: usize,
pub modifier: Option<Data>,
}
impl Entity {
pub fn data(&self) -> &Data {
if self.modifier.is_none() {
&DATA[self.idata]
} else {
self.modifier.as_ref().unwrap()
}
}
}
pub const DATA: [Data; 1] = [Data {
ctype: Type::TYPE2,
int: 1,
}];
fn main() {
let mut itemvec = vec![Entity {
idata: 0,
modifier: None,
}];
eprintln!("vec[0]: {:p} = {:?}", &itemvec[0], itemvec[0]);
eprintln!("removed item 0");
let item = itemvec.remove(0);
eprintln!("item: {:p} = {:?}", &item, item);
eprintln!("modifier: {:p} = {:?}", &item.modifier, item.modifier);
eprintln!("DATA: {:p} = {:?}", &DATA[0], DATA[0]);
let itemdata = item.data();
eprintln!("itemdata: {:p} = {:?}", itemdata, itemdata);
}
Complete code
I can't understand what I'm doing wrong. Why isn't the compiler generating a warning? Is it the removal of the (non-copy) item of the vector? Is it the ambiguous lifetimes?
How to return a reference to a global vector or an internal Option?
By using Option::unwrap_or_else:
impl Entity {
pub fn data(&self) -> &Data {
self.modifier.as_ref().unwrap_or_else(|| &DATA[self.idata])
}
}
but it's instead generating wrong instructions and the program is crashing with SIGILL
The code in your question does not have this behavior on macOS with Rust 1.27.2 or 1.28.0. On Ubuntu I see an issue when running the program in Valgrind, but the problem goes away in Rust 1.28.0.
See also:
Why should I prefer `Option::ok_or_else` instead of `Option::ok_or`?
What is this unwrap thing: sometimes it's unwrap sometimes it's unwrap_or

Manipulating an object from inside a loop that borrows it

I'm writing some code in Rust that connects to a remote server, and depending on the messages sent by that server, computes some statistics or executes actions based on these statistics. But this is more of a learning project for me and I've run into an issue.
Here is the code that I have reduced to a bare minimum to reproduce the problem :
// Repro code for error[E0502]: cannot borrow `*self` as mutable because `self.server` is also borrowed as immutable
use std::collections::HashMap;
struct ServerReader {
server: Vec<u32>, // A vec for demo purposes, but please imagine this is a server object
counters: HashMap<u32, usize>,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6), // Filling my "server" with some messages
counters: HashMap::new(),
}
}
fn run(&mut self) {
println!("Connecting..."); // ... here there should be some code to connect to the server ...
for message in self.server.iter() { // We wait for the network messages sent by the server, and process them as they come
// ----------- immutable borrow occurs here
println!("Received {}", message);
self.process_message(*message); // HOW
// ^^^^ mutable borrow occurs here
}
// - immutable borrow ends here
println!("Disconnected");
}
fn process_message(&mut self, message: u32) {
// Please imagine that this function contains complex stuff
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
fn main() {
let mut reader = ServerReader::new();
reader.run();
println!("Done");
}
While I think I understand why the compiler is unhappy, I'm struggling to come up with a solution. I cannot manipulate my structure outside of the loop, since I have to work while connected and listening to the server. I also could put everything directly in the loop and not call any method, but I don't want to end up with a 1000 line loop (and I'd prefer to understand what an actual solution would look like).
As you've worked out, you can't call a &mut self method while you're borrowing part of self, so you need to restructure somehow.
The way I would do it is to split the state needed by process_message into a separate type (in your example that's basically the HashMap, but in the real application it's likely to contain more), and move the method to that type. This works because you can separately borrow fields from a struct.
struct SomeState {
counters: HashMap<u32, usize>,
}
impl SomeState {
pub fn new() -> SomeState {
SomeState {
counters: HashMap::new(),
}
}
fn process_message(&mut self, message: u32) {
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
struct ServerReader {
server: Vec<u32>,
state: SomeState,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6),
state: SomeState::new(),
}
}
fn run(&mut self) {
println!("Connecting...");
for message in self.server.iter() {
println!("Received {}", message);
self.state.process_message(*message);
}
println!("Disconnected");
}
}
An alternative (which may or may not be possible in your real example) would be to avoid borrowing in the loop, making it more like:
loop {
// if next_message() returns an owned message, ie not still borrowing
// self
let message = self.next_message();
// now no borrow left
self.process_message(message);
}
Given that you don't need the full ServerReader for processing a message, you could make process_message a free function and just pass &mut self.counters to it. Then you have disjoint borrows of server and counters, which is fine.
Or if your non-server part of ServerReader is larger, extract that into its own struct, and make process_message an impl method of that struct.
In order to allow mutability in an Iterator, you should use iter_mut() and work on mutable references (&mut message). Then, to avoid the additional borrow, you could just perform the addition in the body of the loop:
for &mut message in self.server.iter_mut() {
println!("Received {}", message);
*self.counters.entry(message).or_insert(0) += 1;
}

Threaded calling of functions in a vector

I have an EventRegistry which people can use to register event listeners. It then calls the appropriate listeners when an event is broadcast. But, when I try to multithread it, it doesn't compile. How would I get this code working?
use std::collections::HashMap;
use std::thread;
struct EventRegistry<'a> {
event_listeners: HashMap<&'a str, Vec<Box<Fn() + Sync>>>
}
impl<'a> EventRegistry<'a> {
fn new() -> EventRegistry<'a> {
EventRegistry {
event_listeners: HashMap::new()
}
}
fn add_event_listener(&mut self, event: &'a str, listener: Box<Fn() + Sync>) {
match self.event_listeners.get_mut(event) {
Some(listeners) => {
listeners.push(listener);
return
},
None => {}
};
let mut listeners = Vec::with_capacity(1);
listeners.push(listener);
self.event_listeners.insert(event, listeners);
}
fn broadcast_event(&mut self, event: &str) {
match self.event_listeners.get(event) {
Some(listeners) => {
for listener in listeners.iter() {
let _ = thread::spawn(|| {
listener();
});
}
}
None => {}
}
}
}
fn main() {
let mut main_registry = EventRegistry::new();
main_registry.add_event_listener("player_move", Box::new(|| {
println!("Hey, look, the player moved!");
}));
main_registry.broadcast_event("player_move");
}
Playpen (not sure if it's minimal, but it produces the error)
If I use thread::scoped, it works too, but that's unstable, and I think it only works because it immediately joins back to the main thread.
Updated question
I meant "call them in their own thread"
The easiest thing to do is avoid the Fn* traits, if possible. If you know that you are only using full functions, then it's straightforward:
use std::thread;
fn a() { println!("a"); }
fn b() { println!("b"); }
fn main() {
let fns = vec![a as fn(), b as fn()];
for &f in &fns {
thread::spawn(move || f());
}
thread::sleep_ms(500);
}
If you can't use that for some reason (like you want to accept closures), then you will need to be a bit more explicit and use Arc:
use std::thread;
use std::sync::Arc;
fn a() { println!("a"); }
fn b() { println!("b"); }
fn main() {
let fns = vec![
Arc::new(Box::new(a) as Box<Fn() + Send + Sync>),
Arc::new(Box::new(b) as Box<Fn() + Send + Sync>),
];
for f in &fns {
let my_f = f.clone();
thread::spawn(move || my_f());
}
thread::sleep_ms(500);
}
Here, we can create a reference-counted trait object. We can clone the trait object (increasing the reference count) each time we spawn a new thread. Each thread gets its own reference to the trait object.
If I use thread::scoped, it works too
thread::scoped is pretty awesome; it's really unfortunate that it needed to be marked unstable due to some complex interactions that weren't the best.
One of the benefits of a scoped thread is that the thread is guaranteed to end by a specific time: when the JoinGuard is dropped. That means that scoped threads are allowed to contain non-'static references, so long as those references last longer than the thread!
A spawned thread has no such guarantees about how long they live; these threads may live "forever". Any references they take must also live "forever", thus the 'static restriction.
This serves to explain your original problem. You have a vector with a non-'static lifetime, but you are handing references that point into that vector to the thread. If the vector were to be deallocated before the thread exited, you could attempt to access undefined memory, which leads to crashes in C or C++ programs. This is Rust helping you out!
Original question
Call functions in vector without consuming them
The answer is that you just call them:
fn a() { println!("a"); }
fn b() { println!("b"); }
fn main() {
let fns = vec![Box::new(a) as Box<Fn()>, Box::new(b) as Box<Fn()>];
fns[0]();
fns[1]();
fns[0]();
fns[1]();
}
Playpen

Resources