mutation reference for Box closure, but not live long enough - rust

I'm rather new to rust, I have some problems,I don't know how to describe my problem. I think talk is cheap and I want to show my code, but I must add more detail for post.
// programA
struct ClosureTest {
x: u8,
}
type Executor = Box<dyn FnMut() -> ()>;
impl ClosureTest {
fn new(x: u8) -> Self {
ClosureTest { x }
}
}
struct Ftest {}
impl Ftest {
fn exec(&self, mut executor: Executor) {
executor();
}
}
fn receive_test(arg: &mut ClosureTest) {
arg.x = 6;
println!("{}", arg.x);
}
fn main() {
let mut closure_test = ClosureTest::new(5);
let f_test = Ftest {};
let executor = Box::new(|| {
receive_test(&mut closure_test);
});
f_test.exec(executor);
}
I got these errors:
error[E0597]: `closure_test` does not live long enough
--> src/main.rs:30:23
|
29 | let executor = Box::new(|| {
| -- value captured here
30 | receive_test(&mut closure_test);
| ^^^^^^^^^^^^ borrowed value does not live long enough
...
33 | f_test.exec(executor);
| -------- cast requires that `closure_test` is borrowed for `'static`
34 | }
| - `closure_test` dropped here while still borrowed
But if I use mut reference like follows, it works fine.
// programB
struct ClosureTest {
x: u8,
}
type Executor<'a> = &'a mut dyn FnMut() -> ();
impl ClosureTest {
fn new(x: u8) -> Self {
ClosureTest { x }
}
}
struct Ftest {}
impl Ftest {
fn exec(&self, executor: Executor) {
executor();
}
}
fn receive_test(arg: &mut ClosureTest) {
arg.x = 6;
println!("{}", arg.x);
}
fn main() {
let mut closure_test = ClosureTest::new(5);
let f_test = Ftest {};
let mut executor = || {
receive_test(&mut closure_test);
};
f_test.exec(&mut executor);
}
So
What's the reason cause this difference.
Why in sync program "programA" will cause lifetime error?
closure_test is drop before f_test.exec(executor)? or after?

To understand what is happening here you'll need to make a little dive into the Rust Book and learn about lifetimes.
I'll simplify for sake of explanation and comment this:
//type Executor = Box< dyn FnMut() -> ()>;
And change the signature of exec function to this:
fn exec(&self, mut executor: Box<dyn FnMut() -> () >) {
executor();
}
First of all let's notice that if we will comment very last line:
//f_test.exec(executor);
the error is gone. The reason for this is that reference to the closure you created - &Box<dyn FnMut() -> ()> has shorter lifetime than other part of main (it's scoped). It means that when you are creating this closure it exists, but just after the reference to this is dropped. That's why it doesn't complain if you not use it, because it's all okay.
Lifetimes are not variables, they are claims, kinda of similar to generics.
If you would change right now the signature of function exec function to this:
impl Ftest {
fn exec<'a>(&self, mut executor: Box<dyn FnMut() -> () + 'a>) {
executor();
}
}
I would say this:
let's say that lifetime of my function is a,
it would takes some parameters and return Box of function which lifetime would be also a, so it would live as long as I.
By doing it we hinted compiler that the lifetime of closure you'll create should live "in same lifetime" as exec() is executed. That's why even if we bring back the last line of main it would work :D.
There is also an option of adding move keyword:
fn main() {
let mut closure_test = ClosureTest::new(5);
let f_test = Ftest {};
let executor = Box::new(move || {
receive_test(&mut closure_test);
});
//println!("{}", closure_test.x);
f_test.exec(executor);
}
This will work, because by adding move keyword we actually tell that we would rather move closure_test object to our scope, not just a reference, therefore it's not dropped. However you wouldn't be able to use it later as shown if you'll uncomment the line with println!().
I'll be happy if anybody would provide some corrections and more details. This concept in Rust is unique and takes time to fully understand so it would be helpful.

Related

Is there a way to tell rustc when is is being too conservative about certain borrows?

I'm playing around with building a very simple stack based evaluator in Rust and I've come up against an odd situation where I think the borrow checker is being too conservative:
use std::collections::HashMap;
pub type Value = i32;
pub type Result = std::result::Result<(), Error>;
type Op = Box<dyn Fn(&mut Evaluator) -> Result>;
type OpTable = HashMap<String, Op>;
pub struct Evaluator {
stack: Vec<Value>,
ops: OpTable,
}
#[derive(Debug, PartialEq)]
pub enum Error {
DivisionByZero,
StackUnderflow,
UnknownWord,
InvalidWord,
}
impl Evaluator {
fn add(&mut self) -> Result {
if let (Some(x), Some(y)) = (self.stack.pop(), self.stack.pop()) {
self.stack.push(y + x);
Ok(())
} else {
Err(Error::StackUnderflow)
}
}
fn sub(&mut self) -> Result {
if let (Some(x), Some(y)) = (self.stack.pop(), self.stack.pop()) {
self.stack.push(y - x);
Ok(())
} else {
Err(Error::StackUnderflow)
}
}
pub fn new() -> Evaluator {
let stack: Vec<Value> = vec![];
let mut ops: OpTable = HashMap::new();
ops.insert("+".to_string(), Box::new(Evaluator::add));
ops.insert("-".to_string(), Box::new(Evaluator::sub));
Evaluator { stack, ops }
}
pub fn eval(&mut self, input: &str) -> Result {
let symbols = input.split_ascii_whitespace().collect::<Vec<_>>();
// user definition
if let (Some(&":"), Some(&";")) = (symbols.first(), symbols.last()) {
if symbols.len() > 3 {
let statement = symbols[2..symbols.len() - 1].join(" ");
self.ops.insert(
symbols[1].to_string().to_ascii_lowercase(),
Box::new(move |caller: &mut Evaluator| caller.exec(&statement)),
);
return Ok(());
} else {
return Err(Error::InvalidWord);
}
}
self.exec(input)
}
fn exec(&mut self, input: &str) -> Result {
let symbols = input.split_ascii_whitespace().collect::<Vec<_>>();
for sym in symbols {
if let Ok(n) = sym.parse::<i32>() {
self.stack.push(n);
} else {
let s = sym.to_ascii_lowercase();
if let Some(f) = self.ops.get(&s) { // <--------------errors here
f(self)?; // <----------------------------|
} else {
return Err(Error::InvalidWord);
}
}
}
Ok(())
}
}
fn main() {
let mut e = Evaluator::new();
e.eval("1 2 +");
println!("{:?}", e.stack);
e.eval(": plus-1 1 + ;");
e.eval("4 plus-1");
println!("{:?}", e.stack);
}
I'm getting:
error[E0502]: cannot borrow `*self` as mutable because it is also borrowed as immutable
--> src/main.rs:77:21
|
76 | if let Some(f) = self.ops.get(&s) {
| -------- immutable borrow occurs here
77 | f(self)?;
| -^^^^^^
| |
| mutable borrow occurs here
| immutable borrow later used by call
For more information about this error, try `rustc --explain E0502`.
error: could not compile `evaluator` due to previous error
I believe this is because taking part of the hashmap (f) borrows all of self immutably, then I'm passing self mutably to f(). However, there is no real conflict here (I think).
I'm able to get around this by actually removing and reinserting the value:
fn exec(&mut self, input: &str) -> Result {
let symbols = input.split_ascii_whitespace().collect::<Vec<_>>();
for sym in symbols {
if let Ok(n) = sym.parse::<i32>() {
self.stack.push(n);
} else {
let s = sym.to_ascii_lowercase();
if self.ops.contains_key(&s) {
let f = self.ops.remove(&s).unwrap();
if let Err(e) = f(self) {
self.ops.insert(s, f);
return Err(e);
}
self.ops.insert(s, f);
} else {
return Err(Error::InvalidWord);
}
}
}
Ok(())
}
But this feels hacky and is a lot more verbose and inefficient. Am I missing something? Is there a way to tell the compiler the first version is ok?
The compiler is entirely correct, and so is your explanation: The call to get() needs a borrow on self.ops to return an &Op of the same lifetime. You then try to call that FnMut with a mutable borrow of self; this mutable borrow of self aliases with the immutable borrow on self.ops, and in theory it would be possible for an implementation of this FnMut to modify that borrowed Op through self, which is not allowed. The compiler prevented a situation where mutation occurs through an aliased pointer.
This situation often occurs when passing &mut self around, as immutable borrows on members of self which result in more borrows (&self.ops.get() has the same lifetime as &self) "lock" all of self.
While your second example is cumbersome, it is at least correct as proven by the compiler: By removing Op from the hashtable, the FnMut can't reach itself through self anymore, and mutation while aliasing is prevented.
A better approach is usually to avoid taking &mut self as an argument (&mut self as in &mut Executor).

Why does a call to `fn pop(&mut self) -> Result<T, &str>` continue to borrow my data structure?

I am developing some basic data structures to learn the syntax and Rust in general. Here is what I came up with for a stack:
#[allow(dead_code)]
mod stack {
pub struct Stack<T> {
data: Vec<T>,
}
impl<T> Stack<T> {
pub fn new() -> Stack<T> {
return Stack { data: Vec::new() };
}
pub fn pop(&mut self) -> Result<T, &str> {
let len: usize = self.data.len();
if len > 0 {
let idx_to_rmv: usize = len - 1;
let last: T = self.data.remove(idx_to_rmv);
return Result::Ok(last);
} else {
return Result::Err("Empty stack");
}
}
pub fn push(&mut self, elem: T) {
self.data.push(elem);
}
pub fn is_empty(&self) -> bool {
return self.data.len() == 0;
}
}
}
mod stack_tests {
use super::stack::Stack;
#[test]
fn basics() {
let mut s: Stack<i16> = Stack::new();
s.push(16);
s.push(27);
let pop_result = s.pop().expect("");
assert_eq!(s.pop().expect("Empty stack"), 27);
assert_eq!(s.pop().expect("Empty stack"), 16);
let pop_empty_result = s.pop();
match pop_empty_result {
Ok(_) => panic!("Should have had no result"),
Err(_) => {
println!("Empty stack");
}
}
if s.is_empty() {
println!("O");
}
}
}
I get this interesting error:
error[E0502]: cannot borrow `s` as immutable because it is also borrowed as mutable
--> src/main.rs:58:12
|
49 | let pop_empty_result = s.pop();
| - mutable borrow occurs here
...
58 | if s.is_empty() {
| ^ immutable borrow occurs here
...
61 | }
| - mutable borrow ends here
Why can't I just call pop on my mutable struct?
Why does pop borrow the value? If I add a .expect() after it, it is ok, it doesn't trigger that error. I know that is_empty takes an immutable reference, if I switch it to mutable I just get a second mutable borrow.
Your pop function is declared as:
pub fn pop(&mut self) -> Result<T, &str>
Due to lifetime elision, this expands to
pub fn pop<'a>(&'a mut self) -> Result<T, &'a str>
This says that the Result::Err variant is a string that lives as long as the stack you are calling it on. Since the input and output lifetimes are the same, the returned value might be pointing somewhere into the Stack data structure so the returned value must continue to hold the borrow.
If I add a .expect() after it, it is ok, it doesn't trigger that error.
That's because expect consumes the Result, discarding the Err variant without ever putting it into a variable binding. Since that's never stored, the borrow cannot be saved anywhere and it is released.
To solve the problem, you need to have distinct lifetimes between the input reference and output reference. Since you are using a string literal, the easiest solution is to denote that using the 'static lifetime:
pub fn pop(&mut self) -> Result<T, &'static str>
Extra notes:
Don't call return explicitly at the end of the block / method: return Result::Ok(last) => Result::Ok(last).
Result, Result::Ok, and Result::Err are all imported via the prelude, so you don't need to qualify them: Result::Ok(last) => Ok(last).
There's no need to specify types in many cases let len: usize = self.data.len() => let len = self.data.len().
This happens because of lifetimes. When you construct a method which takes a reference the compiler detects that and if no lifetimes are specified it "generates" them:
pub fn pop<'a>(&'a mut self) -> Result<T, &'a str> {
let len: usize = self.data.len();
if len > 0 {
let idx_to_rmv: usize = len - 1;
let last: T = self.data.remove(idx_to_rmv);
return Result::Ok(last);
} else {
return Result::Err("Empty stack");
}
}
This is what compiler sees actually. So, you want to return a static string, then you have to specify the lifetime for a &str explicitly and let the lifetime for the reference to mut self be inferred automatically:
pub fn pop(&mut self) -> Result<T, &'static str> {

How to change the variable from inside Fn closure in Rust?

I have the following code (playground):
struct A {
pub vec: Vec<u64>,
}
impl A {
fn perform_for_all<F: Fn(&mut u64)>(&mut self, f: F) {
for mut i in &mut self.vec {
f(i);
}
}
}
fn main() {
let mut a = A {
vec: vec![1, 3, 44, 2, 4, 5, 6],
};
let mut done = false;
a.perform_for_all(|v| {
println!("value: {:?}", v);
done = true;
});
if !done {
a.perform_for_all(|v| {
println!("value {:?}", v);
});
}
}
The following errors occur:
error[E0594]: cannot assign to `done`, as it is a captured variable in a `Fn` closure
--> src/main.rs:21:9
|
21 | done = true;
| ^^^^^^^^^^^ cannot assign
|
help: consider changing this to accept closures that implement `FnMut`
--> src/main.rs:19:23
|
19 | a.perform_for_all(|v| {
| _______________________^
20 | | println!("value: {:?}", v);
21 | | done = true;
22 | | });
| |_____^
I have a list of loaded objects and a list of objects in a database. I need a function that takes a closure and executes it on the loaded objects and if we don't have the objects in the list, execute it on a list of objects from the database.
That function looks like:
pub fn perform_for_match_with_mark<F>(&mut self, mark: MatchMark, f: F)
where
F: Fn(&mut GameMatch),
{
self.perform_for_all_matches(
|m| {
// runtime list
if let Game::Match(ref mut gm) = *m {
if gm.match_stamp().mark == mark {
f(gm);
}
}
},
None,
);
// if we have called `f` above - don't execute lines below.
let tx = self.match_tx.clone();
GamesDatabase::perform_for_match_with_mark(mark, |ms| {
// database
self.perform_for_all_matches(
|m| {
if let Game::Match(ref gm) = *m {
if gm.match_stamp().id == ms.id {
f(&mut GameMatch::new_with_match_stamp(
tx.clone(),
ms.clone(),
gm.needs_server_set,
gm.server_id,
))
}
}
},
None,
);
});
}
We have to operate on objects from the database only if we were unable to find them in runtime list. That is why I decided to make a variable which says "we already found these objects in the list, leave the database alone".
Change your perform_for_all function to use FnMut instead of Fn:
fn perform_for_all<F>(&mut self, mut f: F)
where
F: FnMut(&mut u64),
{
for mut i in &mut self.vec {
f(&mut i);
}
}
As Peter said, there is some compiler magic going on.
The signature for Fn::call is:
extern "rust-call" fn call(&self, args: Args) -> Self::Output
This takes an immutable reference to self, which is why you can't modify any of the captured variables.
The signature for FnMut::call_mut lets you mutate variables because it takes &mut self:
extern "rust-call" fn call_mut(&mut self, args: Args) -> Self::Output
By changing your closure from Fn to FnMut, you allow it to modify its captured variables, given that the references you pass to it are mutable.
Just to expand a little on SplittyDev's answer.
When you use a closure, the compiler does some magic to let the closure access variables in its environment. Effectively it will create a new struct, whose members are the variables that you tried to access.
It's not exactly this (which won't actually compile), but it's a reasonable approximation conceptually:
struct Closure_1 {
done: bool
}
impl FnMut<&mut u64> for Closure_1 {
fn call_mut(&mut self, v: &mut u64) {
println!("value: {:?}", v);
self.done = true;
}
}
And when you call it, those variables will be borrowed or copied (or moved if you use move keyword).
let mut c1 = Closure_1 { done : done };
a.perform_for_all(|v| c1.call(&v));
done = c1.done;
When the closure modifies its environment, it cannot be a Fn because it must also mutate the variables on itself:
impl Fn<&mut u64> for Closure_1 {
fn call(&self, v: &mut u64) {
println!("value: {:?}", v);
self.done = true; // Can't do this because self is not a mutable ref
}
}
See The Rust Programming Language section on closures and their environment for more information.

borrowed value does not live long enough in this case ( Vec<&Fn(i32) -> i32> )

I am having this error, other times I had something similar and I have been able to solve, in different ways but now is not how to solve in this case:
borrowed value does not live long enough in
I moved the code that fails one more simple, but I can not find the error:
fn main(){
let mut v: Vec<&Fn(i32) -> i32> = Vec::new();
v.push(&ops_code1);
//v.push(&ops_code2);
//v.push(&ops_code3);
}
fn ops_code1(value: i32) -> i32 {
..//
error: borrowed value does not live long enough
v.push(&ops_code1);
play.rust
What you are doing here is creating a Vec of closures. In Rust static functions are treated slightly differently from closures, so when we create the reference a closure is actually created. If we do that after creating the Vec the resulting closure gets a shorter lifetime than the Vec, which is an error. We can instead use a let to create the closure before the Vec, giving a long enough lifetime, outliving the Vec:
fn main() {
let extended = &ops_code1;
let mut v: Vec<&Fn(i32) -> i32> = Vec::new();
// Note that placing it here does not work:
// let extended = &ops_code1;
v.push(extended);
//v.push(&ops_code2);
//v.push(&ops_code3);
}
fn ops_code1(value: i32) -> i32 {
println!("ops_code1 {}", value);
value
}
Rust Playground
However, if you only use static functions - and not closures - the following also works fine, and lets you avoid the extra let:
fn main() {
let mut v: Vec<fn(i32) -> i32> = Vec::new();
v.push(ops_code1);
v.push(ops_code2);
}
fn ops_code1(value: i32) -> i32 {
println!("ops_code1 {}", value);
value
}
fn ops_code2(value: i32) -> i32 {
println!("ops_code2 {}", value);
value
}
Rust Playground
A third option is to use boxed closures, which let's you use both closures and static functions without the extra lets, but with its own trade-offs:
fn main() {
let mut v: Vec<Box<Fn(i32) -> i32>> = Vec::new();
v.push(Box::new(ops_code1));
v.push(Box::new(ops_code2));
for f in v {
f(1);
}
}
fn ops_code1(value: i32) -> i32 {
println!("ops_code1 {}", value);
value
}
fn ops_code2(value: i32) -> i32 {
println!("ops_code2 {}", value);
value
}
Rust Playground

Why does Rust borrow checker reject this code?

I'm getting a Rust compile error from the borrow checker, and I don't understand why. There's probably something about lifetimes I don't fully understand.
I've boiled it down to a short code sample. In main, I want to do this:
fn main() {
let codeToScan = "40 + 2";
let mut scanner = Scanner::new(codeToScan);
let first_token = scanner.consume_till(|c| { ! c.is_digit ()});
println!("first token is: {}", first_token);
// scanner.consume_till(|c| { c.is_whitespace ()}); // WHY DOES THIS LINE FAIL?
}
Trying to call scanner.consume_till a second time gives me this error:
example.rs:64:5: 64:12 error: cannot borrow `scanner` as mutable more than once at a time
example.rs:64 scanner.consume_till(|c| { c.is_whitespace ()}); // WHY DOES THIS LINE FAIL?
^~~~~~~
example.rs:62:23: 62:30 note: previous borrow of `scanner` occurs here; the mutable borrow prevents subsequent moves, borrows, or modification of `scanner` until the borrow ends
example.rs:62 let first_token = scanner.consume_till(|c| { ! c.is_digit ()});
^~~~~~~
example.rs:65:2: 65:2 note: previous borrow ends here
example.rs:59 fn main() {
...
example.rs:65 }
Basically, I've made something like my own iterator, and the equivalent to the "next" method takes &mut self. Because of that, I can't use the method more than once in the same scope.
However, the Rust std library has an iterator which can be used more than once in the same scope, and it also takes a &mut self parameter.
let test = "this is a string";
let mut iterator = test.chars();
iterator.next();
iterator.next(); // This is PERFECTLY LEGAL
So why does the Rust std library code compile, but mine doesn't? (I'm sure the lifetime annotations are at the root of it, but my understanding of lifetimes doesn't lead to me expecting a problem).
Here's my full code (only 60 lines, shortened for this question):
use std::str::{Chars};
use std::iter::{Enumerate};
#[deriving(Show)]
struct ConsumeResult<'lt> {
value: &'lt str,
startIndex: uint,
endIndex: uint,
}
struct Scanner<'lt> {
code: &'lt str,
char_iterator: Enumerate<Chars<'lt>>,
isEof: bool,
}
impl<'lt> Scanner<'lt> {
fn new<'lt>(code: &'lt str) -> Scanner<'lt> {
Scanner{code: code, char_iterator: code.chars().enumerate(), isEof: false}
}
fn assert_not_eof<'lt>(&'lt self) {
if self.isEof {fail!("Scanner is at EOF."); }
}
fn next(&mut self) -> Option<(uint, char)> {
self.assert_not_eof();
let result = self.char_iterator.next();
if result == None { self.isEof = true; }
return result;
}
fn consume_till<'lt>(&'lt mut self, quit: |char| -> bool) -> ConsumeResult<'lt> {
self.assert_not_eof();
let mut startIndex: Option<uint> = None;
let mut endIndex: Option<uint> = None;
loop {
let should_quit = match self.next() {
None => {
endIndex = Some(endIndex.unwrap() + 1);
true
},
Some((i, ch)) => {
if startIndex == None { startIndex = Some(i);}
endIndex = Some(i);
quit (ch)
}
};
if should_quit {
return ConsumeResult{ value: self.code.slice(startIndex.unwrap(), endIndex.unwrap()),
startIndex:startIndex.unwrap(), endIndex: endIndex.unwrap() };
}
}
}
}
fn main() {
let codeToScan = "40 + 2";
let mut scanner = Scanner::new(codeToScan);
let first_token = scanner.consume_till(|c| { ! c.is_digit ()});
println!("first token is: {}", first_token);
// scanner.consume_till(|c| { c.is_whitespace ()}); // WHY DOES THIS LINE FAIL?
}
Here's a simpler example of the same thing:
struct Scanner<'a> {
s: &'a str
}
impl<'a> Scanner<'a> {
fn step_by_3_bytes<'a>(&'a mut self) -> &'a str {
let return_value = self.s.slice_to(3);
self.s = self.s.slice_from(3);
return_value
}
}
fn main() {
let mut scan = Scanner { s: "123456" };
let a = scan.step_by_3_bytes();
println!("{}", a);
let b = scan.step_by_3_bytes();
println!("{}", b);
}
If you compile that, you get errors like the code in the question:
<anon>:19:13: 19:17 error: cannot borrow `scan` as mutable more than once at a time
<anon>:19 let b = scan.step_by_3_bytes();
^~~~
<anon>:16:13: 16:17 note: previous borrow of `scan` occurs here; the mutable borrow prevents subsequent moves, borrows, or modification of `scan` until the borrow ends
<anon>:16 let a = scan.step_by_3_bytes();
^~~~
<anon>:21:2: 21:2 note: previous borrow ends here
<anon>:13 fn main() {
...
<anon>:21 }
^
Now, the first thing to do is to avoid shadowing lifetimes: that is, this code has two lifetimes called 'a and all the 'as in step_by_3_bytes refer to the 'a declare there, none of them actually refer to the 'a in Scanner<'a>. I'll rename the inner one to make it crystal clear what is going on
impl<'a> Scanner<'a> {
fn step_by_3_bytes<'b>(&'b mut self) -> &'b str {
The problem here is the 'b is connecting the self object with the str return value. The compiler has to assume that calling step_by_3_bytes can make arbitrary modifications, including invalidating previous return values, when looking at the definition of step_by_3_bytes from the outside (which is how the compiler works, type checking is purely based on type signatures of things that are called, no introspect). That is, it could be defined like
struct Scanner<'a> {
s: &'a str,
other: String,
count: uint
}
impl<'a> Scanner<'a> {
fn step_by_3_bytes<'b>(&'b mut self) -> &'b str {
self.other.push_str(self.s);
// return a reference into data we own
self.other.as_slice()
}
}
Now, each call to step_by_3_bytes starts modifying the object that previous return values came from. E.g. it could cause the String to reallocate and thus move in memory, leaving any other &str return values as dangling pointers. Rust protects against this by tracking these references and disallowing mutation if it could cause such catastrophic events. Going back to our actual code: the compiler is type checking main just by looking at the type signature of step_by_3_bytes/consume_till and so it can only assume the worst case scenario (i.e. the example I just gave).
How do we solve this?
Let's take a step back: as if we're just starting out and don't know which lifetimes we want for the return values, so we'll just leave them anonymous (not actually valid Rust):
impl<'a> Scanner<'a> {
fn step_by_3_bytes<'b>(&'_ mut self) -> &'_ str {
Now, we get to ask the fun question: which lifetimes do we want where?
It's almost always best to annotate the longest valid lifetimes, and we know our return value lives for 'a (since it comes straight of the s field, and that &str is valid for 'a). That is,
impl<'a> Scanner<'a> {
fn step_by_3_bytes<'b>(&'_ mut self) -> &'a str {
For the other '_, we don't actually care: as API designers, we don't have any particular desire or need to connect the self borrow with any other references (unlike the return value, where we wanted/needed to express which memory it came from). So, we might as well leave it off
impl<'a> Scanner<'a> {
fn step_by_3_bytes<'b>(&mut self) -> &'a str {
The 'b is unused, so it can be killed, leaving us with
impl<'a> Scanner<'a> {
fn step_by_3_bytes(&mut self) -> &'a str {
This expresses that Scanner is referring to some memory that is valid for at least 'a, and then returning references into just that memory. The self object is essentially just a proxy for manipulating those views: once you have the reference it returns, you can discard the Scanner (or call more methods).
In summary, the full, working code is
struct Scanner<'a> {
s: &'a str
}
impl<'a> Scanner<'a> {
fn step_by_3_bytes(&mut self) -> &'a str {
let return_value = self.s.slice_to(3);
self.s = self.s.slice_from(3);
return_value
}
}
fn main() {
let mut scan = Scanner { s: "123456" };
let a = scan.step_by_3_bytes();
println!("{}", a);
let b = scan.step_by_3_bytes();
println!("{}", b);
}
Applying this change to your code is simply adjusting the definition of consume_till.
fn consume_till(&mut self, quit: |char| -> bool) -> ConsumeResult<'lt> {
So why does the Rust std library code compile, but mine doesn't? (I'm sure the lifetime annotations are at the root of it, but my understanding of lifetimes doesn't lead to me expecting a problem).
There's a slight (but not huge) difference here: Chars is just returning a char, i.e. no lifetimes in the return value. The next method (essentially) has signature:
impl<'a> Chars<'a> {
fn next(&mut self) -> Option<char> {
(It's actually in an Iterator trait impl, but that's not important.)
The situation you have here is similar to writing
impl<'a> Chars<'a> {
fn next(&'a mut self) -> Option<char> {
(Similar in terms of "incorrect linking of lifetimes", the details differ.)
Let’s look at consume_till.
It takes &'lt mut self and returns ConsumeResult<'lt>. This means that the lifetime 'lt, the duration of the borrow of the input parameter self, will be that of the output parameter, the return value.
Expressed another way, after calling consume_till, you cannot use self again until its result is out of scope.
That result is placed into first_token, and first_token is still in scope in your last line.
In order to get around this, you must cause first_token to pass out of scope; the insertion of a new block around it will do this:
fn main() {
let code_to_scan = "40 + 2";
let mut scanner = Scanner::new(code_to_scan);
{
let first_token = scanner.consume_till(|c| !c.is_digit());
println!("first token is: {}", first_token);
}
scanner.consume_till(|c| c.is_whitespace());
}
All this does stand to reason: while you have a reference to something inside the Scanner, it is not safe to let you modify it, lest that reference be invalidated. This is the memory safety that Rust provides.

Resources