I'm reading through the "Learn rust" tutorial, and I'm trying to understand lifetimes. Chapter 10-3 has the following non-working example:
fn longest(x: &str, y: &str) -> &str {
if x.len() > y.len() {
x
} else {
y
}
}
The paragraph below explains the error as :
When we’re defining this function, we don’t know the concrete values that will be passed into this function, so we don’t know whether the if case or the else case will execute.
However, if I change the block of code to do something else; say, print x and return y; so that we know what is being returned each time, the same error occurs. Why?
fn longest(x: &str, y: &str) -> &str {
println!("{}", x);
y
}
It book also says :
The borrow checker can’t determine this either, because it doesn’t know how the lifetimes of x and y relate to the lifetime of the return value.
My doubts are:
Is the borrow checker capable of tracking lifetimes across functions? If so, can you please provide an example?
I don't understand the error. x and y are references passed into longest, so the compiler should know that its owner is elsewhere(and that its lifetime would continue beyond longest). When the compiler sees that the return values are either x or y, why is there a confusion on lifetimes?
Think of the function as a black box. Neither you, nor the compiler knows what happens inside. You may say that the compiler "knows", but that's not really true. Imagine that it returns X or Y based on the result of a remote HTTP call. How can it know in advance ?
Yet it needs to provide some guarantee that the returned reference is safe to use. That works by forcing you (i.e. the developer) to explicitly specify the relationships between the input parameters and the returned value.
First you need to specify the lifetimes of the parameters. I'll use 'x, for x, 'y for y and 'r for the result. Thus our function will look like:
fn longest<'x, 'y, 'r>(x: &'x str, y: &'y str) -> &'r str
But this is not enough. We still need to tell the compiler what the relationships are. There are two way to do it (the magic syntax will be explained later):
Inside the <> brackets like that: <'a, 'b: 'a>
In a where clause like that: where 'b: 'a
Both options are the same but the where clause will be more readable if you have a large number of generic parameters.
Back to the problem. We need to tell the compiler that 'r depends on both 'x and 'y and that it will be valid as long as they are valid. We can do that by saying 'long: 'short which translates to "lifetime 'long must be valid at least as long as lifetime 'short".
Thus we need to modify our function like that:
fn longest<'x, 'y, 'r>(x: &'x str, y: &'y str) -> &'r str
where
'x: 'r,
'y: 'r,
{
if x.len() > y.len() {
x
} else {
y
}
}
I.e. we are saying that our returned value will not outlive the function parameters, thus preventing a "use after free" situation.
PS: In this example you can actually do it with only one lifetime parameter, as we are not interested in the relationship between them. In this case the lifetime will be the smaller one of x/y:
fn longest<'a>(x: &'a str, y: &'a str) -> &'a str
Related
I am very confused about lifetimes, in particular an example in the rust book:
https://doc.rust-lang.org/book/ch10-03-lifetime-syntax.html
Says that:
fn longest(x: &str, y: &str) -> &str {
if x.len() > y.len() {
x
} else {
y
}
}
Needs a lifetime specifier so the compiler knows the lifetime? Doesn't the compiler already know the lifetime? x and y are both parameters to the function, so x and y will last for the whole duration of the function so why would we need a lifetime specifier here? I don't think x and y will ever run into lifetime issues based on the code as is.
I have the following code:
struct Solver<'a> {
guesses: Vec<&'a str>,
}
impl<'a> Solver<'a> {
fn register_guess(&mut self, guess: &'a str) {
self.guesses.push(guess);
}
}
fn foo(mut solver: Solver, guess: &str) {
solver.register_guess(guess)
}
It doesn't compile:
|
11 | fn foo(mut solver: Solver, guess: &str) {
| ---------- - let's call the lifetime of this reference `'1`
| |
| has type `Solver<'2>`
12 | solver.register_guess(guess)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ argument requires that `'1` must outlive `'2`
The error message says that the argument guess must outlive solver. It's plainly obvious to me that that's true: the lifetime of solver ends at the end of the function, and the lifetime of guess doesn't. This seems like something the compiler should be able to infer, and compile without error.
Why isn't that the case? Does this code actually have some way for solver to outlive guess? Or is it just that the compiler doesn't try to do this kind of inference at all?
I know how to fix it --- change the function to fn foo<'a>(mut solver: Solver<'a>, guess: &'a str) --- but I'm asking why I should have to do that.
While solver itself can't outlive guess, the lifetime it refers to very well could. For example, imagine invoking foo() with a Solver<'static>. That kind of solver would expect guess to be &'static str and might store the data referred to by guess in a global variable. (Remember that the compiler doesn't consider what register_guess() does while borrow-checking foo(), it just considers its signature.)
More generally, Solver<'a> might contain references to 'a data that outlives solver itself. Nothing stops register_guess() from storing the contents of guess inside such references. If guess isn't guaranteed to live at least as long as 'a, then foo() is simply unsound. For example, take this alternative definition of Solver:
struct Solver<'a> {
guesses: &'a mut Vec<&'a str>,
}
With unchanged signature of register_guess(), foo() would allow unsound code like this:
fn main() {
let mut guesses = vec![];
let solver = Solver { guesses: &mut guesses };
{
let guess = "foo".to_string();
// stores temporary "foo" to guesses, which outlives it
foo(solver, guess.as_str());
}
println!("{}", guesses[0]); // UB: use after free
}
This error comes from rust's rules of lifetime elision. One of this rules states that:
Each elided lifetime in the parameters becomes a distinct lifetime parameter
Rust conservatively assumes that each not specified lifetime is different. If you want some lifetimes to be equal you must specify it explicitly. Your problem is equivalent to simple function that takes two string slices and returns the longer one. You must write the signature of such function as fn longer<'a>(&'a str, &'a str) -> &'a str, or the compiler will give you the same error.
Why does this code compile?
#[derive(Debug)]
pub struct Foo<'a> {
pub x: &'a [&'a str],
}
fn main() {
let s = "s".to_string();
let t: &str = s.as_ref();
{
let v = vec![t];
let foo = Foo { x: &v[..] };
println!("{:?}", &foo);
}
}
In my Foo struct, my x field contains a slice of &strs. My understanding of these lifetime labels is that the slice and the individual &strs have the same lifetime.
However, in my example the &str t (in the outer block) does not have the same lifetime as the container slice (in the inner block).
My understanding of these lifetime labels is that the slice and the individual &strs have the same lifetime.
This is a common misconception. It really means that there has to be a lifetime that applies to both. When instantiated, the Foo struct's 'a corresponds to the lines of the inner block after let v = vec![t]; as that is a lifetime that both variables share.
If this flexibility didn't exist, lifetimes would be super painful to use. Variables defined on two lines have different actual lifetimes (the one defined first outlives the one defined second). If the lifetimes had to actually match, we'd always have to define all the variables on the same line!
Some more detailed information is available in RFC #738 — Variance.
The syntax 'a is actually used in two different situations:
labeling a loop
indicating a bound
The first situation, labeling a loop:
fn main() {
'a: loop {
println!("{}", 3);
break 'a;
}
}
Here, 'a clearly delineates the lifetime of the loop body, and allows breaking from multiple layers of loops in one fell swoop.
The second, and more similar situation, is using 'a to represent a bound:
fn<'a> susbtr(haystack: &'a str, offset: usize) -> &'a str;
In this case, the lifetime 'a does not represent the actual lifetime of the variable, it represents a bound on the lifetime of the referenced variable and allows tying together the bounds of various variables.
Note that the caller and callee interpret the bound differently:
from the perspective of the caller, 'a is an upper-bound, a promise that the return value will live at least as long as the parameter (maybe longer, no guarantee)
from the perspective of the callee (ie, substr), 'a is a lower-bound, a check that any returned value must live at least as long as the parameter (maybe longer, not necessary)
We can have variance since the bound does not represent the actual lifetime, when a single bound is used for multiple lifetimes, the compiler will simply deduce the lowest/highest bound that makes sense for the situation:
the caller gets the lowest upper-bound feasible (ie, the least guarantee)
the caller gets the highest lower-bound feasible (ie, the least constraint)
For example:
fn<'b> either(one: &'b str, two: &'b str, flag: bool) -> &'b str {
if flag { one } else { two }
}
can be called with:
fn<'a> call(o: &'a str, flag: bool) -> &'a str {
either(o, "Hello, World", flag)
}
Here, the lifetime of o is unknown (some 'a) whilst the lifetime of "Hello, World" is known ('static), 'static is by definition the greater of the lifetimes (it lives for all the program).
the caller of call only knows that the return value lives at least as long as o
call must guarantee this, it supplies o and "Hello, World" to either where 'b is deduced to be the lowest bound between 'a and 'static (thus 'a)
either simply must return something that lives as long as either one of its arguments; it's unaware that their lifetime may differ, and cares not
I was reading the lifetimes chapter of the Rust book, and I came across this example for a named/explicit lifetime:
struct Foo<'a> {
x: &'a i32,
}
fn main() {
let x; // -+ x goes into scope
// |
{ // |
let y = &5; // ---+ y goes into scope
let f = Foo { x: y }; // ---+ f goes into scope
x = &f.x; // | | error here
} // ---+ f and y go out of scope
// |
println!("{}", x); // |
} // -+ x goes out of scope
It's quite clear to me that the error being prevented by the compiler is the use-after-free of the reference assigned to x: after the inner scope is done, f and therefore &f.x become invalid, and should not have been assigned to x.
My issue is that the problem could have easily been analyzed away without using the explicit 'a lifetime, for instance by inferring an illegal assignment of a reference to a wider scope (x = &f.x;).
In which cases are explicit lifetimes actually needed to prevent use-after-free (or some other class?) errors?
The other answers all have salient points (fjh's concrete example where an explicit lifetime is needed), but are missing one key thing: why are explicit lifetimes needed when the compiler will tell you you've got them wrong?
This is actually the same question as "why are explicit types needed when the compiler can infer them". A hypothetical example:
fn foo() -> _ {
""
}
Of course, the compiler can see that I'm returning a &'static str, so why does the programmer have to type it?
The main reason is that while the compiler can see what your code does, it doesn't know what your intent was.
Functions are a natural boundary to firewall the effects of changing code. If we were to allow lifetimes to be completely inspected from the code, then an innocent-looking change might affect the lifetimes, which could then cause errors in a function far away. This isn't a hypothetical example. As I understand it, Haskell has this problem when you rely on type inference for top-level functions. Rust nipped that particular problem in the bud.
There is also an efficiency benefit to the compiler — only function signatures need to be parsed in order to verify types and lifetimes. More importantly, it has an efficiency benefit for the programmer. If we didn't have explicit lifetimes, what does this function do:
fn foo(a: &u8, b: &u8) -> &u8
It's impossible to tell without inspecting the source, which would go against a huge number of coding best practices.
by inferring an illegal assignment of a reference to a wider scope
Scopes are lifetimes, essentially. A bit more clearly, a lifetime 'a is a generic lifetime parameter that can be specialized with a specific scope at compile time, based on the call site.
are explicit lifetimes actually needed to prevent [...] errors?
Not at all. Lifetimes are needed to prevent errors, but explicit lifetimes are needed to protect what little sanity programmers have.
Let's have a look at the following example.
fn foo<'a, 'b>(x: &'a u32, y: &'b u32) -> &'a u32 {
x
}
fn main() {
let x = 12;
let z: &u32 = {
let y = 42;
foo(&x, &y)
};
}
Here, the explicit lifetimes are important. This compiles because the result of foo has the same lifetime as its first argument ('a), so it may outlive its second argument. This is expressed by the lifetime names in the signature of foo. If you switched the arguments in the call to foo the compiler would complain that y does not live long enough:
error[E0597]: `y` does not live long enough
--> src/main.rs:10:5
|
9 | foo(&y, &x)
| - borrow occurs here
10 | };
| ^ `y` dropped here while still borrowed
11 | }
| - borrowed value needs to live until here
The lifetime annotation in the following structure:
struct Foo<'a> {
x: &'a i32,
}
specifies that a Foo instance shouldn't outlive the reference it contains (x field).
The example you came across in the Rust book doesn't illustrate this because f and y variables go out of scope at the same time.
A better example would be this:
fn main() {
let f : Foo;
{
let n = 5; // variable that is invalid outside this block
let y = &n;
f = Foo { x: y };
};
println!("{}", f.x);
}
Now, f really outlives the variable pointed to by f.x.
Note that there are no explicit lifetimes in that piece of code, except the structure definition. The compiler is perfectly able to infer lifetimes in main().
In type definitions, however, explicit lifetimes are unavoidable. For example, there is an ambiguity here:
struct RefPair(&u32, &u32);
Should these be different lifetimes or should they be the same? It does matter from the usage perspective, struct RefPair<'a, 'b>(&'a u32, &'b u32) is very different from struct RefPair<'a>(&'a u32, &'a u32).
Now, for simple cases, like the one you provided, the compiler could theoretically elide lifetimes like it does in other places, but such cases are very limited and do not worth extra complexity in the compiler, and this gain in clarity would be at the very least questionable.
If a function receives two references as arguments and returns a reference, then the implementation of the function might sometimes return the first reference and sometimes the second one. It is impossible to predict which reference will be returned for a given call. In this case, it is impossible to infer a lifetime for the returned reference, since each argument reference may refer to a different variable binding with a different lifetime. Explicit lifetimes help to avoid or clarify such a situation.
Likewise, if a structure holds two references (as two member fields) then a member function of the structure may sometimes return the first reference and sometimes the second one. Again explicit lifetimes prevent such ambiguities.
In a few simple situations, there is lifetime elision where the compiler can infer lifetimes.
I've found another great explanation here: http://doc.rust-lang.org/0.12.0/guide-lifetimes.html#returning-references.
In general, it is only possible to return references if they are
derived from a parameter to the procedure. In that case, the pointer
result will always have the same lifetime as one of the parameters;
named lifetimes indicate which parameter that is.
The case from the book is very simple by design. The topic of lifetimes is deemed complex.
The compiler cannot easily infer the lifetime in a function with multiple arguments.
Also, my own optional crate has an OptionBool type with an as_slice method whose signature actually is:
fn as_slice(&self) -> &'static [bool] { ... }
There is absolutely no way the compiler could have figured that one out.
As a newcomer to Rust, my understanding is that explicit lifetimes serve two purposes.
Putting an explicit lifetime annotation on a function restricts the type of code that may appear inside that function. Explicit lifetimes allow the compiler to ensure that your program is doing what you intended.
If you (the compiler) want(s) to check if a piece of code is valid, you (the compiler) will not have to iteratively look inside every function called. It suffices to have a look at the annotations of functions that are directly called by that piece of code. This makes your program much easier to reason about for you (the compiler), and makes compile times managable.
On point 1., Consider the following program written in Python:
import pandas as pd
import numpy as np
def second_row(ar):
return ar[0]
def work(second):
df = pd.DataFrame(data=second)
df.loc[0, 0] = 1
def main():
# .. load data ..
ar = np.array([[0, 0], [0, 0]])
# .. do some work on second row ..
second = second_row(ar)
work(second)
# .. much later ..
print(repr(ar))
if __name__=="__main__":
main()
which will print
array([[1, 0],
[0, 0]])
This type of behaviour always surprises me. What is happening is that df is sharing memory with ar, so when some of the content of df changes in work, that change infects ar as well. However, in some cases this may be exactly what you want, for memory efficiency reasons (no copy). The real problem in this code is that the function second_row is returning the first row instead of the second; good luck debugging that.
Consider instead a similar program written in Rust:
#[derive(Debug)]
struct Array<'a, 'b>(&'a mut [i32], &'b mut [i32]);
impl<'a, 'b> Array<'a, 'b> {
fn second_row(&mut self) -> &mut &'b mut [i32] {
&mut self.0
}
}
fn work(second: &mut [i32]) {
second[0] = 1;
}
fn main() {
// .. load data ..
let ar1 = &mut [0, 0][..];
let ar2 = &mut [0, 0][..];
let mut ar = Array(ar1, ar2);
// .. do some work on second row ..
{
let second = ar.second_row();
work(second);
}
// .. much later ..
println!("{:?}", ar);
}
Compiling this, you get
error[E0308]: mismatched types
--> src/main.rs:6:13
|
6 | &mut self.0
| ^^^^^^^^^^^ lifetime mismatch
|
= note: expected type `&mut &'b mut [i32]`
found type `&mut &'a mut [i32]`
note: the lifetime 'b as defined on the impl at 4:5...
--> src/main.rs:4:5
|
4 | impl<'a, 'b> Array<'a, 'b> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...does not necessarily outlive the lifetime 'a as defined on the impl at 4:5
--> src/main.rs:4:5
|
4 | impl<'a, 'b> Array<'a, 'b> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
In fact you get two errors, there is also one with the roles of 'a and 'b interchanged. Looking at the annotation of second_row, we find that the output should be &mut &'b mut [i32], i.e., the output is supposed to be a reference to a reference with lifetime 'b (the lifetime of the second row of Array). However, because we are returning the first row (which has lifetime 'a), the compiler complains about lifetime mismatch. At the right place. At the right time. Debugging is a breeze.
The reason why your example does not work is simply because Rust only has local lifetime and type inference. What you are suggesting demands global inference. Whenever you have a reference whose lifetime cannot be elided, it must be annotated.
I think of a lifetime annotation as a contract about a given ref been valid in the receiving scope only while it remains valid in the source scope. Declaring more references in the same lifetime kind of merges the scopes, meaning that all the source refs have to satisfy this contract.
Such annotation allow the compiler to check for the fulfillment of the contract.
I believe that this function declaration tells Rust that the lifetime of the function's output is the same as the lifetime of it's s parameter:
fn substr<'a>(s: &'a str, until: u32) -> &'a str;
^^^^
It seems to me that the compiler only needs to know this(1):
fn substr(s: &'a str, until: u32) -> &'a str;
What does the annotation <'a> after the function name mean? Why does the compiler need it, and what does it do with it?
(1): I know it needs to know even less, due to lifetime elision. But this question is about specifying lifetime explicitly.
Let me expand on the previous answers…
What does the annotation <'a> after the function name mean?
I wouldn't use the word "annotation" for that. Much like <T> introduces a generic type parameter, <'a> introduces a generic lifetime parameter. You can't use any generic parameters without introducing them first and for generic functions this introduction happens right after their name. You can think of a generic function as a family of functions. So, essentially, you get one function for every combination of generic parameters. substr::<'x> would be a specific member of that function family for some lifetime 'x.
If you're unclear on when and why we have to be explicit about lifetimes, read on…
A lifetime parameter is always associated with all reference types. When you write
fn main() {
let x = 28374;
let r = &x;
}
the compiler knows that x lives in the main function's scope enclosed with curly braces. Internally, it identifies this scope with some lifetime parameter. For us, it is unnamed. When you take the address of x, you'll get a value of a specific reference type. A reference type is kind of a member of a two dimensional family of reference types. One axis is the type of what the reference points to and the other axis is a lifetime that is used for two constraints:
The lifetime parameter of a reference type represents an upper bound for how long you can hold on to that reference
The lifetime parameter of a reference type represents a lower bound for the lifetime of the things you can make the reference point to.
Together, these constraints play a vital role in Rust's memory safety story. The goal here is to avoid dangling references. We would like to rule out references that point to some memory region we are not allowed to use anymore because that thing it used to point to does not exist anymore.
One potential source of confusion is probably the fact that lifetime parameters are invisible most of the time. But that does not mean they are not there. References always have a lifetime parameter in their type. But such a lifetime parameter does not have to have a name and most of the time we don't need to mention it anyways because the compiler can assign names for lifetime parameters automatically. This is called "lifetime elision". For example, in the following case, you don't see any lifetime parameters being mentioned:
fn substr(s: &str, until: u32) -> &str {…}
But it's okay to write it like this. It's actually a short-cut syntax for the more explicit
fn substr<'a>(s: &'a str, until: u32) -> &'a str {…}
Here, the compiler automatically assigns the same name to the "input lifetime" and the "output lifetime" because it's a very common pattern and most likely exactly what you want. Because this pattern is so common, the compiler lets us get away without saying anything about lifetimes. It assumes that this more explicit form is what we meant based on a couple of "lifetime elision" rules (which are at least documented here)
There are situations in which explicit lifetime parameters are not optional. For example, if you write
fn min<T: Ord>(x: &T, y: &T) -> &T {
if x <= y {
x
} else {
y
}
}
the compiler will complain because it will interpret the above declaration as
fn min<'a, 'b, 'c, T: Ord>(x: &'a T, y: &'b T) -> &'c T { … }
So, for each reference a separate lifetime parameter is introduced. But no information on how the lifetime parameters relate to each other is available in this signature. The user of this generic function could use any lifetimes. And that's a problem inside its body. We're trying to return either x or y. But the type of x is &'a T. That's not compatible with the return type &'c T. The same is true for y. Since the compiler knows nothing about how these lifetimes relate to each other, it's not safe to return these references as a reference of type &'c T.
Can it ever be safe to go from a value of type &'a T to &'c T? Yes. It's safe if the lifetime 'a is equal or greater than the lifetime 'c. Or in other words 'a: 'c. So, we could write this
fn min<'a, 'b, 'c, T: Ord>(x: &'a T, y: &'b T) -> &'c T
where 'a: 'c, 'b: 'c
{ … }
and get away with it without the compiler complaining about the function's body. But it's actually unnecessarily complex. We can also simply write
fn min<'a, T: Ord>(x: &'a T, y: &'a T) -> &'a T { … }
and use a single lifetime parameter for everything. The compiler is able to deduce 'a as the minimum lifetime of the argument references at the call site just because we used the same lifetime name for both parameters. And this lifetime is precisely what we need for the return type.
I hope this answers your question. :)
Cheers!
What does the annotation <'a> after the function name mean?
fn substr<'a>(s: &'a str, until: u32) -> &'a str;
// ^^^^
This is declaring a generic lifetime parameter. It's similar to a generic type parameter (often seen as <T>), in that the caller of the function gets to decide what the lifetime is. Like you said, the lifetime of the result will be the same as the lifetime of the first argument.
All lifetime names are equivalent, except for one: 'static. This lifetime is pre-set to mean "guaranteed to live for the entire life of the program".
The most common lifetime parameter name is probably 'a, but you can use any letter or string. Single letters are most common, but any snake_case identifier is acceptable.
Why does the compiler need it, and what does it do with it?
Rust generally favors things to be explicit, unless there's a very good ergonomic benefit. For lifetimes, lifetime elision takes care of something like 85+% of cases, which seemed like a clear win.
Type parameters live in the same namespace as other types — is T a generic type or did someone name a struct that? Thus type parameters need to have an explicit annotation that shows that T is a parameter and not a real type. However, lifetime parameters don't have this same problem, so that's not the reason.
Instead, the main benefit of explicitly listing type parameters is because you can control how multiple parameters interact. A nonsense example:
fn better_str<'a, 'b, 'c>(a: &'a str, b: &'b str) -> &'c str
where
'a: 'c,
'b: 'c,
{
if a.len() < b.len() {
a
} else {
b
}
}
We have two strings and say that the input strings may have different lifetimes, but must both outlive the lifetime of the result value.
Another example, as pointed out by DK, is that structs can have their own lifetimes. I made this example also a bit of nonsense, but it hopefully conveys the point:
struct Player<'a> {
name: &'a str,
}
fn name<'p, 'n>(player: &'p Player<'n>) -> &'n str {
player.name
}
Lifetimes can be one of the more mind-bending parts of Rust, but they are pretty great when you start to grasp them.
The <'a> annotation just declares the lifetimes used in the function, exactly like generic parameters <T>.
fn subslice<'a, T>(s: &'a [T], until: u32) -> &'a [T] { \\'
&s[..until as usize]
}
Note that in your example, all lifetimes can be inferred.
fn subslice<T>(s: &[T], until: u32) -> &[T] {
&s[..until as usize]
}
fn substr(s: &str, until: u32) -> &str {
&s[..until as usize]
}
playpen example