The following code works fine:
fn get<F: Fn(&[u8]) -> u8>(f: F) -> u8 {
f(&[1, 2, 3])
}
However, when I add explicit lifetime information to it, it doesn't:
fn get<'inp, F: Fn(&'inp [u8]) -> u8>(f: F) -> u8 {
f(&[1, 2, 3])
}
What lifetime does the compiler infer in the working code?
I'm using Rust 1.18.0.
The error message is:
error: borrowed value does not live long enough
--> test.rs:4:8
|
4 | f(&[1, 2, 3])
| ^^^^^^^^^ does not live long enough
5 | }
| - temporary value only lives until here
|
note: borrowed value must be valid for the lifetime 'inp as defined on the body at 3:49...
--> test.rs:3:50
|
3 | fn get<'inp, F: Fn(&'inp [u8]) -> u8>(f: F) -> u8 {
| __________________________________________________^
4 | | f(&[1, 2, 3])
5 | | }
| |_^
Lifetimes in trait bounds are a bit special and the Fn family of traits has a special lifetime elision rule. We'll dive into that, but first, here it the correct explicitly annotated version:
fn get<F: for<'inp> Fn(&'inp [u8]) -> u8>(f: F) -> u8 {
f(&[1, 2, 3])
}
Oh gosh, what is this for<'inp> doing there? It's a so called higher ranked trait bound (HRTB) and it's used here to make 'inp universally quantiefied in regards to f. In order to fully understand that, we need to understand a bit of theory.
Who has the choice?
Let's take a look at an easy example:
fn bar<'a>(x: &'a u8) {}
Here, bar() is generic of the lifetime 'a. The syntax above reads: "choose any 'a and there is a bar() that will work with 'a". This means that we can choose any 'a we want, and bar() works! Who are "we"? "We" are the caller -- the one calling bar. This will be important later: the caller chooses the generic parameters. We can call bar() with a &'static u8 as well as with a reference that doesn't live as long.
Now you might ask: are there situations where the caller doesn't choose the generic parameter, but someone else does? Yes, there are! Sadly, it's a bit more difficult to understand, because it doesn't occur too often in today's Rust code. But let's try:
trait Bar<'a> {
fn bar(&self, x: &'a u8);
}
This is similar to the bar() function above, but now the lifetime parameter is defined on the trait, not the function. Let's use the trait:
fn use_bar<'a, B: Bar<'a>>(b: B) {
let local = 0u8;
b.bar(&local);
}
This doesn't compile, printing the same error as above. Why? The method b.bar() expects a reference of lifetime 'a. But who chooses 'a here? Exactly: the caller -- the caller of use_bar(), not the caller of bar()! So the caller of use_bar() could choose the 'static lifetime; in that case, it's easy to see that our &local doesn't fulfill the lifetime requirements.
Note that the caller of use_bar() chooses 'a as well as B. Once use_bar() is instantiated, B is a fixed type and B::bar() works only for one specific lifetime. This means the caller of bar() can't choose the lifetime, but bar() itself chose it!
What do we want instead? We want use_bar() to choose the lifetime of the bar() call. And we can do that with the for<> syntax:
fn use_bar<B: for<'a> Bar<'a>>(b: B) {
let local = 0u8;
b.bar(&local);
}
This works. What we say here is: "for any lifetime parameter 'a, B has to implement the trait Bar<'a>". Instead of: "there needs to exist a lifetime parameter 'a for which B implements Bar<'a>". It's all about who chooses which parameter.
Let's use the real names for it:
a generic parameter is universally quantified if the caller can choose it
a generic parameter is existentially quantified if the callee can choose it
What does Rust do?
To return to your example:
fn get<'inp, F: Fn(&'inp [u8]) -> u8>(f: F) -> u8 {
f(&[1, 2, 3])
}
Here we have the same problem as above: the lifetime parameter of f is existentially quantified. The caller of f cannot choose the lifetime parameter. We can fix that with the for<> syntax as shown above.
When you omit the lifetimes:
fn get<F: Fn(&[u8]) -> u8>(f: F) -> u8 {
f(&[1, 2, 3])
}
The Rust compiler will do something special for the Fn family of traits. Your F: Fn(&[u8]) desugars to F: for<'a> Fn<(&'a [u8],)>. If you use Fn* traits with parameters that involve lifetimes, those lifetimes are automatically universally quantified, because that's usually what you want with higher order functions.
The infered code seems to be:
fn get<F : for<'inp> Fn(&'inp[u8]) -> u8>(f: F) -> u8 {
f(&[1,2,3])
}
Explanation
Related
The implementation of std::mem::drop is documented to be the following:
pub fn drop<T>(_x: T) { }
As such, I would expect the closure |_| () (colloquially known as the toilet closure) to be a potential 1:1 replacement to drop, in both directions. However, the code below shows that drop isn't compatible with a higher ranked trait bound on the function's parameter, whereas the toilet closure is.
fn foo<F, T>(f: F, x: T)
where
for<'a> F: FnOnce(&'a T),
{
dbg!(f(&x));
}
fn main() {
foo(|_| (), "toilet closure"); // this compiles
foo(drop, "drop"); // this does not!
}
The compiler's error message:
error[E0631]: type mismatch in function arguments
--> src/main.rs:10:5
|
1 | fn foo<F, T>(f: F, x: T)
| ---
2 | where
3 | for<'a> F: FnOnce(&'a T),
| ------------- required by this bound in `foo`
...
10 | foo(drop, "drop"); // this does not!
| ^^^
| |
| expected signature of `for<'a> fn(&'a _) -> _`
| found signature of `fn(_) -> _`
error[E0271]: type mismatch resolving `for<'a> <fn(_) {std::mem::drop::<_>} as std::ops::FnOnce<(&'a _,)>>::Output == ()`
--> src/main.rs:10:5
|
1 | fn foo<F, T>(f: F, x: T)
| ---
2 | where
3 | for<'a> F: FnOnce(&'a T),
| ------------- required by this bound in `foo`
...
10 | foo(drop, "drop"); // this does not!
| ^^^ expected bound lifetime parameter 'a, found concrete lifetime
Considering that drop is supposedly generic with respect to any sized T, it sounds unreasonable that the "more generic" signature fn(_) -> _ is not compatible with for<'a> fn (&'a _) -> _. Why is the compiler not admitting the signature of drop here, and what makes it different when the toilet closure is placed in its stead?
The core of the issue is that drop is not a single function, but rather a parameterized set of functions that each drop some particular type. To satisfy a higher-ranked trait bound (hereafter hrtb), you'd need a single function that can simultaneously take references to a type with any given lifetime.
We'll use drop as our typical example of a generic function, but all this applies more generally too. Here's the code for reference: fn drop<T>(_: T) {}.
Conceptually, drop is not a single function, but rather one function for every possible type T. Any particular instance of drop takes only arguments of a single type. This is called monomorphization. If a different T is used with drop, a different version of drop is compiled. That's why you can't pass a generic function as an argument and use that function in full generality (see this question)
On the other hand, a function like fn pass(x: &i32) -> &i32 {x} satisfies the hrtb for<'a> Fn(&'a i32) -> &'a i32. Unlike drop, we have a single function that simultaneously satisfies Fn(&'a i32) -> &'a i32 for every lifetime 'a. This is reflected in how pass can be used.
fn pass(x: &i32) -> &i32 {
x
}
fn two_uses<F>(f: F)
where
for<'a> F: Fn(&'a i32) -> &'a i32, // By the way, this can simply be written
// F: Fn(&i32) -> &i32 due to lifetime elision rules.
// That applies to your original example too.
{
{
// x has some lifetime 'a
let x = &22;
println!("{}", f(x));
// 'a ends around here
}
{
// y has some lifetime 'b
let y = &23;
println!("{}", f(y));
// 'b ends around here
}
// 'a and 'b are unrelated since they have no overlap
}
fn main() {
two_uses(pass);
}
(playground)
In the example, the lifetimes 'a and 'b have no relation to each other: neither completely encompasses the other. So there isn't some kind of subtyping thing going on here. A single instance of pass is really being used with two different, unrelated lifetimes.
This is why drop doesn't satisfy for<'a> FnOnce(&'a T). Any particular instance of drop can only cover one lifetime (ignoring subtyping). If we passed drop into two_uses from the example above (with slight signature changes and assuming the compiler let us), it would have to choose some particular lifetime 'a and the instance of drop in the scope of two_uses would be Fn(&'a i32) for some concrete lifetime 'a. Since the function would only apply to single lifetime 'a, it wouldn't be possible to use it with two unrelated lifetimes.
So why does the toilet closure get a hrtb? When inferring the type for a closure, if the expected type hints that a higher-ranked trait bound is needed, the compiler will try to make one fit. In this case, it succeeds.
Issue #41078 is closely related to this and in particular, eddyb's comment here gives essentially the explanation above (though in the context of closures, rather than ordinary functions). The issue itself doesn't address the present problem though. It instead addresses what happens if you assign the toilet closure to a variable before using it (try it out!).
It's possible that the situation will change in the future, but it would require a pretty big change in how generic functions are monomorphized.
In short, both lines should fail. But since one step in old way of handling hrtb lifetimes, namely the leak check, currently has some soundness issue, rustc ends up (incorrectly) accepting one and leaving the other with a pretty bad error message.
If you disable the leak check with rustc +nightly -Zno-leak-check, you'll be able to see a more sensible error message:
error[E0308]: mismatched types
--> src/main.rs:10:5
|
10 | foo(drop, "drop");
| ^^^ one type is more general than the other
|
= note: expected type `std::ops::FnOnce<(&'a &str,)>`
found type `std::ops::FnOnce<(&&str,)>`
My interpretation of this error is that the &x in the body of the foo function only has a scope lifetime confined to the said body, so f(&x) also has the same scope lifetime which can't possibly satisfy the for<'a> universal quantification required by the trait bound.
The question you present here is almost identical to issue #57642, which also has two contrasting parts.
The new way to process hrtb lifetimes is by using so-called universes. Niko has a WIP to tackle the leak check with universes. Under this new regime, both parts of issue #57642 linked above is said to all fail with far more clear diagnoses. I suppose the compiler should be able to handle your example code correctly by then, too.
From what I understand, when x implements trait Foo,
the following two lines should be equivalent.
x.foo();
Foo::foo(&x);
However, I am facing a problem where the compiler accepts the first one, and rejects the second one, with a rather strange error message.
As usual, this example is available on the playground.
Consider the following two related traits.
pub trait Bar<'a> {
type BarError: Into<MyError>;
fn bar(&self) -> Result<(), Self::BarError>;
}
pub trait Foo: for<'a> Bar<'a> {
type FooError: Into<MyError>;
fn foo(&self) -> Result<(), Self::FooError>
where
for<'a> <Self as Bar<'a>>::BarError: Into<<Self as Foo>::FooError>;
}
This example is a bit complex, but I do need the lifetime parameter on Bar, and I can't have it on Foo. As a consequence:
I have to resort on Higher-Rank Trait Bounds (HRTB);
I can not rely on Bar::BarError in Foo (there are actually an infinite number of types Bar<'_>::BarError), so Foo must have its own FooError;
and so I need the complex trait bound in the foo method to convert BarErrors to FooErrors.
Now, let's implement Bar and Foo for a concrete type, e.g. Vec<i32>.
impl<'a> Bar<'a> for Vec<i32> {
type BarError = Never;
fn bar(&self) /* ... */
}
impl Foo for Vec<i32> {
type FooError = Never;
fn foo(&self) /* ... */
}
Note that Never is an empty enum, indicating that these implementations never fail. In order to comply with the trait definitions, From<Never> is implemented for MyError.
We can now demonstrate the problem: the following compiles like charm.
let x = vec![1, 2, 3];
let _ = x.foo();
But the following des not.
let x = vec![1, 2, 3];
let _ = Foo::foo(&x);
The error messages says:
error[E0271]: type mismatch resolving `<std::vec::Vec<i32> as Foo>::FooError == MyError`
--> src/main.rs:49:13
|
49 | let _ = Foo::foo(&x);
| ^^^^^^^^ expected enum `Never`, found struct `MyError`
|
= note: expected type `Never`
found type `MyError`
The compiler seems to believe that I wrote something like this (NB: this is not correct Rust, but just to give the idea).
let _ = Foo::<FooError=MyError>::foo(&x);
And this does not work because x implements Foo<FooError=Never>.
Why does the compiler adds this additional constraint? Is it a bug? If not, is it possible to write it otherwise so it compiles?
NB: you may wonder why I don't just stick to the first version (x.foo(&x)). In my actual situation, foo is actually named retain, which is also the name of a method in Vec. So I must use the second form, to avoid the ambiguity.
NB2: if I remove the HRTB in the declaration of method foo, both lines compile. But then I can not call Bar::bar in any implementation of Foo::foo, which is not an option for me. And changing foo to something like fn foo<'a>(&'a self) -> Result<(), <Self as Bar<'a>>::BarError) is not an option either, unfortunately.
From what I understand, when x implements trait Foo, the following two lines should be equivalent.
x.foo();
Foo::foo(&x);
This is true for an inherent method (one that is defined on the type of x itself), but not for a trait method. In your case the equivalent is <Vec<i32> as Foo>::foo(&x);.
Here is a playground link
Considering the following code:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || &t)
}
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
t moves to the closure, so the closure live as long as t
The closure returns a reference to t which was moved to the closure. So the reference is valid as long as the closure exists.
There is no lifetime problem, the code compiles.
What actually happens:
The code does not compile:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
|
note: first, the lifetime cannot outlive the lifetime as defined on the body at 2:14...
--> src/lib.rs:2:14
|
2 | Box::new(move || &t)
| ^^^^^^^^^^
note: ...so that closure can access `t`
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
note: but, the lifetime must be valid for the lifetime 'a as defined on the function body at 1:8...
--> src/lib.rs:1:8
|
1 | fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
| ^^
= note: ...so that the expression is assignable:
expected std::boxed::Box<(dyn std::ops::Fn() -> &'a T + 'a)>
found std::boxed::Box<dyn std::ops::Fn() -> &T>
I do not understand the conflict. How can I fix it?
Very interesting question! I think I understood the problem(s) at play here. Let me try to explain.
tl;dr: closures cannot return references to values captured by moving, because that would be a reference to self. Such a reference cannot be returned because the Fn* traits don't allow us to express that. This is basically the same as the streaming iterator problem and could be fixed via GATs (generic associated types).
Implementing it manually
As you probably know, when you write a closure, the compiler will generate a struct and impl blocks for the appropriate Fn traits, so closures are basically syntax sugar. Let's try to avoid all that sugar and build your type manually.
What you want is a type which owns another type and can return references to that owned type. And you want to have a function which returns a boxed instance of said type.
struct Baz<T>(T);
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
fn make_baz<T>(t: T) -> Box<Baz<T>> {
Box::new(Baz(t))
}
This is pretty equivalent to your boxed closure. Let's try to use it:
let outside = {
let s = "hi".to_string();
let baz = make_baz(s);
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // works too
This works just fine. The string s is moved into the Baz type and that Baz instance is moved into the Box. s is now owned by baz and then by outside.
It gets more interesting when we add a single character:
let outside = {
let s = "hi".to_string();
let baz = make_baz(&s); // <-- NOW BORROWED!
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // doesn't work!
Now we cannot make the lifetime of baz bigger than the lifetime of s, since baz contains a reference to s which would be an dangling reference of s would go out of scope earlier than baz.
The point I wanted to make with this snippet: we didn't need to annotate any lifetimes on the type Baz to make this safe; Rust figured it out on its own and enforces that baz lives no longer than s. This will be important below.
Writing a trait for it
So far we only covered the basics. Let's try to write a trait like Fn to get closer to your original problem:
trait MyFn {
type Output;
fn call(&self) -> Self::Output;
}
In our trait, there are no function parameters, but otherwise it's fairly identical to the real Fn trait.
Let's implement it!
impl<T> MyFn for Baz<T> {
type Output = ???;
fn call(&self) -> Self::Output {
&self.0
}
}
Now we have a problem: what do we write instead of ???? Naively one would write &T... but we need a lifetime parameter for that reference. Where do we get one? What lifetime does the return value even have?
Let's check the function we implemented before:
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
So here we use &T without lifetime parameter too. But this only works because of lifetime elision. Basically, the compiler fills in the blanks so that fn call(&self) -> &T is equivalent to:
fn call<'s>(&'s self) -> &'s T
Aha, so the lifetime of the returned reference is bound to the self lifetime! (more experienced Rust users might already have a feeling where this is going...).
(As a side note: why is the returned reference not dependent on the lifetime of T itself? If T references something non-'static then this has to be accounted for, right? Yes, but it is already accounted for! Remember that no instance of Baz<T> can ever live longer than the thing T might reference. So the self lifetime is already shorter than whatever lifetime T might have. Thus we only need to concentrate on the self lifetime)
But how do we express that in the trait impl? Turns out: we can't (yet). This problem is regularly mentioned in the context of streaming iterators -- that is, iterators that return an item with a lifetime bound to the self lifetime. In today's Rust, it is sadly impossible to implement this; the type system is not strong enough.
What about the future?
Luckily, there is an RFC "Generic Associated Types" which was merged some time ago. This RFC extends the Rust type system to allow associated types of traits to be generic (over other types and lifetimes).
Let's see how we can make your example (kinda) work with GATs (according to the RFC; this stuff doesn't work yet ☹). First we have to change the trait definition:
trait MyFn {
type Output<'a>; // <-- we added <'a> to make it generic
fn call(&self) -> Self::Output;
}
The function signature hasn't changed in the code, but notice that lifetime elision kicks in! The above fn call(&self) -> Self::Output is equivalent to:
fn call<'s>(&'s self) -> Self::Output<'s>
So the lifetime of the associated type is bound to the self lifetime. Just as we wanted! The impl looks like this:
impl<T> MyFn for Baz<T> {
type Output<'a> = &'a T;
fn call(&self) -> Self::Output {
&self.0
}
}
To return a boxed MyFn we would need to write this (according to this section of the RFC:
fn make_baz<T>(t: T) -> Box<for<'a> MyFn<Output<'a> = &'a T>> {
Box::new(Baz(t))
}
And what if we want to use the real Fn trait? As far as I understand, we can't, even with GATs. I think it's impossible to change the existing Fn trait to use GATs in a backwards compatible manner. So it's likely that the standard library will keep the less powerful trait as is. (side note: how to evolve the standard library in backwards incompatible ways to use new language features is something I wondered about a few times already; so far I haven't heard of any real plan in this regards; I hope the Rust team comes up with something...)
Summary
What you want is not technically impossible or unsafe (we implemented it as a simple struct and it works). However, unfortunately it is impossible to express what you want in the form of closures/Fn traits in Rust's type system right now. This is the same problem streaming iterators are dealing with.
With the planned GAT feature, it is possible to express all of this in the type system. However, the standard library would need to catch up somehow to make your exact code possible.
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
This makes no sense. A value cannot "live as long" as a type, because a type doesn't live. "T has lifetime 'a" is a very imprecise statement, easy to misunderstand. What T: 'a really means is "instances of T must stay valid at least as long as lifetime 'a. For example, T must not be a reference with a lifetime shorter than 'a, or a struct containing such a reference. Note that this has nothing to do with forming references to T, i.e. &T.
The value t, then, lives as long as its lexical scope (it's a function parameter) says it does, which has nothing to do with 'a at all.
t moves to the closure, so the closure live as long as t
This is also incorrect. The closure lives as long as the closure does lexically. It is a temporary in the result expression, and therefore lives until the end of the result expression. t's lifetime concerns the closure not at all, since it has its own T variable inside, the capture of t. Since the capture is a copy/move of t, it is not in any way affected by t's lifetime.
The temporary closure is then moved into the box's storage, but that's a new object with its own lifetime. The lifetime of that closure is bound to the lifetime of the box, i.e. it is the return value of the function, and later (if you store the box outside the function) the lifetime of whatever variable you store the box in.
All of that means that a closure that returns a reference to its own capture state must bind the lifetime of that reference to its own reference. Unfortunately, this is not possible.
Here's why:
The Fn trait implies the FnMut trait, which in turn implies the FnOnce trait. That is, every function object in Rust can be called with a by-value self argument. This means that every function object must be still valid being called with a by-value self argument and returning the same thing as always.
In other words, trying to write a closure that returns a reference to its own captures expands to roughly this code:
struct Closure<T> {
captured: T,
}
impl<T> FnOnce<()> for Closure<T> {
type Output = &'??? T; // what do I put as lifetime here?
fn call_once(self, _: ()) -> Self::Output {
&self.captured // returning reference to local variable
// no matter what, the reference would be invalid once we return
}
}
And this is why what you're trying to do is fundamentally impossible. Take a step back, think of what you're actually trying to accomplish with this closure, and find some other way to accomplish it.
You expect the type T to have lifetime 'a, but t is not a reference to a value of type T. The function takes ownership of the variable t by argument passing:
// t is moved here, t lifetime is the scope of the function
fn foo<'a, T: 'a>(t: T)
You should do:
fn foo<'a, T: 'a>(t: &'a T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || t)
}
The other answers are top-notch, but I wanted to chime in with another reason your original code couldn't work. A big problem lies in the signature:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a>
This says that the caller may specify any lifetime when calling foo and the code will be valid and memory-safe. That cannot possibly be true for this code. It wouldn't make sense to call this with 'a set to 'static, but nothing about this signature would prevent that.
I can't call Foo::new(words).split_first() in the following code
fn main() {
let words = "Sometimes think, the greatest sorrow than older";
/*
let foo = Foo::new(words);
let first = foo.split_first();
*/
let first = Foo::new(words).split_first();
println!("{}", first);
}
struct Foo<'a> {
part: &'a str,
}
impl<'a> Foo<'a> {
fn split_first(&'a self) -> &'a str {
self.part.split(',').next().expect("Could not find a ','")
}
fn new(s: &'a str) -> Self {
Foo { part: s }
}
}
the compiler will give me an error message
error[E0716]: temporary value dropped while borrowed
--> src/main.rs:8:17
|
8 | let first = Foo::new(words).split_first();
| ^^^^^^^^^^^^^^^ - temporary value is freed at the end of this statement
| |
| creates a temporary which is freed while still in use
9 |
10 | println!("{}", first);
| ----- borrow later used here
|
= note: consider using a `let` binding to create a longer lived value
If I bind the value of Foo::new(words) first, then call the split_first method there is no problem.
These two methods of calling should intuitively be the same but are somehow different.
Short answer: remove the 'a lifetime for the self parameter of split_first: fn split_first(&self) -> &'a str (playground).
Long answer:
When you write this code:
struct Foo<'a> {
part: &'a str,
}
impl<'a> Foo<'a> {
fn new(s: &'a str) -> Self {
Foo { part: s }
}
}
You are telling the compiler that all Foo instances are related to some lifetime 'a that must be equal to or shorter than the lifetime of the string passed as parameter to Foo::new. That lifetime 'a may be different from the lifetime of each Foo instance. When you then write:
let words = "Sometimes think, the greatest sorrow than older";
Foo::new(words)
The compiler infers that the lifetime 'a must be equal to or shorter than the lifetime of words. Barring any other constraints the compiler will use the lifetime of words, which is 'static so it is valid for the full life of the program.
When you add your definition of split_first:
fn split_first(&'a self) -> &'a str
You are adding an extra constraint: you are saying that 'a must also be equal to or shorter than the lifetime of self. The compiler will therefore take the shorter of the lifetime of words and the lifetime of the temporary Foo instance, which is the lifetime of the temporary. #AndersKaseorg's answer explains why that doesn't work.
By removing the 'a lifetime on the self parameter, I am decorrelating 'a from the lifetime of the temporary, so the compiler can again infer that 'a is the lifetime of words, which is long enough for the program to work.
Foo::new(words).split_first() would be interpreted roughly as
let tmp = Foo::new(words);
let ret = tmp.split_first();
drop(tmp);
ret
If Rust allowed you to do this, the references in ret would point [edit: would be allowed by the type of split_first to point*] into the now dropped value of tmp. So it’s a good thing that Rust disallows this. If you wrote the equivalent one-liner in C++, you’d silently get undefined behavior.
By writing the let binding yourself, you delay the drop until the end of the scope, thus extending the region where it’s safe to have these references.
For more details, see temporary lifetimes in the Rust Reference.
* Edit: As pointed out by Jmb, the real problem in this particular example is that the type
fn split_first(&'a self) -> &'a str
isn’t specific enough, and a better solution is to refine the type to:
fn split_first<'b>(&'b self) -> &'a str
which can be abbreviated:
fn split_first(&self) -> &'a str
This conveys the intended guarantee that the returned references do not point into the Foo<'a> (only into the string itself).
Considering the following code:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || &t)
}
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
t moves to the closure, so the closure live as long as t
The closure returns a reference to t which was moved to the closure. So the reference is valid as long as the closure exists.
There is no lifetime problem, the code compiles.
What actually happens:
The code does not compile:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
|
note: first, the lifetime cannot outlive the lifetime as defined on the body at 2:14...
--> src/lib.rs:2:14
|
2 | Box::new(move || &t)
| ^^^^^^^^^^
note: ...so that closure can access `t`
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
note: but, the lifetime must be valid for the lifetime 'a as defined on the function body at 1:8...
--> src/lib.rs:1:8
|
1 | fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
| ^^
= note: ...so that the expression is assignable:
expected std::boxed::Box<(dyn std::ops::Fn() -> &'a T + 'a)>
found std::boxed::Box<dyn std::ops::Fn() -> &T>
I do not understand the conflict. How can I fix it?
Very interesting question! I think I understood the problem(s) at play here. Let me try to explain.
tl;dr: closures cannot return references to values captured by moving, because that would be a reference to self. Such a reference cannot be returned because the Fn* traits don't allow us to express that. This is basically the same as the streaming iterator problem and could be fixed via GATs (generic associated types).
Implementing it manually
As you probably know, when you write a closure, the compiler will generate a struct and impl blocks for the appropriate Fn traits, so closures are basically syntax sugar. Let's try to avoid all that sugar and build your type manually.
What you want is a type which owns another type and can return references to that owned type. And you want to have a function which returns a boxed instance of said type.
struct Baz<T>(T);
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
fn make_baz<T>(t: T) -> Box<Baz<T>> {
Box::new(Baz(t))
}
This is pretty equivalent to your boxed closure. Let's try to use it:
let outside = {
let s = "hi".to_string();
let baz = make_baz(s);
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // works too
This works just fine. The string s is moved into the Baz type and that Baz instance is moved into the Box. s is now owned by baz and then by outside.
It gets more interesting when we add a single character:
let outside = {
let s = "hi".to_string();
let baz = make_baz(&s); // <-- NOW BORROWED!
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // doesn't work!
Now we cannot make the lifetime of baz bigger than the lifetime of s, since baz contains a reference to s which would be an dangling reference of s would go out of scope earlier than baz.
The point I wanted to make with this snippet: we didn't need to annotate any lifetimes on the type Baz to make this safe; Rust figured it out on its own and enforces that baz lives no longer than s. This will be important below.
Writing a trait for it
So far we only covered the basics. Let's try to write a trait like Fn to get closer to your original problem:
trait MyFn {
type Output;
fn call(&self) -> Self::Output;
}
In our trait, there are no function parameters, but otherwise it's fairly identical to the real Fn trait.
Let's implement it!
impl<T> MyFn for Baz<T> {
type Output = ???;
fn call(&self) -> Self::Output {
&self.0
}
}
Now we have a problem: what do we write instead of ???? Naively one would write &T... but we need a lifetime parameter for that reference. Where do we get one? What lifetime does the return value even have?
Let's check the function we implemented before:
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
So here we use &T without lifetime parameter too. But this only works because of lifetime elision. Basically, the compiler fills in the blanks so that fn call(&self) -> &T is equivalent to:
fn call<'s>(&'s self) -> &'s T
Aha, so the lifetime of the returned reference is bound to the self lifetime! (more experienced Rust users might already have a feeling where this is going...).
(As a side note: why is the returned reference not dependent on the lifetime of T itself? If T references something non-'static then this has to be accounted for, right? Yes, but it is already accounted for! Remember that no instance of Baz<T> can ever live longer than the thing T might reference. So the self lifetime is already shorter than whatever lifetime T might have. Thus we only need to concentrate on the self lifetime)
But how do we express that in the trait impl? Turns out: we can't (yet). This problem is regularly mentioned in the context of streaming iterators -- that is, iterators that return an item with a lifetime bound to the self lifetime. In today's Rust, it is sadly impossible to implement this; the type system is not strong enough.
What about the future?
Luckily, there is an RFC "Generic Associated Types" which was merged some time ago. This RFC extends the Rust type system to allow associated types of traits to be generic (over other types and lifetimes).
Let's see how we can make your example (kinda) work with GATs (according to the RFC; this stuff doesn't work yet ☹). First we have to change the trait definition:
trait MyFn {
type Output<'a>; // <-- we added <'a> to make it generic
fn call(&self) -> Self::Output;
}
The function signature hasn't changed in the code, but notice that lifetime elision kicks in! The above fn call(&self) -> Self::Output is equivalent to:
fn call<'s>(&'s self) -> Self::Output<'s>
So the lifetime of the associated type is bound to the self lifetime. Just as we wanted! The impl looks like this:
impl<T> MyFn for Baz<T> {
type Output<'a> = &'a T;
fn call(&self) -> Self::Output {
&self.0
}
}
To return a boxed MyFn we would need to write this (according to this section of the RFC:
fn make_baz<T>(t: T) -> Box<for<'a> MyFn<Output<'a> = &'a T>> {
Box::new(Baz(t))
}
And what if we want to use the real Fn trait? As far as I understand, we can't, even with GATs. I think it's impossible to change the existing Fn trait to use GATs in a backwards compatible manner. So it's likely that the standard library will keep the less powerful trait as is. (side note: how to evolve the standard library in backwards incompatible ways to use new language features is something I wondered about a few times already; so far I haven't heard of any real plan in this regards; I hope the Rust team comes up with something...)
Summary
What you want is not technically impossible or unsafe (we implemented it as a simple struct and it works). However, unfortunately it is impossible to express what you want in the form of closures/Fn traits in Rust's type system right now. This is the same problem streaming iterators are dealing with.
With the planned GAT feature, it is possible to express all of this in the type system. However, the standard library would need to catch up somehow to make your exact code possible.
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
This makes no sense. A value cannot "live as long" as a type, because a type doesn't live. "T has lifetime 'a" is a very imprecise statement, easy to misunderstand. What T: 'a really means is "instances of T must stay valid at least as long as lifetime 'a. For example, T must not be a reference with a lifetime shorter than 'a, or a struct containing such a reference. Note that this has nothing to do with forming references to T, i.e. &T.
The value t, then, lives as long as its lexical scope (it's a function parameter) says it does, which has nothing to do with 'a at all.
t moves to the closure, so the closure live as long as t
This is also incorrect. The closure lives as long as the closure does lexically. It is a temporary in the result expression, and therefore lives until the end of the result expression. t's lifetime concerns the closure not at all, since it has its own T variable inside, the capture of t. Since the capture is a copy/move of t, it is not in any way affected by t's lifetime.
The temporary closure is then moved into the box's storage, but that's a new object with its own lifetime. The lifetime of that closure is bound to the lifetime of the box, i.e. it is the return value of the function, and later (if you store the box outside the function) the lifetime of whatever variable you store the box in.
All of that means that a closure that returns a reference to its own capture state must bind the lifetime of that reference to its own reference. Unfortunately, this is not possible.
Here's why:
The Fn trait implies the FnMut trait, which in turn implies the FnOnce trait. That is, every function object in Rust can be called with a by-value self argument. This means that every function object must be still valid being called with a by-value self argument and returning the same thing as always.
In other words, trying to write a closure that returns a reference to its own captures expands to roughly this code:
struct Closure<T> {
captured: T,
}
impl<T> FnOnce<()> for Closure<T> {
type Output = &'??? T; // what do I put as lifetime here?
fn call_once(self, _: ()) -> Self::Output {
&self.captured // returning reference to local variable
// no matter what, the reference would be invalid once we return
}
}
And this is why what you're trying to do is fundamentally impossible. Take a step back, think of what you're actually trying to accomplish with this closure, and find some other way to accomplish it.
You expect the type T to have lifetime 'a, but t is not a reference to a value of type T. The function takes ownership of the variable t by argument passing:
// t is moved here, t lifetime is the scope of the function
fn foo<'a, T: 'a>(t: T)
You should do:
fn foo<'a, T: 'a>(t: &'a T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || t)
}
The other answers are top-notch, but I wanted to chime in with another reason your original code couldn't work. A big problem lies in the signature:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a>
This says that the caller may specify any lifetime when calling foo and the code will be valid and memory-safe. That cannot possibly be true for this code. It wouldn't make sense to call this with 'a set to 'static, but nothing about this signature would prevent that.