Why does this compile:
fn func<T>(
callback: impl FnOnce(&mut i64) -> T,
) -> T {
let v = 42;
callback(&mut 42)
}
but this not?:
fn func<'a, T>(
callback: impl FnOnce(&'a mut i64) -> T,
) -> T {
let v = 42;
callback(&mut 42)
}
Even this doesn't compile:
fn func<'a, T: 'static>(
callback: impl FnOnce(&'a mut i64) -> T,
) -> T {
let v = 42;
callback(&mut 42)
}
Is there a way to tell Rust that the T returned from the callback doesn't hold any references to 'a? I thought : 'static would forbid references in general but it doesn't seem to work. Unfortunately, I do need to give 'a a name because I'm using the lifetime elsewhere, the actual code is somewhat more complicated than this minimal example.
In your first snippet, the lifetimes are elided via Higher-Ranked Trait Bounds:
fn func<T>(
callback: impl for<'a> FnOnce(&'a mut i64) -> T,
) -> T {
let v = 42;
callback(&mut 42)
}
That means the closure can be called with any lifetime, and that includes the lifetime of v.
In your second snippet, the lifetime is chosen by the caller. This is not related to T: for example, the caller can choose 'static, then store the parameter in a static. There is just no way to satisfy this requirement with a variable defined in the function.
Related
I have a function that expects a short lived object. I would expect that I would be able to always pass it a long lived object. But I am getting a strange error when I try to encode that:
type F<'arg> = Box<dyn FnOnce(&'arg ())>;
fn contravar<'small, 'large: 'small>(f: F<'small>) -> F<'large> {
f
}
playground
Particularly:
error: lifetime may not live long enough
--> src/lib.rs:3:5
|
2 | fn contravar<'small, 'large: 'small>(f: F<'small>) -> F<'large> {
| ------ ------ lifetime `'large` defined here
| |
| lifetime `'small` defined here
3 | f
| ^ function was supposed to return data with lifetime `'large` but it is returning data with lifetime `'small`
|
= help: consider adding the following bound: `'small: 'large`
It seems like F is invariant for its argument but I would have guessed that it's contravariant. Am I missing something? Is there a way to make F<'arg> really contravariant for 'arg?
Edit: it looks like the "problem" is that rust wants to treat all generic traits the same (including Fn/FnMut/FnOnce). My opinion is that those 3 are and should be treated special especially given that they are the only way to refer to closures. For that reason I opened an issue
The Rust Reference's page on Subtyping and Variance documents that, as of Rust 1.63.0, fn(T) -> () is contravariant over T and that dyn Trait<T> + 'a is invariant over T.
FnOnce, FnMut and Fn are traits, so that means dyn FnOnce(&'a ()) is unfortunately invariant over &'a ().
// Compiles
pub fn contravariant<'a, 'b: 'a>(x: fn(&'a ())) -> fn(&'b ()) { x }
// Doesn't compile
pub fn contravariant2<'a, 'b: 'a>(x: Box<dyn FnOnce(&'a ())>) -> Box<dyn FnOnce(&'b ())> { x }
Is there a way to wrap FnOnce somehow to convince the compiler of the correct variance?
Here's what I could come up with using unsafe code. Note that I'm not making any guarantees as to whether this is sound or not. I don't know of any way to do this without unsafe code.
use std::marker::PhantomData;
trait Erased {}
impl<T> Erased for T {}
pub struct VariantBoxedFnOnce<Arg, Output> {
boxed_real_fn: Box<dyn Erased + 'static>,
_phantom_fn: PhantomData<fn(Arg) -> Output>,
}
impl<Arg, Output> VariantBoxedFnOnce<Arg, Output> {
pub fn new(real_fn: Box<dyn FnOnce(Arg) -> Output>) -> Self {
let boxed_real_fn: Box<dyn Erased + '_> = Box::new(real_fn);
let boxed_real_fn: Box<dyn Erased + 'static> = unsafe {
// Step through *const T because *mut T is invariant over T
Box::from_raw(Box::into_raw(boxed_real_fn) as *const (dyn Erased + '_) as *mut (dyn Erased + 'static))
};
Self {
boxed_real_fn,
_phantom_fn: PhantomData,
}
}
pub fn call_once(self, arg: Arg) -> Output {
let boxed_real_fn: Box<Box<dyn FnOnce(Arg) -> Output>> = unsafe {
// Based on Box<dyn Any>::downcast()
Box::from_raw(Box::into_raw(self.boxed_real_fn) as *mut Box<dyn FnOnce(Arg) -> Output>)
};
boxed_real_fn(arg)
}
}
pub fn contravariant<'a, 'b: 'a>(x: VariantBoxedFnOnce<&'a (), ()>) -> VariantBoxedFnOnce<&'b (), ()> { x }
#[cfg(test)]
mod tests {
use super::*;
fn foo(_x: &()) {}
#[test]
pub fn check_fn_does_not_require_static() {
let f = VariantBoxedFnOnce::new(Box::new(foo));
let x = ();
f.call_once(&x);
}
#[test]
pub fn check_fn_arg_is_contravariant() {
let f = VariantBoxedFnOnce::new(Box::new(foo));
let g = contravariant(f);
let x = ();
g.call_once(&x);
}
}
Here, VariantBoxedFnOnce is limited to functions taking one argument.
The trick is to store a type-erased version of the Box<dyn FnOnce(Arg) -> Output> such that Arg disappears, because we don't want the variance of VariantBoxedFnOnce<Arg, Output> to depend on Box<dyn FnOnce(Arg) -> Output> (which is invariant over Arg). However, there's also a PhantomData<fn(Arg) -> Output> field to provide the proper contravariance over Arg (and covariance over Output).
We can't use Any as our erased type, because only 'static types implement Any, and we have a step in VariantBoxedFnOnce::new() where we have a Box<dyn Erased + '_> where '_ is not guaranteed to be 'static. We then immediately "transmute" it into 'static, to avoid having a redundant lifetime parameter on VariantBoxedFnOnce, but that 'static is a lie (hence the unsafe code). call_once "downcasts" the erased type to the "original" Box<dyn FnOnce(Arg) -> Output>, except that Arg and Output may be different from the original due to variance.
I am seeking help to understand why the borrow checker fails for the following minimal non-working example, and I would be very happy to learn how to correctly implement what I was trying to do:
use std::collections::HashSet;
struct Foo {
data: HashSet<usize>
}
impl Foo {
fn test<'a, F, T>(&mut self, _operation: F) -> ()
where F: Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
T: Iterator<Item=&'a usize>
{
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation(&self.data, &update).copied().collect();
}
fn new() -> Self {
Foo { data: HashSet::new() }
}
}
fn main() {
let mut foo: Foo = Foo::new();
foo.test(HashSet::intersection);
}
My main source of confusion is that, if I replace the call to _operation with HashSet::intersection, the code compiles. I thought that the type of the parameter _operation would allow me to pass both HashSet::intersection and HashSet::union as operations here.
For the record, this is the error I receive:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> src\main.rs:13:32
|
13 | self.data = _operation(&self.data, &update).copied().collect();
| ^^^^^^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime defined on the method body at 8:23...
--> src\main.rs:8:23
|
8 | fn test<'a, F, T>(&mut self, _operation: F) -> ()
| ^^^^^^^^^
note: ...so that reference does not outlive borrowed content
--> src\main.rs:13:32
|
13 | self.data = _operation(&self.data, &update).copied().collect();
| ^^^^^^^^^^
note: but, the lifetime must be valid for the lifetime `'a` as defined on the method body at 8:13...
--> src\main.rs:8:13
|
8 | fn test<'a, F, T>(&mut self, _operation: F) -> ()
| ^^
note: ...so that reference does not outlive borrowed content
--> src\main.rs:13:32
|
13 | self.data = _operation(&self.data, &update).copied().collect();
| ^^^^^^^^^^
For more information about this error, try `rustc --explain E0495`.
error: could not compile `aoc06` due to previous error
The issue, as the (albeit cryptic) compiler message suggests, is that there is a lifetime mismatch: _operation expects HashSet references that live as long as 'a, but &self.data has a lifetime 'b, the elided lifetime of &mut self, and &update has a different lifetime that lasts the duration of the test function body.
To resolve this issue, we must specify that the function type F takes in HashMap references of arbitrary lifetimes, not just the specific lifetime 'a -- this lets the compiler infer the appropriate lifetime when _operation is invoked. This is why we need Higher-Rank Trait Bounds (HRTBs):
fn test<F, T>(&mut self, _operation: F) -> ()
where F: for<'a> Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
However, this raises another issue. How do we apply the higher-ranked lifetime 'a to the type parameter T? Unfortunately Rust does not support higher-kinded types, but we can get away with "abstracting" out the the function type F and the higher-kinded type T to a trait and an associated type on said trait.
trait Operation<'a, T: 'a> {
type Output: Iterator<Item = &'a T>;
fn operate(self, a: &'a HashSet<T>, b: &'a HashSet<T>) -> Self::Output;
}
The Operation trait represents an operation on two HashSets that returns an iterator over references, equivalent to the functions HashSet::union, HashSet::intersection, and the like. We can achieve this using the following impl, which ensures that HashSet::intersection and the like implement Operation:
impl<'a, T: 'a, I, F> Operation<'a, T> for F
where
I: Iterator<Item = &'a T>,
F: FnOnce(&'a HashSet<T>, &'a HashSet<T>) -> I,
{
type Output = I;
fn operate(self, a: &'a HashSet<T>, b: &'a HashSet<T>) -> Self::Output {
self(a, b)
}
}
We can then use a HRTB on the Operation trait instead, which does not require any nested higher-kinded types:
fn test(&mut self, _operation: impl for<'a> Operation<'a, usize>) -> () {
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation.operate(&self.data, &update).copied().collect();
println!("{:?}", self.data);
}
Playground
The arguments you are passing do not match the lifetime which you declared the Fn bound with.
fn test<'a, F, T>(&mut self, _operation: F) -> ()
'a is some arbitrary lifetime that may be specified by the caller,
F: Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
which must be adequate for the references given to _operation,
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation(&self.data, &update).copied().collect();
but here you pass in a borrow from self (whose lifetime is not specified to outlive 'a) and a borrow from update (which is a local variable, which cannot outlive 'a).
In order to correctly write this, you need to specify that _operation may be called with any lifetime (which thus includes the lifetimes of borrows of local variables). That's simple, by itself:
fn test<F, T>(&mut self, _operation: F) -> ()
where
F: for<'a> Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
Note that 'a is no longer a lifetime parameter of test. Instead it's part of the bound on F: you can read this for<'a> notation as “for any lifetime, which we will call 'a, F may be called as a function with references &'a ...”.
However, this isn't actually a solution, because you also have T: Iterator<Item = &'a usize>, which uses 'a again. It isn't currently possible to write a where clause that expresses this relationship, particularly as even without the item being a reference, the iterator will be borrowing the &'a HashSets.
This is an unfortunate limitation of current Rust — it also comes up in trying to write a function that takes an async function which borrows an input (which is structurally the same as your situation, with Future in place of Iterator). However, there is a workaround: you can define a trait for the function, which has a single lifetime parameter that links everything together. (This doesn't impose any extra work on the caller of your function, because the trait is blanket implemented for all suitable functions.)
Here's your code with such a trait added and the needed modifications to fn test():
use std::collections::HashSet;
trait IteratorCallback<'a> {
type Output: Iterator<Item = &'a usize> + 'a;
fn call(self, a: &'a HashSet<usize>, b: &'a HashSet<usize>) -> Self::Output;
}
impl<'a, F, T> IteratorCallback<'a> for F
where
F: FnOnce(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
T: Iterator<Item = &'a usize> + 'a,
{
type Output = T;
fn call(self, a: &'a HashSet<usize>, b: &'a HashSet<usize>) -> T {
// Delegate to FnOnce
self(a, b)
}
}
struct Foo {
data: HashSet<usize>,
}
impl Foo {
fn test<F>(&mut self, _operation: F) -> ()
where
F: for<'a> IteratorCallback<'a>,
{
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation.call(&self.data, &update).copied().collect();
}
fn new() -> Self {
Foo {
data: HashSet::new(),
}
}
}
fn main() {
let mut foo: Foo = Foo::new();
foo.test(HashSet::intersection);
}
Note: I changed the function bound to FnOnce because that's more permissive than Fn and all you need in this case, but the same technique will work with Fn as long as you change fn call(self, to fn call(&self,.
Credit: I used this Reddit comment by user Lej77 as an example to work from for the trait technique.
Sometimes I struggle with lifetimes. I'm still learning and I don't know what's happening here:
use std::future::Future;
use futures::future::{BoxFuture, FutureExt};
struct M{}
struct Client{}
impl Client {
async fn send_and_expect<'a>(
&'a mut self,
m: &M
) -> std::result::Result<(), ()> {
Ok(())
}
pub fn connection_retrier<'a, T>(
f: fn(&'a mut Self, &M) -> T,
f_self: &'a mut Self,
f_m: &'a M,
)-> BoxFuture<'a, std::result::Result<(),()>>
where
T: Future<Output = std::result::Result<(), ()>> + 'a
{
async move {
Client::send_and_expect(f_self, f_m).await
}.boxed()
}
async fn send_with_retry<'a>(&'a mut self) -> std::result::Result<(), ()> {
let m = M{};
Client::connection_retrier(
Client::send_and_expect,
&mut *self, &m).await
}
}
Error:
Compiling playground v0.0.1 (/playground)
error[E0308]: mismatched types
--> src/lib.rs:31:21
|
11 | ) -> std::result::Result<(), ()> {
| --------------------------- the `Output` of this `async fn`'s found opaque type
...
31 | Client::send_and_expect,
| ^^^^^^^^^^^^^^^^^^^^^^^ one type is more general than the other
|
= note: expected fn pointer `for<'r> fn(&mut Client, &'r M) -> impl futures::Future`
found fn pointer `for<'a, '_> fn(&'a mut Client, &M) -> impl futures::Future
Playground
I'm now completely confused about why in for<'r> fn(&mut Client, &'r M) -> impl futures::Future, &mut Client has no lifetime. And what does _ means in for<'a, '_> fn(&'a mut Client, &M) -> impl futures::Future`?
I'm very intersted in learning what's happening here.
Cleaning up the code
In order to see this code from the compiler's perspective, let's change all the lifetime parameters in this example to be explicit and have distinct names, expand the async fn sugar, and then look at how the error changes.
The above example is equivalent to the following after lifetime inference and desugaring.
// name the second lifetime, and expand the async fn sugar.
fn send_and_expect<'a, 'b>(&'a mut self, m: &'b M) -> impl Future<Item=Result<(), ()>> + 'a + 'b
{ ... }
// rename 'a to 'c to avoid ambiguous lifetime names
pub fn connection_retrier<'c, T>(
f: for<'d> fn(&'c mut Self, &'d M) -> T, // name the implicit higher-ranked lifetime here
f_self: &'c mut Self,
f_m: &'c M,
)-> BoxFuture<'c, std::result::Result<(), ()>>
where T: Future<Output = std::result::Result<(), ()>> + 'c
{ ... }
// rename 'a to 'e to avoid ambiguous lifetime names
async fn send_with_retry<'e>(&'e mut self) -> std::result::Result<(), ()> {
After making this change, the error becomes:
Compiling playground v0.0.1 (/playground)
error[E0308]: mismatched types
--> src/lib.rs:31:21
|
11 | ) -> std::result::Result<(), ()> {
| --------------------------- the `Output` of this `async fn`'s found opaque type
...
31 | Client::send_and_expect,
| ^^^^^^^^^^^^^^^^^^^^^^^ one type is more general than the other
|
= note: expected fn pointer `for<'d> fn(&mut Client, &'d M) -> impl futures::Future`
found fn pointer `for<'a, 'b> fn(&'a mut Client, &'b M) -> impl futures::Future`
This should clarify your question about '_: it's just the name the compiler gave the inferred second lifetime parameter of send_and_expect. As for the missing lifetime on &mut Client, you can see that it's still missing here. For reasons I do not completely understand, and in ways that depend on the exact error message given, the compiler will sometimes omit concrete lifetimes when printing the type of references, but make no mistake, the lifetime of that reference is 'c.
Solving the error
Onto the actual problem. The signature of f indicates that connection_retrier is expecting a function which (1) takes a reference of lifetime &'c and (2) a reference of any other lifetime, and (3) returns a Future type which will remain valid as long as 'c does, as specified by your where bound.
When we pass send_and_expect to connection_retrier, in order for the signatures to match the compiler is coercing it to the type for<'d> send_and_expect::<'c, 'd>. But that type doesn't meet condition (3) above! Unlike a regular function, the default behavior of an async function is to capture all input lifetimes in its return type, so the return type of for<'d> send_and_expect::<'c, 'd> is in fact impl Future<Item=Result<(), ()>> + 'c + 'd, as you can tell by looking at send_and_expect's expanded signature.
Since this type borrows from the two lifetimes 'c and 'd, and there is no constraint that 'd: 'c (read: 'd outlives 'c), it may not remain valid for the entirety of the lifetime 'c, if 'd ends first. It is this mismatch that results in the rather cryptic lifetime error you received. There are two ways you can solve this problem, depending on your preferred semantics. You can either:
Remove the higher-ranked bound from f entirely, and specify exactly the lifetimes you will be calling it with in connection_retrier (both &'c.)
pub fn connection_retrier<'c, T>(
f: fn(&'c mut Self, &'c M) -> T
f_self: &'c mut Self,
f_m: &'c M,
) -> BoxFuture<'c, std::result::Result<(), ()>>
where T: Future<Output = std::result::Result<(), ()>> + 'c
{ ... }
Keep the signature of connection_retrier the same and specify that the future returned by send_and_expect only borrows from its first argument. To do this, you will need to drop the async fn sugar on the signature and wrap the body in an async move block.
fn send_and_expect<'a, 'b>(&'a mut self, m: &'b M) -> impl Future<Output=Result<(), ()>> + 'a
{ async move { ... } }
Note that you cannot solve this by writing the type of f as for<'d: 'c> fn(&'c mut Self, &'d M) -> T, as bounds are currently not permitted for universally quantified lifetimes.
Consider next code:
fn get_ref<'a, R>(slice: &'a Vec<i32>, f: fn(&'a Vec<i32>) -> R) -> R
where
R: 'a,
{
f(slice)
}
fn main() {
let v = [1,2,3,4,5,6];
let iter = get_ref(&v, |x| x.iter().skip(1).take(2));
println!("{:?}", iter.collect::<Vec<_>>());
}
I create some static variable, then apply some function to its reference and get a result.
It seems to work totally fine. At least it successfully compiles.
Now I am trying to add next level of abstraction. And things are getting weird...
fn owned<'a, R>(owner: Vec<i32>, f: fn(&'a Vec<i32>) -> R)
where
R: 'a,
{
let _ = get_ref(&owner, f); // error occurs here
// `owner` does not live long enough.
}
// get_ref is the same as in the first example
fn get_ref<'a, R>(slice: &'a Vec<i32>, f: fn(&'a Vec<i32>) -> R) -> R
where
R: 'a,
{
f(slice)
}
fn main() {
let v = [1,2,3,4,5,6];
owned(v, |x| x.iter().skip(1).take(2));
}
For me it looks like pretty the same code. But Rust fails to compile it. I really don't understand why this is happening and how should I rewrite my code to compile.
Imagine if I decide to call the function owned<'static, i32>(Vec<i32>, foo) with foo defined as:
fn foo(vec: &'static Vec<i32>) -> i32 { ... }
This satisfies the bounds for owned, since i32: 'static. However this means that you must have a static reference to call f, but owner does not live forever, since it is destroyed at the end of owned.
One way to fix it is to use the following:
fn owned<R>(owner: Vec<i32>, f: for<'a> fn(&'a Vec<i32>) -> R) {
let _ = get_ref(&owner, f);
}
It says that f must be callable with any lifetime, rather than just some specific lifetime. However, this has the consequence that R cannot borrow from the argument, since R is declared in a larger scope than 'a. There isn't any way to fix that while keeping the generics as is.
This answer was taken from my response to this thread on URLO.
Can somebody help me to rewrite this piece of code with new unboxed closures:
struct Builder;
pub fn build(rules: |params: &mut Builder|) -> Builder {
let mut builder = Builder::new();
rules(&mut builder);
builder
}
I tried to write like this, but I got a lifetime error:
pub fn build<F>(rules: F) -> Builder where F: FnOnce<(&mut Builder,), ()> {
let mut builder = Builder::new();
rules(&mut builder);
builder
}
valico/src/builder.rs:48:59: 48:71 error: missing lifetime specifier [E0106]
valico/src/builder.rs:48 pub fn build<F>(rules: F) -> Builder where F: FnOnce<(&mut Builder,), ()> {
^~~~~~~~~~~~
What lifetime I need to specify? Simplified example in the sandbox.
This requires higher rank trait bounds, specifically, higher rank lifetimes. The full unsugared syntax would be F: for<'a> FnOnce<(&'a mut Builder,), ()>.
Using a lifetime on the function can't work, e.g. if we had
pub fn build<'b, F>(rules: F) -> Builder where F: FnOnce<(&'b mut Builder,), ()>
This says that build works with whatever lifetime the caller wishes (e.g. they could chose 'b == 'static), but this is invalid, because there is a specific concrete lifetime that needs to be the used: the lifetime of the &mut builder inside the function. Using F: for<'a> ... in a bound says that F works with any lifetime 'a, so the compiler sees that it is legal to substitute in the one of &mut builder.
As I hinted above, that's the really ugly unsugared syntax. There's two successive ways this can be made much nicer. Firstly, the canonical way to use the closure traits is the () sugar: for<'a> FnOnce(&'a mut Builder) -> (), or, like with the rest of Rust, the -> () can be dropped: for<'a> FnOnce(&'a mut Builder). (NB. this is just sugar for FnOnce<...>, but only the sugared syntax will be stabilised for interacting with these traits at 1.0.)
Then, the paren syntax has a little extra rule: it automatically inserts lifetimes that act like for<'a> (specifically, it undergoes lifetime elision with any inserted lifetime placed into a for on the trait), so just F: FnOnce(&mut Builder) is equivalent to F: for<'a> FnOnce(&'a mut Builder), and it's the recommended version.
Applying these fixes to your playpen example:
pub fn initialize_with_closure<F>(rules: F) -> uint where F: FnOnce(&mut uint) {
let mut i = 0;
rules(&mut i);
i
}
// equivalently
pub fn initialize_with_closure_explicit<F>(rules: F) -> uint where F: for<'a> FnOnce(&'a mut uint) -> () {
let mut i = 0;
rules(&mut i);
i
}
pub fn main() {
initialize_with_closure(|i: &mut uint| *i = *i + 20);
initialize_with_closure_explicit(|i: &mut uint| *i = *i + 20);
}
playpen