Related
Rust reference object-safety confused me for a while, and says:
Explicitly non-dispatchable functions require:
Have a where Self: Sized bound (receiver type of Self (i.e. self) implies this).
But I found code::iter::Iterator has dozen of methods are declared as explicitly non-dispatchable functions, one of them below:
pub trait Iterator {
...
fn count(self) -> usize
where
Self: Sized,
{
self.fold(
0,
#[rustc_inherit_overflow_checks]
|count, _| count + 1,
)
}
...
}
However, all of them are dispatchable by trait-object at rust-playground:
fn main() {
let it: &mut dyn Iterator<Item = u32> = &mut [1, 2, 3].into_iter();
assert_eq!(3, it.count()); // ok
}
That is good, I start try to implements a worked example, but it can not be dispatched at rust-playground, and report compiler error: "the dispatch method cannot be invoked on a trait object" that is expected:
fn main() {
pub trait Sup {
fn dispatch(self) -> String
where
Self: Sized,
{
"sup".to_string()
}
}
struct Sub;
impl Sup for Sub {
fn dispatch(self) -> String {
"sub".to_string()
}
}
let it: &mut dyn Sup = &mut Sub;
assert_eq!("trait", it.dispatch());
}
Why explicitly non-dispatchable methods in code::iter::Iterator are dispatchable? Is there any magic which I didn't found?
The reason is simple, if we think of this: what type we're invoking the method count on?
Is it dyn Iterator<Item = u32>? Let's check:
assert_eq!(3, <dyn Iterator<Item = u32>>::count(it));
Nope, there are two errors:
error[E0308]: mismatched types
--> src/main.rs:3:53
|
3 | assert_eq!(3, <dyn Iterator<Item = u32>>::count(it));
| ^^ expected trait object `dyn Iterator`, found mutable reference
|
= note: expected trait object `dyn Iterator<Item = u32>`
found mutable reference `&mut dyn Iterator<Item = u32>`
error[E0277]: the size for values of type `dyn Iterator<Item = u32>` cannot be known at compilation time
--> src/main.rs:3:53
|
3 | assert_eq!(3, <dyn Iterator<Item = u32>>::count(it));
| --------------------------------- ^^ doesn't have a size known at compile-time
| |
| required by a bound introduced by this call
|
= help: the trait `Sized` is not implemented for `dyn Iterator<Item = u32>`
note: required by a bound in `count`
OK, well... is it &mut dyn Iterator, then?
assert_eq!(3, <&mut dyn Iterator<Item = u32>>::count(it));
Now it compiles. It's understandable that the second error goes away - &mut T is always Sized. But why do the &mut dyn Iterator has access to the method of Iterator?
The answer is in the documentation. First, dyn Iterator obviously implements Iterator - that's true for any trait. Second, there's implementation of Iterator for any &mut I, where I: Iterator + ?Sized - which our dyn Iterator satisfies.
Now, one may ask - what code is executed when we use this implementation? After all, count requires consuming self, so calling it on reference can't delegate to the dyn Iterator - otherwise we'd be back to square one, dispatching non-dispatchable.
Here, the answer lies in the structure of the Iterator trait. As one can see, it has only a single required method, namely next, which takes &mut self; all other methods are provided, that is, they have some default implementations using next - for example, here's it for count:
fn count(self) -> usize
where
Self: Sized,
{
self.fold(
0,
#[rustc_inherit_overflow_checks]
|count, _| count + 1,
)
}
where fold, in turn, is the following:
fn fold<B, F>(mut self, init: B, mut f: F) -> B
where
Self: Sized,
F: FnMut(B, Self::Item) -> B,
{
let mut accum = init;
while let Some(x) = self.next() {
accum = f(accum, x);
}
accum
}
As you can see, knowing just the next, compiler can derive fold and then count.
Now, back to our &mut dyn Iterators. Let's check how, exactly, this implementation looks like; it appears to be quite simple:
#[stable(feature = "rust1", since = "1.0.0")]
impl<I: Iterator + ?Sized> Iterator for &mut I {
type Item = I::Item;
#[inline]
fn next(&mut self) -> Option<I::Item> {
(**self).next()
}
fn size_hint(&self) -> (usize, Option<usize>) {
(**self).size_hint()
}
fn advance_by(&mut self, n: usize) -> Result<(), usize> {
(**self).advance_by(n)
}
fn nth(&mut self, n: usize) -> Option<Self::Item> {
(**self).nth(n)
}
}
You can see that the &self and &mut self methods, i.e. the ones which can be called on the trait object, are forwarded by the reference to the inner value and dispatched dynamically.
The self methods, i.e. the ones which cannot use the trait object, are dispached statically using their default implementation, which consume the reference and pass it, eventually, into one of these - non-consuming, dynamically-dispatched - methods.
The following error goes away if I do as rustc tells me to, and change the bound to
where F: Fn() -> () + 'static
pub struct Struct(Box<dyn Fn() -> ()>);
pub fn example<F>(f: F)
where
F: Fn() -> ()
{
Struct(Box::new(|| ())); // ok
Struct(Box::new(f)); // error: The parameter type `F` may not live long eneough
// help: consider adding an explicit lifetime bound: `F: 'static`
}
However, I just don't understand what 'static means here. It doesn't seem to mean that the closure itself lives forever.
At least, it doesn't seem like the closure lives forever. If I have the closure own some data, that data gets dropped at some point, which makes me suspect the closure is dropped:
pub struct Struct(Box<dyn Fn() -> ()>);
#[derive(Debug)]
struct DropMe;
impl Drop for DropMe {
fn drop(&mut self) {
println!("dropped");
}
}
/// prints:
/// "got DropMe"
/// "got DropMe"
/// "dropped"
/// "end of program"
pub fn main() {
let d = DropMe;
example(move || {
println!("got {:?}", d);
});
println!("end of program")
}
pub fn example<F>(f: F)
where
F: Fn() -> () + 'static
{
let s = Struct(Box::new(f));
s.0();
s.0();
}
Is 'static an upper bound on the lifetime of the closure rather than a lower bound? 'static makes sense as an upper bound, then example would be saying "give me a function that I can hold on to as long as I want", which in practice can be less than the 'static lifetime.
How can I tell when + 'lifetime adds 'lifetime as an upper bound vs. a lower bound?
The Rustonomicon chapter on subtyping+variance doesn't seem to cover this, and I couldn't find the information I was looking for in the Rust Book or Rust Reference.
T: 'a isn't a constraint on the lifetime of instances of T, but of things that T borrows (if any): that is, all borrows in T must outlive 'a.
Thus F: Trait + 'static requires that any borrows in F be for 'static lifetime (or longer, which doesn't exist), regardless that Trait in this case is Fn() -> ().
In your case, the closure takes ownership of d (and borrows the &'static str literal); hence it satisfies F: 'static. But if instead of move || ... the closure merely borrowed d with || ..., then it would not be able to satisfy 'static (as the lifetime of the borrow of d cannot exceed the scope of the call to main).
Yes, 'static is an upper bound.
In fact, all lifetimes constraints are upper bounds. For the callee (that is, whom uses the lifetime).
For the caller (i.e. the provider of the lifetime), on the other hand, they're usually lower bounds: give me something that lives at least as 'static (of course, nothing lives more than 'static, so it actually means "give me something 'static". But it does matter when talking about other lifetimes).
Variance is about changing the caller's respect regarding the lifetime: whether it can pass a longer lifetime (covariance, i.e. a lower bound), a shorter lifetime (contravariance, i.e. an upper bound), or only exactly this lifetime (invariance).
The 'static variant is not obligatory. It seems like it's just what compiler proposes by default.
The two edits below should hopefully illustrate the capture of lifetime of variables from environment.
//rustc 1.62.1 (e092d0b6b)
pub struct Struct<'a>(Box<dyn Fn() -> () + 'a>);
#[derive(Debug)]
struct DropMe;
impl Drop for DropMe {
fn drop(&mut self) {
println!("dropped");
}
}
pub fn main() {
let d = DropMe;
let x = 43;
example(move || {
println!("got {:?} {:?}", d, &x);
});
println!("end of program")
}
pub fn example<'env_borrow, F>(f: F)
where
F: Fn() -> () + 'env_borrow
{
let s = Struct(Box::new(f));
s.0();
s.0();
// drop(s);
println!("end of higher bound func");
}
// rustc 1.62.1 (e092d0b6b)
pub struct Struct<'a>(Box<dyn FnMut() -> () + 'a>);
#[derive(Debug)]
struct DropMe;
impl Drop for DropMe {
fn drop(&mut self) {
println!("dropped");
}
}
pub fn main() {
let d = DropMe;
let mut x = 43;
let y = &mut x;
let closure_boxed = example(move || {
println!("got {:?} , *&mut x = {:?}", d, y);
*y += 1;
});
// drop(x);
// ^ ERROR: try uncomment
drop(closure_boxed);
// ^ ERROR: try comment
println!("x is now {}", x);
println!("end of program")
}
pub fn example<'env_borrow, F>(f: F) -> Struct<'env_borrow>
where
F: FnMut() -> () + 'env_borrow
{
let mut s = Struct(Box::new(f));
s.0();
s.0();
s
}
I believe the issue is that Struct has no generic lifetime parameters, which means the lifetime of one of its instances has no name. The function an instance of Struct boxes (obviously) has to live as long as the instance, but the only way to guarantee that to the compiler is to give it a named lifetime. But because Struct has no lifetime parameters, the only named lifetime that is guaranteed to last sufficient long is 'static.
To fix this, you can give Struct a generic lifetime parameter and then constrain the boxed function to also only live that long.
pub struct Struct<'a>(Box<dyn 'a + Fn() -> ()>);
pub fn example<F>(f: F)
where
F: Fn() -> (),
{
Struct(Box::new(|| ())); // ok
Struct(Box::new(f)); // also ok; 'a will be inferred from the lifetime of f
}
I am seeking help to understand why the borrow checker fails for the following minimal non-working example, and I would be very happy to learn how to correctly implement what I was trying to do:
use std::collections::HashSet;
struct Foo {
data: HashSet<usize>
}
impl Foo {
fn test<'a, F, T>(&mut self, _operation: F) -> ()
where F: Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
T: Iterator<Item=&'a usize>
{
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation(&self.data, &update).copied().collect();
}
fn new() -> Self {
Foo { data: HashSet::new() }
}
}
fn main() {
let mut foo: Foo = Foo::new();
foo.test(HashSet::intersection);
}
My main source of confusion is that, if I replace the call to _operation with HashSet::intersection, the code compiles. I thought that the type of the parameter _operation would allow me to pass both HashSet::intersection and HashSet::union as operations here.
For the record, this is the error I receive:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> src\main.rs:13:32
|
13 | self.data = _operation(&self.data, &update).copied().collect();
| ^^^^^^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime defined on the method body at 8:23...
--> src\main.rs:8:23
|
8 | fn test<'a, F, T>(&mut self, _operation: F) -> ()
| ^^^^^^^^^
note: ...so that reference does not outlive borrowed content
--> src\main.rs:13:32
|
13 | self.data = _operation(&self.data, &update).copied().collect();
| ^^^^^^^^^^
note: but, the lifetime must be valid for the lifetime `'a` as defined on the method body at 8:13...
--> src\main.rs:8:13
|
8 | fn test<'a, F, T>(&mut self, _operation: F) -> ()
| ^^
note: ...so that reference does not outlive borrowed content
--> src\main.rs:13:32
|
13 | self.data = _operation(&self.data, &update).copied().collect();
| ^^^^^^^^^^
For more information about this error, try `rustc --explain E0495`.
error: could not compile `aoc06` due to previous error
The issue, as the (albeit cryptic) compiler message suggests, is that there is a lifetime mismatch: _operation expects HashSet references that live as long as 'a, but &self.data has a lifetime 'b, the elided lifetime of &mut self, and &update has a different lifetime that lasts the duration of the test function body.
To resolve this issue, we must specify that the function type F takes in HashMap references of arbitrary lifetimes, not just the specific lifetime 'a -- this lets the compiler infer the appropriate lifetime when _operation is invoked. This is why we need Higher-Rank Trait Bounds (HRTBs):
fn test<F, T>(&mut self, _operation: F) -> ()
where F: for<'a> Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
However, this raises another issue. How do we apply the higher-ranked lifetime 'a to the type parameter T? Unfortunately Rust does not support higher-kinded types, but we can get away with "abstracting" out the the function type F and the higher-kinded type T to a trait and an associated type on said trait.
trait Operation<'a, T: 'a> {
type Output: Iterator<Item = &'a T>;
fn operate(self, a: &'a HashSet<T>, b: &'a HashSet<T>) -> Self::Output;
}
The Operation trait represents an operation on two HashSets that returns an iterator over references, equivalent to the functions HashSet::union, HashSet::intersection, and the like. We can achieve this using the following impl, which ensures that HashSet::intersection and the like implement Operation:
impl<'a, T: 'a, I, F> Operation<'a, T> for F
where
I: Iterator<Item = &'a T>,
F: FnOnce(&'a HashSet<T>, &'a HashSet<T>) -> I,
{
type Output = I;
fn operate(self, a: &'a HashSet<T>, b: &'a HashSet<T>) -> Self::Output {
self(a, b)
}
}
We can then use a HRTB on the Operation trait instead, which does not require any nested higher-kinded types:
fn test(&mut self, _operation: impl for<'a> Operation<'a, usize>) -> () {
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation.operate(&self.data, &update).copied().collect();
println!("{:?}", self.data);
}
Playground
The arguments you are passing do not match the lifetime which you declared the Fn bound with.
fn test<'a, F, T>(&mut self, _operation: F) -> ()
'a is some arbitrary lifetime that may be specified by the caller,
F: Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
which must be adequate for the references given to _operation,
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation(&self.data, &update).copied().collect();
but here you pass in a borrow from self (whose lifetime is not specified to outlive 'a) and a borrow from update (which is a local variable, which cannot outlive 'a).
In order to correctly write this, you need to specify that _operation may be called with any lifetime (which thus includes the lifetimes of borrows of local variables). That's simple, by itself:
fn test<F, T>(&mut self, _operation: F) -> ()
where
F: for<'a> Fn(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
Note that 'a is no longer a lifetime parameter of test. Instead it's part of the bound on F: you can read this for<'a> notation as “for any lifetime, which we will call 'a, F may be called as a function with references &'a ...”.
However, this isn't actually a solution, because you also have T: Iterator<Item = &'a usize>, which uses 'a again. It isn't currently possible to write a where clause that expresses this relationship, particularly as even without the item being a reference, the iterator will be borrowing the &'a HashSets.
This is an unfortunate limitation of current Rust — it also comes up in trying to write a function that takes an async function which borrows an input (which is structurally the same as your situation, with Future in place of Iterator). However, there is a workaround: you can define a trait for the function, which has a single lifetime parameter that links everything together. (This doesn't impose any extra work on the caller of your function, because the trait is blanket implemented for all suitable functions.)
Here's your code with such a trait added and the needed modifications to fn test():
use std::collections::HashSet;
trait IteratorCallback<'a> {
type Output: Iterator<Item = &'a usize> + 'a;
fn call(self, a: &'a HashSet<usize>, b: &'a HashSet<usize>) -> Self::Output;
}
impl<'a, F, T> IteratorCallback<'a> for F
where
F: FnOnce(&'a HashSet<usize>, &'a HashSet<usize>) -> T,
T: Iterator<Item = &'a usize> + 'a,
{
type Output = T;
fn call(self, a: &'a HashSet<usize>, b: &'a HashSet<usize>) -> T {
// Delegate to FnOnce
self(a, b)
}
}
struct Foo {
data: HashSet<usize>,
}
impl Foo {
fn test<F>(&mut self, _operation: F) -> ()
where
F: for<'a> IteratorCallback<'a>,
{
let update: HashSet<usize> = vec![4, 2, 9].into_iter().collect();
self.data = _operation.call(&self.data, &update).copied().collect();
}
fn new() -> Self {
Foo {
data: HashSet::new(),
}
}
}
fn main() {
let mut foo: Foo = Foo::new();
foo.test(HashSet::intersection);
}
Note: I changed the function bound to FnOnce because that's more permissive than Fn and all you need in this case, but the same technique will work with Fn as long as you change fn call(self, to fn call(&self,.
Credit: I used this Reddit comment by user Lej77 as an example to work from for the trait technique.
I'm trying to create a struct that will manage a Tokio task with one tokio::sync::mpsc::Sender that sends input to the task, one tokio::sync::mpsc::Receiver that receives output from the task, and a handle that I can join at the end.
use tokio::sync::mpsc;
use tokio::task::JoinHandle;
// A type that implements BlockFunctionality consumes instances of T and
// produces either Ok(Some(U)) if an output is ready, Ok(None) if an output
// is not ready, or an Err(_) if the operation fails
pub trait BlockFunctionality<T, U> {
fn apply(&mut self, input: T) -> Result<Option<U>, &'static str>;
}
pub struct Block<T, U> {
pub tx_input: mpsc::Sender<T>,
pub rx_output: mpsc::Receiver<U>,
pub handle: JoinHandle<Result<(), &'static str>>,
}
impl<T: Send, U: Send> Block<T, U> {
pub fn from<B: BlockFunctionality<T, U> + Send>(b: B) -> Self {
let (tx_input, mut rx_input) = mpsc::channel(10);
let (mut tx_output, rx_output) = mpsc::channel(10);
let handle: JoinHandle<Result<(), &'static str>> = tokio::spawn(async move {
let mut owned_b = b;
while let Some(t) = rx_input.recv().await {
match owned_b.apply(t)? {
Some(u) => tx_output
.send(u)
.await
.map_err(|_| "Unable to send output")?,
None => (),
}
}
Ok(())
});
Block {
tx_input,
rx_output,
handle,
}
}
}
When I try to compile this, I get this error for B and a similar one for the other two type parameters:
|
22 | pub fn from<B: BlockFunctionality<T, U> + Send>(b:B) -> Self {
| -- help: consider adding an explicit lifetime bound...: `B: 'static +`
...
27 | let handle:JoinHandle<Result<(), &'static str>> = tokio::spawn(async move {
| ^^^^^^^^^^^^ ...so that the type `impl std::future::Future` will meet its required lifetime bounds
I'm having a hard time understanding where the problem is with lifetimes. The way I understand it, lifetime problems usually come from references that don't live long enough, but I'm moving values, not using references. I move b, rx_input, and tx_output into the closure and I keep tx_input, rx_output, and handle in the calling scope. Does anyone know how to satisfy the compiler in this case?
Those values may be references or contain references. Reference types are valid types: B could be &'a str. Or B could be SomeType<'a>, a type with a lifetime parameter, that itself contains a &'a str.
To say that B: 'static means that all lifetime parameters of B outlive 'static (ref). For example, types which own their own data and thus have no lifetime parameters (e.g. String) satisfy this bound. But &'static str also satisfies the bound.
Because tokio::spawn creates something whose lifetime is not statically scoped, it requires a 'static argument.
So to satisfy the compiler, add the 'static bounds:
impl<T: 'static + Send, U: 'static + Send> Block<T, U> {
pub fn from<B: 'static + BlockFunctionality<T, U> + Send>(b:B) -> Self {
Broadly speaking my goal is this:
For some known type Bar...
Have a trait Foo with a function: get_iterator<T>() -> T where T: Iterator<Item = Bar>
The instance of the iterator borrows the original object Foo is implemented on.
I imagine it working like this:
let mut foo = Foo;
let bar = foo.get_iterator();
foo.mutable_call(); // <-- This fails, because foo is borrowed in bar
for x in bar {
...
}
So, that's the goal, and here's my attempt, which I can't seem to get working:
struct ValsFromT<'a, T: 'a> {
parent:&'a T,
offset: usize,
}
struct Val;
trait HasValsIterator<T> {
fn val_iterator(&self) -> T where T: Iterator<Item = Val>;
}
struct Foo;
impl<'a> Iterator for ValsFromT<'a, Foo> {
type Item = Val;
fn next(&mut self) -> Option<Val> {
return None;
}
}
impl<'a> HasValsIterator<ValsFromT<'a, Foo>> for Foo {
fn val_iterator(&'a self) -> ValsFromT<'a, Foo> {
return ValsFromT {
offset: 0usize,
parent: self
};
}
}
fn takes_vals<T>(instance:T) where T: HasValsIterator<T> {
// ...
}
#[test]
fn test_foo() {
let x = Foo;
takes_vals(x);
}
(playpen: http://is.gd/wys3fx)
We're getting the dreaded concrete/bound lifetime error here, because of trying to return an iterator instance that references self from the trait function:
<anon>:22:3: 27:4 error: method `val_iterator` has an incompatible type for trait:
expected bound lifetime parameter ,
found concrete lifetime [E0053]
<anon>:22 fn val_iterator(&'a self) -> ValsFromT<'a, Foo> {
<anon>:23 return ValsFromT {
<anon>:24 offset: 0usize,
<anon>:25 parent: self
<anon>:26 };
<anon>:27 }
<anon>:22:3: 27:4 help: see the detailed explanation for E0053
Is there some way of doing this?
Unfortunately, Veedrac's suggestion doesn't work directly. You will get the following error if you'd try to use val_iterator() method on instance inside takes_vals():
<anon>:31:25: 31:39 error: the trait `core::iter::Iterator` is not implemented for the type `U` [E0277]
<anon>:31 let iter = instance.val_iterator();
^~~~~~~~~~~~~~
<anon>:31:25: 31:39 help: see the detailed explanation for E0277
<anon>:31:25: 31:39 note: `U` is not an iterator; maybe try calling `.iter()` or a similar method
error: aborting due to previous error
playpen: application terminated with error code 101
This (and some other further errors) requires changing the signature of the function to this one:
fn takes_vals<'a, T: 'a, U: Iterator<Item=Val>+'a>(instance: T) where T: HasValsIterator<'a, U>
However, even this doesn't work yet:
<anon>:31:16: 31:24 error: `instance` does not live long enough
<anon>:31 let iter = instance.val_iterator();
^~~~~~~~
<anon>:30:97: 32:2 note: reference must be valid for the lifetime 'a as defined on the block at 30:96...
<anon>:30 fn takes_vals<'a, T: 'a, U: Iterator<Item=Val>+'a>(instance: T) where T: HasValsIterator<'a, U> {
<anon>:31 let iter = instance.val_iterator();
<anon>:32 }
<anon>:30:97: 32:2 note: ...but borrowed value is only valid for the scope of parameters for function at 30:96
<anon>:30 fn takes_vals<'a, T: 'a, U: Iterator<Item=Val>+'a>(instance: T) where T: HasValsIterator<'a, U> {
<anon>:31 let iter = instance.val_iterator();
<anon>:32 }
Remember that the trait requires that val_iterator() accepts the target by reference with lifetime 'a. This lifetime in this function is an input parameter. However, when val_iterator() is called on instance, the only lifetime which can be specified for the reference is the one of instance which is strictly smaller than any possible 'a as a parameter, because it is a local variable. Therefore, it is not possible to pass instance by value; you can only pass it by reference for lifetimes to match:
fn takes_vals<'a, T: 'a, U: Iterator<Item=Val>+'a>(instance: &'a T) where T: HasValsIterator<'a, U> {
let iter = instance.val_iterator();
}
This works.
I'd like to add that using associated types instead of type parameters would be more correct semantically:
trait HasValsIterator<'a> {
type Iter: Iterator<Item=Val> + 'a;
fn val_iterator(&'a self) -> Self::Iter;
}
impl<'a> HasValsIterator<'a> for Foo {
type Iter = ValsFromT<'a, Foo>;
fn val_iterator(&'a self) -> ValsFromT<'a, Foo> { ... }
}
fn takes_vals<'a, T: 'a>(instance: &'a T) where T: HasValsIterator<'a> {
...
}
I say that this is more correct because the type of the iterator is determined by the implementor, that is, it is "output" type, which are modeled by associated types. As you can see, takes_vals() signature also shrank considerably.
Ideally, HasValsIterator trait should have been defined like this:
trait HasValsIterator {
type Iter<'a>: Iterator<Item=Val> + 'a
fn val_iterator<'a>(&'a self) -> Iter<'a>;
}
This way, val_iterator() would in any situation, including when HasValsIterator implementor is passed by value. However, Rust is not there yet.