This whole lifetime thing in Rust is still dark magic for me. I have a general idea of how it works but whenever I have to define lifetimes myself I have a hard time figuring out what to do. Here's what I want to do:
I have a [&str, 100] that comes from a submodule and I want to write a very simple randomized iterator that uses data from this submodule. Here's roughly what I do:
use rand::distributions::{Distribution, Uniform};
use super::data:Data;
struct RandomData {
range: Uniform<usize>,
rng: rand::rngs::ThreadRng,
}
impl RandomData {
fn new () -> RandomData {
RandomData {
range:·Uniform::new(0, Data.len()),
rng: rand::thread_rng(),
}
}
}
impl Iterator for RandomData {
type Item = &str;
fn next(next(&mut self) -> Option<Self::Item> {
let index = self.range.sample(&mut self.rng);
Some(Data[index])
}
}
Now, obviously the compiler is asking for lifetimes here because of the &str and the easiest way would be to simply use a static lifetime &'static str. But I wondered how to do this right, so I tried the real deal.
I started with the following changes to the iterator implementation:
impl<'a> Iterator for RandomData {
type Item = &'a str;
fn next(next(&mut self) -> Option<Self::Item> { .. }
}
Now the compiler says: error[E0207]: the lifetime parameter 'a is not constrained by the impl trait, self type, or predicates and suggest to read more about this error E0207, which I did. I think the gist is, that the lifetime parameter needs to appear either in the Trait or implementing type. Both is not the case because I don't need it there and in this case the documentation suggests to use PhantomData. But it also only talks about types and I don't really get it to work.
If I try to do:
struct RandomData<'a> {
range: Uniform<usize>,
rng: rand::rngs::ThreadRng,
phantom: PhantomData<&'a str>
}
I get a whole new bunch of messages about anonymous lifetimes, so I added them, but then get stuck with:
error[E0106]: missing lifetime specifier --> src/epcs/random_epc.rs:12:22
|
12 | pub fn new () -> RandomEPC {
| ^^^^^^^^^ help: consider giving it a 'static lifetime: `RandomEPC + 'static`
|
= help: this function's return type contains a borrowed value, but there is no value for it to be borrowed from
and I'm not sure where to go from here.
Edit
Thanks phimuemue for the suggestion. I created a simplified example here:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=bddde9310da5cf838dafee83e05cd78a
Lifetime is Rust describe how long will the data live before being drop.
In your case, the item of your Iterator is a reference to the data hold by Data. Therefore its lifetime correspond to the lifetime of Data. If Data is of lifetime static (it will live during the whole life of the process) then the right way to do your impl is to output Item with lifetime `static.
The idea with Associated Types is that the trait should be implemented only once. You can't implement twice Iterator, once with String as Item and once with &'static str. (See the book). Therefore, in your case you should implement Iterator only once with &'static str and not try to implement it for every lifetime 'l with &'l str.
Related
Given the following struct and impl:
use std::slice::Iter;
use std::cell::RefCell;
struct Foo {
bar: RefCell<Vec<u32>>,
}
impl Foo {
pub fn iter(&self) -> Iter<u32> {
self.bar.borrow().iter()
}
}
fn main() {}
I get an error message about a lifetime issue:
error: borrowed value does not live long enough
--> src/main.rs:9:9
|
9 | self.bar.borrow().iter()
| ^^^^^^^^^^^^^^^^^ does not live long enough
10 | }
| - temporary value only lives until here
|
note: borrowed value must be valid for the anonymous lifetime #1 defined on the body at 8:36...
--> src/main.rs:8:37
|
8 | pub fn iter(&self) -> Iter<u32> {
| _____________________________________^ starting here...
9 | | self.bar.borrow().iter()
10 | | }
| |_____^ ...ending here
How am I able to return and use bars iterator?
You cannot do this because it would allow you to circumvent runtime checks for uniqueness violations.
RefCell provides you a way to "defer" mutability exclusiveness checks to runtime, in exchange allowing mutation of the data it holds inside through shared references. This is done using RAII guards: you can obtain a guard object using a shared reference to RefCell, and then access the data inside RefCell using this guard object:
&'a RefCell<T> -> Ref<'a, T> (with borrow) or RefMut<'a, T> (with borrow_mut)
&'b Ref<'a, T> -> &'b T
&'b mut RefMut<'a, T> -> &'b mut T
The key point here is that 'b is different from 'a, which allows one to obtain &mut T references without having a &mut reference to the RefCell. However, these references will be linked to the guard instead and can't live longer than the guard. This is done intentionally: Ref and RefMut destructors toggle various flags inside their RefCell to force mutability checks and to force borrow() and borrow_mut() panic if these checks fail.
The simplest thing you can do is to return a wrapper around Ref, a reference to which would implement IntoIterator:
use std::cell::Ref;
struct VecRefWrapper<'a, T: 'a> {
r: Ref<'a, Vec<T>>
}
impl<'a, 'b: 'a, T: 'a> IntoIterator for &'b VecRefWrapper<'a, T> {
type IntoIter = Iter<'a, T>;
type Item = &'a T;
fn into_iter(self) -> Iter<'a, T> {
self.r.iter()
}
}
(try it on playground)
You can't implement IntoIterator for VecRefWrapper directly because then the internal Ref will be consumed by into_iter(), giving you essentially the same situation you're in now.
Alternate Solution
Here is an alternate solution that uses interior mutability as it was intended. Instead of creating an iterator for &T values, we should create an iterator for Ref<T> values, which deference automatically.
struct Iter<'a, T> {
inner: Option<Ref<'a, [T]>>,
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = Ref<'a, T>;
fn next(&mut self) -> Option<Self::Item> {
match self.inner.take() {
Some(borrow) => match *borrow {
[] => None,
[_, ..] => {
let (head, tail) = Ref::map_split(borrow, |slice| {
(&slice[0], &slice[1..])
});
self.inner.replace(tail);
Some(head)
}
},
None => None,
}
}
}
Playground
Explanation
The accepted answer has a few significant drawbacks that may confuse those new to Rust. I will explain how, in my personal experience, the accepted answer might actually be harmful to a beginner, and why I believe this alternative uses interior mutability and iterators as they were intended.
As the previous answer importantly highlights, using RefCell creates a divergent type hierarchy that isolates mutable and immutable access to a shared value, but you do not have to worry about lifetimes to solve the iteration problem:
RefCell<T> .borrow() -> Ref<T> .deref() -> &T
RefCell<T> .borrow_mut() -> RefMut<T> .deref_mut() -> &mut T
The key to solving this without lifetimes is the Ref::map method, which is critically missed in the book. Ref::map "makes a new reference to a component of the borrowed data", or in other words converts a Ref<T> of the outer type to a Ref<U> of some inner value:
Ref::map(Ref<T>, ...) -> Ref<U>
Ref::map and its counterpart RefMut::map are the real stars of the interior mutability pattern, not borrow() and borrow_mut().
Why? Because unlike borrow() and borrow_mut(), Ref::mut and RefMut::map, allow you to create references to interior values that can be "returned".
Consider adding a first() method to the Foo struct described in the question:
fn first(&self) -> &u32 {
&self.bar.borrow()[0]
}
Nope, .borrow() makes a temporary Ref that only lives until the method returns:
error[E0515]: cannot return value referencing temporary value
--> src/main.rs:9:11
|
9 | &self.bar.borrow()[0]
| ^-----------------^^^
| ||
| |temporary value created here
| returns a value referencing data owned by the current function
error: aborting due to previous error; 1 warning emitted
We can make it more obvious what is happening if we break it up and make the implicit deference explicit:
fn first(&self) -> &u32 {
let borrow: Ref<_> = self.bar.borrow();
let bar: &Vec<u32> = borrow.deref();
&bar[0]
}
Now we can see that .borrow() creates a Ref<T> that is owned by the method's scope, and isn't returned and therefore dropped even before the reference it provided can be used. So, what we really need is to return an owned type instead of a reference. We want to return a Ref<T>, as it implements Deref for us!
Ref::map will help us do just that for component (internal) values:
fn first(&self) -> Ref<u32> {
Ref::map(self.bar.borrow(), |bar| &bar[0])
}
Of course, the .deref() will still happen automatically, and Ref<u32> will be mostly be referentially transparent as &u32.
Gotcha. One easy mistake to make when using Ref::map is to try to create an owned value in the closure, which is not possible as when we tried to use borrow(). Consider the type signature of the second parameter, the function: FnOnce(&T) -> &U,. It returns a reference, not an owned type!
This is why we use a slice in the answer &v[..] instead of trying to use the vector's .iter() method, which returns an owned std::slice::Iter<'a, T>. Slices are a reference type.
Additional Thoughts
Alright, so now I will attempt to justify why this solution is better than the accepted answer.
First, the use of IntoIterator is inconsistent with the Rust standard library, and arguably the purpose and intent of the trait. The trait method consumes self: fn into_iter(self) -> ....
let v = vec![1,2,3,4];
let i = v.into_iter();
// v is no longer valid, it was moved into the iterator
Using IntoIterator indirectly for a wrapper is inconsistent as you consume the wrapper and not the collection. In my experience, beginners will benefit from sticking with the conventions. We should use a regular Iterator.
Next, the IntoIterator trait is implemented for the reference &VecRefWrapper and not the owned type VecRefWrapper.
Suppose you are implementing a library. The consumers of your API will have to seemingly arbitrarily decorate owned values with reference operators, as is demonstrated in the example on the playground:
for &i in &foo.iter() {
println!("{}", i);
}
This is a subtle and confusing distinction if you are new to Rust. Why do we have to take a reference to the value when it is anonymously owned by - and should only exist for - the scope of the loop?
Finally, the solution above shows how it is possible to drill all they way into your data with interior mutability, and makes the path forward for implementing a mutable iterator clear as well. Use RefMut.
From my research there is currently no solution to this problem. The biggest problem here is self-referentiality and the fact that rust cannot prove your code to be safe. Or at least not in the generic fashion.
I think it's safe to assume that crates like ouroboros, self-cell and owning_ref are solution if you know that your struct (T in Ref<T>) does not contain any smart pointers nor anything which could invalidate any pointers you might obtain in your "dependent" struct.
Note that self-cell does this safely with extra heap allocation which might be ok in some cases.
There was also RFC for adding map_value to Ref<T> but as you can see, there is always some way to invalidate pointers in general (which does not mean your specific case is wrong it's just that it probably will never be added to the core library/language because it cannot be guaranteed for any T)
Yeah, so no answer, sorry. impl IntoIterator for &T works but I think it's rather hack and it forces you to write for x in &iter instead of for x in iter
I was writing some code where I want to use an iterator, or its reversed version depending on a flag, but the straightforward code gives an error
pub fn eggs<I,T>(iter:I)->Box<dyn Iterator<Item=T>>
where I:Iterator<Item=T>+DoubleEndedIterator
{
Box::new(iter.rev())
}
pub fn bacon<I,T>(iter:I, reverse:bool) -> Box<dyn Iterator<Item=T>>
where I:Iterator<Item=T>+DoubleEndedIterator
{
if reverse {
Box::new(iter.rev())
} else {
Box::new(iter)
}
}
fn main()
{
let pants:String = "pants".into();
eggs(pants.chars());
}
fails to compile:
error[E0310]: the parameter type `I` may not live long enough
--> src/main.rs:5:5
|
2 | pub fn eggs<I,T>(iter:I)->Box<dyn Iterator<Item=T>>
| - help: consider adding an explicit lifetime bound...: `I: 'static`
...
5 | Box::new(iter.rev())
| ^^^^^^^^^^^^^^^^^^^^ ...so that the type `Rev<I>` will meet its required lifetime bounds
With my limited understanding of Rust, I'm not sure where those lifetime bounds are coming from. There aren't any on the Iterator trait, or the Rev struct, and the parameter is being moved.
What is the proper way to declare these sorts of functions given that 'static isn't really an option.
rust playground
This doesn't have to do with .rev() at all, but with returning Box<dyn Iterator>:
// error[E0310]: the parameter type `I` may not live long enough
fn boxed_iter<I, T>(iter: I) -> Box<dyn Iterator<Item = T>>
// - help: consider adding an explicit lifetime bound...: `I: 'static`
where
I: Iterator<Item = T>,
{
Box::new(iter)
// ^^^^^^^^^^^^^^ ...so that the type `I` will meet its required lifetime bounds
}
The reason for this is that trait objects like Box<dyn Trait> have an implicit 'static lifetime if not specified. So when the compiler tries to cast Box<I> to Box<dyn Iterator>, it fails if I is does not also have a 'static lifetime. (There are some more specific rules if the trait contains lifetimes itself; you can read about those in more detail here.)
If you instead want a shorter lifetime, you need to specify it explicitly as Box<dyn 'a + Trait>. So for example:
fn boxed_iter<'a, I, T>(iter: I) -> Box<dyn 'a + Iterator<Item = T>>
where
I: 'a + Iterator<Item = T>,
{
Box::new(iter)
}
Frxstrem's answer is excellent. I just want to add that, if you know that the return value of your function has a specific concrete type, you can use the special impl trait syntax.
In the case of your eggs function, the return type is probably something like Rev<I>. That type, on its own, isn't very illuminating, but we know that such a type exists and the only thing we care about is that it's an iterator, so we can write
pub fn eggs<I,T>(iter:I) -> impl Iterator<Item=T> + DoubleEndedIterator
where I: Iterator<Item=T> + DoubleEndedIterator {
iter.rev()
}
Now the compiler still understands that there is a single concrete type and will act accordingly (no need to box the value or have dynamic dispatch), but we as the programmers still only have to care about the Iterator and DoubleEndedIterator aspects of it. Zero-cost abstraction at its finest.
Your bacon function can't benefit from this, as it could return either an I or a Rev<I> depending on input, so the dynamic dispatch is actually necessary. In that case, you'll need to follow Frxstrem's answer to correctly box your iterator.
Returning a reference to an owned field is allowed in other scenarios. For instance, the code below compiles fine.
struct Charlie<T> {
delta: T,
}
impl<T> Charlie<T> {
fn delta(&self) -> &T
{
&self.delta
}
}
I'd like to return a reference to an owned field from a trait method in similar fashion. Assume that Alpha is a trait in the standard library, so I can't modify it.
Is there a way to implement this?
trait Alpha {
type Item;
fn bravo(&mut self) -> Self::Item;
}
impl<T> Alpha for Charlie<T> {
type Item = &T;
fn bravo(&mut self) -> Self::Item
{
&self.delta
}
}
The above doesn't compile and the error messages don't seem to apply. If I try following the error message suggestions, things just get complicated and I run into a series of errors with suggestions that also don't pan out.
Compiling playground v0.0.1 (/playground)
error[E0106]: missing lifetime specifier
--> src/main.rs:22:17
|
22 | type Item = &T;
| ^ expected named lifetime parameter
|
help: consider introducing a named lifetime parameter
|
22 | type Item<'a> = &'a T;
| ^^^^ ^^^
I've tried the above suggestion and went so far as to declare 'a on Charlie's struct and add a PhantomData field, and thoroughly peppered the rest of my sources with 'as. And the compiler continues whining and complaining and nagging throughout the whole process.
I managed to find this post on StackOverflow, but the solutions there seem to all require modifying the trait.
I'm thinking what I'm trying to do may not be possible. But I don't really understand why not.
The Alpha trait method that I was struggling with is actually
type Item = Take<&I>;
fn next(&mut self) -> Option<Self::Item> { ... }
I was trying to return another wrapped iterator type, where I is the wrapped iterator. The Iterator trait itself doesn't define any lifetimes I can leverage.
I do know a way to implement around this limitation using the smart pointer classes to encapsulate the field I want to share in Charlie. Then make that my Item type. I was just hoping for something with less overhead.
The contract for that trait unfortunately doesn't allow it.
trait Alpha {
type Item;
fn bravo(&mut self) -> Self::Item;
}
This says that, if Self is an Alpha, then "there exists some single Self::Item which I can get from any &mut self with any lifetime". You want "there exists a class of Self::Item whose lifetimes relate to &mut self in a nontrivial way".
The easiest way to get around this is to make bravo take self by value.
trait Alpha {
type Item;
fn bravo(self) -> Self::Item;
}
Now, the contract says "there's some way to get a Self::Item from a self", which is much simpler. We can implement it as
impl<'a, T> Alpha for Charlie<&'a mut T> {
type Item = &'a T;
fn bravo(self) -> Self::Item {
&self.delta
}
}
I ran into an issue while trying to define and use a trait with methods that borrow self mutably.
Some context that might make it easier: I am working on a toy compiler, and the problem I was trying to solve was to define a trait for code nodes, which are either statements or expressions. The trait was meant to be used for traversing code mutably (for rewriting purposes). The abstraction I was trying to create was a "code node" that may have any number of children that are either statements or expressions. This is how it went:
// Actually these are enums with different payload types for different kinds of exprs/stmts,
// but this is not relevant.
struct Expression;
struct Statement;
trait CodeNode<'a>
where
Self::ExprIter: Iterator<Item = &'a mut Expression>,
Self::StmtIter: Iterator<Item = &'a mut Statement>,
{
type ExprIter;
type StmtIter;
fn child_exprs(&'a mut self) -> Self::ExprIter;
fn child_stmts(&'a mut self) -> Self::StmtIter;
}
This trait would be then implemented for quite a few types (I have a separate type for different kinds of statements and expressions).
The way I tried to use it was:
fn process<'a>(node: &'a mut impl CodeNode<'a>) {
for _stmt in node.child_stmts() {
// ...
}
for _expr in node.child_exprs() {
// ...
}
}
And this is where the problem lies. Rust compiler treats a call to node.child_stmts as a mutable borrow of node for the entire lifetime 'a, and so it does not allow a call to node.child_exprs later in the same function. Here is how the error looks:
error[E0499]: cannot borrow `*node` as mutable more than once at a time
--> src/main.rs:21:18
|
16 | fn process<'a>(node: &'a mut impl CodeNode<'a>) {
| -- lifetime `'a` defined here
17 | for _stmt in node.child_stmts() {
| ------------------
| |
| first mutable borrow occurs here
| argument requires that `*node` is borrowed for `'a`
...
21 | for _expr in node.child_exprs() {
| ^^^^ second mutable borrow occurs here
What I want to do is to somehow make compiler aware of the fact that node implements CodeNode<'a> for any lifetime parameter, and so it should use two separate lifetimes for two
calls, but I can't quite figure out a way to do it.
Any suggestions are welcome, I don't have a lot of experience with Rust, so maybe I am missing some more high-level solution to the original problem.
Your lifetime 'a is constrained by the CodeNode so both functions will be called with the same lifetime, but what you want are two lifetimes constrained by the two functions. So why not do something like this.
struct Expression;
struct Statement;
trait CodeNode
{
type ExprIter<'a> : Iterator<Item = &'a mut Expression>; //unstable
type StmtIter<'a> : Iterator<Item = &'a mut Statement>; //unstable
fn child_exprs<'a>(&'a mut self) -> Self::ExprIter<'a>;
fn child_stmts<'a>(&'a mut self) -> Self::StmtIter<'a>;
}
fn process(node: &mut impl CodeNode) {
for _stmt in node.child_stmts() {
// ...
}
for _expr in node.child_exprs() {
// ...
}
}
Unfortunately I had to use the unstable feature of generic associated types, but I believe this is what you want.
I also want to express that iterating over mutable references might not be a good idea and maybe you should change your program structure if that is possible.
EDIT:
#pretzelhammer proposed in the comments the following link which might be interesting: Solving the generalized streaming iterator problem without gats
References to wrapper types like &Rc<T> and &Box<T> are invariant in T (&Rc<T> is not a &Rc<U> even if T is a U). A concrete example of the issue (Rust Playground):
use std::rc::Rc;
use std::rc::Weak;
trait MyTrait {}
struct MyStruct {
}
impl MyTrait for MyStruct {}
fn foo(rc_trait: Weak<MyTrait>) {}
fn main() {
let a = Rc::new(MyStruct {});
foo(Rc::downgrade(&a));
}
This code results in the following error:
<anon>:15:23: 15:25 error: mismatched types:
expected `&alloc::rc::Rc<MyTrait>`,
found `&alloc::rc::Rc<MyStruct>`
Similar example (with similar error) with Box<T> (Rust Playground):
trait MyTrait {}
struct MyStruct {
}
impl MyTrait for MyStruct {}
fn foo(rc_trait: &Box<MyTrait>) {}
fn main() {
let a = Box::new(MyStruct {});
foo(&a);
}
In these cases I could of course just annotate a with the desired type, but in many cases that won't be possible because the original type is needed as well. So what do I do then?
What you see here is not related to variance and subtyping at all.
First, the most informative read on subtyping in Rust is this chapter of Nomicon. You can find there that in Rust subtyping relationship (i.e. when you can pass a value of one type to a function or a variable which expects a variable of different type) is very limited. It can only be observed when you're working with lifetimes.
For example, the following piece of code shows how exactly &Box<T> is (co)variant:
fn test<'a>(x: &'a Box<&'a i32>) {}
fn main() {
static X: i32 = 12;
let xr: &'static i32 = &X;
let xb: Box<&'static i32> = Box::new(xr); // <---- start of box lifetime
let xbr: &Box<&'static i32> = &xb;
test(xbr); // Covariance in action: since 'static is longer than or the
// same as any 'a, &Box<&'static i32> can be passed to
// a function which expects &'a Box<&'a i32>
//
// Note that it is important that both "inner" and "outer"
// references in the function signature are defined with
// the same lifetime parameter, and thus in `test(xbr)` call
// 'a gets instantiated with the lifetime associated with
// the scope I've marked with <----, but nevertheless we are
// able to pass &'static i32 as &'a i32 because the
// aforementioned scope is less than 'static, therefore any
// shared reference type with 'static lifetime is a subtype of
// a reference type with the lifetime of that scope
} // <---- end of box lifetime
This program compiles, which means that both & and Box are covariant over their respective type and lifetime parameters.
Unlike most of "conventional" OOP languages which have classes/interfaces like C++ and Java, in Rust traits do not introduce subtyping relationship. Even though, say,
trait Show {
fn show(&self) -> String;
}
highly resembles
interface Show {
String show();
}
in some language like Java, they are quite different in semantics. In Rust bare trait, when used as a type, is never a supertype of any type which implements this trait:
impl Show for i32 { ... }
// the above does not mean that i32 <: Show
Show, while being a trait, indeed can be used in type position, but it denotes a special unsized type which can only be used to form trait objects. You cannot have values of the bare trait type, therefore it does not even make sense to talk about subtyping and variance with bare trait types.
Trait objects take form of &SomeTrait or &mut SomeTrait or SmartPointer<SomeTrait>, and they can be passed around and stored in variables and they are needed to abstract away the actual implementation of the trait. However, &T where T: SomeTrait is not a subtype of &SomeTrait, and these types do not participate in variance at all.
Trait objects and regular pointers have incompatible internal structure: &T is just a regular pointer to a concrete type T, while &SomeTrait is a fat pointer which contains a pointer to the original value of a type which implements SomeTrait and also a second pointer to a vtable for the implementation of SomeTrait of the aforementioned type.
The fact that passing &T as &SomeTrait or Rc<T> as Rc<SomeTrait> works happens because Rust does automatic coercion for references and smart pointers: it is able to construct a fat pointer &SomeTrait for a regular reference &T if it knows T; this is quite natural, I believe. For instance, your example with Rc::downgrade() works because Rc::downgrade() returns a value of type Weak<MyStruct> which gets coerced to Weak<MyTrait>.
However, constructing &Box<SomeTrait> out of &Box<T> if T: SomeTrait is much more complex: for one, the compiler would need to allocate a new temporary value because Box<T> and Box<SomeTrait> has different memory representations. If you have, say, Box<Box<T>>, getting Box<Box<SomeTrait>> out of it is even more complex, because it would need creating a new allocation on the heap to store Box<SomeTrait>. Thus, there are no automatic coercions for nested references and smart pointers, and again, this is not connected with subtyping and variance at all.
In the case of Rc::downgrade this is actually just a failure of the type inference in this particular case, and will work if it is done as a separate let:
fn foo(rc_trait: Weak<MyTrait>) {}
fn main() {
let a = Rc::new(MyStruct {});
let b = Rc::downgrade(&a);
foo(b);
}
Playground
For Box<T> it is very likely you don't actually want a reference to the box as the argument, but a reference to the contents. In which case there is no invariance to deal with:
fn foo(rc_trait: &MyTrait) {}
fn main() {
let a = Box::new(MyStruct {});
foo(a.as_ref());
}
Playground
Similarly, for the case with Rc<T>, if you write a function that takes an Rc<T> you probably want a clone (i.e. a reference counted reference), and not a normal reference:
fn foo(rc_trait: Rc<MyTrait>) {}
fn main() {
let a = Rc::new(MyStruct {});
foo(a.clone());
}
Playground