Related
The general setup is I have an array of values I'd like to map() and then chain() with 1 additional value. I've learned from this answer that the proper way to construct that final value is to use std::iter::once. This works and eliminated the below problem, but I would still like to understand it better.
In my broken, likely rust-anti-pattern-riddled example, I was using an array of a single element and then calling into_iter(). This produced a value / reference type-mismatch in the chain.
Question: What is the Rust-idiomatic mechanism for correcting this value / reference mismatch? Particularly if clone and copy are unavailable.
Background: Why is there a type mis-match to begin with?
This much I believe I understand. Based on the definition of std::iter::Map, the item type for the iterator is type Item = B where B is constrained by F: FnMut(<I as Iterator>::Item) -> B (i.e. the mapped type). However array defines the following 2 IntoIterator implementations, both of which appear to produce references.
impl<'a, const N: usize, T> IntoIterator for &'a [T; N] where
[T; N]: LengthAtMost32,
type Item = &'a T
impl<'a, const N: usize, T> IntoIterator for &'a mut [T; N] where
[T; N]: LengthAtMost32,
type Item = &'a mut T
Example demonstrating the issue:
#[derive(PartialEq, Eq, Clone, Copy)]
enum Enum1 {
A, B, C
}
#[derive(PartialEq, Eq, Clone, Copy)]
enum Enum2 {
X, Y, Z
}
struct Data {
// Other data omitted
e1: Enum1,
e2: Enum2
}
struct Consumer {
// Other data omitted
/** Predicate which evaluates if this consumer can consume given Data */
consumes: Box<dyn Fn(&Data) -> bool>
}
fn main() {
// Objective: 3 consumers which consume data with A, B, and X respectively
let v: Vec<Consumer> = [Enum1::A, Enum1::B].iter()
.map(|&e1| Consumer { consumes: Box::new(move |data| data.e1 == e1) })
// This chain results in an iterator type-mismatch:
// expected &Consumer, found Consumer
.chain([Consumer { consumes: Box::new(move |data| data.e2 == Enum2::X) }].into_iter())
.collect(); // Fails as well due to the chain failure
}
Error:
error[E0271]: type mismatch resolving `<std::slice::Iter<'_, Consumer> as std::iter::IntoIterator>::Item == Consumer`
--> src/main.rs:52:10
|
52 | .chain([Consumer { consumes: Box::new(move |data| data.e2 == Enum2::X) }].into_iter())
| ^^^^^ expected reference, found struct `Consumer`
|
= note: expected type `&Consumer`
found type `Consumer`
Rust playground example.
There is a long-standing issue regarding this. The technical details are a bit heavy, but essentially, due to underlying, technical reasons, you cannot take ownership of a fixed-size array and return owned references without a lot of hocus pocus. This becomes obvious when you think about what a fixed-size array is and how it is stored in memory, and how you can get elements out without cloning them.
As a result, due to the implementations you found already, you can only get borrowed references. You can bypass this with arrayvec (as they have a sound implementation of IntoIterator for ArrayVec with owned types), or you can require that all your T: Clone and deal with it that way, at a cost of extra items in memory (temporarily; 90% of the time the compiler optimizes this away).
Considering the following code:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || &t)
}
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
t moves to the closure, so the closure live as long as t
The closure returns a reference to t which was moved to the closure. So the reference is valid as long as the closure exists.
There is no lifetime problem, the code compiles.
What actually happens:
The code does not compile:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
|
note: first, the lifetime cannot outlive the lifetime as defined on the body at 2:14...
--> src/lib.rs:2:14
|
2 | Box::new(move || &t)
| ^^^^^^^^^^
note: ...so that closure can access `t`
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
note: but, the lifetime must be valid for the lifetime 'a as defined on the function body at 1:8...
--> src/lib.rs:1:8
|
1 | fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
| ^^
= note: ...so that the expression is assignable:
expected std::boxed::Box<(dyn std::ops::Fn() -> &'a T + 'a)>
found std::boxed::Box<dyn std::ops::Fn() -> &T>
I do not understand the conflict. How can I fix it?
Very interesting question! I think I understood the problem(s) at play here. Let me try to explain.
tl;dr: closures cannot return references to values captured by moving, because that would be a reference to self. Such a reference cannot be returned because the Fn* traits don't allow us to express that. This is basically the same as the streaming iterator problem and could be fixed via GATs (generic associated types).
Implementing it manually
As you probably know, when you write a closure, the compiler will generate a struct and impl blocks for the appropriate Fn traits, so closures are basically syntax sugar. Let's try to avoid all that sugar and build your type manually.
What you want is a type which owns another type and can return references to that owned type. And you want to have a function which returns a boxed instance of said type.
struct Baz<T>(T);
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
fn make_baz<T>(t: T) -> Box<Baz<T>> {
Box::new(Baz(t))
}
This is pretty equivalent to your boxed closure. Let's try to use it:
let outside = {
let s = "hi".to_string();
let baz = make_baz(s);
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // works too
This works just fine. The string s is moved into the Baz type and that Baz instance is moved into the Box. s is now owned by baz and then by outside.
It gets more interesting when we add a single character:
let outside = {
let s = "hi".to_string();
let baz = make_baz(&s); // <-- NOW BORROWED!
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // doesn't work!
Now we cannot make the lifetime of baz bigger than the lifetime of s, since baz contains a reference to s which would be an dangling reference of s would go out of scope earlier than baz.
The point I wanted to make with this snippet: we didn't need to annotate any lifetimes on the type Baz to make this safe; Rust figured it out on its own and enforces that baz lives no longer than s. This will be important below.
Writing a trait for it
So far we only covered the basics. Let's try to write a trait like Fn to get closer to your original problem:
trait MyFn {
type Output;
fn call(&self) -> Self::Output;
}
In our trait, there are no function parameters, but otherwise it's fairly identical to the real Fn trait.
Let's implement it!
impl<T> MyFn for Baz<T> {
type Output = ???;
fn call(&self) -> Self::Output {
&self.0
}
}
Now we have a problem: what do we write instead of ???? Naively one would write &T... but we need a lifetime parameter for that reference. Where do we get one? What lifetime does the return value even have?
Let's check the function we implemented before:
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
So here we use &T without lifetime parameter too. But this only works because of lifetime elision. Basically, the compiler fills in the blanks so that fn call(&self) -> &T is equivalent to:
fn call<'s>(&'s self) -> &'s T
Aha, so the lifetime of the returned reference is bound to the self lifetime! (more experienced Rust users might already have a feeling where this is going...).
(As a side note: why is the returned reference not dependent on the lifetime of T itself? If T references something non-'static then this has to be accounted for, right? Yes, but it is already accounted for! Remember that no instance of Baz<T> can ever live longer than the thing T might reference. So the self lifetime is already shorter than whatever lifetime T might have. Thus we only need to concentrate on the self lifetime)
But how do we express that in the trait impl? Turns out: we can't (yet). This problem is regularly mentioned in the context of streaming iterators -- that is, iterators that return an item with a lifetime bound to the self lifetime. In today's Rust, it is sadly impossible to implement this; the type system is not strong enough.
What about the future?
Luckily, there is an RFC "Generic Associated Types" which was merged some time ago. This RFC extends the Rust type system to allow associated types of traits to be generic (over other types and lifetimes).
Let's see how we can make your example (kinda) work with GATs (according to the RFC; this stuff doesn't work yet ☹). First we have to change the trait definition:
trait MyFn {
type Output<'a>; // <-- we added <'a> to make it generic
fn call(&self) -> Self::Output;
}
The function signature hasn't changed in the code, but notice that lifetime elision kicks in! The above fn call(&self) -> Self::Output is equivalent to:
fn call<'s>(&'s self) -> Self::Output<'s>
So the lifetime of the associated type is bound to the self lifetime. Just as we wanted! The impl looks like this:
impl<T> MyFn for Baz<T> {
type Output<'a> = &'a T;
fn call(&self) -> Self::Output {
&self.0
}
}
To return a boxed MyFn we would need to write this (according to this section of the RFC:
fn make_baz<T>(t: T) -> Box<for<'a> MyFn<Output<'a> = &'a T>> {
Box::new(Baz(t))
}
And what if we want to use the real Fn trait? As far as I understand, we can't, even with GATs. I think it's impossible to change the existing Fn trait to use GATs in a backwards compatible manner. So it's likely that the standard library will keep the less powerful trait as is. (side note: how to evolve the standard library in backwards incompatible ways to use new language features is something I wondered about a few times already; so far I haven't heard of any real plan in this regards; I hope the Rust team comes up with something...)
Summary
What you want is not technically impossible or unsafe (we implemented it as a simple struct and it works). However, unfortunately it is impossible to express what you want in the form of closures/Fn traits in Rust's type system right now. This is the same problem streaming iterators are dealing with.
With the planned GAT feature, it is possible to express all of this in the type system. However, the standard library would need to catch up somehow to make your exact code possible.
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
This makes no sense. A value cannot "live as long" as a type, because a type doesn't live. "T has lifetime 'a" is a very imprecise statement, easy to misunderstand. What T: 'a really means is "instances of T must stay valid at least as long as lifetime 'a. For example, T must not be a reference with a lifetime shorter than 'a, or a struct containing such a reference. Note that this has nothing to do with forming references to T, i.e. &T.
The value t, then, lives as long as its lexical scope (it's a function parameter) says it does, which has nothing to do with 'a at all.
t moves to the closure, so the closure live as long as t
This is also incorrect. The closure lives as long as the closure does lexically. It is a temporary in the result expression, and therefore lives until the end of the result expression. t's lifetime concerns the closure not at all, since it has its own T variable inside, the capture of t. Since the capture is a copy/move of t, it is not in any way affected by t's lifetime.
The temporary closure is then moved into the box's storage, but that's a new object with its own lifetime. The lifetime of that closure is bound to the lifetime of the box, i.e. it is the return value of the function, and later (if you store the box outside the function) the lifetime of whatever variable you store the box in.
All of that means that a closure that returns a reference to its own capture state must bind the lifetime of that reference to its own reference. Unfortunately, this is not possible.
Here's why:
The Fn trait implies the FnMut trait, which in turn implies the FnOnce trait. That is, every function object in Rust can be called with a by-value self argument. This means that every function object must be still valid being called with a by-value self argument and returning the same thing as always.
In other words, trying to write a closure that returns a reference to its own captures expands to roughly this code:
struct Closure<T> {
captured: T,
}
impl<T> FnOnce<()> for Closure<T> {
type Output = &'??? T; // what do I put as lifetime here?
fn call_once(self, _: ()) -> Self::Output {
&self.captured // returning reference to local variable
// no matter what, the reference would be invalid once we return
}
}
And this is why what you're trying to do is fundamentally impossible. Take a step back, think of what you're actually trying to accomplish with this closure, and find some other way to accomplish it.
You expect the type T to have lifetime 'a, but t is not a reference to a value of type T. The function takes ownership of the variable t by argument passing:
// t is moved here, t lifetime is the scope of the function
fn foo<'a, T: 'a>(t: T)
You should do:
fn foo<'a, T: 'a>(t: &'a T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || t)
}
The other answers are top-notch, but I wanted to chime in with another reason your original code couldn't work. A big problem lies in the signature:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a>
This says that the caller may specify any lifetime when calling foo and the code will be valid and memory-safe. That cannot possibly be true for this code. It wouldn't make sense to call this with 'a set to 'static, but nothing about this signature would prevent that.
Considering the following code:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || &t)
}
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
t moves to the closure, so the closure live as long as t
The closure returns a reference to t which was moved to the closure. So the reference is valid as long as the closure exists.
There is no lifetime problem, the code compiles.
What actually happens:
The code does not compile:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
|
note: first, the lifetime cannot outlive the lifetime as defined on the body at 2:14...
--> src/lib.rs:2:14
|
2 | Box::new(move || &t)
| ^^^^^^^^^^
note: ...so that closure can access `t`
--> src/lib.rs:2:22
|
2 | Box::new(move || &t)
| ^^
note: but, the lifetime must be valid for the lifetime 'a as defined on the function body at 1:8...
--> src/lib.rs:1:8
|
1 | fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a> {
| ^^
= note: ...so that the expression is assignable:
expected std::boxed::Box<(dyn std::ops::Fn() -> &'a T + 'a)>
found std::boxed::Box<dyn std::ops::Fn() -> &T>
I do not understand the conflict. How can I fix it?
Very interesting question! I think I understood the problem(s) at play here. Let me try to explain.
tl;dr: closures cannot return references to values captured by moving, because that would be a reference to self. Such a reference cannot be returned because the Fn* traits don't allow us to express that. This is basically the same as the streaming iterator problem and could be fixed via GATs (generic associated types).
Implementing it manually
As you probably know, when you write a closure, the compiler will generate a struct and impl blocks for the appropriate Fn traits, so closures are basically syntax sugar. Let's try to avoid all that sugar and build your type manually.
What you want is a type which owns another type and can return references to that owned type. And you want to have a function which returns a boxed instance of said type.
struct Baz<T>(T);
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
fn make_baz<T>(t: T) -> Box<Baz<T>> {
Box::new(Baz(t))
}
This is pretty equivalent to your boxed closure. Let's try to use it:
let outside = {
let s = "hi".to_string();
let baz = make_baz(s);
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // works too
This works just fine. The string s is moved into the Baz type and that Baz instance is moved into the Box. s is now owned by baz and then by outside.
It gets more interesting when we add a single character:
let outside = {
let s = "hi".to_string();
let baz = make_baz(&s); // <-- NOW BORROWED!
println!("{}", baz.call()); // works
baz
};
println!("{}", outside.call()); // doesn't work!
Now we cannot make the lifetime of baz bigger than the lifetime of s, since baz contains a reference to s which would be an dangling reference of s would go out of scope earlier than baz.
The point I wanted to make with this snippet: we didn't need to annotate any lifetimes on the type Baz to make this safe; Rust figured it out on its own and enforces that baz lives no longer than s. This will be important below.
Writing a trait for it
So far we only covered the basics. Let's try to write a trait like Fn to get closer to your original problem:
trait MyFn {
type Output;
fn call(&self) -> Self::Output;
}
In our trait, there are no function parameters, but otherwise it's fairly identical to the real Fn trait.
Let's implement it!
impl<T> MyFn for Baz<T> {
type Output = ???;
fn call(&self) -> Self::Output {
&self.0
}
}
Now we have a problem: what do we write instead of ???? Naively one would write &T... but we need a lifetime parameter for that reference. Where do we get one? What lifetime does the return value even have?
Let's check the function we implemented before:
impl<T> Baz<T> {
fn call(&self) -> &T {
&self.0
}
}
So here we use &T without lifetime parameter too. But this only works because of lifetime elision. Basically, the compiler fills in the blanks so that fn call(&self) -> &T is equivalent to:
fn call<'s>(&'s self) -> &'s T
Aha, so the lifetime of the returned reference is bound to the self lifetime! (more experienced Rust users might already have a feeling where this is going...).
(As a side note: why is the returned reference not dependent on the lifetime of T itself? If T references something non-'static then this has to be accounted for, right? Yes, but it is already accounted for! Remember that no instance of Baz<T> can ever live longer than the thing T might reference. So the self lifetime is already shorter than whatever lifetime T might have. Thus we only need to concentrate on the self lifetime)
But how do we express that in the trait impl? Turns out: we can't (yet). This problem is regularly mentioned in the context of streaming iterators -- that is, iterators that return an item with a lifetime bound to the self lifetime. In today's Rust, it is sadly impossible to implement this; the type system is not strong enough.
What about the future?
Luckily, there is an RFC "Generic Associated Types" which was merged some time ago. This RFC extends the Rust type system to allow associated types of traits to be generic (over other types and lifetimes).
Let's see how we can make your example (kinda) work with GATs (according to the RFC; this stuff doesn't work yet ☹). First we have to change the trait definition:
trait MyFn {
type Output<'a>; // <-- we added <'a> to make it generic
fn call(&self) -> Self::Output;
}
The function signature hasn't changed in the code, but notice that lifetime elision kicks in! The above fn call(&self) -> Self::Output is equivalent to:
fn call<'s>(&'s self) -> Self::Output<'s>
So the lifetime of the associated type is bound to the self lifetime. Just as we wanted! The impl looks like this:
impl<T> MyFn for Baz<T> {
type Output<'a> = &'a T;
fn call(&self) -> Self::Output {
&self.0
}
}
To return a boxed MyFn we would need to write this (according to this section of the RFC:
fn make_baz<T>(t: T) -> Box<for<'a> MyFn<Output<'a> = &'a T>> {
Box::new(Baz(t))
}
And what if we want to use the real Fn trait? As far as I understand, we can't, even with GATs. I think it's impossible to change the existing Fn trait to use GATs in a backwards compatible manner. So it's likely that the standard library will keep the less powerful trait as is. (side note: how to evolve the standard library in backwards incompatible ways to use new language features is something I wondered about a few times already; so far I haven't heard of any real plan in this regards; I hope the Rust team comes up with something...)
Summary
What you want is not technically impossible or unsafe (we implemented it as a simple struct and it works). However, unfortunately it is impossible to express what you want in the form of closures/Fn traits in Rust's type system right now. This is the same problem streaming iterators are dealing with.
With the planned GAT feature, it is possible to express all of this in the type system. However, the standard library would need to catch up somehow to make your exact code possible.
What I expect:
The type T has lifetime 'a.
The value t live as long as T.
This makes no sense. A value cannot "live as long" as a type, because a type doesn't live. "T has lifetime 'a" is a very imprecise statement, easy to misunderstand. What T: 'a really means is "instances of T must stay valid at least as long as lifetime 'a. For example, T must not be a reference with a lifetime shorter than 'a, or a struct containing such a reference. Note that this has nothing to do with forming references to T, i.e. &T.
The value t, then, lives as long as its lexical scope (it's a function parameter) says it does, which has nothing to do with 'a at all.
t moves to the closure, so the closure live as long as t
This is also incorrect. The closure lives as long as the closure does lexically. It is a temporary in the result expression, and therefore lives until the end of the result expression. t's lifetime concerns the closure not at all, since it has its own T variable inside, the capture of t. Since the capture is a copy/move of t, it is not in any way affected by t's lifetime.
The temporary closure is then moved into the box's storage, but that's a new object with its own lifetime. The lifetime of that closure is bound to the lifetime of the box, i.e. it is the return value of the function, and later (if you store the box outside the function) the lifetime of whatever variable you store the box in.
All of that means that a closure that returns a reference to its own capture state must bind the lifetime of that reference to its own reference. Unfortunately, this is not possible.
Here's why:
The Fn trait implies the FnMut trait, which in turn implies the FnOnce trait. That is, every function object in Rust can be called with a by-value self argument. This means that every function object must be still valid being called with a by-value self argument and returning the same thing as always.
In other words, trying to write a closure that returns a reference to its own captures expands to roughly this code:
struct Closure<T> {
captured: T,
}
impl<T> FnOnce<()> for Closure<T> {
type Output = &'??? T; // what do I put as lifetime here?
fn call_once(self, _: ()) -> Self::Output {
&self.captured // returning reference to local variable
// no matter what, the reference would be invalid once we return
}
}
And this is why what you're trying to do is fundamentally impossible. Take a step back, think of what you're actually trying to accomplish with this closure, and find some other way to accomplish it.
You expect the type T to have lifetime 'a, but t is not a reference to a value of type T. The function takes ownership of the variable t by argument passing:
// t is moved here, t lifetime is the scope of the function
fn foo<'a, T: 'a>(t: T)
You should do:
fn foo<'a, T: 'a>(t: &'a T) -> Box<Fn() -> &'a T + 'a> {
Box::new(move || t)
}
The other answers are top-notch, but I wanted to chime in with another reason your original code couldn't work. A big problem lies in the signature:
fn foo<'a, T: 'a>(t: T) -> Box<Fn() -> &'a T + 'a>
This says that the caller may specify any lifetime when calling foo and the code will be valid and memory-safe. That cannot possibly be true for this code. It wouldn't make sense to call this with 'a set to 'static, but nothing about this signature would prevent that.
Given the following struct and impl:
use std::slice::Iter;
use std::cell::RefCell;
struct Foo {
bar: RefCell<Vec<u32>>,
}
impl Foo {
pub fn iter(&self) -> Iter<u32> {
self.bar.borrow().iter()
}
}
fn main() {}
I get an error message about a lifetime issue:
error: borrowed value does not live long enough
--> src/main.rs:9:9
|
9 | self.bar.borrow().iter()
| ^^^^^^^^^^^^^^^^^ does not live long enough
10 | }
| - temporary value only lives until here
|
note: borrowed value must be valid for the anonymous lifetime #1 defined on the body at 8:36...
--> src/main.rs:8:37
|
8 | pub fn iter(&self) -> Iter<u32> {
| _____________________________________^ starting here...
9 | | self.bar.borrow().iter()
10 | | }
| |_____^ ...ending here
How am I able to return and use bars iterator?
You cannot do this because it would allow you to circumvent runtime checks for uniqueness violations.
RefCell provides you a way to "defer" mutability exclusiveness checks to runtime, in exchange allowing mutation of the data it holds inside through shared references. This is done using RAII guards: you can obtain a guard object using a shared reference to RefCell, and then access the data inside RefCell using this guard object:
&'a RefCell<T> -> Ref<'a, T> (with borrow) or RefMut<'a, T> (with borrow_mut)
&'b Ref<'a, T> -> &'b T
&'b mut RefMut<'a, T> -> &'b mut T
The key point here is that 'b is different from 'a, which allows one to obtain &mut T references without having a &mut reference to the RefCell. However, these references will be linked to the guard instead and can't live longer than the guard. This is done intentionally: Ref and RefMut destructors toggle various flags inside their RefCell to force mutability checks and to force borrow() and borrow_mut() panic if these checks fail.
The simplest thing you can do is to return a wrapper around Ref, a reference to which would implement IntoIterator:
use std::cell::Ref;
struct VecRefWrapper<'a, T: 'a> {
r: Ref<'a, Vec<T>>
}
impl<'a, 'b: 'a, T: 'a> IntoIterator for &'b VecRefWrapper<'a, T> {
type IntoIter = Iter<'a, T>;
type Item = &'a T;
fn into_iter(self) -> Iter<'a, T> {
self.r.iter()
}
}
(try it on playground)
You can't implement IntoIterator for VecRefWrapper directly because then the internal Ref will be consumed by into_iter(), giving you essentially the same situation you're in now.
Alternate Solution
Here is an alternate solution that uses interior mutability as it was intended. Instead of creating an iterator for &T values, we should create an iterator for Ref<T> values, which deference automatically.
struct Iter<'a, T> {
inner: Option<Ref<'a, [T]>>,
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = Ref<'a, T>;
fn next(&mut self) -> Option<Self::Item> {
match self.inner.take() {
Some(borrow) => match *borrow {
[] => None,
[_, ..] => {
let (head, tail) = Ref::map_split(borrow, |slice| {
(&slice[0], &slice[1..])
});
self.inner.replace(tail);
Some(head)
}
},
None => None,
}
}
}
Playground
Explanation
The accepted answer has a few significant drawbacks that may confuse those new to Rust. I will explain how, in my personal experience, the accepted answer might actually be harmful to a beginner, and why I believe this alternative uses interior mutability and iterators as they were intended.
As the previous answer importantly highlights, using RefCell creates a divergent type hierarchy that isolates mutable and immutable access to a shared value, but you do not have to worry about lifetimes to solve the iteration problem:
RefCell<T> .borrow() -> Ref<T> .deref() -> &T
RefCell<T> .borrow_mut() -> RefMut<T> .deref_mut() -> &mut T
The key to solving this without lifetimes is the Ref::map method, which is critically missed in the book. Ref::map "makes a new reference to a component of the borrowed data", or in other words converts a Ref<T> of the outer type to a Ref<U> of some inner value:
Ref::map(Ref<T>, ...) -> Ref<U>
Ref::map and its counterpart RefMut::map are the real stars of the interior mutability pattern, not borrow() and borrow_mut().
Why? Because unlike borrow() and borrow_mut(), Ref::mut and RefMut::map, allow you to create references to interior values that can be "returned".
Consider adding a first() method to the Foo struct described in the question:
fn first(&self) -> &u32 {
&self.bar.borrow()[0]
}
Nope, .borrow() makes a temporary Ref that only lives until the method returns:
error[E0515]: cannot return value referencing temporary value
--> src/main.rs:9:11
|
9 | &self.bar.borrow()[0]
| ^-----------------^^^
| ||
| |temporary value created here
| returns a value referencing data owned by the current function
error: aborting due to previous error; 1 warning emitted
We can make it more obvious what is happening if we break it up and make the implicit deference explicit:
fn first(&self) -> &u32 {
let borrow: Ref<_> = self.bar.borrow();
let bar: &Vec<u32> = borrow.deref();
&bar[0]
}
Now we can see that .borrow() creates a Ref<T> that is owned by the method's scope, and isn't returned and therefore dropped even before the reference it provided can be used. So, what we really need is to return an owned type instead of a reference. We want to return a Ref<T>, as it implements Deref for us!
Ref::map will help us do just that for component (internal) values:
fn first(&self) -> Ref<u32> {
Ref::map(self.bar.borrow(), |bar| &bar[0])
}
Of course, the .deref() will still happen automatically, and Ref<u32> will be mostly be referentially transparent as &u32.
Gotcha. One easy mistake to make when using Ref::map is to try to create an owned value in the closure, which is not possible as when we tried to use borrow(). Consider the type signature of the second parameter, the function: FnOnce(&T) -> &U,. It returns a reference, not an owned type!
This is why we use a slice in the answer &v[..] instead of trying to use the vector's .iter() method, which returns an owned std::slice::Iter<'a, T>. Slices are a reference type.
Additional Thoughts
Alright, so now I will attempt to justify why this solution is better than the accepted answer.
First, the use of IntoIterator is inconsistent with the Rust standard library, and arguably the purpose and intent of the trait. The trait method consumes self: fn into_iter(self) -> ....
let v = vec![1,2,3,4];
let i = v.into_iter();
// v is no longer valid, it was moved into the iterator
Using IntoIterator indirectly for a wrapper is inconsistent as you consume the wrapper and not the collection. In my experience, beginners will benefit from sticking with the conventions. We should use a regular Iterator.
Next, the IntoIterator trait is implemented for the reference &VecRefWrapper and not the owned type VecRefWrapper.
Suppose you are implementing a library. The consumers of your API will have to seemingly arbitrarily decorate owned values with reference operators, as is demonstrated in the example on the playground:
for &i in &foo.iter() {
println!("{}", i);
}
This is a subtle and confusing distinction if you are new to Rust. Why do we have to take a reference to the value when it is anonymously owned by - and should only exist for - the scope of the loop?
Finally, the solution above shows how it is possible to drill all they way into your data with interior mutability, and makes the path forward for implementing a mutable iterator clear as well. Use RefMut.
From my research there is currently no solution to this problem. The biggest problem here is self-referentiality and the fact that rust cannot prove your code to be safe. Or at least not in the generic fashion.
I think it's safe to assume that crates like ouroboros, self-cell and owning_ref are solution if you know that your struct (T in Ref<T>) does not contain any smart pointers nor anything which could invalidate any pointers you might obtain in your "dependent" struct.
Note that self-cell does this safely with extra heap allocation which might be ok in some cases.
There was also RFC for adding map_value to Ref<T> but as you can see, there is always some way to invalidate pointers in general (which does not mean your specific case is wrong it's just that it probably will never be added to the core library/language because it cannot be guaranteed for any T)
Yeah, so no answer, sorry. impl IntoIterator for &T works but I think it's rather hack and it forces you to write for x in &iter instead of for x in iter
I have managed to make the Rust type checker go into an infinite loop. A very similar program compiles with no trouble. Why does the program I want not compile?
To save your time and effort, I have made minimal versions of the two programs that isolate the problem. Of course, the minimal version is a pointless program. You'll have to use your imagination to see my motivation.
Success
Let me start with the version that works. The struct F<T> wraps a T. The type Target can be converted from an F<T> provided T can.
struct F<T>(T);
impl<T> From<F<T>> for Target where Target: From<T> {
fn from(a: F<T>) -> Target {
let b = Target::from(a.0);
f(&b)
}
}
Here's an example caller:
fn main() {
let x = Target;
let y = F(F(F(x)));
let z = Target::from(y);
println!("{:?}", z);
}
This runs and prints "Target".
Failure
The function f does not consume its argument. I would prefer it if the From conversion also did not consume its argument, because the type F<T> could be expensive or impossible to clone. I can write a custom trait FromRef that differs from std::convert::From by accepting an immutable borrow instead of an owned value:
trait FromRef<T> {
fn from_ref(a: &T) -> Self;
}
Of course, I ultimately want to use From<&'a T>, but by defining my own trait I can ask my question more clearly, without messing around with lifetime parameters. (The behaviour of the type-checker is the same using From<&'a T>).
Here's my implementation:
impl<T> FromRef<F<T>> for Target where Target: FromRef<T> {
fn from_ref(a: &F<T>) -> Target {
let b = Target::from_ref(&a.0);
f(&b)
}
}
This compiles. However, the main() function doesn't:
fn main() {
let x = Target;
let y = F(F(F(x)));
let z = Target::from_ref(y);
println!("{:?}", z);
}
It gives a huge error message beginning:
error[E0275]: overflow evaluating the requirement `_: std::marker::Sized`
--> <anon>:26:13
|
26 | let z = Target::from_ref(y);
| ^^^^^^^^^^^^^^^^
|
= note: consider adding a `#![recursion_limit="128"]` attribute to your crate
= note: required because of the requirements on the impl of `FromRef<F<_>>` for `Target`
= note: required because of the requirements on the impl of `FromRef<F<F<_>>>` for `Target`
= note: required because of the requirements on the impl of `FromRef<F<F<F<_>>>>` for `Target`
etc...
What am I doing wrong?
Update
I've randomly fixed it!
The problem was that I forgot to implement FromRef<Target> for Target.
So I would now like to know: what was the compiler thinking? I still can't relate the problem to the error message.
You can't avoid consuming the input in the standard From/Into traits.
They are defined to always consume the input. Their definition specifies both input and output as owned types, with unrelated lifetimes, so you can't even "cheat" by trying to consume a reference.
If you're returning a reference, you can implement AsRef<T> instead. Or if your type is a thin wrapper/smart pointer, Deref<T>. You can provide methods as_foo()
If you're returning a new (owned) object, the convention is to provide to_foo() methods.