Background: I'm creating an iterator that returns a reference to a slice &[T], but the data vector needs to remain immutable. The iterator cannot modify the original data, but must return the same slice pointer repeatedly after it has modified it. I've considered having my iterator own a Vec<T>, but I would like to avoid that (and it didn't seem to work). I avoid allocations since I plan to use it mostly in realtime audio and allocations could potentially block. Code:
pub struct Windower<'a, 'b, T: 'a + 'b> {
window_type: WindowType,
hop_size: usize,
bin_size: usize,
current_index: usize,
data: &'a [T],
out_data: &'b mut [T]
}
impl<'a, 'b, T: Float + FromPrimitive> Iterator for Windower<'a, 'b, T> {
type Item = &'b [T];
fn next(&mut self) -> Option<Self::Item> {
if self.current_index < (self.len() - 1) {
let start = self.current_index * self.hop_size;
let end = start + self.bin_size;
self.current_index += 1;
let window = self.window();
let data_iter = self.data[start..end].iter();
for &mut v in self.out_data {
let val: T = window.next().unwrap() *
*data_iter.next().unwrap();
v = val;
}
Some(self.out_data)
} else {
None
}
}
}
Returns the error:
src/waves.rs:160:18: 160:31 error: cannot infer an appropriate lifetime for automatic coercion due to conflicting requirements [E0495]
src/waves.rs:160 Some(self.out_data)
^~~~~~~~~~~~~
src/waves.rs:146:5: 164:6 help: consider using an explicit lifetime parameter as shown: fn next(&'b mut self) -> Option<Self::Item>
I cannot figure out how to fix this. I can't make the change suggested, because the trait implementation for Iterator does not have an explicit lifetime parameter.
Rust prevents you from having more than one alias to an object if one of them is a mutable alias.
Here, Windower::out_data is a mutable alias to some slice, and you're trying to return an immutable alias to the same data from your next method. In order for this to be safe, Rust must prevent you from being able to use Windower::out_data for as long as the slice returned by next is in scope. This means that the signature fn next(&'b mut self) -> Option<Self::Item> is indeed required, which means you simply cannot implement Iterator with your current implementation.
Related
I'm having difficulties understanding lifetime parameters in the following code snippet.
struct C {
data: Vec<u32>,
cols: usize
}
trait M<'s> {
fn get(&'s self, r: usize, c: usize) -> u32;
fn get_mut(&'s mut self, r: usize, c: usize) -> &'s mut u32;
}
impl<'s> M<'s> for C {
fn get(&'s self, r: usize, c: usize) -> u32 {
return self.data[self.cols*r+c];
}
fn get_mut(&'s mut self, r: usize, c: usize) -> &'s mut u32 {
return &mut self.data[self.cols*r+c];
}
}
#[cfg(test)]
mod tests {
use super::*;
fn create() -> C {
let data = vec![0u32,1u32,2u32,3u32,4u32,5u32];
return C{data, cols: 3};
}
fn select<'s, 'r: 's>(data: &'r mut dyn M<'s>) {
let mut _val: u32 = 0;
for r in 0..2 {
for c in 0..3 {
_val += *data.get_mut(r,c);
}
}
}
#[test]
fn test_select() {
let mut data = create();
select(&mut data);
}
}
The code snippet does not compile, because it complains that *data is borrowed multiple times in the function fn select<'s, 'r: 's>(data: &'r mut dyn M<'s>) {} when calling get_mut (once in every loop iteration). Even safeguarding the questionable line with curly braces (and thus creating a new context) does not help. My expectation (in both cases) would be, that the mutable borrow of &mut data should end right after the execution of that line.
On the other hand, when I remove all lifetime parameters, everything works as expected.
Can anyone explain what's the difference between the two versions (with and without explicit lifetimes)?
I've also tried to find information about additional lifetime parameters for traits, in particular specifying their meaning, but I have found none. So I assume, that they are just a declaration of the used labels inside the trait. But if that is so, then I would assume that leaving out the lifetime parameters completely and applying the eliding rules would lead to the same result.
There are two things to consider. The first is when you use a generic lifetime for a function, that lifetime must be larger than the life of the function call simply by construction. And the second is since the lifetime self is tied to the lifetime parameter of the trait, when you call .get_mut(), data is borrowed for the lifetime of 's. Combining those two principles, data is borrowed for longer than the function call so you can't call it again (its already mutably borrowed).
On the other hand, when I remove all lifetime parameters, everything works as expected. Can anyone explain what's the difference between the two versions (with and without explicit lifetimes)?
Without a generic lifetime on M, the methods will behave as if defined as so:
impl M for C {
fn get<'a>(&'a self, r: usize, c: usize) -> u32 {
return self.data[self.cols * r + c];
}
fn get_mut<'a>(&'a mut self, r: usize, c: usize) -> &'a mut u32 {
return &mut self.data[self.cols * r + c];
}
}
Thus there is no lifetime associated with the trait; the lifetimes given and returned from the function are generic only to those method calls. And since the compiler can choose a new lifetime 'a for each call and it will always pick the shorted lifetime to satisfy its usage, you can then call data.get_mut() multiple times without worry. And I'll be honest, having the lifetime on the trait didn't make much sense with the original code; as mentioned, the code works with all lifetime annotations removed: playground.
This question already has an answer here:
How can I create my own data structure with an iterator that returns mutable references?
(1 answer)
Closed 1 year ago.
I am trying to implement a simple lookup iterator:
pub struct LookupIterMut<'a, D> {
data : &'a mut [D],
indices : &'a [usize],
i: usize
}
impl<'a, D> Iterator for LookupIterMut<'a, D> {
type Item = &'a mut D;
fn next(&mut self) -> Option<Self::Item> {
if self.i >= self.indices.len() {
None
} else {
let index = self.indices[self.i] as usize;
self.i += 1;
Some(&mut self.data[index]) // error here
}
}
}
The idea was to allow a caller consecutive mutable access to an internal storage. However I am getting the error cannot infer an appropriate lifetime for lifetime parameter in function call due to conflicting requirements.
As far as I understand I would have to change the function signature to next(&'a mut self) -> .. but this would not be an Iterator anymore.
I also discovered that I could simply use raw pointers, though I am not sure if this is appropriate here:
// ...
type Item = *mut D;
// ...
Thanks for your help
Your code is invalid because you try to return multiple mutable references to the same slice with the same lifetime 'a.
For such a thing to work, you would need a different lifetime for each returned Item so that you wouldn't hold 2 mutable references to the same slice. You cannot do that for now because it requires Generic Associated Types:
type Item<'item> = &'item mut D; // Does not work today
One solution is to check that the indices are unique and to rebind the lifetime of the referenced item to 'a in an unsafe block. This is safe because all the indices are unique, so the user cannot hold 2 mutable references to the same item.
Don't forget to encapsulate the whole code inside a module, so that the struct cannot be build without the check in new:
mod my_mod {
pub struct LookupIterMut<'a, D> {
data: &'a mut [D],
indices: &'a [usize],
i: usize,
}
impl<'a, D> LookupIterMut<'a, D> {
pub fn new(data: &'a mut [D], indices: &'a [usize]) -> Result<Self, ()> {
let mut uniq = std::collections::HashSet::new();
let all_distinct = indices.iter().all(move |&x| uniq.insert(x));
if all_distinct {
Ok(LookupIterMut {
data,
indices,
i: 0,
})
} else {
Err(())
}
}
}
impl<'a, D> Iterator for LookupIterMut<'a, D> {
type Item = &'a mut D;
fn next(&mut self) -> Option<Self::Item> {
self.indices.get(self.i).map(|&index| {
self.i += 1;
unsafe { std::mem::transmute(&mut self.data[index]) }
})
}
}
}
Note that your code will panic if one index is out of bounds.
Using unsafe
Reminder: it is unsound to have, at any time, two accessible mutable references to the same underlying value.
The crux of the problem is that the language cannot guarantee that the code abides by the above rule, should indices contain any duplicate, then the iterator as implemented would allow obtaining concurrently two mutable references to the same item in the slice, which is unsound.
When the language cannot make the guarantee on its own, then you either need to find an alternative approach or you need to do your due diligence and then use unsafe.
In this case, on the Playground:
impl<'a, D> LookupIterMut<'a, D> {
pub fn new(data: &'a mut [D], indices: &'a [usize]) -> Self {
let set: HashSet<usize> = indices.iter().copied().collect();
assert!(indices.len() == set.len(), "Duplicate indices!");
Self { data, indices, i: 0 }
}
}
impl<'a, D> Iterator for LookupIterMut<'a, D> {
type Item = &'a mut D;
fn next(&mut self) -> Option<Self::Item> {
if self.i >= self.indices.len() {
None
} else {
let index = self.indices[self.i];
assert!(index < self.data.len());
self.i += 1;
// Safety:
// - index is guaranteed to be within bounds.
// - indices is guaranteed not to contain duplicates.
Some(unsafe { &mut *self.data.as_mut_ptr().offset(index as isize) })
}
}
}
Performance wise, the construction of a HashSet in the constructor is rather unsatisfying but cannot really be avoided in general. If indices was guaranteed to be sorted for example, then the check could be performed without allocation.
I've run into a problem implementing Index on a type with a lifetime constraint. I have a SubImage struct which contains a reference to an Image. I can't find any way to satisfy the compiler.
impl<'a, P> Index<usize> for SubImage<'a, P> {
type Output = [P];
fn index<'b> (&'b self, y: usize) -> &'b [P] {
let start = (self.rect.y0 + y) * self.image.size.x + self.rect.x0;
let end = start + self.rect.width();
&self.image.buf[start..end]
}
}
'a is the lifetime of the referenced image, so slicing its buffer requires this constraint. The code here compiles, but it is ambiguous. All calls to the index operator result an an error message such as:
src/image.rs:179:13: 179:32 error: cannot infer an appropriate lifetime for lifetime parameter 'a in function call due to conflicting requirements
src/image.rs:179 Some(&self.sub[self.back])
^~~~~~~~~~~~~~~~~~~
src/image.rs:174:3: 181:4 help: consider using an explicit lifetime parameter as shown: fn next_back(&'a mut self) -> Option<&'a [P]>
src/image.rs:174 fn next_back (&mut self) -> Option<&'a [P]> {
src/image.rs:175 if self.front == self.back {
src/image.rs:176 None
src/image.rs:177 } else {
src/image.rs:178 self.back -= 1;
src/image.rs:179 Some(&self.sub[self.back])
Is there any possible way to ensure that the return value is constrained to both 'a and 'b or some other way to implement Index properly in a situation like this? The compiler's suggestion doesn't work because the function signature doesn't match that of the trait.
There's actually more to your problem and it is not the Index trait implementation. Also, your example is not really a minimal, complete, and verifiable example (MCVE), so I have to guess as to what your exact problem is here.
The core of your problem is, that you cannot have your iterator return a reference, if it owns the content without borrowing the iterator itself. Your implementation of the Index trait for you SubImage is fine.
I will try to simulate your problem. Let's say we have a struct Julmond and it borrows some slice of integers (similar to your SubImage).
struct Julmond<'a>(&'a [i32]);
impl<'a> Index<usize> for Julmond<'a> {
type Output = [i32];
fn index<'b>(&'b self, idx: usize) -> &'b [i32] {
if idx < self.0.len() {
&self.0[idx..] // we always take a subslice from idx until the end
} else {
panic!("Index out of bounds!")
}
}
}
The Index trait requires that we borrow self. That is fine, since some implementors might own the data you're indexing into. This borrowing of self is expressed by linking the named lifetimes of self and the outgoing reference in the traits method signature:
fn index(&'a self, index: Idx) -> &'a Self::Output;
If we index into a Julmond, that value is considered borrowed as long as we hold on to the resulting reference into the Julmond:
let array = [1, 2, 3, 4, 5, 6];
let mut j = Julmond(&array);
let r = &j[3];
&mut j; // Error: r is still in scope and therefore j is still borrowed
What I can read from your example code is that you have some type that owns your SubImage and implements the Iterator trait. We will try to mimic that with another struct Nebelung implementing the Iterator trait on the way:
struct Nebelung<'a> {
j: Julmond<'a>,
pos: usize,
}
impl<'a> Iterator for Nebelung<'a> {
type Item = &'a [i32];
fn next(&mut self) -> Option<&'a [i32]> {
if self.pos < self.j.0.len() {
let tmp_pos = self.pos;
self.pos += 1;
Some(&self.j[tmp_pos]) // problematic line
} else {
None
}
}
}
This implementation returns an ever shrinking slice of the array from the underlying Julmond struct. We can test it like this:
fn main() {
let array = [1, 2, 3, 4, 5, 6];
let j = Julmond(&array);
let n = Nebelung { j: &j, pos: 0 };
for s in n {
println!("{:?}", s);
}
}
But this doesn't work. The compiler will complain (like in your example) that it cannot infer an appropriate lifetime for 'a. The reason is the borrow of self in the index method. When we call the index operator with j[tmp_pos] we are borrowing j. But j is owned by self of type Nebelung and so borrowing from j means we are borrowing from self. We are trying to return a reference to something that is owned by self and that requires that self has to be borrowed as well. The compiler suggests the right thing: linking the lifetimes of self and the outgoing reference. However, this violates the method signature of next.
If we want to return a reference from an iterator, that iterator cannot own the returned value. Otherwise, we would have to borrow the iterator in the call but that is not possible with next.
The only way around this, is having the iterator NOT own the value. So we modify the Nebelung struct to hold a reference to a Julmond:
struct Nebelung<'a: 'b, 'b> {
j: &'b Julmond<'a>,
pos: usize,
}
The 'a: 'b means that "'a outlives 'b" and it is required here. Since our reference j to a Julmond must not outlive the borrowed content of the Julmond. Ok great, our Nebelung is not the owner of the Julmond anymore. Just a borrower. Now we can implement the Iterator trait for it like this:
impl<'a, 'b> Iterator for Nebelung<'a, 'b> {
type Item = &'b [i32];
fn next(&mut self) -> Option<&'b [i32]> {
if self.pos < self.j.0.len() {
let tmp_pos = self.pos;
self.pos += 1;
Some(&self.j[tmp_pos])
} else {
None
}
}
}
The lifetimes of self and the outgoing reference are not required to be linked, since we are just returning a reference to some value which we are not the owner of. So the call to &self.j[tmp_pos] is not a borrow from self anymore. It is a borrow from the Julmond (via the index implementation).
Complete example
Whatever type you are implementing the Iterator trait for. You cannot have next (or next_back) return a reference if the type owns the value. Have your type borrow the SubImage instead.
I have a lifetime issue, I'm trying to implement an iterator returning its items by reference, here is the code:
struct Foo {
d: [u8; 42],
pos: usize
}
impl<'a> Iterator<&'a u8> for Foo {
fn next<'a>(&'a mut self) -> Option<&'a u8> {
let r = self.d.get(self.pos);
if r.is_some() {
self.pos += 1;
}
r
}
}
fn main() {
let mut x = Foo {
d: [1; 42],
pos: 0
};
for i in x {
println!("{}", i);
}
}
However this code doesn't compile properly, I get an issue related to the lifetime of parameters, here is the corresponding error:
$ rustc test.rs
test.rs:8:5: 14:6 error: method `next` has an incompatible type for trait: expected concrete lifetime, but found bound lifetime parameter
test.rs:8 fn next<'a>(&'a mut self) -> Option<&'a u8> {
test.rs:9 let r = self.d.get(self.pos);
test.rs:10 if r.is_some() {
test.rs:11 self.pos += 1;
test.rs:12 }
test.rs:13 r
...
test.rs:8:49: 14:6 note: expected concrete lifetime is the lifetime 'a as defined on the block at 8:48
test.rs:8 fn next<'a>(&'a mut self) -> Option<&'a u8> {
test.rs:9 let r = self.d.get(self.pos);
test.rs:10 if r.is_some() {
test.rs:11 self.pos += 1;
test.rs:12 }
test.rs:13 r
...
error: aborting due to previous error
Does somebody has an idea how to fix this issue and still returning items by reference?
At least what does this message means: expected concrete lifetime, but found bound lifetime parameter ?
Note on the version of Rust used: at the time this question and answer were written, the Iterator trait used generics; it has changed to use associated types and is now defined thus:
pub trait Iterator {
type Item;
fn next(&mut self) -> Option<Self::Item>;
…
}
And so the incorrect implementation shown here would be like this:
impl<'a> Iterator for Foo {
type Item = &'a u8;
fn next<'a>(&'a mut self) -> Option<&'a u8>;
}
In practical terms this affects nothing; it is merely that A becomes Self::Item.
The definition of the Iterator trait is thus:
pub trait Iterator<A> {
fn next(&mut self) -> Option<A>;
…
}
Note carefully: fn next(&mut self) -> Option<A>.
Here is what you have:
impl<'a> Iterator<&'a u8> for Foo {
fn next<'a>(&'a mut self) -> Option<&'a u8>;
}
Note carefully: fn next<'a>(&'a mut self) -> Option<&'a u8>.
There are several problems here:
You have introduced a new generic parameter <'a> which should not be there. For convenience’s sake and to emphasise what has happened here, I shall dub the 'a defined on the impl block ρ₀ and the 'a defined on the method ρ₁. They are not the same.
The lifetime of &mut self is different from that of the trait.
The lifetime of the return type is different to the trait: where A is &'ρ₀ u8, the return type uses in the place of A &'ρ₁ u8. It expected the concrete lifetime ρ₀ but found instead the lifetime ρ₁. (I’m not certain precisely what the “bound” bit means, so I’ll keep quiet on it lest I be wrong.)
Here’s what this amounts to: you cannot connect the lifetime of the object you are iterating over to &mut self. Instead, it must be bound to something in the type you are implementing the trait for. To take an example, iterating over items in a slice is done by creating a new iterator object connected to the base slice, impl<'a, T> Iterator<&'a T> for Items<'a, T>. Expressed in another way, the way the iteration traits are designed is not, if you are producing references, for you to return something inside self, but rather to return something inside another object that you have a reference to.
For your specific, presumably simple example, you should either stop yielding references, or alter it so that your iterator object does not contain the data that you are iterating over—let it merely contain a reference to it, e.g. &'a [T] or even something like Items<'a, T>.
I've been working on a multi-dimensional array library, toying around with different interfaces, and ran into an issue I can't seem to solve. This may be a simple misunderstanding of lifetimes, but I've tried just about every solution I can think of, to no success.
The goal: implement the Index and IndexMut traits to return a borrowed vector from a 2d matrix, so this syntax can be used mat[rowind][colind].
A (very simplified) version of the data structure definition is below.
pub struct Matrix<T> {
shape: [uint, ..2],
dat: Vec<T>
}
impl<T: FromPrimitive+Clone> Matrix<T> {
pub fn new(shape: [uint, ..2]) -> Matrix<T> {
let size = shape.iter().fold(1, |a, &b| { a * b});
// println!("Creating MD array of size: {} and shape: {}", size, shape)
Matrix{
shape: shape,
dat: Vec::<T>::from_elem(size, FromPrimitive::from_uint(0u).expect("0 must be convertible to parameter type"))
}
}
pub fn mut_index(&mut self, index: uint) -> &mut [T] {
let base = index*self.shape[1];
self.dat.mut_slice(base, base + self.shape[1])
}
}
fn main(){
let mut m = Matrix::<f32>::new([4u,4]);
println!("{}", m.dat)
println!("{}", m.mut_index(3)[0])
}
The mut_index method works exactly as I would like the IndexMut trait to work, except of course that it doesn't have the syntax sugar. The first attempt at implementing IndexMut made me wonder, since it returns a borrowed reference to the specified type, I really want to specify [T] as a type, but it isn't a valid type. So the only option is to specify &mut [T] like this.
impl<T: FromPrimitive+Clone> IndexMut<uint, &mut [T]> for Matrix<T> {
fn index_mut(&mut self, index: &uint) -> &mut(&mut[T]) {
let base = index*self.shape[1];
&mut self.dat.mut_slice(base, base + self.shape[1])
}
}
This complains about a missing lifetime specifier on the trait impl line. So I try adding one.
impl<'a, T: FromPrimitive+Clone> IndexMut<uint, &'a mut [T]> for Matrix<T> {
fn index_mut(&'a mut self, index: &uint) -> &mut(&'a mut[T]) {
let base = index*self.shape[1];
&mut self.dat.mut_slice(base, base + self.shape[1])
}
}
Now I get method `index_mut` has an incompatible type for trait: expected concrete lifetime, but found bound lifetime parameter 'a [E0053]. Aside from this I've tried just about every combination of one and two lifetimes I can think of, as well as creating a secondary structure to hold a reference that is stored in the outer structure during the indexing operation so a reference to that can be returned instead, but that's not possible for Index. The final answer may just be that this isn't possible, given the response on this old github issue, but that would seem to be a problematic limitation of the Index and IndexMut traits. Is there something I'm missing?
At present, this is not possible, but when Dynamically Sized Types lands I believe it will become possible.
Let’s look at the signature:
pub trait IndexMut<Index, Result> {
fn index_mut<'a>(&'a mut self, index: &Index) -> &'a mut Result;
}
(Note the addition of the <'a> compared with what the docs say; I’ve filed #16228 about that.)
'a is an arbitrary lifetime, but it is important that it is specified on the method, not on the impl as a whole: it is in absolute truth a generic parameter to the method. I’ll show how it all comes out here with the names 'ρ₀ and 'ρ₁. So then, in this attempt:
impl<'ρ₀, T: FromPrimitive + Clone> IndexMut<uint, &'ρ₀ mut [T]> for Matrix<T> {
fn index_mut<'ρ₁>(&'ρ₁ mut self, index: &uint) -> &'ρ₁ mut &'ρ₀ mut [T] {
let base = index * self.shape[1];
&mut self.dat.mut_slice(base, base + self.shape[1])
}
}
This satisfies the requirements that (a) all lifetimes must be explicit in the impl header, and (b) that the method signature matches the trait definition: Index is uint and Result is &'ρ₀ mut [T]. Because 'ρ₀ is defined on the impl block (so that it can be used as a parameter there) and 'ρ₁ on the method (because that’s what the trait defines), 'ρ₀ and 'ρ₁ cannot be combined into a single named lifetime. (You could call them both 'a, but this is shadowing and does not change anything except for the introduction of a bit more confusion!)
However, this is not enough to have it all work, and it will indeed not compile, because 'ρ₀ is not tied to anything, nor is there to tie it to in the signature. And so you cannot cast self.data.mut_slice(…), which is of type &'ρ₁ mut [T], to &'ρ₀ mut [T] as the lifetimes do not match, nor is there any known subtyping relationship between them (that is, it cannot structurally be demonstrated that the lifetime 'ρ₀ is less than—a subtype of—'ρ₁; although the return type of the method would make that clear, it is not so at the basic type level, and so it is not permitted).
Now as it happens, IndexMut isn’t as useful as it should be anyway owing to #12825, as matrix[1] would always use IndexMut and never Index if you have implemented both. I’m not sure if that’s any consolation, though!
The solution comes in Dynamically Sized Types. When that is here, [T] will be a legitimate unsized type which can be used as the type for Result and so this will be the way to write it:
impl<T: FromPrimitive + Clone> IndexMut<uint, [T]> for Matrix<T> {
fn index_mut<'a>(&'a mut self, index: &uint) -> &'a mut [T] {
let base = index * self.shape[1];
&mut self.dat.mut_slice(base, base + self.shape[1])
}
}
… but that’s not here yet.
This code works in Rust 1.25.0 (and probably has for quite a while)
extern crate num;
use num::Zero;
pub struct Matrix<T> {
shape: [usize; 2],
dat: Vec<T>,
}
impl<T: Zero + Clone> Matrix<T> {
pub fn new(shape: [usize; 2]) -> Matrix<T> {
let size = shape.iter().product();
Matrix {
shape: shape,
dat: vec![T::zero(); size],
}
}
pub fn mut_index(&mut self, index: usize) -> &mut [T] {
let base = index * self.shape[1];
&mut self.dat[base..][..self.shape[1]]
}
}
fn main() {
let mut m = Matrix::<f32>::new([4; 2]);
println!("{:?}", m.dat);
println!("{}", m.mut_index(3)[0]);
}
You can enhance it to support Index and IndexMut:
use std::ops::{Index, IndexMut};
impl<T> Index<usize> for Matrix<T> {
type Output = [T];
fn index(&self, index: usize) -> &[T] {
let base = index * self.shape[1];
&self.dat[base..][..self.shape[1]]
}
}
impl<T> IndexMut<usize> for Matrix<T> {
fn index_mut(&mut self, index: usize) -> &mut [T] {
let base = index * self.shape[1];
&mut self.dat[base..][..self.shape[1]]
}
}
fn main() {
let mut m = Matrix::<f32>::new([4; 2]);
println!("{:?}", m.dat);
println!("{}", m[3][0]);
m[3][0] = 42.42;
println!("{:?}", m.dat);
println!("{}", m[3][0]);
}