Related
The Read trait is implemented for &[u8]. How can I get a Read trait over several concatenated u8 slices without actually doing any concatenation first?
If I concatenate first, there will be two copies -- multiple arrays into a single array followed by copying from single array to destination via the Read trait. I would like to avoid the first copying.
I want a Read trait over &[&[u8]] that treats multiple slices as a single continuous slice.
fn foo<R: std::io::Read + Send>(data: R) {
// ...
}
let a: &[u8] = &[1, 2, 3, 4, 5];
let b: &[u8] = &[1, 2];
let c: &[&[u8]] = &[a, b];
foo(c); // <- this won't compile because `c` is not a slice of bytes.
You could use the multi_reader crate, which can concatenate any number of values that implement Read:
let a: &[u8] = &[1, 2, 3, 4, 5];
let b: &[u8] = &[1, 2];
let c: &[&[u8]] = &[a, b];
foo(multi_reader::MultiReader::new(c.iter().copied()));
If you don't want to depend on an external crate, you can wrap the slices in a struct of your own and implement Read for it:
struct MultiRead<'a> {
sources: &'a [&'a [u8]],
pos_in_current: usize,
}
impl<'a> MultiRead<'a> {
fn new(sources: &'a [&'a [u8]]) -> MultiRead<'a> {
MultiRead {
sources,
pos_in_current: 0,
}
}
}
impl Read for MultiRead<'_> {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
let current = loop {
if self.sources.is_empty() {
return Ok(0); // EOF
}
let current = self.sources[0];
if self.pos_in_current < current.len() {
break current;
}
self.pos_in_current = 0;
self.sources = &self.sources[1..];
};
let read_size = buf.len().min(current.len() - self.pos_in_current);
buf[..read_size].copy_from_slice(¤t[self.pos_in_current..][..read_size]);
self.pos_in_current += read_size;
Ok(read_size)
}
}
Playground
Create a wrapper type around the slices and implement Read for it. Compared to user4815162342's answer, I delegate down to the implementation of Read for slices:
use std::{io::Read, mem};
struct Wrapper<'a, 'b>(&'a mut [&'b [u8]]);
impl<'a, 'b> Read for Wrapper<'a, 'b> {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
let slices = mem::take(&mut self.0);
match slices {
[head, ..] => {
let n_bytes = head.read(buf)?;
if head.is_empty() {
// Advance the child slice
self.0 = &mut slices[1..];
} else {
// More to read, put back all the child slices
self.0 = slices;
}
Ok(n_bytes)
}
_ => Ok(0),
}
}
}
fn main() {
let parts: &mut [&[u8]] = &mut [b"hello ", b"world"];
let mut w = Wrapper(parts);
let mut buf = Vec::new();
w.read_to_end(&mut buf).unwrap();
assert_eq!(b"hello world", &*buf);
}
A more efficient implementation would implement further methods from Read, such as read_to_end or read_vectored.
See also:
How do I implement a trait I don't own for a type I don't own?
I am practicing rust and decided to create a Matrix ops/factorization project.
Basically I want to be able to process the underlying vector in multiple threads. Since I will be providing each thread non-overlapping indexes (which may or may not be contiguous) and the threads will be joined before the end of whatever function created them, there is no need for a lock /synchronization.
I know that there are several crates that can do this, but I would like to know if there is a relatively idiomatic crate-free way to implement it on my own.
The best I could come up with is (simplified the code a bit):
use std::thread;
//This represents the Matrix
#[derive(Debug, Clone)]
pub struct MainStruct {
pub data: Vec<f64>,
}
//This is the bit that will be shared by the threads,
//ideally it should have its lifetime tied to that of MainStruct
//but i have no idea how to make phantomdata work in this case
#[derive(Debug, Clone)]
pub struct SliceTest {
pub data: Vec<SubSlice>,
}
//This struct is to hide *mut f64 to allow it to be shared to other threads
#[derive(Debug, Clone)]
pub struct SubSlice {
pub data: *mut f64,
}
impl MainStruct {
pub fn slice(&mut self) -> (SliceTest, SliceTest) {
let mut out_vec_odd: Vec<SubSlice> = Vec::new();
let mut out_vec_even: Vec<SubSlice> = Vec::new();
unsafe {
let ptr = self.data.as_mut_ptr();
for i in 0..self.data.len() {
let ptr_to_push = ptr.add(i);
//Non contiguous idxs
if i % 2 == 0 {
out_vec_even.push(SubSlice{data:ptr_to_push});
} else {
out_vec_odd.push(SubSlice{data:ptr_to_push});
}
}
}
(SliceTest{data: out_vec_even}, SliceTest{data: out_vec_odd})
}
}
impl SubSlice {
pub fn set(&self, val: f64) {
unsafe {*(self.data) = val;}
}
}
unsafe impl Send for SliceTest {}
unsafe impl Send for SubSlice {}
fn main() {
let mut maindata = MainStruct {
data: vec![0.0, 1.0, 2.0, 3.0, 4.0, 5.0],
};
let (mut outvec1, mut outvec2) = maindata.slice();
let mut threads = Vec::new();
threads.push(
thread::spawn(move || {
for i in 0..outvec1.data.len() {
outvec1.data[i].set(999.9);
}
})
);
threads.push(
thread::spawn(move || {
for i in 0..outvec2.data.len() {
outvec2.data[i].set(999.9);
}
})
);
for handles in threads {
handles.join();
}
println!("maindata = {:?}", maindata.data);
}
EDIT:
Following kmdreko suggestion below, got the code to work exactly how I wanted it without using unsafe code, yay!
Of course in terms of performance it may be cheaper to copy the f64 slices than to create mutable reference vectors unless your struct is filled with other structs instead of f64
extern crate crossbeam;
use crossbeam::thread;
#[derive(Debug, Clone)]
pub struct Matrix {
data: Vec<f64>,
m: usize, //number of rows
n: usize, //number of cols
}
...
impl Matrix {
...
pub fn get_data_mut(&mut self) -> &mut Vec<f64> {
&mut self.data
}
pub fn calculate_idx(max_cols: usize, i: usize, j: usize) -> usize {
let actual_idx = j + max_cols * i;
actual_idx
}
//Get individual mutable references for contiguous indexes (rows)
pub fn get_all_row_slices(&mut self) -> Vec<Vec<&mut f64>> {
let max_cols = self.max_cols();
let max_rows = self.max_rows();
let inner_data = self.get_data_mut().chunks_mut(max_cols);
let mut out_vec: Vec<Vec<&mut f64>> = Vec::with_capacity(max_rows);
for chunk in inner_data {
let row_vec = chunk.iter_mut().collect();
out_vec.push(row_vec);
}
out_vec
}
//Get mutable references for disjoint indexes (columns)
pub fn get_all_col_slices(&mut self) -> Vec<Vec<&mut f64>> {
let max_cols = self.max_cols();
let max_rows = self.max_rows();
let inner_data = self.get_data_mut().chunks_mut(max_cols);
let mut out_vec: Vec<Vec<&mut f64>> = Vec::with_capacity(max_cols);
for _ in 0..max_cols {
out_vec.push(Vec::with_capacity(max_rows));
}
let mut inner_idx = 0;
for chunk in inner_data {
let row_vec_it = chunk.iter_mut();
for elem in row_vec_it {
out_vec[inner_idx].push(elem);
inner_idx += 1;
}
inner_idx = 0;
}
out_vec
}
...
}
fn test_multithreading() {
fn test(in_vec: Vec<&mut f64>) {
for elem in in_vec {
*elem = 33.3;
}
}
fn launch_task(mat: &mut Matrix, f: fn(Vec<&mut f64>)) {
let test_vec = mat.get_all_row_slices();
thread::scope(|s| {
for elem in test_vec.into_iter() {
s.spawn(move |_| {
println!("Spawning thread...");
f(elem);
});
}
}).unwrap();
}
let rows = 4;
let cols = 3;
//new function code omitted, returns Result<Self, MatrixError>
let mut mat = Matrix::new(rows, cols).unwrap()
launch_task(&mut mat, test);
for i in 0..rows {
for j in 0..cols {
//Requires index trait implemented for matrix
assert_eq!(mat[(i, j)], 33.3);
}
}
}
This API is unsound. Since there is no lifetime annotation binding SliceTest and SubSlice to the MainStruct, they can be preserved after the data has been destroyed and if used would result in use-after-free errors.
Its easy to make it safe though; you can use .iter_mut() to get distinct mutable references to your elements:
pub fn slice(&mut self) -> (Vec<&mut f64>, Vec<&mut f64>) {
let mut out_vec_even = vec![];
let mut out_vec_odd = vec![];
for (i, item_ref) in self.data.iter_mut().enumerate() {
if i % 2 == 0 {
out_vec_even.push(item_ref);
} else {
out_vec_odd.push(item_ref);
}
}
(out_vec_even, out_vec_odd)
}
However, this surfaces another problem: thread::spawn cannot hold references to local variables. The threads created are allowed to live beyond the scope they're created in, so even though you did .join() them, you aren't required to. This was a potential issue in your original code as well, just the compiler couldn't warn about it.
There's no easy way to solve this. You'd need to use a non-referential way to use data on the other threads, but that would be using Arc, which doesn't allow mutating its data, so you'd have to resort to a Mutex, which is what you've tried to avoid.
I would suggest reaching for scope from the crossbeam crate, which does allow you to spawn threads that reference local data. I know you've wanted to avoid using crates, but this is the best solution in my opinion.
See a working version on the playground.
See:
How to get multiple mutable references to elements in a Vec?
Can you specify a non-static lifetime for threads?
I am trying to create an mutable iterator for a vector of type: Vec<Vec<(K, V)>>
The iterator code:
pub struct IterMut<'a, K: 'a, V: 'a> {
iter: &'a mut Vec<Vec<(K, V)>>,
ix: usize,
inner_ix: usize,
}
impl<'a, K, V> Iterator for IterMut<'a, K, V> {
type Item = (&'a K, &'a mut V);
#[inline]
fn next(&mut self) -> Option<(&'a K, &'a mut V)> {
while self.iter.len() < self.ix {
while self.iter[self.ix].len() < self.inner_ix {
self.inner_ix += 1;
let (ref k, ref mut v) = self.iter[self.ix][self.inner_ix];
return Some((&k, &mut v));
}
self.ix += 1;
}
return None;
}
}
The error I get is:
error[E0495]: cannot infer an appropriate lifetime for lifetime parameter in function call due to conflicting requirements
--> src/main.rs:16:42
|
16 | let (ref k, ref mut v) = self.iter[self.ix][self.inner_ix];
| ^^^^^^^^^^^^^^^^^^
|
help: consider using an explicit lifetime parameter as shown: fn next(&'a mut self) -> Option<(&'a K, &'a mut V)>
--> src/main.rs:11:5
|
11 | fn next(&mut self) -> Option<(&'a K, &'a mut V)> {
| ^
Apparently I have lifetime problems, but I don't know how to tell the compiler that this should work.
Is this how you should implement the mutable iterator or is there a better way?
When debugging cryptic error messages, I've found it easier to try and isolate the issue as much as possible.
The first step is to break the expression into its essential constituents, let's start by splitting the indexing steps:
fn next(&mut self) -> Option<(&'a K, &'a mut V)> {
while self.iter.len() < self.ix {
while self.iter[self.ix].len() < self.inner_ix {
self.inner_ix += 1;
let outer: &'a mut Vec<_> = self.iter;
let inner: &'a mut Vec<_> = &mut outer[self.ix];
let (ref k, ref mut v) = inner[self.inner_ix];
return Some((&k, &mut v));
}
self.ix += 1;
}
return None;
}
The Index trait assumes that the lifetime of its output is linked to that of its receiver, so to get a 'a lifetime we need the receiver to have a &'a lifetime, and it propagates upward, leading to the above code.
However there's an issue here: let outer: &'a mut Vec<_> = self.iter; will not compile because mutable references are not Copy.
So, how does one get a mutable reference from a mutable reference (which must be possible since IndexMut gets a mutable reference)?
One uses re-borrowing: let outer: &'a mut Vec<_> = &mut *self.iter;.
And, oh:
error[E0495]: cannot infer an appropriate lifetime for borrow expression due to conflicting requirements
--> <anon>:16:45
|
16 | let outer: &'a mut Vec<_> = &mut *self.iter;
| ^^^^^^^^^^^^^^^
|
The reborrowed reference is not valid for 'a, it's valid only for the (unnamed) lifetime of self!
Why Rust? Why?
Because doing otherwise would be unsafe.
&mut T is guaranteed NOT to be aliasing, however your method could create aliasing references (if you forgot to advance the index):
#[inline]
fn next(&mut self) -> Option<(&'a K, &'a mut V)> {
let (ref k, ref mut v) = self.iter[self.ix][self.inner_ix];
return Some((&k, &mut v));
}
And even if you don't, there's not guarantee that you don't have a rewind method that would allow "stepping back".
TL;DR: You were about to step on a landmine, you were steered toward Stack Overflow instead ;)
Alright, but how do you implement the iterator!.
Well, using iterators, of course. As Shepmaster (briefly) answers, there is the equivalent in the standard library already in the guise of FlatMap. The trick is to use existing iterators for the nitty-gritty details!
Something like:
use std::slice::IterMut;
pub struct MyIterMut<'a, K: 'a, V: 'a> {
outer: IterMut<'a, Vec<(K, V)>>,
inner: IterMut<'a, (K, V)>,
}
Then you consume from inner as long as it provides items, and when empty you refill it from outer.
impl<'a, K, V> MyIterMut<'a, K, V> {
fn new(v: &'a mut Vec<Vec<(K, V)>>) -> MyIterMut<'a, K, V> {
let mut outer = v.iter_mut();
let inner = outer.next()
.map(|v| v.iter_mut())
.unwrap_or_else(|| (&mut []).iter_mut());
MyIterMut { outer: outer, inner: inner }
}
}
impl<'a, K, V> Iterator for MyIterMut<'a, K, V> {
type Item = (&'a K, &'a mut V);
#[inline]
fn next(&mut self) -> Option<(&'a K, &'a mut V)> {
loop {
match self.inner.next() {
Some(r) => return Some((&r.0, &mut r.1)),
None => (),
}
match self.outer.next() {
Some(v) => self.inner = v.iter_mut(),
None => return None,
}
}
}
}
A quick test case:
fn main() {
let mut v = vec![
vec![(1, "1"), (2, "2")],
vec![],
vec![(3, "3")]
];
let iter = MyIterMut::new(&mut v);
let c: Vec<_> = iter.collect();
println!("{:?}", c);
}
Prints:
[(1, "1"), (2, "2"), (3, "3")]
as expected, so it's not completely broken, but I wish I did not have to rely on the &[] is 'static trick (ie, that std::slice::IterMut implemented Default).
You've provided no reason that you are reimplementing the standard Iterator::flat_map, so I'd just use that and another map to remove the mutability you don't need:
fn main() {
let mut a: Vec<Vec<(u8, u8)>> = Default::default();
let c = a.iter_mut()
.flat_map(|x| x.iter_mut())
.map(|&mut (ref a, ref mut b)| (a, b))
.count();
println!("{}", c);
}
Once you have that, you can just return the iterator in one of the many ways.
#[derive(Debug, Default)]
struct Thing<K, V>(Vec<Vec<(K, V)>>);
impl<K, V> Thing<K, V> {
fn iter_mut<'a>(&'a mut self) -> Box<Iterator<Item = (&'a K, &'a mut V)> + 'a> {
Box::new(self.0
.iter_mut()
.flat_map(|x| x.iter_mut())
.map(|&mut (ref a, ref mut b)| (a, b)))
}
}
fn main() {
let mut a = Thing::<u8, u8>::default();
let c = a.iter_mut().count();
println!("{}", c);
}
I have some object that I want to split into two parts via a mutable borrow, then combine those back together into the original object when the split references go out of scope.
The simplified example below is for a Count struct that holds a single i32, which we want to split into two &mut i32s, who are both incorporated back into the original Count when the two mutable references go out of scope.
The approach I am taking below is to use an intermediate object CountSplit which holds a mutable reference to the original Count object and has the Drop trait implemented to do the re-combination logic.
This approach feels kludgy. In particular, this is awkward:
let mut ms = c.make_split();
let (x, y) = ms.split();
Doing this in one line like let (x, y) = c.make_split().split(); is not allowed because the intermediate object must have a longer lifetime. Ideally I would be able to do something like let (x, y) = c.magic_split(); and avoid exposing the intermediate object altogether.
Is there a way to do this which doesn't require doing two let's every time, or some other way to tackle this pattern that would be more idiomatic?
#[derive(Debug)]
struct Count {
val: i32,
}
trait MakeSplit<'a> {
type S: Split<'a>;
fn make_split(&'a mut self) -> Self::S;
}
impl<'a> MakeSplit<'a> for Count {
type S = CountSplit<'a>;
fn make_split(&mut self) -> CountSplit {
CountSplit {
top: self,
second: 0,
}
}
}
struct CountSplit<'a> {
top: &'a mut Count,
second: i32,
}
trait Split<'a> {
fn split(&'a mut self) -> (&'a mut i32, &'a mut i32);
}
impl<'a, 'b> Split<'a> for CountSplit<'b> {
fn split(&mut self) -> (&mut i32, &mut i32) {
(&mut self.top.val, &mut self.second)
}
}
impl<'a> Drop for CountSplit<'a> {
fn drop(&mut self) {
println!("custom drop occurs here");
self.top.val += self.second;
}
}
fn main() {
let mut c = Count { val: 2 };
println!("{:?}", c); // Count { val: 2 }
{
let mut ms = c.make_split();
let (x, y) = ms.split();
println!("split: {} {}", x, y); // split: 2 0
// each of these lines correctly gives a compile-time error
// c.make_split(); // can't borrow c as mutable
// println!("{:?}", c); // or immutable
// ms.split(); // also can't borrow ms
*x += 100;
*y += 5000;
println!("split: {} {}", x, y); // split: 102 5000
} // custom drop occurs here
println!("{:?}", c); // Count { val: 5102 }
}
playground:
I don't think a reference to a temporary value like yours can be made to work in today's Rust.
If it's any help, if you specifically want to call a function with two &mut i32 parameters like you mentioned in the comments, e.g.
fn foo(a: &mut i32, b: &mut i32) {
*a += 1;
*b += 2;
println!("split: {} {}", a, b);
}
you can already do that with the same number of lines as you'd have if your chaining worked.
With the chaining, you'd call
let (x, y) = c.make_split().split();
foo(x, y);
And if you just leave out the conversion to a tuple, it looks like this:
let mut ms = c.make_split();
foo(&mut ms.top.val, &mut ms.second);
You can make it a little prettier by e.g. storing the mutable reference to val directly in CountSplit as first, so that it becomes foo(&mut ms.first, &mut ms.second);. If you want it to feel even more like a tuple, I think you can use DerefMut to be able to write foo(&mut ms.0, &mut ms.1);.
Alternatively, you can of course formulate this as a function taking a function
impl Count {
fn as_split<F: FnMut(&mut i32, &mut i32)>(&mut self, mut f: F) {
let mut second = 0;
f(&mut self.val, &mut second);
self.val += second;
}
}
and then just call
c.as_split(foo);
I have a function f that accepts two references, one mut and one not mut. I have values for f inside a HashMap:
use std::collections::HashMap;
fn f(a: &i32, b: &mut i32) {}
fn main() {
let mut map = HashMap::new();
map.insert("1", 1);
map.insert("2", 2);
{
let a: &i32 = map.get("1").unwrap();
println!("a: {}", a);
let b: &mut i32 = map.get_mut("2").unwrap();
println!("b: {}", b);
*b = 5;
}
println!("Results: {:?}", map)
}
This doesn't work because HashMap::get and HashMap::get_mut attempt to mutably borrow and immutably borrow at the same time:
error[E0502]: cannot borrow `map` as mutable because it is also borrowed as immutable
--> src/main.rs:15:27
|
12 | let a: &i32 = map.get("1").unwrap();
| --- immutable borrow occurs here
...
15 | let b: &mut i32 = map.get_mut("2").unwrap();
| ^^^ mutable borrow occurs here
...
18 | }
| - immutable borrow ends here
In my real code I'm using a large, complex structure instead of a i32 so it is not a good idea to clone it.
In fact, I'm borrowing two different things mutably/immutably, like:
struct HashMap {
a: i32,
b: i32,
}
let mut map = HashMap { a: 1, b: 2 };
let a = &map.a;
let b = &mut map.b;
Is there any way to explain to the compiler that this is actually safe code?
I see how it possible to solve in the concrete case with iter_mut:
{
let mut a: &i32 = unsafe { mem::uninitialized() };
let mut b: &mut i32 = unsafe { mem::uninitialized() };
for (k, mut v) in &mut map {
match *k {
"1" => {
a = v;
}
"2" => {
b = v;
}
_ => {}
}
}
f(a, b);
}
But this is slow in comparison with HashMap::get/get_mut
TL;DR: You will need to change the type of HashMap
When using a method, the compiler does not inspect the interior of a method, or perform any runtime simulation: it only bases its ownership/borrow-checking analysis on the signature of the method.
In your case, this means that:
using get will borrow the entire HashMap for as long as the reference lives,
using get_mut will mutably borrow the entire HashMap for as long as the reference lives.
And therefore, it is not possible with a HashMap<K, V> to obtain both a &V and &mut V at the same time.
The work-around, therefore, is to avoid the need for a &mut V entirely.
This can be accomplished by using Cell or RefCell:
Turn your HashMap into HashMap<K, RefCell<V>>,
Use get in both cases,
Use borrow() to get a reference and borrow_mut() to get a mutable reference.
use std::{cell::RefCell, collections::HashMap};
fn main() {
let mut map = HashMap::new();
map.insert("1", RefCell::new(1));
map.insert("2", RefCell::new(2));
{
let a = map.get("1").unwrap();
println!("a: {}", a.borrow());
let b = map.get("2").unwrap();
println!("b: {}", b.borrow());
*b.borrow_mut() = 5;
}
println!("Results: {:?}", map);
}
This will add a runtime check each time you call borrow() or borrow_mut(), and will panic if you ever attempt to use them incorrectly (if the two keys are equal, unlike your expectations).
As for using fields: this works because the compiler can reason about borrowing status on a per-field basis.
Something appears to have changed since the question was asked. In Rust 1.38.0 (possibly earlier), the following compiles and works:
use std::collections::HashMap;
fn f(a: &i32, b: &mut i32) {}
fn main() {
let mut map = HashMap::new();
map.insert("1", 1);
map.insert("2", 2);
let a: &i32 = map.get("1").unwrap();
println!("a: {}", a);
let b: &mut i32 = map.get_mut("2").unwrap();
println!("b: {}", b);
*b = 5;
println!("Results: {:?}", map)
}
playground
There is no need for RefCell, nor is there even a need for the inner scope.