I'm currently looking into doing more stuff with arrays, but I think the performance of those operations could be even better if we were allowed to somehow transmute into a Leaked<T> the array up front, only to un-leak it when the function ends. This would let us use leak amplification without a) introducing unsafety and b) setting up a catch_panic(_). Is this somehow possible in Rust?
For example, creating a generic array from an iterator (this obviously does not work):
#[inline]
fn map_inner<I, S, F, T, N>(list: I, f: F) -> GenericArray<T, N>
where I: IntoIterator<Item=S>, F: Fn(&S) -> T, N: ArrayLength<T> {
unsafe {
// pre-leak the whole array, it's uninitialized anyway
let mut res : GenericArray<Leaked<T>, N> = std::mem::uninitialized();
let i = list.into_iter();
for r in res.iter_mut() {
// this could panic anytime
std::ptr::write(r, Leaked::new(f(i.next().unwrap())))
}
// transmuting un-leaks the array
std::mem::transmute::<GenericArray<Leaked<T>, N>,
GenericArray<T, N>>(res)
}
}
I should note that if we either had compile-time access to the size of T or a type that can hide its innards from borrowck (like Leaked<T> in the example), this is perfectly feasible.
It is possible using nodrop, but it could leak.
fn map_inner<I, S, F, T, N>(list: I, f: F) -> GenericArray<T, N>
where I: IntoIterator<Item=S>, F: Fn(&S) -> T, N: ArrayLength<T> {
unsafe {
// pre-leak the whole array, it's uninitialized anyway
let mut res : NoDrop<GenericArray<T, N>> = NoDrop::new(std::mem::uninitialized());
let i = list.into_iter();
for r in res.iter_mut() {
// this could panic anytime
std::ptr::write(r, f(i.next().unwrap()))
}
res.into_inner()
}
}
Let's suppose that after the first item (a) is consumed from i and written to r, a panic happens. The remaining items from i would be drop, but the item a would not. Although leaking memory is not considered unsafe, it is not desirable.
I think that the approach described in the question link is the way to go. It is similar to the Vec and ArrayVec implementations. I'm using a similar approach in array library that I'm writing.
Related
I want to be able to create a new vector c: Vec<T> from a: Vec<T> and b: T where c is equal to b appended to a, without using mutables. Say I have this code-
fn concat_vec<T>(a: Vec<T>, b: T) -> Vec<T> {
return Vec::<T>::new(a, b);
}
I am aware this does not work, because Vec::new() has no input parameters. What I am wondering is if there is an function or macro in std::vec that can initialize a vector from another vector and a set of additional values to append onto the vector. I know a function can be created very easily to do this, I am just wondering if functionality exists in the standard library to work with vectors without them being mutable in the first place.
Note that you can use the tap crate to push b to a in an expression that evaluates to a:
use tap::Tap;
fn concat_vec<T>(a: Vec<T>, b: T) -> Vec<T> {
a.tap_mut(|a| a.push(b))
}
This combines the performance benefits of push() (in comparison to building a brand new vector) with the elegance of having a short expression.
Vec::from_iter should work if you're willing to use iterators.
pub fn concat_vec<T>(a: Vec<T>, b: T) -> Vec<T> {
return Vec::from_iter(a.into_iter().chain(std::iter::once(b)));
}
Edit(s): Also as far as I have seen: the most common way to work with vectors, slices, and arrays is through the use of the Iterator trait and the functionality it provides.
Also on the topic of speed: the above approach is slightly faster if you avoid cloning. On my computer iterators take about 1.571µs per call whereas cloning and pushing to a vector took 1.580µs per call. (Tested by running the functions 300 000 times on a debug build.)
The universal form of the inefficient solution should look like this:
fn concat_vec<T>(a: Vec<T>, b: T) -> Vec<T> {
Vec::from_iter(a.into_iter().chain(std::iter::once(b)))
}
…but its strongly recommended to use push:
fn concat_vec<T: std::clone::Clone>(a: Vec<T>, b: T) -> Vec<T> {
let mut rslt = a.to_vec();
rslt.push(b);
rslt
}
Generalized Question
How can I implement a general function pinned_array_of_default in stable Rust where [T; N] is too large to fit on the stack?
fn pinned_array_of_default<T: Default, const N: usize>() -> Pin<Box<[T; N]>> {
unimplemented!()
}
Alternatively, T can implement Copy if that makes the process easier.
fn pinned_array_of_element<T: Copy, const N: usize>(x: T) -> Pin<Box<[T; N]>> {
unimplemented!()
}
Keeping the solution in safe Rust would have been preferable, but it seems unlikely that it is possible.
Approaches
Initially I was hopping that by implementing Default I might be able to get Default to handle the initial allocation, however it still creates it on the stack so this will not work for large values of N.
let boxed: Box<[T; N]> = Box::default();
let foo = Pin::new(boxed);
I suspect I need to use MaybeUninit to achieve this and there is a Box::new_uninit() function, but it is currently unstable and I would ideally like to keep this within stable Rust. I also somewhat unsure if transmuting Pin<Box<MaybeUninit<B>>> to Pin<Box<B>> could somehow have negative effects on the Pin.
Background
The purpose behind using a Pin<Box<[T; N]>> is to hold a block of pointers where N is some constant factor/multiple of the page size.
#[repr(C)]
#[derive(Copy, Clone)]
pub union Foo<R: ?Sized> {
assigned: NonNull<R>,
next_unused: Option<NonNull<Self>>,
}
Each pointer may or may not be in use at a given point in time. An in-use Foo points to R, and an unused/empty Foo has a pointer to either the next empty Foo in the block or None. A pointer to the first unused Foo in the block is stored separately. When a block is full, a new block is created and then pointer chain of unused positions continues through the next block.
The box needs to be pinned since it will contain self referential pointers as well as outside structs holding pointers into assigned positions in each block.
I know that Foo is wildly unsafe by Rust standards, but the general question of creating a Pin<Box<[T; N]>> still stands
A way to construct a large array on the heap and avoid creating it on the stack is to proxy through a Vec. You can construct the elements and use .into_boxed_slice() to get a Box<[T]>. You can then use .try_into() to convert it to a Box<[T; N]>. And then use .into() to convert it to a Pin<Box<[T; N]>>:
fn pinned_array_of_default<T: Default, const N: usize>() -> Pin<Box<[T; N]>> {
let mut vec = vec![];
vec.resize_with(N, T::default);
let boxed: Box<[T; N]> = match vec.into_boxed_slice().try_into() {
Ok(boxed) => boxed,
Err(_) => unreachable!(),
};
boxed.into()
}
You can optionally make this look more straight-forward if you add T: Clone so that you can do vec![T::default(); N] and/or add T: Debug so you can use .unwrap() or .expect().
See also:
Creating a fixed-size array on heap in Rust
Recently, I wrote the following:
use std::ptr;
fn modify_mut_ret<T,R,F> (ptr: &mut T, f: F) -> R
where F: FnOnce(T) -> (T,R)
{
unsafe {
let (t,r) = f(ptr::read(ptr));
ptr::write(ptr,t);
r
}
}
This is a simple utility, so I expected it was in the standard library, but I couldn't find it (at least in std::mem). If we assume, for example, T: Default, we can safely implement this with an extra drop overhead:
use std::mem;
#[inline]
fn modify_mut_ret<T,R,F>(ptr: &mut T, f: F) -> R
where F: FnOnce(T) -> (T,R),
T: Default
{
let mut t = T::default();
mem::swap(ptr, &mut t);
let (t,r) = f(t);
*ptr = t;
r
}
I don't think the first implementation contains any undefined behavior: we have no alignment issue, and we, with ptr::write, eliminate one of the two ownerships duplicated with ptr::read. However I'm anxious about the fact that std seemingly doesn't contain a function with this behavior. Have I got anything wrong or have I forgot something? Does the unsafe code above contain any UB?
This code contains only one instance of UB, which is because the function can return early. Let's take a closer look at it (I moved some things around to make it easier to take apart):
fn modify_mut_ret<T, R, F: FnOnce(T) -> (T, R)>(x: &mut T, f: F) -> R {
unsafe {
let old_val = ptr::read(x); // Copied from original value, two copies of the
// same non-Copy object exist now
let (t, r) = f(old_val); // Supplied one copy to the closure
ptr::write(x, t); // Erased the second copy by writing without dropping it
r
}
}
If the closure runs fine, the outer function will progress as normal and the total number of copies of the old value of x is going to stay at just one copy, which will be owned by the closure, which it may or may not store for later in an Rc<RefCell<...>>/Arc<RwLock<...>> or a global variable.
If it panics, however, and the panic is caught by the code calling modify_mut_ret using std::panic::catch_unwind, there would be two copies of the old value of x, because the ptr::write wasn't reached yet but ptr::read already was.
What you need to do is handle the panicking by aborting the process:
use std::{ptr, panic::{catch_unwind, AssertUnwindSafe}};
fn modify_mut_ret<T, R, F>(x: &mut T, f: F) -> R
where F: FnOnce(T) -> (T, R) {
unsafe {
let old_val = ptr::read(x);
let (t, r) = catch_unwind(AssertUnwindSafe(|| f(old_val)))
.unwrap_or_else(|_| std::process::abort());
ptr::write(x, t); // Erased the second copy by writing without dropping it
r
}
}
This way, panicking in the closure will never leave the function since it will catch the panic and abort the process immediately, before any other code can observe the duplicated value.
The AssertUnwindSafe is there because we have to ensure that we're not gonna observe logically invalid values created as a result of a panic, since we will always abort after a panic. See UnwindSafe's documentation for more on that.
I have the following:
enum SomeType {
VariantA(String),
VariantB(String, i32),
}
fn transform(x: SomeType) -> SomeType {
// very complicated transformation, reusing parts of x in order to produce result:
match x {
SomeType::VariantA(s) => SomeType::VariantB(s, 0),
SomeType::VariantB(s, i) => SomeType::VariantB(s, 2 * i),
}
}
fn main() {
let mut data = vec![
SomeType::VariantA("hello".to_string()),
SomeType::VariantA("bye".to_string()),
SomeType::VariantB("asdf".to_string(), 34),
];
}
I would now like to call transform on each element of data and store the resulting value back in data. I could do something like data.into_iter().map(transform).collect(), but this will allocate a new Vec. Is there a way to do this in-place, reusing the allocated memory of data? There once was Vec::map_in_place in Rust but it has been removed some time ago.
As a work-around, I've added a Dummy variant to SomeType and then do the following:
for x in &mut data {
let original = ::std::mem::replace(x, SomeType::Dummy);
*x = transform(original);
}
This does not feel right, and I have to deal with SomeType::Dummy everywhere else in the code, although it should never be visible outside of this loop. Is there a better way of doing this?
Your first problem is not map, it's transform.
transform takes ownership of its argument, while Vec has ownership of its arguments. Either one has to give, and poking a hole in the Vec would be a bad idea: what if transform panics?
The best fix, thus, is to change the signature of transform to:
fn transform(x: &mut SomeType) { ... }
then you can just do:
for x in &mut data { transform(x) }
Other solutions will be clunky, as they will need to deal with the fact that transform might panic.
No, it is not possible in general because the size of each element might change as the mapping is performed (fn transform(u8) -> u32).
Even when the sizes are the same, it's non-trivial.
In this case, you don't need to create a Dummy variant because creating an empty String is cheap; only 3 pointer-sized values and no heap allocation:
impl SomeType {
fn transform(&mut self) {
use SomeType::*;
let old = std::mem::replace(self, VariantA(String::new()));
// Note this line for the detailed explanation
*self = match old {
VariantA(s) => VariantB(s, 0),
VariantB(s, i) => VariantB(s, 2 * i),
};
}
}
for x in &mut data {
x.transform();
}
An alternate implementation that just replaces the String:
impl SomeType {
fn transform(&mut self) {
use SomeType::*;
*self = match self {
VariantA(s) => {
let s = std::mem::replace(s, String::new());
VariantB(s, 0)
}
VariantB(s, i) => {
let s = std::mem::replace(s, String::new());
VariantB(s, 2 * *i)
}
};
}
}
In general, yes, you have to create some dummy value to do this generically and with safe code. Many times, you can wrap your whole element in Option and call Option::take to achieve the same effect .
See also:
Change enum variant while moving the field to the new variant
Why is it so complicated?
See this proposed and now-closed RFC for lots of related discussion. My understanding of that RFC (and the complexities behind it) is that there's an time period where your value would have an undefined value, which is not safe. If a panic were to happen at that exact second, then when your value is dropped, you might trigger undefined behavior, a bad thing.
If your code were to panic at the commented line, then the value of self is a concrete, known value. If it were some unknown value, dropping that string would try to drop that unknown value, and we are back in C. This is the purpose of the Dummy value - to always have a known-good value stored.
You even hinted at this (emphasis mine):
I have to deal with SomeType::Dummy everywhere else in the code, although it should never be visible outside of this loop
That "should" is the problem. During a panic, that dummy value is visible.
See also:
How can I swap in a new value for a field in a mutable reference to a structure?
Temporarily move out of borrowed content
How do I move out of a struct field that is an Option?
The now-removed implementation of Vec::map_in_place spans almost 175 lines of code, most of having to deal with unsafe code and reasoning why it is actually safe! Some crates have re-implemented this concept and attempted to make it safe; you can see an example in Sebastian Redl's answer.
You can write a map_in_place in terms of the take_mut or replace_with crates:
fn map_in_place<T, F>(v: &mut [T], f: F)
where
F: Fn(T) -> T,
{
for e in v {
take_mut::take(e, f);
}
}
However, if this panics in the supplied function, the program aborts completely; you cannot recover from the panic.
Alternatively, you could supply a placeholder element that sits in the empty spot while the inner function executes:
use std::mem;
fn map_in_place_with_placeholder<T, F>(v: &mut [T], f: F, mut placeholder: T)
where
F: Fn(T) -> T,
{
for e in v {
let mut tmp = mem::replace(e, placeholder);
tmp = f(tmp);
placeholder = mem::replace(e, tmp);
}
}
If this panics, the placeholder you supplied will sit in the panicked slot.
Finally, you could produce the placeholder on-demand; basically replace take_mut::take with take_mut::take_or_recover in the first version.
I'm trying to initialize a boxed slice of None values, such that the underlying type T does not need to implement Clone or Copy. Here a few ideal solutions:
fn by_vec<T>() -> Box<[Option<T>]> {
vec![None; 5].into_boxed_slice()
}
fn by_arr<T>() -> Box<[Option<T>]> {
Box::new([None; 5])
}
Unfortunately, the by_vec implementation requires T: Clone and the by_arr implemenation requires T: Copy. I've experimented with a few more approaches:
fn by_vec2<T>() -> Box<[Option<T>]> {
let v = &mut Vec::with_capacity(5);
for i in 0..v.len() {
v[i] = None;
}
v.into_boxed_slice() // Doesn't work: cannot move out of borrowed content
}
fn by_iter<T>() -> Box<[Option<T>]> {
(0..5).map(|_| None).collect::<Vec<Option<T>>>().into_boxed_slice()
}
by_vec2 doesn't get past the compiler (I'm not sure I understand why), but by_iter does. I'm concerned about the performance of collect -- will it need to resize the vector it is collecting into as it iterates, or can it allocate the correct sized vector to begin with?
Maybe I'm going about this all wrong -- I'm very new to Rust, so any tips would be appreciated!
Let's start with by_vec2. You are taking a &mut reference to a Vec. You shouldn't do that, work directly with the Vec and make the v binding mutable.
Then you are iterating over the length of a Vec with a capacity of 5 and a length of 0. That means your loop never gets executed. What you wanted was to iterate over 0..v.cap().
Since your v is still of length 0, accessing v[i] in the loop will panic at runtime. What you actually want is v.push(None). This would normally cause reallocations, but in your case you already allocated with Vec::with_capacity, so pushing 5 times will not allocate.
This time around we did not take a reference to the Vec so into_boxed_slice will actually work.
fn by_vec2<T>() -> Box<[Option<T>]> {
let mut v = Vec::with_capacity(5);
for _ in 0..v.capacity() {
v.push(None);
}
v.into_boxed_slice()
}
Your by_iter function actually only allocates once. The Range iterator created by 0..5 knows that is exactly 5 elements long. So collect will in fact check that length and allocate only once.