In GLSL, a vec3 is required to be aligned on a 16-byte boundary when used in a uniform block. In order to represent this requirement, I have a Rust type defined as follows:
#[repr(C, align(16))]
struct UniformVec3([f32; 3]);
This makes it easier to construct uniform data on the CPU side without having to manually specify padding. However, in order to upload the data to the GPU, the struct has to be transmuted to a &[u8], like this:
let v = UniformVec3([1.0, 0.0, 0.0]);
unsafe {
let as_bytes = std::slice::from_raw_parts(
(&v as *const UniformVec3) as *const u8,
std::size_of::<UniformVec3>(),
);
// do stuff with as_bytes...
}
As it stands, this is undefined behavior because the bit pattern of the four bytes of padding is undefined. My question is: is there some way to initialize this struct (with its current definition!) without invoking UB upon transmuting it to a &[u8]? Or do I have to insert explicit padding in the struct, which will be defined at initialization?
Related
Using crate dxgcap how can i save response of capture_frame fn to file using image crate.
I would also like to crop the images so it would be interesting to convert to a Vec
Note: capture_frame returns Vec<dxgcap::BGRA8> BGRA8 is struct with r,g,b,a all are u8.
Converting of types inside Vec is tricky, because Vec interacts with memory allocator and promises to report accurate allocation size and exact same alignment, which could differ between types. It's cumbersome and potentially unsafe.
If you only need to read the elements as a contiguous u8 slice, then:
let bytes = std::slice::from_raw_parts(vec.as_ptr() as *const u8, vec.len() * 4);
If you control the types used, then check out the bytemuck crate which has traits for casting to and from bytes.
I saw the following code for returning a byte array to C:
#[repr(C)]
struct Buffer {
data: *mut u8,
len: usize,
}
extern "C" fn generate_data() -> Buffer {
let mut buf = vec![0; 512].into_boxed_slice();
let data = buf.as_mut_ptr();
let len = buf.len();
std::mem::forget(buf);
Buffer { data, len }
}
extern "C" fn free_buf(buf: Buffer) {
let s = unsafe { std::slice::from_raw_parts_mut(buf.data, buf.len) };
let s = s.as_mut_ptr();
unsafe {
Box::from_raw(s);
}
}
I notice that the free_buf function takes a Buffer, instead of a *mut u8. Is this intentional?
Can the free_buf function be reduced to:
unsafe extern "C" fn free_buf(ptr: *mut u8) {
Box::from_raw(ptr);
}
You are correct to note that the C runtime free function takes only a pointer to the memory region to be freed as an argument.
However, you don't call this directly. In fact Rust has a layer that abstracts away the actual memory allocator being used: std::alloc::GlobalAlloc.
The reason for providing such an abstraction is to allow other allocators to be used, and in fact it is quite easy to swap out the default OS provided allocator.
It would be quite limiting to require that any allocator keeps track of the length of blocks to allow them to be freed without supplying the length to the deallocation function, so the general deallocation function requires the length as well.
You might be interested to know that C++ has a similar abstraction. This answer provides some more discussion about why it could be preferable to require the application to keep track of the lengths of allocated memory regions rather than the heap manager.
If we check the type of Box::from_raw, we see that it construct a Box<u8> from a raw *mut u8. One would need a *mut [u8] (fat pointer to slice) in order to construct a Box<[u8]> (which is what we have in the very beginning).
And dropping a Box<u8> will (at best) only release one byte of memory (if not causing a runtime error), while dropping a Box<[u8]> correctly releases all the memory.
No, what you do is undefined behavior, it's mandatory that the type between into_raw() and from_raw() match. Rust alloc API doesn't require the allocator to remember any information, and so the allocation implementation will expect correctness of all information pass to it.
In your example, *mut u8 and *mut [u8] are a totally different type and so have different layout.
Also, mismatch the type could prevent destructor to run properly.
You can't use from_raw() to destruct any pointer like C free() using void *.
I have complex number data filled into a Vec<f64> by an external C library (prefer not to change) in the form [i_0_real, i_0_imag, i_1_real, i_1_imag, ...] and it appears that this Vec<f64> has the same memory layout as a Vec<num_complex::Complex<f64>> of half the length would be, given that num_complex::Complex<f64>'s data structure is memory-layout compatible with [f64; 2] as documented here. I'd like to use it as such without needing a re-allocation of a potentially large buffer.
I'm assuming that it's valid to use from_raw_parts() in std::vec::Vec to fake a new Vec that takes ownership of the old Vec's memory (by forgetting the old Vec) and use size / 2 and capacity / 2, but that requires unsafe code. Is there a "safe" way to do this kind of data re-interpretation?
The Vec is allocated in Rust as a Vec<f64> and is populated by a C function using .as_mut_ptr() that fills in the Vec<f64>.
My current compiling unsafe implementation:
extern crate num_complex;
pub fn convert_to_complex_unsafe(mut buffer: Vec<f64>) -> Vec<num_complex::Complex<f64>> {
let new_vec = unsafe {
Vec::from_raw_parts(
buffer.as_mut_ptr() as *mut num_complex::Complex<f64>,
buffer.len() / 2,
buffer.capacity() / 2,
)
};
std::mem::forget(buffer);
return new_vec;
}
fn main() {
println!(
"Converted vector: {:?}",
convert_to_complex_unsafe(vec![3.0, 4.0, 5.0, 6.0])
);
}
Is there a "safe" way to do this kind of data re-interpretation?
No. At the very least, this is because the information you need to know is not expressed in the Rust type system but is expressed via prose (a.k.a. the docs):
Complex<T> is memory layout compatible with an array [T; 2].
— Complex docs
If a Vec has allocated memory, then [...] its pointer points to len initialized, contiguous elements in order (what you would see if you coerced it to a slice),
— Vec docs
Arrays coerce to slices ([T])
— Array docs
Since a Complex is memory-compatible with an array, an array's data is memory-compatible with a slice, and a Vec's data is memory-compatible with a slice, this transformation should be safe, even though the compiler cannot tell this.
This information should be attached (via a comment) to your unsafe block.
I would make some small tweaks to your function:
Having two Vecs at the same time pointing to the same data makes me very nervous. This can be trivially avoided by introducing some variables and forgetting one before creating the other.
Remove the return keyword to be more idiomatic
Add some asserts that the starting length of the data is a multiple of two.
As rodrigo points out, the capacity could easily be an odd number. To attempt to avoid this, we call shrink_to_fit. This has the downside that the Vec may need to reallocate and copy the memory, depending on the implementation.
Expand the unsafe block to cover all of the related code that is required to ensure that the safety invariants are upheld.
pub fn convert_to_complex(mut buffer: Vec<f64>) -> Vec<num_complex::Complex<f64>> {
// This is where I'd put the rationale for why this `unsafe` block
// upholds the guarantees that I must ensure. Too bad I
// copy-and-pasted from Stack Overflow without reading this comment!
unsafe {
buffer.shrink_to_fit();
let ptr = buffer.as_mut_ptr() as *mut num_complex::Complex<f64>;
let len = buffer.len();
let cap = buffer.capacity();
assert!(len % 2 == 0);
assert!(cap % 2 == 0);
std::mem::forget(buffer);
Vec::from_raw_parts(ptr, len / 2, cap / 2)
}
}
To avoid all the worrying about the capacity, you could just convert a slice into the Vec. This also doesn't have any extra memory allocation. It's simpler because we can "lose" any odd trailing values because the Vec still maintains them.
pub fn convert_to_complex(buffer: &[f64]) -> &[num_complex::Complex<f64>] {
// This is where I'd put the rationale for why this `unsafe` block
// upholds the guarantees that I must ensure. Too bad I
// copy-and-pasted from Stack Overflow without reading this comment!
unsafe {
let ptr = buffer.as_ptr() as *mut num_complex::Complex<f64>;
let len = buffer.len();
assert!(len % 2 == 0);
std::slice::from_raw_parts(ptr, len / 2)
}
}
Using the Piston image crate, I can write an image by feeding it a Vec<u8>, but my actual data is Vec<Rgb<u8>> (because that is a lot easier to deal with, and I want to grow it dynamically).
How can I convert Vec<Rgb<u8>> to Vec<u8>? Rgb<u8> is really [u8; 3]. Does this have to be an unsafe conversion?
The answer depends on whether you are fine with copying the data. If copying is not an issue for you, you can do something like this:
let img: Vec<Rgb<u8>> = ...;
let buf: Vec<u8> = img.iter().flat_map(|rgb| rgb.data.iter()).cloned().collect();
If you want to perform the conversion without copying, though, we first need to make sure that your source and destination types actually have the same memory layout. Rust makes very few guarantees about the memory layout of structs. It currently does not even guarantee that a struct with a single member has the same memory layout as the member itself.
In this particular case, the Rust memory layout is not relevant though, since Rgb is defined as
#[repr(C)]
pub struct Rgb<T: Primitive> {
pub data: [T; 3],
}
The #[repr(C)] attribute specifies that the memory layout of the struct should be the same as an equivalent C struct. The C memory layout is not fully specified in the C standard, but according to the unsafe code guidelines, there are some rules that hold for "most" platforms:
Field order is preserved.
The first field begins at offset 0.
Assuming the struct is not packed, each field's offset is aligned to the ABI-mandated alignment for that field's type, possibly creating unused padding bits.
The total size of the struct is rounded up to its overall alignment.
As pointed out in the comments, the C standard theoretically allows additional padding at the end of the struct. However, the Piston image library itself makes the assumption that a slice of channel data has the same memory layout as the Rgb struct, so if you are on a platform where this assumption does not hold, all bets are off anyway (and I couldnt' find any evidence that such a platform exists).
Rust does guarantee that arrays, slices and vectors are densely packed, and that structs and arrays have an alignment equal to the maximum alignment of their elements. Together with the assumption that the layout of Rgb is as specified by the rules I quotes above, this guarantees that Rgb<u8> is indeed laid out as three consecutive bytes in memory, and that Vec<Rgb<u8>> is indeed a consecutive, densely packed buffer of RGB values, so our conversion is safe. We still need to use unsafe code to write it:
let p = img.as_mut_ptr();
let len = img.len() * mem::size_of::<Rgb<u8>>();
let cap = img.capacity() * mem::size_of::<Rgb<u8>>();
mem::forget(img);
let buf: Vec<u8> = unsafe { Vec::from_raw_parts(p as *mut u8, len, cap) };
If you want to protect against the case that there is additional padding at the end of Rgb, you can check whether size_of::<Rgb<u8>>() is indeed 3. If it is, you can use the unsafe non-copying version, otherwise you have to use the first version above.
You choose the Vec<Rgb<u8>> storage format because it's easier to deal with and you want it to grow dynamically. But as you noticed, there's no guarantee of compatibility of its storage with a Vec<u8>, and no safe conversion.
Why not take the problem the other way and build a convenient facade for a Vec<u8> ?
type Rgb = [u8; 3];
#[derive(Debug)]
struct Img(Vec<u8>);
impl Img {
fn new() -> Img {
Img(Vec::new())
}
fn push(&mut self, rgb: &Rgb) {
self.0.push(rgb[0]);
self.0.push(rgb[1]);
self.0.push(rgb[2]);
}
// other convenient methods
}
fn main() {
let mut img = Img::new();
let rgb : Rgb = [1, 2, 3];
img.push(&rgb);
img.push(&rgb);
println!("{:?}", img);
}
I want to return a vector in a pub extern "C" fn. Since a vector has an arbitrary length, I guess I need to return a struct with
the pointer to the vector, and
the number of elements in the vector
My current code is:
extern crate libc;
use self::libc::{size_t, int32_t, int64_t};
// struct to represent an array and its size
#[repr(C)]
pub struct array_and_size {
values: int64_t, // this is probably not how you denote a pointer, right?
size: int32_t,
}
// The vector I want to return the address of is already in a Boxed struct,
// which I have a pointer to, so I guess the vector is on the heap already.
// Dunno if this changes/simplifies anything?
#[no_mangle]
pub extern "C" fn rle_show_values(ptr: *mut Rle) -> array_and_size {
let rle = unsafe {
assert!(!ptr.is_null());
&mut *ptr
};
// this is the Vec<i32> I want to return
// the address and length of
let values = rle.values;
let length = values.len();
array_and_size {
values: Box::into_raw(Box::new(values)),
size: length as i32,
}
}
#[derive(Debug, PartialEq)]
pub struct Rle {
pub values: Vec<i32>,
}
The error I get is
$ cargo test
Compiling ranges v0.1.0 (file:///Users/users/havpryd/code/rust-ranges)
error[E0308]: mismatched types
--> src/rle.rs:52:17
|
52 | values: Box::into_raw(Box::new(values)),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected i64, found *-ptr
|
= note: expected type `i64`
= note: found type `*mut std::vec::Vec<i32>`
error: aborting due to previous error
error: Could not compile `ranges`.
To learn more, run the command again with --verbose.
-> exit code: 101
I posted the whole thing because I could not find an example of returning arrays/vectors in the eminently useful Rust FFI Omnibus.
Is this the best way to return a vector of unknown size from Rust? How do I fix my remaining compile error? Thanks!
Bonus q: if the fact that my vector is in a struct changes the answer, perhaps you could also show how to do this if the vector was not in a Boxed struct already (which I think means the vector it owns is on the heap too)? I guess many people looking up this q will not have their vectors boxed already.
Bonus q2: I only return the vector to view its values (in Python), but I do not want to let the calling code change the vector. But I guess there is no way to make the memory read-only and ensure the calling code does not fudge with the vector? const is just for showing intent, right?
Ps: I do not know C or Rust well, so my attempt might be completely WTF.
pub struct array_and_size {
values: int64_t, // this is probably not how you denote a pointer, right?
size: int32_t,
}
First of all, you're correct. The type you want for values is *mut int32_t.
In general, and note that there are a variety of C coding styles, C often doesn't "like" returning ad-hoc sized array structs like this. The more common C API would be
int32_t rle_values_size(RLE *rle);
int32_t *rle_values(RLE *rle);
(Note: many internal programs do in fact use sized array structs, but this is by far the most common for user-facing libraries because it's automatically compatible with the most basic way of representing arrays in C).
In Rust, this would translate to:
extern "C" fn rle_values_size(rle: *mut RLE) -> int32_t
extern "C" fn rle_values(rle: *mut RLE) -> *mut int32_t
The size function is straightforward, to return the array, simply do
extern "C" fn rle_values(rle: *mut RLE) -> *mut int32_t {
unsafe { &mut (*rle).values[0] }
}
This gives a raw pointer to the first element of the Vec's underlying buffer, which is all C-style arrays really are.
If, instead of giving C a reference to your data you want to give C the data, the most common option would be to allow the user to pass in a buffer that you clone the data into:
extern "C" fn rle_values_buf(rle: *mut RLE, buf: *mut int32_t, len: int32_t) {
use std::{slice,ptr}
unsafe {
// Make sure we don't overrun our buffer's length
if len > (*rle).values.len() {
len = (*rle).values.len()
}
ptr::copy_nonoverlapping(&(*rle).values[0], buf, len as usize);
}
}
Which, from C, looks like
void rle_values_buf(RLE *rle, int32_t *buf, int32_t len);
This (shallowly) copies your data into the presumably C-allocated buffer, which the C user is then responsible for destroying. It also prevents multiple mutable copies of your array from floating around at the same time (assuming you don't implement the version that returns a pointer).
Note that you could sort of "move" the array into C as well, but it's not particularly recommended and involves the use mem::forget and expecting the C user to explicitly call a destruction function, as well as requiring both you and the user to obey some discipline that may be difficult to structure the program around.
If you want to receive an array from C, you essentially just ask for both a *mut i32 and i32 corresponding to the buffer start and length. You can assemble this into a slice using the from_raw_parts function, and then use the to_vec function to create an owned Vector containing the values allocated from the Rust side. If you don't plan on needing to own the values, you can simply pass around the slice you produced via from_raw_parts.
However, it is imperative that all values be initialized from either side, typically to zero. Otherwise you invoke legitimately undefined behavior which often results in segmentation faults (which tend to frustratingly disappear when inspected with GDB).
There are multiple ways to pass an array to C.
First of all, while C has the concept of fixed-size arrays (int a[5] has type int[5] and sizeof(a) will return 5 * sizeof(int)), it is not possible to directly pass an array to a function or return an array from it.
On the other hand, it is possible to wrap a fixed size array in a struct and return that struct.
Furthermore, when using an array, all elements must be initialized, otherwise a memcpy technically has undefined behavior (as it is reading from undefined values) and valgrind will definitely report the issue.
Using a dynamic array
A dynamic array is an array whose length is unknown at compile-time.
One may chose to return a dynamic array if no reasonable upper-bound is known, or this bound is deemed too large for passing by value.
There are two ways to handle this situation:
ask C to pass a suitably sized buffer
allocate a buffer and return it to C
They differ in who allocates the memory: the former is simpler, but may require to either have a way to hint at a suitable size or to be able to "rewind" if the size proves unsuitable.
Ask C to pass a suitable sized buffer
// file.h
int rust_func(int32_t* buffer, size_t buffer_length);
// file.rs
#[no_mangle]
pub extern fn rust_func(buffer: *mut libc::int32_t, buffer_length: libc::size_t) -> libc::c_int {
// your code here
}
Note the existence of std::slice::from_raw_parts_mut to transform this pointer + length into a mutable slice (do initialize it with 0s before making it a slice or ask the client to).
Allocate a buffer and return it to C
// file.h
struct DynArray {
int32_t* array;
size_t length;
}
DynArray rust_alloc();
void rust_free(DynArray);
// file.rs
#[repr(C)]
struct DynArray {
array: *mut libc::int32_t,
length: libc::size_t,
}
#[no_mangle]
pub extern fn rust_alloc() -> DynArray {
let mut v: Vec<i32> = vec!(...);
let result = DynArray {
array: v.as_mut_ptr(),
length: v.len() as _,
};
std::mem::forget(v);
result
}
#[no_mangle]
pub extern fn rust_free(array: DynArray) {
if !array.array.is_null() {
unsafe { Box::from_raw(array.array); }
}
}
Using a fixed-size array
Similarly, a struct containing a fixed size array can be used. Note that both in Rust and C all elements should be initialized, even if unused; zeroing them works well.
Similarly to the dynamic case, it can be either passed by mutable pointer or returned by value.
// file.h
struct FixedArray {
int32_t array[32];
};
// file.rs
#[repr(C)]
struct FixedArray {
array: [libc::int32_t; 32],
}