Using crate dxgcap how can i save response of capture_frame fn to file using image crate.
I would also like to crop the images so it would be interesting to convert to a Vec
Note: capture_frame returns Vec<dxgcap::BGRA8> BGRA8 is struct with r,g,b,a all are u8.
Converting of types inside Vec is tricky, because Vec interacts with memory allocator and promises to report accurate allocation size and exact same alignment, which could differ between types. It's cumbersome and potentially unsafe.
If you only need to read the elements as a contiguous u8 slice, then:
let bytes = std::slice::from_raw_parts(vec.as_ptr() as *const u8, vec.len() * 4);
If you control the types used, then check out the bytemuck crate which has traits for casting to and from bytes.
Related
I've read the term "fat pointer" in several contexts already, but I'm not sure what exactly it means and when it is used in Rust. The pointer seems to be twice as large as a normal pointer, but I don't understand why. It also seems to have something to do with trait objects.
The term "fat pointer" is used to refer to references and raw pointers to dynamically sized types (DSTs) – slices or trait objects. A fat pointer contains a pointer plus some information that makes the DST "complete" (e.g. the length).
Most commonly used types in Rust are not DSTs but have a fixed size known at compile time. These types implement the Sized trait. Even types that manage a heap buffer of dynamic size (like Vec<T>) are Sized, as the compiler knows the exact number of bytes a Vec<T> instance will take up on the stack. There are currently four different kinds of DSTs in Rust.
Slices ([T] and str)
The type [T] (for any T) is dynamically sized (so is the special "string slice" type str). That's why you usually only see it as &[T] or &mut [T], i.e. behind a reference. This reference is a so-called "fat pointer". Let's check:
dbg!(size_of::<&u32>());
dbg!(size_of::<&[u32; 2]>());
dbg!(size_of::<&[u32]>());
This prints (with some cleanup):
size_of::<&u32>() = 8
size_of::<&[u32; 2]>() = 8
size_of::<&[u32]>() = 16
So we see that a reference to a normal type like u32 is 8 bytes large, as is a reference to an array [u32; 2]. Those two types are not DSTs. But as [u32] is a DST, the reference to it is twice as large. In the case of slices, the additional data that "completes" the DST is simply the length. So one could say the representation of &[u32] is something like this:
struct SliceRef {
ptr: *const u32,
len: usize,
}
Trait objects (dyn Trait)
When using traits as trait objects (i.e. type erased, dynamically dispatched), these trait objects are DSTs. Example:
trait Animal {
fn speak(&self);
}
struct Cat;
impl Animal for Cat {
fn speak(&self) {
println!("meow");
}
}
dbg!(size_of::<&Cat>());
dbg!(size_of::<&dyn Animal>());
This prints (with some cleanup):
size_of::<&Cat>() = 8
size_of::<&dyn Animal>() = 16
Again, &Cat is only 8 bytes large because Cat is a normal type. But dyn Animal is a trait object and therefore dynamically sized. As such, &dyn Animal is 16 bytes large.
In the case of trait objects, the additional data that completes the DST is a pointer to the vtable (the vptr). I cannot fully explain the concept of vtables and vptrs here, but they are used to call the correct method implementation in this virtual dispatch context. The vtable is a static piece of data that basically only contains a function pointer for each method. With that, a reference to a trait object is basically represented as:
struct TraitObjectRef {
data_ptr: *const (),
vptr: *const (),
}
(This is different from C++, where the vptr for abstract classes is stored within the object. Both approaches have advantages and disadvantages.)
Custom DSTs
It's actually possible to create your own DSTs by having a struct where the last field is a DST. This is rather rare, though. One prominent example is std::path::Path.
A reference or pointer to the custom DST is also a fat pointer. The additional data depends on the kind of DST inside the struct.
Exception: Extern types
In RFC 1861, the extern type feature was introduced. Extern types are also DSTs, but pointers to them are not fat pointers. Or more exactly, as the RFC puts it:
In Rust, pointers to DSTs carry metadata about the object being pointed to. For strings and slices this is the length of the buffer, for trait objects this is the object's vtable. For extern types the metadata is simply (). This means that a pointer to an extern type has the same size as a usize (ie. it is not a "fat pointer").
But if you are not interacting with a C interface, you probably won't ever have to deal with these extern types.
Above, we've seen the sizes for immutable references. Fat pointers work the same for mutable references, immutable raw pointers and mutable raw pointers:
size_of::<&[u32]>() = 16
size_of::<&mut [u32]>() = 16
size_of::<*const [u32]>() = 16
size_of::<*mut [u32]>() = 16
I've read the term "fat pointer" in several contexts already, but I'm not sure what exactly it means and when it is used in Rust. The pointer seems to be twice as large as a normal pointer, but I don't understand why. It also seems to have something to do with trait objects.
The term "fat pointer" is used to refer to references and raw pointers to dynamically sized types (DSTs) – slices or trait objects. A fat pointer contains a pointer plus some information that makes the DST "complete" (e.g. the length).
Most commonly used types in Rust are not DSTs but have a fixed size known at compile time. These types implement the Sized trait. Even types that manage a heap buffer of dynamic size (like Vec<T>) are Sized, as the compiler knows the exact number of bytes a Vec<T> instance will take up on the stack. There are currently four different kinds of DSTs in Rust.
Slices ([T] and str)
The type [T] (for any T) is dynamically sized (so is the special "string slice" type str). That's why you usually only see it as &[T] or &mut [T], i.e. behind a reference. This reference is a so-called "fat pointer". Let's check:
dbg!(size_of::<&u32>());
dbg!(size_of::<&[u32; 2]>());
dbg!(size_of::<&[u32]>());
This prints (with some cleanup):
size_of::<&u32>() = 8
size_of::<&[u32; 2]>() = 8
size_of::<&[u32]>() = 16
So we see that a reference to a normal type like u32 is 8 bytes large, as is a reference to an array [u32; 2]. Those two types are not DSTs. But as [u32] is a DST, the reference to it is twice as large. In the case of slices, the additional data that "completes" the DST is simply the length. So one could say the representation of &[u32] is something like this:
struct SliceRef {
ptr: *const u32,
len: usize,
}
Trait objects (dyn Trait)
When using traits as trait objects (i.e. type erased, dynamically dispatched), these trait objects are DSTs. Example:
trait Animal {
fn speak(&self);
}
struct Cat;
impl Animal for Cat {
fn speak(&self) {
println!("meow");
}
}
dbg!(size_of::<&Cat>());
dbg!(size_of::<&dyn Animal>());
This prints (with some cleanup):
size_of::<&Cat>() = 8
size_of::<&dyn Animal>() = 16
Again, &Cat is only 8 bytes large because Cat is a normal type. But dyn Animal is a trait object and therefore dynamically sized. As such, &dyn Animal is 16 bytes large.
In the case of trait objects, the additional data that completes the DST is a pointer to the vtable (the vptr). I cannot fully explain the concept of vtables and vptrs here, but they are used to call the correct method implementation in this virtual dispatch context. The vtable is a static piece of data that basically only contains a function pointer for each method. With that, a reference to a trait object is basically represented as:
struct TraitObjectRef {
data_ptr: *const (),
vptr: *const (),
}
(This is different from C++, where the vptr for abstract classes is stored within the object. Both approaches have advantages and disadvantages.)
Custom DSTs
It's actually possible to create your own DSTs by having a struct where the last field is a DST. This is rather rare, though. One prominent example is std::path::Path.
A reference or pointer to the custom DST is also a fat pointer. The additional data depends on the kind of DST inside the struct.
Exception: Extern types
In RFC 1861, the extern type feature was introduced. Extern types are also DSTs, but pointers to them are not fat pointers. Or more exactly, as the RFC puts it:
In Rust, pointers to DSTs carry metadata about the object being pointed to. For strings and slices this is the length of the buffer, for trait objects this is the object's vtable. For extern types the metadata is simply (). This means that a pointer to an extern type has the same size as a usize (ie. it is not a "fat pointer").
But if you are not interacting with a C interface, you probably won't ever have to deal with these extern types.
Above, we've seen the sizes for immutable references. Fat pointers work the same for mutable references, immutable raw pointers and mutable raw pointers:
size_of::<&[u32]>() = 16
size_of::<&mut [u32]>() = 16
size_of::<*const [u32]>() = 16
size_of::<*mut [u32]>() = 16
I am trying to implement std::io::Read trait for a struct.
My objective is to convert obj to byte array and read it through the implementation of Read trait.
Following is the code I have written so far.
use chrono::{DateTime, Utc};
use std::io::Error;
use std::io::Read;
use std::vec::Vec;
use std::str;
use super::{Chain, Transaction};
// The struct I need to convert to byte array and add the Read impl.
#[derive(Debug)]
pub struct Block {
index: u64,
timestamp: DateTime<Utc>,
transactions: Vec<Transaction>,
proof: i64,
previous_hash: String,
}
// The Read trait implementation for Block
impl Read for Block {
fn read(&mut self, buf: &mut [u8]) -> std::result::Result<usize, Error> {
let bytes: &[u8] = unsafe { any_as_u8_slice(&self) };
buf.clone_from_slice(bytes);
Ok(bytes.len())
}
}
// Function that converts to byte array. (found on stackoverflow)
unsafe fn any_as_u8_slice<T: Sized>(p: &T) -> &[u8] {
::std::slice::from_raw_parts((p as *const T) as *const u8, ::std::mem::size_of::<T>())
}
I get an error when I execute the code this way.
let mut buffer: Vec<u8> = Vec::new();
let result = block.read(buffer.as_mut());
ERROR
thread 'main' panicked at 'destination and source slices have
different lengths',
/Users/harsh/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/slice/mod.rs:2554:9
I am new to Rust, trying to learn by porting another program in Rust.
How do I copy &[u8] to another &mut [u8] which is a vec. (Fix the Read impl for Block)?
And is there a better way to do this?
Convert object to byte array and return it from the Read implementation.
There's a few different problems here:
What you're trying to do here won't be sound in general. Rust structs might include padding bytes or otherwise initialized bytes, which means that reading them from a [u8] is undefined behavior. The name for what you're trying to do here is a Transmute and they are famously very difficult to do correctly.
It's not clear to me why specifically you're doing this in terms of the Read trait. Read is generally for i/o devices, like files or stdin, or in-memory buffers that behave like i/o devices. Even if we assume that a direct, inplace transmute to a byte slice is appropriate here, it would make more sense to just have a method on Block resembling fn as_byte_slice(&self) -> &[u8] { ... }.
Even if we set aside the above issues, it's still not clear to me that you're going to get the outcome you expect. Transmuting a struct to a byte array will convert only the raw bytes in the struct, which will work fine for primitive types like u64, but for types like Vec<T> and String will only return the underlying pointer to the allocated storage.
I'm guessing that what you actually want to have happen here is that all of the contents of the Block– including the list of transactions and the previous_hash– be converted into the byte slice. This is called serialization, and the de-facto way to do it in Rust is with serde. Serde is an abstract library that connects types (like Vec and your own Block) to data formats like json and bincode.
In your question, you've said you want to "convert the [object] to a byte array". This is a bit nonspecific; it's likely that there is actually a specific data format into which you're suppose to convert this Block. Your specific application will likely describe which specific data format is in use, and you can then look into whether there already exists a serde Serializer for that data format, or whether you'll need to write your own.
Using the Piston image crate, I can write an image by feeding it a Vec<u8>, but my actual data is Vec<Rgb<u8>> (because that is a lot easier to deal with, and I want to grow it dynamically).
How can I convert Vec<Rgb<u8>> to Vec<u8>? Rgb<u8> is really [u8; 3]. Does this have to be an unsafe conversion?
The answer depends on whether you are fine with copying the data. If copying is not an issue for you, you can do something like this:
let img: Vec<Rgb<u8>> = ...;
let buf: Vec<u8> = img.iter().flat_map(|rgb| rgb.data.iter()).cloned().collect();
If you want to perform the conversion without copying, though, we first need to make sure that your source and destination types actually have the same memory layout. Rust makes very few guarantees about the memory layout of structs. It currently does not even guarantee that a struct with a single member has the same memory layout as the member itself.
In this particular case, the Rust memory layout is not relevant though, since Rgb is defined as
#[repr(C)]
pub struct Rgb<T: Primitive> {
pub data: [T; 3],
}
The #[repr(C)] attribute specifies that the memory layout of the struct should be the same as an equivalent C struct. The C memory layout is not fully specified in the C standard, but according to the unsafe code guidelines, there are some rules that hold for "most" platforms:
Field order is preserved.
The first field begins at offset 0.
Assuming the struct is not packed, each field's offset is aligned to the ABI-mandated alignment for that field's type, possibly creating unused padding bits.
The total size of the struct is rounded up to its overall alignment.
As pointed out in the comments, the C standard theoretically allows additional padding at the end of the struct. However, the Piston image library itself makes the assumption that a slice of channel data has the same memory layout as the Rgb struct, so if you are on a platform where this assumption does not hold, all bets are off anyway (and I couldnt' find any evidence that such a platform exists).
Rust does guarantee that arrays, slices and vectors are densely packed, and that structs and arrays have an alignment equal to the maximum alignment of their elements. Together with the assumption that the layout of Rgb is as specified by the rules I quotes above, this guarantees that Rgb<u8> is indeed laid out as three consecutive bytes in memory, and that Vec<Rgb<u8>> is indeed a consecutive, densely packed buffer of RGB values, so our conversion is safe. We still need to use unsafe code to write it:
let p = img.as_mut_ptr();
let len = img.len() * mem::size_of::<Rgb<u8>>();
let cap = img.capacity() * mem::size_of::<Rgb<u8>>();
mem::forget(img);
let buf: Vec<u8> = unsafe { Vec::from_raw_parts(p as *mut u8, len, cap) };
If you want to protect against the case that there is additional padding at the end of Rgb, you can check whether size_of::<Rgb<u8>>() is indeed 3. If it is, you can use the unsafe non-copying version, otherwise you have to use the first version above.
You choose the Vec<Rgb<u8>> storage format because it's easier to deal with and you want it to grow dynamically. But as you noticed, there's no guarantee of compatibility of its storage with a Vec<u8>, and no safe conversion.
Why not take the problem the other way and build a convenient facade for a Vec<u8> ?
type Rgb = [u8; 3];
#[derive(Debug)]
struct Img(Vec<u8>);
impl Img {
fn new() -> Img {
Img(Vec::new())
}
fn push(&mut self, rgb: &Rgb) {
self.0.push(rgb[0]);
self.0.push(rgb[1]);
self.0.push(rgb[2]);
}
// other convenient methods
}
fn main() {
let mut img = Img::new();
let rgb : Rgb = [1, 2, 3];
img.push(&rgb);
img.push(&rgb);
println!("{:?}", img);
}
I'm trying to write a little buffer-thing for parsing so I can pull records off the front of as I parse them out, ideally without making any copies and just transferring ownership of chunks of the front of the buffer off as I run. Here's my implementation:
struct BufferThing {
buf: Vec<u8>,
}
impl BufferThing {
fn extract(&mut self, size: usize) -> Vec<u8> {
assert!(size <= self.buf.len());
let remaining: usize = self.buf.len() - size;
let ptr: *mut u8 = self.buf.as_mut_ptr();
unsafe {
self.buf = Vec::from_raw_parts(ptr.offset(size as isize), remaining, remaining);
Vec::from_raw_parts(ptr, size, size)
}
}
}
This compiles, but panics with a signal: 11, SIGSEGV: invalid memory reference as it starts running. This is mostly the same code as the example in the Nomicon, but I'm trying to do it on Vec's and I'm trying to split a field instead of the object itself.
Is it possible to do this without copying out one of the Vecs? And is there some section of the Nomicon or other documentation that explains why I'm blowing everything up in the unsafe block?
Unfortunately, that's not how memory allocators work. It might have been possible in the past, when memory was at a premium, but today's allocators are geared for speed rather than memory preservation.
A common implementation of memory allocators is to use slabs. Basically, it's:
struct Allocator {
less_than_32_bytes: List<[u8; 32]>,
less_than_64_bytes: List<[u8; 64]>,
less_than_128_bytes: List<[u8; 128]>,
less_than_256_bytes: List<[u8; 256]>,
less_than_512_bytes: List<[u8; 512]>,
...
}
When you request 96 bytes, it takes an element from less_than_128_bytes.
When you free that element, it frees all of it, not just the first N bytes, and the whole block is now re-usable. Any pointer inside the block is now dangling and should NOT be dereferenced.
Furthermore, trying to free a pointer in the middle of a block will only confuse the allocator: it won't find it, because the contract is that you address blocks by their first byte.
You violated the contract using unsafe code, BOOM.
The solution I propose is simple:
use a single Vec<u8> containing the whole buffer to parse
use slices into this Vec for parsing
Rust will check the lifetimes, so your slices cannot outlive the buffer, and slicing a slice further (s[..offset], s[offset..]) does not allocate.
If you don't mind one allocation, there's Vec::split_off which allocates a new Vec big enough for the split part.