I've been writing a hobbyist AES library in Rust and I've run into the problem of eccessive boilerplate code. Basically I represent an AES key with the following struct:
pub struct AesKey<const N: usize> {
data: [u32; N],
}
N can be any of the following values:
4 for 128 bits
6 for 192 bits
8 for 256 bits
This struct then needs to implement the following trait
pub trait AesKeyOps<const N: usize, const N_ROUND_KEYS: usize> {
// ...
}
for each of the possible values of N.
And for each of those implementations it needs to implement even more traits for padding, block modes and other implementation logic.
I originally used const generics in my code to make components more reusable, but at a certain point the impl blocks just started to grow exponentially, even though they all contained basically the same code with just some minor tweaks (9 impl blocks for the different padding schemes [3 schemes * 3 key sizes] and the only difference between them are the values for the generics). I tried using an enum as a generic, as that would be able to contain all of the necessary data, but they aren't allowed as any type of generic.
What can I do to avoid repeating code this much while using generics in rust?
Related
I'm developing a chess engine in Rust. I have a Move struct with from and to fields, which are Square. Square is a struct containing a usize, so that I can use it directly when accessing board elements of the position. Since in Rust indexing must be done with usize, I'm wondering what's the fastest way to handle this situation (note that move generation should be as fast as possible). I understand it's more memory friendly to store u8 and cast them every time I need to use them as an index, but is it faster? What would be the idiomatic way to approach this?
I have:
struct Square {index: usize}
fn position.at(square: Square) -> Option<Piece> {
position.board[square.index]
}
I've tried migrating to u8 and casting every time with mixed results:
struct Square(u8)
fn position.at(square: Square) -> Option<Piece> {
position.board[square.0 as usize]
}
Pro u8 casting:
better cache utilization (objects are smaller); but might be only interesting when there are a lot of objects
Con u8:
casting might require additional instructions on some platforms; but these are usually only register operations which are optimized by the cpu
Idiomatic way to avoid the as usize: implement a wrapper
impl Square {
#[inline]
pub fn index(&self) -> usize {
self.0 as usize
}
}
Or, when you want to make it really typesafe, implement std::ops::Index:
struct Piece;
struct Board([Piece; 64]);
struct Square(u8);
impl std::ops::Index<Square> for Board {
type Output = Piece;
fn index(&self, index: Square) -> &Self::Output {
&self.0[index.0 as usize]
}
}
When you only cast for indexing rusts compiler is smart enough to notice that, so they produce the exact same assembly.
see playground
even if that wouldn't be the case, casting from u8 to usize wouldnt be much more than a single instruction (which is pretty much no overhead)
on the other hand usize takes 8 times as much space than u8 (on a 64bit machine)
So if you plan on having A LOT of square instances that might be a factor to consider and go with the casting option, if not it pretty much doesn't matter at all
I am currently learning Rust for fun. I have some experience in C / C++ and other experience in other programming languages that use more complex paradigms like generics.
Background
For my first project (after the tutorial), I wanted to create a N-Dimensional array (or Matrix) data structure to practice development in Rust.
Here is what I have so far for my Matrix struct and a basic fill and new initializations.
Forgive the absent bound checking and parameter testing
pub struct Matrix<'a, T> {
data: Vec<Option<T>>,
dimensions: &'a [usize],
}
impl<'a, T: Clone> Matrix<'a, T> {
pub fn fill(dimensions: &'a [usize], fill: T) -> Matrix<'a, T> {
let mut total = if dimensions.len() > 0 { 1 } else { 0 };
for dim in dimensions.iter() {
total *= dim;
}
Matrix {
data: vec![Some(fill); total],
dimensions: dimensions,
}
}
pub fn new(dimensions: &'a [usize]) -> Matrix<'a, T> {
...
Matrix {
data: vec![None; total],
dimensions: dimensions,
}
}
}
I wanted the ability to create an "empty" N-Dimensional array using the New fn. I thought using the Option enum would be the best way to accomplish this, as I can fill the N-Dimensional with None and it would allocate space for this T generic automatically.
So then it comes down to being able to set the entries for this. I found the IndexMut and Index traits that looked like I could do something like m[&[2, 3]] = 23. Since the logic is similar to each other here is the IndexMut impl for Matrix.
impl<'a, T> ops::IndexMut<&[usize]> for Matrix<'a, T> {
fn index_mut(&mut self, indices: &[usize]) -> &mut Self::Output {
match self.data[get_matrix_index(self.dimensions, indices)].as_mut() {
Some(x) => x,
None => {
NOT SURE WHAT TO DO HERE.
}
}
}
}
Ideally what would happen is that the value (if there) would be changed i.e.
let mut mat = Matrix::fill(&[4, 4], 0)
mat[&[2, 3]] = 23
This would set the value from 0 to 23 (which the above fn does via returning &mut x from Some(x)). But I also want None to set the value i.e.
let mut mat = Matrix::new(&[4, 4])
mat[&[2, 3]] = 23
Question
Finally, is there a way to make m[&[2,3]] = 23 possible with what the Vec struct requires to allocate the memory? If not what should I change and how can I still have an array with "empty" spots. Open to any suggestions as I am trying to learn. :)
Final Thoughts
Through my research, the Vec struct impls I see that the type T is typed and has to be Sized. This could be useful as to allocate the Vec with the appropriate size via vec![pointer of T that is null but of size of T; total]. But I am unsure of how to do this.
So there are a few ways to make this more similar to idiomatic rust, but first, let's look at why the none branch doesn't make sense.
So the Output type for IndexMut I'm going to assume is &mut T as you don't show the index definition but I feel safe in that assumption. The type &mut T means a mutable reference to an initialized T, unlike pointers in C/C++ where they can point to initialized or uninitialized memory. What this means is that you have to return an initialized T which the none branch cannot because there is no initialized value. This leads to the first of the more idiomatic ways.
Return an Option<T>
The easiest way would be to change Index::Output to be an Option<T>. This is better because the user can decide what to do if there was no value there before and is close to what you are actually storing. Then you can also remove the panic in your index method and allow the caller to choose what to do if there is no value. At this point, I think you can go a little further with gentrifying the structure in the next option.
Store a T directly
This method allows the caller to directly change what the type is that's stored rather than wrapping it in an option. This cleans up most of your index code nicely as you just have to access what's already stored. The main problem is now initialization, how do you represent uninitialized values? You were correct that option is the best way to do this1, but now the caller can decide to have this optional initialization capability by storing an Option themselves. So that means we can always store initialized Ts without losing functionality. This only really changes your new function to instead not fill with None values. My suggestion here is to make a bound T: Default for the new function2:
impl<'a, T: Default> Matrix<'a, T> {
pub fn new(dimensions: &'a [usize]) -> Matrix<'a, T> {
Matrix {
data: (0..total).into_iter().map(|_|Default::default()).collect(),
dimensions: dimensions,
}
}
}
This method is much more common in the rust world and allows the caller to choose whether to allow for uninitialized values. Option<T> also implements default for all T and returns None So the functionality is very similar to what you have currently.
Aditional Info
As you're new to rust there are a few comments that I can make about traps that I've fallen into before. To start your struct contains a reference to the dimensions with a lifetime. What this means is that your structs cannot exist longer than the dimension object that created them. This hasn't caused you a problem so far as all you've been passing is statically created dimensions, dimensions that are typed into the code and stored in static memory. This gives your object a lifetime of 'static, but this won't occur if you use dynamic dimensions.
How else can you store these dimensions so that your object always has a 'static lifetime (same as no lifetime)? Since you want an N-dimensional array stack allocation is out of the question since stack arrays must be deterministic at compile time (otherwise known as const in rust). This means you have to use the heap. This leaves two real options Box<[usize]> or Vec<usize>. Box is just another way of saying this is on the heap and adds Sized to values that are ?Sized. Vec is a little more self-explanatory and adds the ability to be resized at the cost of a little overhead. Either would allow your matrix object to always have a 'static lifetime.
1. The other way to represent this without Option<T>'s discriminate is MaybeUninit<T> which is unsafe territory. This allows you to have a chunk of initialized memory big enough to hold a T and then assume it's initialized unsafely. This can cause a lot of problems and is usually not worth it as Option is already heavily optimized in that if it stores a type with a pointer it uses compiler magic to store the discriminate in whether or not that value is a null pointer.
2. The reason this section doesn't just use vec![Default::default(); total] is that this requires T: Clone as the way this macro works the first part is called once and cloned until there are enough values. This is an extra requirement that we don't need to have so the interface is smoother without it.
The main goal is to implement a computation graph, that handles nodes with values and nodes with operators (think of simple arithmetic operators like add, subtract, multiply etc..). An operator node can take up to two value nodes, and "produces" a resulting value node.
Up to now, I'm using an enum to differentiate between a value and operator node:
pub enum Node<'a, T> where T : Copy + Clone {
Value(ValueNode<'a, T>),
Operator(OperatorNode)
}
pub struct ValueNode<'a, T> {
id: usize,
value_object : &'a dyn ValueType<T>
}
Update: Node::Value contains a struct, which itself contains a reference to a trait object ValueType, which is being implemented by a variety of concrete types.
But here comes the problem. During compililation, the generic types will be elided, and replaced by the actual types. The generic type T is also being propagated throughout the computation graph (obviously):
pub struct ComputationGraph<T> where T : Copy + Clone {
nodes: Vec<Node<T>>
}
This actually restricts the usage of ComputeGraph to one specific ValueType.
Furthermore the generic type T cannot be Sized, since a value node can be an opqaue type handled by a different backend not available through rust (think of C opqaue types made available through FFI).
One possible solution to this problem would be to introduce an additional enum type, that "mirrors" the concrete implementation of the valuetype trait mentioned above. this approach would be similiar, that enum dispatch does.
Is there anything I haven't thought of to use multiple implementations of ValueType?
update:
What i want to achive is following code:
pub struct Scalar<T> where T : Copy + Clone{
data : T
}
fn main() {
let cg = ComputeGraph::new();
// a new scalar type. doesn't have to be a tuple struct
let a = Scalar::new::<f32>(1.0);
let b_size = 32;
let b = Container::new::<opaque_type>(32);
let op = OperatorAdd::new();
// cg.insert_operator_node constructs four nodes: 3 value nodes
// and one operator nodes internally.
let result = cg.insert_operator_node::<Container>(&op, &a, &b);
}
update
ValueType<T> looks like this
pub trait ValueType<T> {
fn get_size(&self) -> usize;
fn get_value(&self) -> T;
}
update
To further increase the clarity of my question think of a small BLAS library backed by OpenCL. The memory management and device interaction shall be transparent to the user. A Matrix type allocates space on an OpenCL device with types as a primitive type buffer, and the appropriate call will return a pointer to that specific region of memory. Think of an operation that will scale the matrix by a scalar type, that is being represented by a primitive value. Both the (pointer to the) buffer and the scalar can be passed to a kernel function. Going back to the ComputeGraph, it seems obvious, that all BLAS operations form some type of computational graph, which can be reduced to a linear list of instructions ( think here of setting kernel arguments, allocating buffers, enqueue the kernel, storing the result, etc... ). Having said all that, a computation graph needs to be able to store value nodes with a variety of types.
As always the answer to the problem posed in the question is obvious. The graph expects one generic type (with trait bounds). Using an enum to "cluster" various subtypes was the solution, as already sketched out in the question.
An example to illustrate the solution. Consider following "subtypes":
struct Buffer<T> {
// fields
}
struct Scalar<T> {
// fields
}
struct Kernel {
// fields
}
The value containing types can be packed into an enum:
enum MemType {
Buffer(Buffer<f32>);
Scalar(Scalar<f32>);
// more enum variants ..
}
Now MemType and Kernel can now be packed in an enum as well
enum Node {
Value(MemType);
Operator(Kernel);
}
Node can now be used as the main type for nodes/vertices inside the graph. The solution might not be very elegant, but it does the trick for now. Maybe some code restructuring might be done in the future.
I'm trying to create a generic struct which uses an "integer type" for references into an array. For performance reasons I'd like to be able to specify easily whether to use u16, u32 or u64. Something like this (which obviously isn't valid Rust code):
struct Foo<T: u16 or u32 or u64> { ... }
Is there any way to express this?
For references into an array usually you'd just use a usize rather than different integer types.
However, to do what you are after you can create a new trait, implement that trait for u16, u32 and u64 and then restrict T to your new trait.
pub trait MyNewTrait {}
impl MyNewTrait for u16 {}
impl MyNewTrait for u32 {}
impl MyNewTrait for u64 {}
struct Foo<T: MyNewTrait> { ... }
You may then also add methods onto MyNewTrait and the impls to encapsulate the logic specific to u16, u32 and u64.
Sometimes you may want to use an enum rather than a generic type with a trait bound. For example:
enum Unsigned {
U16(u16),
U32(u32),
U64(u64),
}
struct Foo { x: Unsigned, ... };
One advantage of making a new type over implementing a new trait for existing types is that you can add foreign traits and inherent behavior to the new type. You can implement any traits you like for Unsigned, like Add, Mul, etc. When Foo contains an Unsigned, implementing traits on Unsigned doesn't affect the signature of Foo like it would to add them as bounds on Foo's parameter (e.g. Foo<T: Add<Output=Self> + PartialCmp + ...>). On the other hand, you do still have to implement each trait.
Another thing to note: while you can generally always make a new type and implement a trait for it, an enum is "closed": you can't add new types to Unsigned without touching the rest of its implementation, like you could if you used a trait. This may be a good thing or a bad thing depending on what your design calls for.
"Performance reasons" is a bit ambiguous, but if you're thinking of storing a lot of Unsigneds that will all be the same internal type, and this:
struct Foo([Unsigned; 1_000_000]);
would waste a ton of space over storing a million u16s, you can still make Foo generic! Just implement From<u16>, From<u32>, and From<u64> for Unsigned and write this instead:
struct Foo<T: Into<Unsigned>>([T; 1_000_000]);
Now you only have one simple trait bound on T, you're not wasting space for tags and padding, and functions that deal with T can always convert it to Unsigned to do calculations with. The cost of the conversion may even be optimized away entirely.
See Also
Should I use enum to emulate the polymorphism or use trait with Box<trait> instead?
Consider the following two structs:
pub struct BitVector<S: BitStorage> {
data: Vec<S>,
capacity: usize,
storage_size: usize
}
pub struct BitSlice<'a, S: BitStorage> {
data: &'a [S],
storage_size: usize
}
Where BitStorage is practically a type that is restricted to all unsigned integers (u8, u16, u32, u64, usize).
How to implement the Deref trait? (BitVector<S> derefs to BitSlice<S> similar to how Vec<S> derefs to &[S])
I have tried the following (Note that it doesn't compile due to issues with lifetimes, but more importantly because I try to return a value on the stack as a reference):
impl<'b, S: BitStorage> Deref for BitVector<S> {
type Target = BitSlice<'b, S>;
fn deref<'a>(&'a self) -> &'a BitSlice<'b, S> {
let slice = BitSlice {
data: self.data,
storage_size: self.storage_size,
};
&slice
}
}
I am aware that it is possible to return a field of a struct by reference, so for example I could return &Vec<S> or &usize in the Deref trait, but is it possible to return a BitSlice noting that I essentially have all the data in the BitVector already as Vec<S> can be transformed into &[S] and storage_size is already there?
I would think this is possible if I could create a struct using both values and somehow tell the compiler to ignore the fact that it is a struct that is created on the stack and instead just use the existing values, but I have got no clue how.
Deref is required to return a reference. A reference always points to some existing memory, and any local variable will not exist long enough. While there are, in theory, some sick tricks you could play to create a new object in deref and return a reference to it, all that I'm aware of result in a memory leak. Let's ignore these technicalities and just say it's plain impossible.
Now what? You'll have to change your API. Vec can implement Deref because it derefs to [T], not to &[T] or anything like that. You may have success with the same strategy: Make BitSlice<S> an unsized type containing only a slice [S], so that the return type is &'a BitSlice<S>. This assume the storage_size member is not needed. But it seems that this refers to the number of bits that are logically valid (i.e., can be accessed without extending the bit vector) — if so, that seems unavoidable1.
The other alternative, of course, is to not implement a Deref. Inconvenient, but if your slice data type is too far from an actual slice, it may be the only option.
RFC PR #1524 that proposed custom dynamically-sized types, then you could have a type BitSlice<S> that is like a slice but can have additional contents such as storage_size. However, this doesn't exist yet and it's far from certain if it ever will.
1 The capacity member on BitVector, however, seems pointless. Isn't that just sizeof S * 8?