sodiumoxide defines PublicKey as:
new_type! {
/// `PublicKey` for signatures
public PublicKey(PUBLICKEYBYTES);
}
The new_type macro expands to:
pub struct $name(pub [u8; $bytes]);
Thus, PublicKey is defined as a simple wrapper of 32 bytes.
When I define my own wrapper of 32 bytes (MyPubKey) it bincode serialises to 32 bytes.
When I bincode serialise PublicKey, it is 40 bytes - the 32 bytes prefixed with a little-endian u64 containing the length.
#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate bincode;
extern crate sodiumoxide;
use sodiumoxide::crypto::{sign, box_};
use bincode::{serialize, deserialize, Infinite};
#[derive(Serialize, Deserialize, PartialEq, Debug)]
pub struct MyPubKey(pub [u8; 32]);
fn main() {
let (pk, sk) = sign::gen_keypair();
let arr: [u8; 32] = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31];
let mpk = MyPubKey(arr);
let encoded_pk: Vec<u8> = serialize(&pk, Infinite).unwrap();
let encoded_arr: Vec<u8> = serialize(&arr, Infinite).unwrap();
let encoded_mpk: Vec<u8> = serialize(&mpk, Infinite).unwrap();
println!("encoded_pk len:{:?} {:?}", encoded_pk.len(), encoded_pk);
println!("encoded_arr len:{:?} {:?}", encoded_arr.len(), encoded_arr);
println!("encoded_mpk len:{:?} {:?}", encoded_mpk.len(), encoded_mpk);
}
Results:
encoded_pk len:40 [32, 0, 0, 0, 0, 0, 0, 0, 7, 199, 134, 217, 109, 46, 34, 155, 89, 232, 171, 185, 199, 190, 253, 88, 15, 202, 58, 211, 198, 49, 46, 225, 213, 233, 114, 253, 61, 182, 123, 181]
encoded_arr len:32 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
encoded_mpk len:32 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
What is the difference between the PublicKey type, created with sodiumoxide's new_type! macro and the MyPublicKey type?
How can I get the 32 bytes out of a PublicKey so that I can serialise them efficiently?
It's up to the implementation of the serialization. sodiumoxide has chosen to implement all serialization by converting the types to a slice and then serializing that:
#[cfg(feature = "serde")]
impl ::serde::Serialize for $newtype {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where S: ::serde::Serializer
{
serializer.serialize_bytes(&self[..])
}
}
Since slices do not have a size known at compile time, serialization must include the length so that deserialization can occur.
You can probably implement your own serialization for a remote type or even just serialize the inner field directly:
serialize(&pk.0, Infinite)
Related
There has a 3-dimensional array x of shape (2000,60,5). If we think it represents a video, the 2000 can represent 2000 frames. I would like to randomly sample it along with the first dimension, i.e., get a set of frame samples. For instance, how to get an array of (500,60,5) which is randomly sampled from x along with the first dimension?
You can pass x as the first argument of the choice method. If you don't want repeated frames in your sample, use replace=False.
For example,
In [10]: x = np.arange(72).reshape(9, 2, 4) # Small array for the demo.
In [11]: x
Out[11]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[16, 17, 18, 19],
[20, 21, 22, 23]],
[[24, 25, 26, 27],
[28, 29, 30, 31]],
[[32, 33, 34, 35],
[36, 37, 38, 39]],
[[40, 41, 42, 43],
[44, 45, 46, 47]],
[[48, 49, 50, 51],
[52, 53, 54, 55]],
[[56, 57, 58, 59],
[60, 61, 62, 63]],
[[64, 65, 66, 67],
[68, 69, 70, 71]]])
Sample "frames" from x with the choice method of NumPy random generator instance.
In [12]: rng = np.random.default_rng()
In [13]: rng.choice(x, size=3)
Out[13]:
array([[[40, 41, 42, 43],
[44, 45, 46, 47]],
[[40, 41, 42, 43],
[44, 45, 46, 47]],
[[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [14]: rng.choice(x, size=3, replace=False)
Out[14]:
array([[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[32, 33, 34, 35],
[36, 37, 38, 39]],
[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]]])
Note that the frames will be in random order; if you want to preserve the order, you could use choice to generate an array of indices, then use the sorted indices to pull the frames out of x.
I'm trying to create efficient SIMD version of dot product to implement 2D convolution for i16 type for FIR filter.
#[cfg(target_arch = "x86_64")]
use std::arch::x86_64::*;
#[target_feature(enable = "avx2")]
unsafe fn dot_product(a: &[i16], b: &[i16]) {
let a = a.as_ptr() as *const [i16; 16];
let b = b.as_ptr() as *const [i16; 16];
let a = std::mem::transmute(*a);
let b = std::mem::transmute(*b);
let ms_256 = _mm256_mullo_epi16(a, b);
dbg!(std::mem::transmute::<_, [i16; 16]>(ms_256));
let hi_128 = _mm256_castsi256_si128(ms_256);
let lo_128 = _mm256_extracti128_si256(ms_256, 1);
dbg!(std::mem::transmute::<_, [i16; 8]>(hi_128));
dbg!(std::mem::transmute::<_, [i16; 8]>(lo_128));
let temp = _mm_add_epi16(hi_128, lo_128);
}
fn main() {
let a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15];
let b = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15];
unsafe {
dot_product(&a, &b);
}
}
I] ~/c/simd (master|…) $ env RUSTFLAGS="-C target-cpu=native" cargo run --release | wl-copy
warning: unused variable: `temp`
--> src/main.rs:16:9
|
16 | let temp = _mm_add_epi16(hi_128, lo_128);
| ^^^^ help: if this is intentional, prefix it with an underscore: `_temp`
|
= note: `#[warn(unused_variables)]` on by default
warning: 1 warning emitted
Finished release [optimized] target(s) in 0.00s
Running `target/release/simd`
[src/main.rs:11] std::mem::transmute::<_, [i16; 16]>(ms_256) = [
0,
1,
4,
9,
16,
25,
36,
49,
64,
81,
100,
121,
144,
169,
196,
225,
]
[src/main.rs:14] std::mem::transmute::<_, [i16; 8]>(hi_128) = [
0,
1,
4,
9,
16,
25,
36,
49,
]
[src/main.rs:15] std::mem::transmute::<_, [i16; 8]>(lo_128) = [
64,
81,
100,
121,
144,
169,
196,
225,
]
While I understand SIMD conceptually I'm not familiar with exact instructions and intrinsics.
I know what I need to multiply two vectors and then horizontally sum then by halving them and using instructions to vertically add two halved of lower size.
I've found madd instruction which supposedly should do one such summation after multiplications right away, but not sure what to do with the result.
If using mul instead of madd I'm not sure which instructions to use to reduce the result further.
Any help is welcome!
PS
I've tried packed_simd but it seems like it doesn't work on stable rust.
import numpy as np
arr = np.array(range(60)).reshape(6,10)
arr
> array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
> [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
> [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
> [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
> [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
> [50, 51, 52, 53, 54, 55, 56, 57, 58, 59]])
What I need:
select_random_windows(arr, number_of windows= 3, window_size=3)
> array([[[ 1, 2, 3],
> [11, 12, 13],
> [21, 22, 23]],
>
> [37, 38, 39],
> [47, 48, 49],
> [57, 58, 59]],
>
> [31, 32, 33],
> [41, 42, 43],
> [51, 52, 53]]])
In this hypothetical case I'm selecting 3 windows of 3x3 within the main array (arr).
My actual array is a raster and I basically need a bunch (on the thousands) of little 3x3 windows.
Any help or even a hint will be much appreciated.
I actually haven't found any practical solution yet...since many many hours
THX!
We can leverage np.lib.stride_tricks.as_strided based scikit-image's view_as_windows to get sliding windows. More info on use of as_strided based view_as_windows.
from skimage.util.shape import view_as_windows
def select_random_windows(arr, number_of_windows, window_size):
# Get sliding windows
w = view_as_windows(arr,window_size)
# Store shape info
m,n = w.shape[:2]
# Get random row, col indices for indexing into windows array
lidx = np.random.choice(m*n,number_of_windows,replace=False)
r,c = np.unravel_index(lidx,(m,n))
# If duplicate windows are allowed, use replace=True or np.random.randint
# Finally index into windows and return output
return w[r,c]
Sample run -
In [209]: arr
Out[209]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59]])
In [210]: np.random.seed(0)
In [211]: select_random_windows(arr, number_of_windows=3, window_size=(2,4))
Out[211]:
array([[[41, 42, 43, 44],
[51, 52, 53, 54]],
[[26, 27, 28, 29],
[36, 37, 38, 39]],
[[22, 23, 24, 25],
[32, 33, 34, 35]]])
You can try [numpy.random.choice()][1]. It takes a 1D or an ndarray and creates a single element or an ndarray by sampling the elements from the given ndarray. You also have an option of providing the size of the array you want as the output.
I'm currently trying to generate an ED25519 keypair from a SHA256 hash (via rust-crypto crate):
extern crate crypto; // rust-crypto = "0.2.36"
use crypto::ed25519;
use crypto::sha2::Sha256;
use crypto::digest::Digest;
fn main() {
let phrase = "purchase hobby popular celery evil fantasy someone party position gossip host gather";
let mut seed = Sha256::new();
seed.input_str(&phrase);
let (_priv, _publ) = ed25519::keypair(&seed); // expects slice
}
However, I totally fail to understand how to correctly pass the SHA256 to the ed25519::keypair() function. I traced down that &seed.result_str() results in:
"fc37862cb425ca4368e8e368c54bb6ea0a1f305a225978564d1bdabdc7d99bdb"
This is the correct hash, while &seed.result_str().as_bytes() results in:
[102, 99, 51, 55, 56, 54, 50, 99, 98, 52, 50, 53, 99, 97, 52, 51, 54, 56, 101, 56, 101, 51, 54, 56, 99, 53, 52, 98, 98, 54, 101, 97, 48, 97, 49, 102, 51, 48, 53, 97, 50, 50, 53, 57, 55, 56, 53, 54, 52, 100, 49, 98, 100, 97, 98, 100, 99, 55, 100, 57, 57, 98, 100, 98]
Which is something I do not want, something entirely different. The question now breaks down to:
|
36 | let (_priv, _publ) = ed25519::keypair(&seed);
| ^^^^^ expected slice, found struct `crypto::sha2::Sha256`
|
= note: expected type `&[u8]`
found type `&crypto::sha2::Sha256`
How to correctly convert the crypto::sha2::Sha256 hash into a [u8] representation?
The Sha256 API may be a little confusing at first because it is designed so that it doesn't allocate any new memory for the data. That's to avoid wasting a memory allocation, in case you want to allocate it yourself. Instead, you give it a buffer to write to:
// Create a buffer in which to write the bytes, making sure it's
// big enough for the size of the hash
let mut bytes = vec![0; seed.output_bytes()];
// Write the raw bytes from the hash into the buffer
seed.result(&mut bytes);
// A reference to a Vec can be coerced to a slice
let (_priv, _publ) = ed25519::keypair(&bytes);
my code:
def originalList = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
def newList = orginalList.percent(0.05,0.95) //I have no idea what I'm doing here
println newList
I have an original list of numbers, they are 1 - 100 and i want to make a new list from the original list however the new list must only have data that belongs to the sub-range 5%- 95% of the original list
so the new list must be like [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18....95]. How do i do that? i know my newList code is wrong
You mean like:
originalList[ 4..94 ] // zero starting pos
Or do you need percentages?
You could do:
originalList[ (originalList.size() * 0.05 - 1)..<(originalList.size() * 0.95) ]
You could also use the metaClass:
List.metaClass.percent { double lower, double upper ->
int d = lower * delegate.size() - 1
int t = upper * delegate.size()
delegate.take( t ).drop( d )
}
originalList.percent( 0.05, 0.95 )