Deserialize JSON list of hex strings as bytes - rust

Question

Iʼm trying to read a JSON stream, part of which looks like
"data": [
"c1a8f800a4393e0cacd05a5bc60ae3e0",
"bbac4013c1ca3482155b584d35dac185",
"685f237d4fcbd191c981b94ef6986cde",
"a08898e81f1ddb6612aa12641b856aa9"
]
(there are more entries in the data list and each each entry is longer, but this should be illustrative; both the length of the list and the length of each hex string is known at compile time)
Ideally Iʼd want a single [u8; 64] (the actual size is known at compile time), or failing that, a Vec<u8>, but I imagine itʼs easier to deseriazie it as a Vec<[u8; 16]> and merge them later. However, Iʼm having trouble doing even that.
The hex crate has a way to deserialize a single hex string as a Vec or array of u8, but I canʼt figure out how to tell Serde to do that for each entry of the list. Is there a simple way to do that Iʼm overlooking, or do I need to write my own list deserializer?

Answer 1

Serde has the power to use serializers and deserializers from other crates in a nested fashion using #[serde(with = "...")]. Since hex has a serde feature, this can be easily done.
Here is a simple example using serde_json and hex.
cargo.toml
serde = { version = "1.0.133", features = ["derive"] }
serde_json = "1.0.74"
hex = { version = "0.4", features = ["serde"] }
main.rs
use serde::{Deserialize, Serialize};
use serde_json::Result;
#[derive(Serialize, Deserialize, Debug)]
struct MyData {
data: Vec<MyHex>,
}
#[derive(Serialize, Deserialize, Debug)]
#[serde(transparent)]
struct MyHex {
#[serde(with = "hex::serde")]
hex: Vec<u8>,
}
fn main() -> Result<()> {
let data = r#"
{
"data": [
"c1a8f800a4393e0cacd05a5bc60ae3e0",
"bbac4013c1ca3482155b584d35dac185",
"685f237d4fcbd191c981b94ef6986cde",
"a08898e81f1ddb6612aa12641b856aa9"
]
}
"#';
let my_data: MyData = serde_json::from_str(data)?;
println!("{:?}", my_data); // MyData { data: [MyHex { hex: [193, 168, 248, 0, 164, 57, 62, 12, 172, 208, 90, 91, 198, 10, 227, 224] }, MyHex { hex: [187, 172, 64, 19, 193, 202, 52, 130, 21, 91, 88, 77, 53, 218, 193, 133] }, MyHex { hex: [104, 95, 35, 125, 79, 203, 209, 145, 201, 129, 185, 78, 246, 152, 108, 222] }, MyHex { hex: [160, 136, 152, 232, 31, 29, 219, 102, 18, 170, 18, 100, 27, 133, 106, 169] }] }
return Ok(());
}
Serde With Reference
Hex Serde Reference

Answer 2

In some performance-critical situations, it may be advantageous to implement your own deserializer and use it with serde(deserialize_with = …).
If you go that route, you have to:
Implement a deserialziation function for data
Implement a visitor which takes a sequence of precisely 4 blocks
These blocks then need another deserialization function that turns a string into [u8; 16]
use serde::{Deserialize, Deserializer};
#[derive(Deserialize, Debug)]
pub struct Foo {
#[serde(deserialize_with = "deserialize_array_of_hex")]
pub data: [u8; 64],
}
fn deserialize_array_of_hex<'de, D: Deserializer<'de>>(d: D) -> Result<[u8; 64], D::Error> {
use serde::de;
use std::fmt;
#[derive(serde_derive::Deserialize)]
struct Block(#[serde(with="hex::serde")] [u8; 16]);
struct VecVisitor;
impl<'de> de::Visitor<'de> for VecVisitor {
type Value = [u8; 64];
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
write!(formatter, "a list containing 4 hex strings")
}
fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
where
A: de::SeqAccess<'de>,
{
let mut data = [0; 64];
for i in 0..4 {
let block = seq
.next_element::<Block>()?
.ok_or_else(|| de::Error::custom("too short"))?;
for j in 0..16 {
data[i * 16 + j] = block.0[j];
}
}
if seq.next_element::<String>()?.is_some() {
return Err(de::Error::custom("too long"))
}
Ok(data)
}
}
d.deserialize_seq(VecVisitor)
}
Full example playground. One could also implement DeserializeSeed for Block and only pass a reference to the [u8; 64] to be written into, but I suspect that copying 16 bytes is negligibly cheap. (Edit: I measured it, it turns out about 10% faster than the other two solutions in this post (when using hex::decode_to_slice in visit_str).)
Actually, nevermind having to implement your own deserializer for performance, the above solution is about equal in performance to
use serde::Deserialize;
#[derive(Deserialize, Debug)]
#[serde(from = "MyDataPre")]
pub struct MyData {
pub data: [u8; 64],
}
impl From<MyDataPre> for MyData {
fn from(p: MyDataPre) -> Self {
let mut data = [0; 64];
for b in 0..4 {
for j in 0..16 {
data[b * 16 + j] = p.data[b].0[j];
}
}
MyData { data }
}
}
#[derive(Deserialize, Debug)]
pub struct MyDataPre {
data: [MyHex; 4],
}
#[derive(Deserialize, Debug)]
struct MyHex (#[serde(with = "hex::serde")] [u8; 16]);
The trick here is the use of #[serde(from = …)], which allows you to deserialize to some other struct, and then tell serde how to convert that to the struct you originally wanted.