Can one specify serde's rename_all rule at runtime? - rust

I have a data model that I would like to be deserialized from "camelCase" to the rust standard "snake_case" when reading from a source, X. But I'd like to leave it in "snake_case" when reading or writing to another source, Y.
For example, the following code,
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
struct Data {
foo_bar: String,
hello_word: String,
}
can only be encoded and decoded in camel case. Even if I manually defined my Serialize and Deserialize implementations, I can't define multiple for the same struct. I could define a second struct that's a copy/paste of the other and then derive but that method would get tedious with multiple large structs. What I would really like to do is specify that rename_all attribute at run-time. But I'm not seeing any way to do that in serde's API.

I think the best way sigh is to just write out one struct Data_ per #[serde(rename_all = ...)], then write one additional struct Data that will be the in-memory representation (which won't be serializable, to remove ambiguity), then implement From in both directions for the Data_s and Data so that they're interconvertible.
Thankfully, we can use a macro so that we only have to specify the fields once. (It is incredibly disgusting nonetheless.)
This playground available here.
use serde::{Deserialize, Serialize}; // 1.0.130
use serde_json; // 1.0.69
macro_rules! interconvertible {
($T:ident <-> $U:ident, $($field_name:ident),*) => {
impl From<$T> for $U {
fn from(t: $T) -> Self {
let $T { $($field_name),* } = t;
Self { $($field_name),* }
}
}
impl From<$U> for $T {
fn from(u: $U) -> Self {
let $U { $($field_name),* } = u;
Self { $($field_name),* }
}
}
};
}
macro_rules! create_data_structs {
($($field_name:ident: $field_type:ty),* $(,)?) => {
#[derive(Serialize, Deserialize, Debug)]
#[serde(rename_all = "camelCase")]
struct DataX {
$($field_name: $field_type),*
}
#[derive(Serialize, Deserialize, Debug)]
#[serde(rename_all = "snake_case")]
struct DataY {
$($field_name: $field_type),*
}
#[derive(Debug)]
struct Data {
$($field_name: $field_type),*
}
interconvertible!(DataX <-> Data, $($field_name),*);
interconvertible!(DataY <-> Data, $($field_name),*);
}
}
create_data_structs!(foo_bar: String, hello_world: String);
fn main() -> serde_json::Result<()> {
let x1: DataX = serde_json::from_str(r#"{"fooBar": "a", "helloWorld": "b"}"#)?;
let y1: DataY = serde_json::from_str(r#"{"foo_bar": "a", "hello_world": "b"}"#)?;
println!("{:?}, {:?}", x1, y1);
let x2: Data = x1.into();
let y2: Data = y1.into();
println!("{:?}, {:?}", x2, y2);
let x_string = serde_json::to_string(&DataX::from(x2))?;
let y_string = serde_json::to_string(&DataY::from(y2))?;
println!("{:?}, {:?}", x_string, y_string);
Ok(())
}
The output is:
DataX { foo_bar: "a", hello_world: "b" }, DataY { foo_bar: "a", hello_world: "b" }
[Data { foo_bar: "a", hello_world: "b" }, Data { foo_bar: "a", hello_world: "b" }]
"{\"fooBar\":\"a\",\"helloWorld\":\"b\"}", "{\"foo_bar\":\"a\",\"hello_world\":\"b\"}"

Since I'm only every decoding from source X I can utilize the #[serde(alias = ???)] macro. So my above use case would be
#[derive(Serialize, Deserialize)]
struct Data {
#[serde(alias="fooBar")]
foo_bar: String,
#[serde(alias="helloWorld")]
hello_word: String,
}
It's still a little tedious but better than an intermediate struct. It won't work though if I want to decode or encode to different cases.
(I'm not going to mark this as an answer because it's a work-around for my specific use case. If anyone has a more generic solution feel free to answer.)

Related

Deserialize using a function of the tag

An API with this internally tagged field structure, with "topic" being the tag:
{
"topic": "Car"
"name": "BMW"
"HP": 250
}
This can be deserialized with
#[derive(Serialize, Deserialize)]
#[serde(tag = "topic")]
pub enum catalog {
CarEntry(Car),
... (other types)
}
#[derive(Serialize, Deserialize)]
pub struct Car {
pub name: String
pub HP: i32
}
It turns out that instead of reporting the topic as just Car, the API actually sends Car.product1 or Car.product2 etc.
This breaks the deserialization, because the deserializer doesn't know what the type is based on the string. Is there a way to supply a function to chop off the type string so that the correct model is found?
I don't think serde provides a way to mangle the tag before using it (at least I don't see anything relevant). And the generated serializers for tagged enums are relatively complex, with internal caching if the tag isn't the first field, and whatnot, so I wouldn't want to reproduce that in a custom deserializer.
The cheapest (but not necessarily most efficient) shot at this is to deserialize to serde_json::Value first, manually process the tag, and then deserialize the serde_json::Values to whatever struct you want.
Do that in a custom deserializer, and it starts looking reasonable:
impl<'de> Deserialize<'de> for Catalog {
fn deserialize<D>(d: D) -> Result<Self, <D as Deserializer<'de>>::Error>
where
D: Deserializer<'de>,
{
use serde_json::{Map, Value};
#[derive(Deserialize)]
struct Pre {
topic: String,
#[serde(flatten)]
data: Map<String, Value>,
}
let v = Pre::deserialize(d)?;
// Now you can mangle Pre any way you want to get your final structs.
match v.topic.as_bytes() {
[b'C', b'a', b'r', b'.', _rest # ..] => Ok(Catalog::CarEntry(
serde_json::from_value(v.data.into()).map_err(de::Error::custom)?,
)),
[b'B', b'a', b'r', b'.', _rest # ..] => Ok(Catalog::BarEntry(
serde_json::from_value(v.data.into()).map_err(de::Error::custom)?,
)),
_ => return Err(de::Error::unknown_variant(&v.topic, &["Car.…", "Bar.…"])),
}
}
}
Playground
Btw, what do you want to do with the suffix of topic? Throw it away? How do you plan on handling serialization if you do throw it away?
You can directly use enum instead of defining extra struct type.
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag = "topic")]
pub enum Catalog {
Car { name: String, hp: i32 }
}
fn main() {
let car = Catalog::Car { name: String::from("BMW"), hp: 2000 };
// Convert the Car to a JSON string.
let serialized = serde_json::to_string(&car).unwrap();
// Prints serialized = {"topic":"Car","name":"BMW","hp":2000}
println!("serialized = {}", serialized);
// Convert the JSON string back to a Car.
let deserialized: Catalog = serde_json::from_str(&serialized).unwrap();
// Prints deserialized = Car { name: "BMW", hp: 2000 }
println!("deserialized = {:?}", deserialized);
}
Playground
You can use #[serde(rename()] to rename type in output
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag = "topic")]
pub enum Catalog {
#[serde(rename(serialize = "Car", deserialize = "CarEntry"))]
CarEntry(Car),
}
#[derive(Serialize, Deserialize, Debug)]
pub struct Car {
pub name: String,
pub hp: i32
}
fn main() {
let car = Car { name: String::from("BMW"), hp: 2000 };
let catalog = Catalog::CarEntry(car);
// Convert the Car to a JSON string.
let serialized = serde_json::to_string(&catalog).unwrap();
// Prints serialized = {"topic":"Car","name":"BMW","hp":2000}
println!("serialized = {}", serialized);
// Convert the JSON string back to a Car.
let deserialized: Car = serde_json::from_str(&serialized).unwrap();
// Prints deserialized = Car { name: "BMW", hp: 2000 }
println!("deserialized = {:?}", deserialized);
}
Playground

Custom serde serialization for enum type

I have the following data structure which should be able to hold either a String, a u64 value, a boolean value, or a String vector.
#[derive(Serialize, Deserialize)]
pub enum JsonRpcParam {
String(String),
Int(u64),
Bool(bool),
Vec(Vec<String>)
}
The use case for this data structure is to build JSON RPC parameters which can have multiple types, so I would be able to build a parameter list like this:
let mut params : Vec<JsonRpcParam> = Vec::new();
params.push(JsonRpcParam::String("Test".to_string()));
params.push(JsonRpcParam::Bool(true));
params.push(JsonRpcParam::Int(64));
params.push(JsonRpcParam::Vec(vec![String::from("abc"), String::from("cde")]));
My problem is now the serialization. I am using serde_json for the serialization part. The default serialization of the vector posted above yields:
[
{
"String":"Test"
},
{
"Bool":true
},
{
"Int":64
},
{
"Vec":[
"abc",
"cde"
]
}
]
Instead, I would like the serialization to look like this:
[
"Test",
true,
64,
["abc","cde"]
]
I attempted to implement a custom serialize method for this type, but dont't know how to achieve what I want, my attempt looks like this:
impl Serialize for JsonRpcParam {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer {
match *self {
JsonRpcParam::String(x) => serializer.serialize_str(x),
JsonRpcParam::Int(x) => serializer.serialize_u64(x),
JsonRpcParam::Bool(x) => serializer.serialize_bool(x),
JsonRpcParam::Vec(x) => _
}
}
}
Instead of manually implementing Serialize you can instead use #[serde(untagged)].
In your case that will work perfectly fine. However, be warned that if the enum variant isn't unique and can't be clearly identified from the JSON, then it will deserialize into the first variant that matches. In short if you also have e.g. a subsequent JsonRpcParam::OtherString(String), then that will deserialize into JsonRpcParam::String(String).
#[derive(Serialize, Deserialize, Debug)]
#[serde(untagged)]
pub enum JsonRpcParam {
String(String),
Int(u64),
Bool(bool),
Vec(Vec<String>),
}
If you now use e.g. serde_json::to_string_pretty(), then it will yield an output in your desired format:
fn main() {
let mut params: Vec<JsonRpcParam> = Vec::new();
params.push(JsonRpcParam::String("Test".to_string()));
params.push(JsonRpcParam::Bool(true));
params.push(JsonRpcParam::Int(64));
params.push(JsonRpcParam::Vec(vec![
String::from("abc"),
String::from("cde"),
]));
let json = serde_json::to_string_pretty(&params).unwrap();
println!("{}", json);
}
Output:
[
"Test",
true,
64,
[
"abc",
"cde"
]
]

Deserialization of json with serde by a numerical value as type identifier

I'm quite new to rust and come from an OOP background. So, maybe I misunderstood some rust basics.
I want to parse a fixed json-structure with serde. This structure represents one of different messages types. Each message has a numeric type attribute to distinguish it. The exact structure of the individual message types varies mostly, but they can also be the same.
{"type": 1, "sender_id": 4, "name": "sender", ...}
{"type": 2, "sender_id": 5, "measurement": 3.1415, ...}
{"type": 3, "sender_id": 6, "measurement": 13.37, ...}
...
First of all, I defined an enum to distinguish between message types also a struct for each type of message without a field storing the type.
#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "type")]
enum Message {
T1(Type1),
T2(Type2),
T3(Type3),
// ...
}
#[derive(Debug, Serialize, Deserialize)]
struct Type1 {
sender_id: u32,
name: String,
// ...
}
#[derive(Debug, Serialize, Deserialize)]
struct Type2 {
sender_id: u32,
measurement: f64,
// ...
}
#[derive(Debug, Serialize, Deserialize)]
struct Type3 {
sender_id: u32,
measurement: f64,
// ...
}
// ...
When I try to turn a string to a Message object, I get an error.
let message = r#"{"type":1,"sender_id":123456789,"name":"sender"}"#;
let message: Message = serde_json::from_str(message)?; // error here
// Error: Custom { kind: InvalidData, error: Error("invalid type: integer `1`, expected variant identifier", line: 1, column: 9) }
So, as I understood, serde tries to figure out the type of the current message but it needs a string
for that. I also tried to write my own deserialize()-function. I tried to get the numerical value
of the corresponding type-key and wanted to create the specific object by the type value.
How I have to implement the deserialize() to extract the type of the message and create the specific message object? Is it possible to write this without writing a deserialize()-function for each Type1/2/3/... struct?
impl<'de> Deserialize<'de> for Message {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where D: Deserializer<'de>,
{
// which functions I have to call?
}
Or is there a better solution to achieve my deserialization?
I prepared a playground for this issue: Playground
Serde doesn't support integer tags yet (see issue #745).
If you're able to change what's producing the data, then if you're able to change type into a string, i.e. "1" instead of 1. Then you can get it working simply using #[serde(rename)].
#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "type")]
enum Message {
#[serde(rename = "1")]
T1(Type1),
#[serde(rename = "2")]
T2(Type2),
#[serde(rename = "3")]
T3(Type3),
// ...
}
If that's not an option, then you indeed need to create a custom deserializer. The shortest in terms of code, is likely to deserialize into a serde_json::Value, and then match on the type, and deserialize the serde_json::Value into the correct Type{1,2,3}.
use serde_json::Value;
impl<'de> serde::Deserialize<'de> for Message {
fn deserialize<D: serde::Deserializer<'de>>(d: D) -> Result<Self, D::Error> {
let value = Value::deserialize(d)?;
Ok(match value.get("type").and_then(Value::as_u64).unwrap() {
1 => Message::T1(Type1::deserialize(value).unwrap()),
2 => Message::T2(Type2::deserialize(value).unwrap()),
3 => Message::T3(Type3::deserialize(value).unwrap()),
type_ => panic!("unsupported type {:?}", type_),
})
}
}
You'll probably want to perform some proper error handling, instead of unwrapping and panicking.
If you need serialization as well, then you will likewise need a custom serializer. For this you could create a new type to serialize into, as you cannot use Message.
use serde::Serializer;
impl Serialize for Message {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
#[derive(Serialize)]
#[serde(untagged)]
enum Message_<'a> {
T1(&'a Type1),
T2(&'a Type2),
T3(&'a Type3),
}
#[derive(Serialize)]
struct TypedMessage<'a> {
#[serde(rename = "type")]
t: u64,
#[serde(flatten)]
msg: Message_<'a>,
}
let msg = match self {
Message::T1(t) => TypedMessage { t: 1, msg: Message_::T1(t) },
Message::T2(t) => TypedMessage { t: 2, msg: Message_::T2(t) },
Message::T3(t) => TypedMessage { t: 3, msg: Message_::T3(t) },
};
msg.serialize(serializer)
}
}
When using #[serde(flatten)], then it uses serde::private::ser::FlatMapSerializer, which is hidden from the documentation. In place of creating new types, you could use SerializeMap and FlatMapSerializer.
However, be warned, given it's undocumented, then any future release of serde could break your code if you're using FlatMapSerializer directly.
use serde::{private::ser::FlatMapSerializer, ser::SerializeMap, Serializer};
impl Serialize for Message {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
let mut s = serializer.serialize_map(None)?;
let type_ = &match self {
Message::T1(_) => 1,
Message::T2(_) => 2,
Message::T3(_) => 3,
};
s.serialize_entry("type", &type_)?;
match self {
Message::T1(t) => t.serialize(FlatMapSerializer(&mut s))?,
Message::T2(t) => t.serialize(FlatMapSerializer(&mut s))?,
Message::T3(t) => t.serialize(FlatMapSerializer(&mut s))?,
}
s.end()
}
}

srml_system::Trait::Hash equality comparison

How can I perform an equality comparison on substrate Hash trait?
Say I have the following code, with owned_vec contains a vector of Hash:
use support::{decl_module, decl_storage, decl_event, dispatch::Result,
StorageValue, StorageMap, ensure, traits::Currency };
use system::ensure_signed;
// this is needed when you want to use Vec and Box
use rstd::prelude::*;
use runtime_primitives::traits::{ As, Hash };
use parity_codec::{ Encode, Decode };
// Our own Cat struct
#[derive(Encode, Decode, Default, Clone, PartialEq, Debug)]
pub struct Kitty<Hash, Balance> {
id: Hash,
name: Option<Vec<u8>>,
base_price: Balance, // when 0, it is not for sale
}
// This module's storage items.
decl_storage! {
trait Store for Module<T: Trait> as CatAuction {
Kitties get(kitties): map T::Hash => Kitty<T::Hash, T::Balance>;
KittyOwner get(owner_of): map T::Hash => Option<T::AccountId>;
OwnedKitties get(kitties_owned): map T::AccountId => Vec<T::Hash> = Vec::new();
}
}
decl_module! {
pub struct Module<T: Trait> for enum Call where origin: T::Origin {
fn deposit_event<T>() = default;
pub fn transaction(origin, kitty_id: T::Hash) -> Result {
let sender = ensure_signed(origin)?;
let kitty_owner = Self::owner_of(kitty_id).ok_or("Kitty has no owner.")?;
let mut kitty = Self::kitties(kitty_id);
<OwnedKitties<T>>::mutate(kitty_owner, |owned_vec| {
let kitty_index = 0;
for (i, el) in owned_vec.iter().enumerate() {
// This is where the compilation error occurs!
if el != kitty.id { continue }
kitty_index = i;
}
owned_vec.remove(kitty_index);
});
}
}
}
It gives me compiler error that:
no implementation for `&<T as srml_system::Trait>::Hash == <T as srml_system::Trait>::Hash
help: the trait `core::cmp::PartialEq<<T as srml_system::Trait>::Hash>` is not implemented for `&<T as srml_system::Trait>::Hash`
help: consider adding a `where &<T as srml_system::Trait>::Hash: core::cmp::PartialEq<<T as srml_system::Trait>::Hash>` bound
Thank you!
p.s: Aware the tutorial says that looping through a vector is not encouraged in runtime module implementation.

How can I create enums with constant values in Rust?

I can do this:
enum MyEnum {
A(i32),
B(i32),
}
but not this:
enum MyEnum {
A(123), // 123 is a constant
B(456), // 456 is a constant
}
I can create the structures for A and B with a single field and then implement that field, but I think there might be an easier way. Is there any?
The best way to answer this is working out why you want constants in an enum: are you associating a value with each variant, or do you want each variant to be that value (like an enum in C or C++)?
For the first case, it probably makes more sense to just leave the enum variants with no data, and make a function:
enum MyEnum {
A,
B,
}
impl MyEnum {
fn value(&self) -> i32 {
match *self {
MyEnum::A => 123,
MyEnum::B => 456,
}
}
}
// call like some_myenum_value.value()
This approach can be applied many times, to associate many separate pieces of information with each variant, e.g. maybe you want a .name() -> &'static str method too. In the future, these functions can even be marked as const functions.
For the second case, you can assign explicit integer tag values, just like C/C++:
enum MyEnum {
A = 123,
B = 456,
}
This can be matched on in all the same ways, but can also be cast to an integer MyEnum::A as i32. (Note that computations like MyEnum::A | MyEnum::B are not automatically legal in Rust: enums have specific values, they're not bit-flags.)
Creating an "enum" with constant values, can be augmented using structs and associated constants.
This is similar to how crates like bitflags works and what it would generate.
Additionally, to prevent direct instantiation of MyEnum you can tag it with #[non_exhaustive].
#[non_exhaustive]
struct MyEnum;
impl MyEnum {
pub const A: i32 = 123;
pub const B: i32 = 456;
}
Then you simply use the "enum" as you otherwise would, by accessing MyEnum::A and MyEnum::B.
People looking at this may stumble upon the introduction and deprecation of FromPrimitive. A possible replacement which might also be useful here is enum_primitive. It allows you to use C-like enums and have them cast between numeric and logical representation:
#[macro_use]
extern crate enum_primitive;
extern crate num;
use num::FromPrimitive;
enum_from_primitive! {
#[derive(Debug, PartialEq)]
enum FooBar {
Foo = 17,
Bar = 42,
Baz,
}
}
fn main() {
assert_eq!(FooBar::from_i32(17), Some(FooBar::Foo));
assert_eq!(FooBar::from_i32(42), Some(FooBar::Bar));
assert_eq!(FooBar::from_i32(43), Some(FooBar::Baz));
assert_eq!(FooBar::from_i32(91), None);
}
The enum-map crate provides the ability to assign a value to the enum record. What is more, you can use this macro with different value types.
use enum_map::{enum_map, Enum}; // 0.6.2
#[derive(Debug, Enum)]
enum Example {
A,
B,
C,
}
fn main() {
let mut map = enum_map! {
Example::A => 1,
Example::B => 2,
Example::C => 3,
};
map[Example::C] = 4;
assert_eq!(map[Example::A], 1);
for (key, &value) in &map {
println!("{:?} has {} as value.", key, value);
}
}
How about this?
enum MyEnum {
A = 123,
B = 456,
}
assert_eq!(MyEnum::A as i32, 123i32);
assert_eq!(MyEnum::B as i32, 456i32);
Just to give another idea.
#[allow(non_snake_case, non_upper_case_globals)]
mod MyEnum {
pub const A: i32 = 123;
pub const B: i32 = 456;
}
Then you can simply use it by accessing MyEnum::A and MyEnum::B or use MyEnum::*.
The advantage of doing this over associated constants is that you can even nest more enums.
#[allow(non_snake_case, non_upper_case_globals)]
mod MyEnum {
pub const A: i32 = 123;
pub const B: i32 = 456;
#[allow(non_snake_case, non_upper_case_globals)]
mod SubEnum {
pub const C: i32 = 789;
}
}
For my project I wrote a macro that automatically generates indexes and sets initial values.
#[macro_export]
macro_rules! cnum {
(#step $_idx:expr,) => {};
(#step $idx:expr, $head:ident, $($tail:ident,)*) => {
pub const $head: usize = $idx;
cnum!(#step $idx + 1usize, $($tail,)*);
};
($name:ident; $($n:ident),* $(,)* $({ $($i:item)* })?) => {
cnum!($name; 0usize; $($n),* $({ $($i)* })?);
};
($name:ident; $start:expr; $($n:ident),* $(,)* $({ $($i:item)* })?) => {
#[macro_use]
#[allow(dead_code, non_snake_case, non_upper_case_globals)]
pub mod $name {
use crate::cnum;
$($($i)*)?
cnum!(#step $start, $($n,)*);
}
};
}
Then you can use it like this,
cnum! { Tokens;
EOF,
WhiteSpace,
Identifier,
{
cnum! { Literal; 100;
Numeric,
String,
True,
False,
Nil,
}
cnum! { Keyword; 200;
For,
If,
Return,
}
}
}
I have created a crate enumeration just for this.
Example using my crate:
use enumeration::prelude::*;
enumerate!(MyEnum(u8; i32)
A = 123
B = 456
);
pub fn main() {
assert_eq!(*MyEnum::A.value(), 123);
assert_eq!(*MyEnum::B.value(), 456);
}

Resources