Related
In my rust project, I'm trying to make a db like implementation that can store the state of a user from different integrations that all implement same functions through a trait, and can be written to a file later using serde with this code:
#[derive(Serialize, Deserialize, Debug)]
pub struct State {
pub current: HashMap<Bytes, Box<dyn UserState>>
pub changes: HashMap<u64, Vec<UserStateChange>>
}
pub trait UserState {
fn id(&self) -> Bytes;
fn apply_change(&mut self, change: UserStateChange);
}
#[derive(Serialize, Deserialize, Debug)]
pub enum UserStateChange {
Foo(FooChange),
Bar(BarChange)
}
#[derive(Serialize, Deserialize, Debug)]
pub struct FooState {
id: Bytes,
count: u64,
start: u64
}
#[derive(Serialize, Deserialize, Debug)]
pub struct FooChange {
count_delta: i64
}
impl UserState for FooState {
fn id(&self) -> Bytes {self.id.clone()}
fb apply_change(&mut self, change: UserStateChange) {
let UserStateChange::Foo(change) = change;
todo!();
}
}
#[derive(Serialize, Deserialize, Debug)]
pub struct BarState {
id: Bytes,
attempts: u64
}
#[derive(Serialize, Deserialize, Debug)]
pub struct BarChange {
attempt_delta: i64
}
impl UserState for BarState {
fn id(&self) -> Bytes {self.id.clone()}
fb apply_change(&mut self, change: UserStateChange) {
let UserStateChange::Bar(change) = change;
// apply logic
}
}
How can I make it so that it can store and serialize the dynamic trait, while ensuring that it supports the apply_change function, and can use its own structure for storing changes with relevant fields?
I have JSON objects with the following format:
{
"name": "foo",
"value": 1234,
"upper_bound": 5000,
"lower_bound": 1000
}
I'd like to use serde to work with these objects, with a struct like
struct MyObject {
name: String,
value: i32,
bound: Range<i32>,
}
Without any modifications, serializing one of these structs yields
{
"name": "foo",
"value": 1234,
"bound": {
"start": 1000,
"end": 5000
}
}
I can apply #[serde(flatten)] to get closer, yielding
{
"name": "foo",
"value": 1234,
"start": 1000,
"end": 5000
}
But adding #[serde(rename...)] doesn't seem to change anything, no matter what kind of arguments I try giving to the rename. Is it possible to flatten the range, and rename the args?
You can use serde attribute with and just use a intermediate structure letting the real implementation to serde:
use core::ops::Range;
use serde::{Deserialize, Serialize};
use serde_json::Error;
#[derive(Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
struct Foo {
name: String,
value: i32,
#[serde(with = "range_aux", flatten)]
bound: Range<i32>,
}
mod range_aux {
use core::ops::Range;
use serde::{Deserialize, Deserializer, Serialize, Serializer};
#[derive(Serialize, Deserialize)]
struct RangeAux {
upper_bound: i32,
lower_bound: i32,
}
pub fn serialize<S>(range: &Range<i32>, ser: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
RangeAux::serialize(
&RangeAux {
upper_bound: range.end,
lower_bound: range.start,
},
ser,
)
}
pub fn deserialize<'de, D>(d: D) -> Result<Range<i32>, D::Error>
where
D: Deserializer<'de>,
{
let range_aux: RangeAux = RangeAux::deserialize(d)?;
Ok(Range {
start: range_aux.lower_bound,
end: range_aux.upper_bound,
})
}
}
fn main() -> Result<(), Error> {
let data = r#"{"name":"foo","value":1234,"upper_bound":5000,"lower_bound":1000}"#;
let foo: Foo = serde_json::from_str(data)?;
assert_eq!(
foo,
Foo {
name: "foo".to_string(),
value: 1234,
bound: 1000..5000
}
);
let output = serde_json::to_string(&foo)?;
assert_eq!(data, output);
Ok(())
}
That very close to remote pattern but this doesn't work with generic see serde#1844.
A possible generic version:
use core::ops::Range;
use serde::{Deserialize, Serialize};
use serde_json::Error;
#[derive(Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
struct Foo {
name: String,
value: i32,
#[serde(with = "range_aux", flatten)]
bound: Range<i32>,
}
mod range_aux {
use core::ops::Range;
use serde::{Deserialize, Deserializer, Serialize, Serializer};
pub fn serialize<S, Idx: Serialize>(range: &Range<Idx>, ser: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
// could require Idx to be Copy or Clone instead of borrowing Idx
#[derive(Serialize)]
struct RangeAux<'a, Idx> {
upper_bound: &'a Idx,
lower_bound: &'a Idx,
}
RangeAux::serialize(
&RangeAux {
upper_bound: &range.end,
lower_bound: &range.start,
},
ser,
)
}
pub fn deserialize<'de, D, Idx: Deserialize<'de>>(d: D) -> Result<Range<Idx>, D::Error>
where
D: Deserializer<'de>,
{
#[derive(Deserialize)]
struct RangeAux<Idx> {
upper_bound: Idx,
lower_bound: Idx,
}
let range_aux: RangeAux<Idx> = RangeAux::deserialize(d)?;
Ok(Range {
start: range_aux.lower_bound,
end: range_aux.upper_bound,
})
}
}
fn main() -> Result<(), Error> {
let data = r#"{"name":"foo","value":1234,"upper_bound":5000,"lower_bound":1000}"#;
let foo: Foo = serde_json::from_str(data)?;
assert_eq!(
foo,
Foo {
name: "foo".to_string(),
value: 1234,
bound: 1000..5000
}
);
let output = serde_json::to_string(&foo)?;
assert_eq!(data, output);
Ok(())
}
Not necessarily more concise than a custom serializer, but certainly a good bit more trivial is a solution with [serde(from and into)]. (I feel like I'm posting this on every serde question. :/)
You define an auxiliary, serializable struct that has the JSON structure you want:
#[derive(Deserialize, Serialize, Clone)]
struct AuxMyObject {
name: String,
value: i32,
upper_bound: i32,
lower_bound: i32,
}
Then you explain to rust how your auxiliary struct relates to the original struct. It's a bit tedious (but easy), there may be some macro crates that help lessen the typing load:
impl From<MyObject> for AuxMyObject {
fn from(from: MyObject) -> Self {
Self {
name: from.name,
value: from.value,
lower_bound: from.bound.start,
upper_bound: from.bound.end,
}
}
}
impl From<AuxMyObject> for MyObject {
fn from(from: AuxMyObject) -> Self {
Self {
name: from.name,
value: from.value,
bound: Range {
start: from.lower_bound,
end: from.upper_bound,
},
}
}
}
Lastly, you tell serde to replace your main struct with the auxiliary struct when serializing:
#[derive(Serialize, Deserialize, Debug, PartialEq, Clone)]
#[serde(from = "AuxMyObject", into = "AuxMyObject")]
struct MyObject { … }
Playground
This example code:
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Foo {
bar: String,
baz: Baz
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
enum Baz {
Quux(u32),
Flob,
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Bish {
bash: u16,
bosh: i8
}
fn main() -> std::io::Result<()> {
let mut btree: BTreeMap<Foo, Bish> = BTreeMap::new();
let foo = Foo {
bar: "thud".to_string(),
baz: Baz::Flob
};
let bish = Bish {
bash: 1,
bosh: 2
};
println!("foo: {}", serde_json::to_string(&foo)?);
println!("bish: {}", serde_json::to_string(&bish)?);
btree.insert(foo, bish);
println!("btree: {}", serde_json::to_string(&btree)?);
Ok(())
}
gives the runtime output/error:
foo: {"bar":"thud","baz":"Flob"}
bish: {"bash":1,"bosh":2}
Error: Custom { kind: InvalidData, error: Error("key must be a string", line: 0, column: 0) }
I've googled this, and found that the problem is that the serialiser would be trying to write:
{{"bar":"thud","baz":"Flob"}:{"bash":1,"bosh":2}}}
which is not valid JSON, as keys must be strings.
The internet tells me to write custom serialisers.
This is not a practical option, as I have a large number of different non-string keys.
How can I make serde_json serialise to (and deserialise from):
{"{\"bar\":\"thud\",\"baz\":\"Flob\"}":{"bash":1,"bosh":2}}
for arbitrary non-string keys in BTreeMap and HashMap?
Although OP decided not to use JSON in the end, I have written a crate that does exactly what the original question asked for: https://crates.io/crates/serde_json_any_key. Using it is as simple as a single function call.
Because this is StackOverflow and just a link is not a sufficient answer, here is a complete implementation, combining code from v1.1 of the crate with OP's main function, replacing only the final call to serde_json::to_string:
extern crate serde;
extern crate serde_json;
use serde::{Serialize, Deserialize};
use std::collections::BTreeMap;
mod serde_json_any_key {
use std::any::{Any, TypeId};
use serde::ser::{Serialize, Serializer, SerializeMap, Error};
use std::cell::RefCell;
struct SerializeMapIterWrapper<'a, K, V>
{
pub iter: RefCell<&'a mut (dyn Iterator<Item=(&'a K, &'a V)> + 'a)>
}
impl<'a, K, V> Serialize for SerializeMapIterWrapper<'a, K, V> where
K: Serialize + Any,
V: Serialize
{
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error> where
S: Serializer
{
let mut ser_map = serializer.serialize_map(None)?;
let mut iter = self.iter.borrow_mut();
// handle strings specially so they don't get escaped and wrapped inside another string
if TypeId::of::<K>() == TypeId::of::<String>() {
while let Some((k, v)) = iter.next() {
let s = (k as &dyn Any).downcast_ref::<String>().ok_or(S::Error::custom("Failed to serialize String as string"))?;
ser_map.serialize_entry(s, &v)?;
}
} else {
while let Some((k, v)) = iter.next() {
ser_map.serialize_entry(match &serde_json::to_string(&k)
{
Ok(key_string) => key_string,
Err(e) => { return Err(e).map_err(S::Error::custom); }
}, &v)?;
}
}
ser_map.end()
}
}
pub fn map_iter_to_json<'a, K, V>(iter: &'a mut dyn Iterator<Item=(&'a K, &'a V)>) -> Result<String, serde_json::Error> where
K: Serialize + Any,
V: Serialize
{
serde_json::to_string(&SerializeMapIterWrapper {
iter: RefCell::new(iter)
})
}
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Foo {
bar: String,
baz: Baz
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
enum Baz {
Quux(u32),
Flob,
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
struct Bish {
bash: u16,
bosh: i8
}
fn main() -> std::io::Result<()> {
let mut btree: BTreeMap<Foo, Bish> = BTreeMap::new();
let foo = Foo {
bar: "thud".to_string(),
baz: Baz::Flob
};
let bish = Bish {
bash: 1,
bosh: 2
};
println!("foo: {}", serde_json::to_string(&foo)?);
println!("bish: {}", serde_json::to_string(&bish)?);
btree.insert(foo, bish);
println!("btree: {}", serde_json_any_key::map_iter_to_json(&mut btree.iter())?);
Ok(())
}
Output:
foo: {"bar":"thud","baz":"Flob"}
bish: {"bash":1,"bosh":2}
btree: {"{\"bar\":\"thud\",\"baz\":\"Flob\"}":{"bash":1,"bosh":2}}
After discovering Rusty Object Notation, I realised that I was pushing a RON-shaped peg into a JSON-shaped hole.
The correct solution was to use JSON for the interface with the outside world, and RON for human-readable local data storage.
I have been trying to setup the following configuration for the serialport crate in Rust with serde, so I can intuitively supply 7 in my config for data_bits, but it will be deserialized as serialport::DataBits::Seven. Unfortunately, it seemingly fails the moment I want it to be a number (7) and not a string (seven).
Test case
cargo.toml
[package]
name = "serde_error"
version = "0.1.0"
authors = ["Jason Miller"]
edition = "2018"
[dependencies]
serialport = "3.3.0"
serde = { version = "1.0", features = ["derive"] }
ron = "0.5.1"
The following results in the error:
6:16: Expected identifier
main.rs
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(remote = "serialport::DataBits")]
pub enum DataBitsDef {
#[serde(rename = "5")]
Five,
#[serde(rename = "6")]
Six,
#[serde(rename = "7")]
Seven,
#[serde(rename = "8")]
Eight,
}
fn default_data_bits() -> serialport::DataBits {
serialport::DataBits::Eight
}
#[derive(Debug, Serialize, Deserialize)]
pub struct TransceiverSettings {
pub vid: u16,
pub pid: u16,
pub baud_rate: u32,
#[serde(default = "default_data_bits", with = "DataBitsDef")]
pub data_bits: serialport::DataBits,
}
impl Default for TransceiverSettings {
fn default() -> Self {
Self {
vid: 0x2341,
pid: 0x0043,
baud_rate: 115_200,
data_bits: serialport::DataBits::Eight,
}
}
}
const TRX_CONFIG: &str = "
(
vid: 0x2341,
pid: 0x0043,
baud_rate: 9600,
data_bits: 7,
)
";
fn main() {
match ron::de::from_str::<TransceiverSettings>(&TRX_CONFIG) {
Err(e) => eprintln!("{}", e),
Ok(c) => println!("{:?}", c),
}
}
Oddly enough, writing 7 as seven succeeds and returns:
TransceiverSettings { vid: 9025, pid: 67, baud_rate: 9600, data_bits: Seven }
main.rs
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(remote = "serialport::DataBits")]
pub enum DataBitsDef {
#[serde(rename = "5")]
Five,
#[serde(rename = "6")]
Six,
#[serde(rename = "seven")]
Seven,
#[serde(rename = "8")]
Eight,
}
fn default_data_bits() -> serialport::DataBits {
serialport::DataBits::Eight
}
#[derive(Debug, Serialize, Deserialize)]
pub struct TransceiverSettings {
pub vid: u16,
pub pid: u16,
pub baud_rate: u32,
#[serde(default = "default_data_bits", with = "DataBitsDef")]
pub data_bits: serialport::DataBits,
}
impl Default for TransceiverSettings {
fn default() -> Self {
Self {
vid: 0x2341,
pid: 0x0043,
baud_rate: 115_200,
data_bits: serialport::DataBits::Eight,
}
}
}
const TRX_CONFIG: &str = "
(
vid: 0x2341,
pid: 0x0043,
baud_rate: 9600,
data_bits: seven,
)
";
fn main() {
match ron::de::from_str::<TransceiverSettings>(&TRX_CONFIG) {
Err(e) => eprintln!("{}", e),
Ok(c) => println!("{:?}", c),
}
}
serde_repr
One of the given examples in the serde documentation seems relevant to my case, but I haven't managed to get it working with my setup.
Serialize enum as number
The serde_repr crate provides alternative derive macros that derive the same Serialize and Deserialize traits but delegate to the underlying representation of a C-like enum. This allows C-like enums to be formatted as integers rather than strings in JSON
#[derive(Serialize_repr, Deserialize_repr, PartialEq, Debug)]
#[repr(u8)]
enum SmallPrime {
Two = 2,
Three = 3,
Five = 5,
Seven = 7,
}
Support for #[serde(remote)] is not present in serde_repr 0.1.5. You will need to submit a pull request or issue to add support for it.
Instead, follow the advice in How to transform fields during deserialization using Serde? and How to transform fields during serialization using Serde?:
use serde::{Deserialize, Serialize};
mod shim {
use serde::{de::Error, Deserialize, Deserializer, Serialize, Serializer};
use serialport::DataBits;
pub fn serialize<S>(v: &DataBits, s: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
use DataBits::*;
let v: u8 = match v {
Five => 5,
Six => 6,
Seven => 7,
Eight => 8,
};
v.serialize(s)
}
pub fn deserialize<'de, D>(d: D) -> Result<DataBits, D::Error>
where
D: Deserializer<'de>,
{
use DataBits::*;
match u8::deserialize(d)? {
5 => Ok(Five),
6 => Ok(Six),
7 => Ok(Seven),
8 => Ok(Eight),
o => Err(D::Error::custom(format_args!("Invalid value {}", o))),
}
}
}
#[derive(Debug, Serialize, Deserialize)]
pub struct TransceiverSettings {
pub vid: u16,
pub pid: u16,
pub baud_rate: u32,
#[serde(default = "default_data_bits", with = "shim")]
pub data_bits: serialport::DataBits,
}
I have a group of different messages that come in as JSON and can be distinguished based on a single field, but then each variant has a different collection of secondary fields:
#[derive(Debug, Serialize, Deserialize)]
struct MessageOne {
///op will always be "one"
op: String,
x: f64,
y: f64,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageTwo {
///op will always be "two"
op: String,
a: f64,
b: i64,
}
The different message types are routed to different processing functions (e.g. process_message_one, process_message_two, etc). Is there an elegant or idiomatic way to automatically select the correct message sub-type? Currently I've defined a generic message:
#[derive(Debug, Serialize, Deserialize)]
struct MessageGeneric {
op: String,
}
then parse the incoming JSON into the MessageGeneric, read the op field and then deserialize again, matching on op to select the correct message type. Full example:
#![allow(unused)]
extern crate serde; // 1.0.78
extern crate serde_json; // 1.0.27
#[macro_use]
extern crate serde_derive;
use std::collections::HashMap;
#[derive(Debug, Serialize, Deserialize)]
struct MessageGeneric {
op: String,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageOne {
///op will always be "one"
op: String,
x: f64,
y: f64,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageTwo {
///op will always be "two"
op: String,
a: f64,
b: f64,
}
fn process_message_one(m: &MessageOne) {
println!("Processing a MessageOne: {:?}", m);
}
fn process_message_two(m: &MessageTwo) {
println!("Processing a MessageTwo: {:?}", m);
}
fn main() {
let data = r#"{
"op": "one",
"x": 1.0,
"y": 2.0
}"#;
let z: MessageGeneric = serde_json::from_str(data).unwrap();
match z.op.as_ref() {
"one" => {
let zp: MessageOne = serde_json::from_str(data).unwrap();
process_message_one(&zp);
},
"two" => {
let zp: MessageTwo = serde_json::from_str(data).unwrap();
process_message_two(&zp);
},
_ => println!("Unknown Message Type")
}
}
I've seen Serde's enum representations but it was unclear to me if/how that would be applied in this case. The messages coming in are defined by an external API, so I can't control their content beyond knowing what the variants are.
There is no point to keep "one" or "two" in your structure MessageOne and MessageTwo: if you have constructed this structure you already know if it is message one or message two.
extern crate serde; // 1.0.78
extern crate serde_json; // 1.0.27
#[macro_use]
extern crate serde_derive;
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag = "op")]
enum Message {
#[serde(rename = "one")]
One { x: f64, y: f64 },
#[serde(rename = "two")]
Two { a: f64, b: f64 },
}
fn process_message(message: &Message) {
println!("Processing a : {:?}", message);
}
use serde_json::Error;
fn main() -> Result<(), Error> {
let data = r#"{
"op": "one",
"x": 1.0,
"y": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
let data = r#"{
"op": "two",
"a": 1.0,
"b": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
let data = r#"{
"op": "42",
"i": 1.0,
"j": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
Ok(())
}
Standard Output
Processing a : One { x: 1.0, y: 2.0 }
Processing a : Two { a: 1.0, b: 2.0 }
Standard Error
Error: Error("unknown variant `42`, expected `one` or `two`", line: 2, column: 18)