Using serde to deserialize a HashMap with a Enum key - rust

I have the following Rust code which models a configuration file which includes a HashMap keyed with an enum.
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Hash)]
enum Source {
#[serde(rename = "foo")]
Foo,
#[serde(rename = "bar")]
Bar
}
#[derive(Debug, Clone, Serialize, Deserialize)]
struct SourceDetails {
name: String,
address: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
struct Config {
name: String,
main_source: Source,
sources: HashMap<Source, SourceDetails>,
}
fn main() {
let config_str = std::fs::read_to_string("testdata.toml").unwrap();
match toml::from_str::<Config>(&config_str) {
Ok(config) => println!("toml: {:?}", config),
Err(err) => eprintln!("toml: {:?}", err),
}
let config_str = std::fs::read_to_string("testdata.json").unwrap();
match serde_json::from_str::<Config>(&config_str) {
Ok(config) => println!("json: {:?}", config),
Err(err) => eprintln!("json: {:?}", err),
}
}
This is the Toml representation:
name = "big test"
main_source = "foo"
[sources]
foo = { name = "fooname", address = "fooaddr" }
[sources.bar]
name = "barname"
address = "baraddr"
This is the JSON representation:
{
"name": "big test",
"main_source": "foo",
"sources": {
"foo": {
"name": "fooname",
"address": "fooaddr"
},
"bar": {
"name": "barname",
"address": "baraddr"
}
}
}
Deserializing the JSON with serde_json works perfectly, but deserializing the Toml with toml gives the error.
Error: Error { inner: ErrorInner { kind: Custom, line: Some(5), col: 0, at: Some(77), message: "invalid type: string \"foo\", expected enum Source", key: ["sources"] } }
If I change the sources HashMap to be keyed on String instead of Source, both the JSON and the Toml deserialize correctly.
I'm pretty new to serde and toml, so I'm looking for suggestions on how to I would properly de-serialize the toml variant.

As others have said in the comments, the Toml deserializer doesn't support enums as keys.
You can use serde attributes to convert them to String first:
use std::convert::TryFrom;
use std::fmt;
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Hash)]
#[serde(try_from = "String")]
enum Source {
Foo,
Bar
}
And then implement a conversion from String:
struct SourceFromStrError;
impl fmt::Display for SourceFromStrError {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.write_str("SourceFromStrError")
}
}
impl TryFrom<String> for Source {
type Error = SourceFromStrError;
fn try_from(s: String) -> Result<Self, Self::Error> {
match s.as_str() {
"foo" => Ok(Source::Foo),
"bar" => Ok(Source::Bar),
_ => Err(SourceFromStrError),
}
}
}
If you only need this for the HashMap in question, you could also follow the suggestion in the Toml issue, which is to keep the definition of Source the same and use the crate, serde_with, to modify how the HashMap is serialized instead:
use serde_with::{serde_as, DisplayFromStr};
use std::collections::HashMap;
#[serde_as]
#[derive(Debug, Clone, Serialize, Deserialize)]
struct Config {
name: String,
main_source: Source,
#[serde_as(as = "HashMap<DisplayFromStr, _>")]
sources: HashMap<Source, SourceDetails>,
}
This requires a FromStr implementation for Source, rather than TryFrom<String>:
impl FromStr for Source {
type Err = SourceFromStrError;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"foo" => Ok(Source::Foo),
"bar" => Ok(Source::Bar),
_ => Err(SourceFromStrError),
}
}
}

Related

How to properly use associated types and trait objects together

Here is my setup
#[derive(Copy, Clone, Debug, Default, Deserialize, Serialize)]
pub struct Foo {}
#[derive(Copy, Clone, Debug, Default, Deserialize, Serialize)]
pub struct Bar {}
pub trait MyTrait {
type CreateMsg;
type DeleteMsg;
}
impl MyTrait for Foo {
type CreateMsg = foo::CreateMsg;
type DeleteMsg = foo::DeleteMsg;
}
impl MyTrait for Bar {
type CreateMsg = bar::CreateMsg;
type DeleteMsg = bar::DeleteMsg;
}
pub fn contains_create_msg(my_trait: impl MyTrait, string: String) -> bool {
my_trait::CreateMsg::contains(string)
}
lazy_static! {
pub static ref MY_TRAIT_OBJECTS:[Box<dyn ContractTrait + Sync>; 2] = [
Box::new(Foo::default()),
Box::new(Bar::default()),
];
}
In foo.rs:
#[derive(Copy, Clone, Debug, Default, Deserialize, Serialize)]
pub enum CreateMsg {
CreateThis { a_field: String },
CreateThat { a_field: String }
}
#[derive(Copy, Clone, Debug, Default, Deserialize, Serialize)]
pub enum DeleteMsg {
DeleteThis { a_field: String },
DeleteThat { a_field: String }
}
In bar.rs:
#[derive(Copy, Clone, Debug, Default, Deserialize, Serialize)]
pub enum CreateMsg {
CreateOne { a_different_field: String },
CreateTwo { a_different_field: String }
}
#[derive(Copy, Clone, Debug, Default, Deserialize, Serialize)]
pub enum DeleteMsg {
DeleteOne { a_different_field: String },
DeleteTwo { a_different_field: String }
}
I get an error, "the value of the associated types must be specified". How can I pass a list of MyTrait objects to other places in my program? My intention is to iterate of these My Traits, and perform different functions using the types that they are associated to. Here is an example of something I would like to do:
let msg = "hello".to_string();
for object in MY_TRAIT_OBJECTS {
println!("{}", object.contains_create_msg(msg.clone()));
}
Sorry if there are syntax errors in there, wrote without an editor

How to rename `start` and `end` range values with serde?

I have JSON objects with the following format:
{
"name": "foo",
"value": 1234,
"upper_bound": 5000,
"lower_bound": 1000
}
I'd like to use serde to work with these objects, with a struct like
struct MyObject {
name: String,
value: i32,
bound: Range<i32>,
}
Without any modifications, serializing one of these structs yields
{
"name": "foo",
"value": 1234,
"bound": {
"start": 1000,
"end": 5000
}
}
I can apply #[serde(flatten)] to get closer, yielding
{
"name": "foo",
"value": 1234,
"start": 1000,
"end": 5000
}
But adding #[serde(rename...)] doesn't seem to change anything, no matter what kind of arguments I try giving to the rename. Is it possible to flatten the range, and rename the args?
You can use serde attribute with and just use a intermediate structure letting the real implementation to serde:
use core::ops::Range;
use serde::{Deserialize, Serialize};
use serde_json::Error;
#[derive(Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
struct Foo {
name: String,
value: i32,
#[serde(with = "range_aux", flatten)]
bound: Range<i32>,
}
mod range_aux {
use core::ops::Range;
use serde::{Deserialize, Deserializer, Serialize, Serializer};
#[derive(Serialize, Deserialize)]
struct RangeAux {
upper_bound: i32,
lower_bound: i32,
}
pub fn serialize<S>(range: &Range<i32>, ser: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
RangeAux::serialize(
&RangeAux {
upper_bound: range.end,
lower_bound: range.start,
},
ser,
)
}
pub fn deserialize<'de, D>(d: D) -> Result<Range<i32>, D::Error>
where
D: Deserializer<'de>,
{
let range_aux: RangeAux = RangeAux::deserialize(d)?;
Ok(Range {
start: range_aux.lower_bound,
end: range_aux.upper_bound,
})
}
}
fn main() -> Result<(), Error> {
let data = r#"{"name":"foo","value":1234,"upper_bound":5000,"lower_bound":1000}"#;
let foo: Foo = serde_json::from_str(data)?;
assert_eq!(
foo,
Foo {
name: "foo".to_string(),
value: 1234,
bound: 1000..5000
}
);
let output = serde_json::to_string(&foo)?;
assert_eq!(data, output);
Ok(())
}
That very close to remote pattern but this doesn't work with generic see serde#1844.
A possible generic version:
use core::ops::Range;
use serde::{Deserialize, Serialize};
use serde_json::Error;
#[derive(Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
struct Foo {
name: String,
value: i32,
#[serde(with = "range_aux", flatten)]
bound: Range<i32>,
}
mod range_aux {
use core::ops::Range;
use serde::{Deserialize, Deserializer, Serialize, Serializer};
pub fn serialize<S, Idx: Serialize>(range: &Range<Idx>, ser: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
// could require Idx to be Copy or Clone instead of borrowing Idx
#[derive(Serialize)]
struct RangeAux<'a, Idx> {
upper_bound: &'a Idx,
lower_bound: &'a Idx,
}
RangeAux::serialize(
&RangeAux {
upper_bound: &range.end,
lower_bound: &range.start,
},
ser,
)
}
pub fn deserialize<'de, D, Idx: Deserialize<'de>>(d: D) -> Result<Range<Idx>, D::Error>
where
D: Deserializer<'de>,
{
#[derive(Deserialize)]
struct RangeAux<Idx> {
upper_bound: Idx,
lower_bound: Idx,
}
let range_aux: RangeAux<Idx> = RangeAux::deserialize(d)?;
Ok(Range {
start: range_aux.lower_bound,
end: range_aux.upper_bound,
})
}
}
fn main() -> Result<(), Error> {
let data = r#"{"name":"foo","value":1234,"upper_bound":5000,"lower_bound":1000}"#;
let foo: Foo = serde_json::from_str(data)?;
assert_eq!(
foo,
Foo {
name: "foo".to_string(),
value: 1234,
bound: 1000..5000
}
);
let output = serde_json::to_string(&foo)?;
assert_eq!(data, output);
Ok(())
}
Not necessarily more concise than a custom serializer, but certainly a good bit more trivial is a solution with [serde(from and into)]. (I feel like I'm posting this on every serde question. :/)
You define an auxiliary, serializable struct that has the JSON structure you want:
#[derive(Deserialize, Serialize, Clone)]
struct AuxMyObject {
name: String,
value: i32,
upper_bound: i32,
lower_bound: i32,
}
Then you explain to rust how your auxiliary struct relates to the original struct. It's a bit tedious (but easy), there may be some macro crates that help lessen the typing load:
impl From<MyObject> for AuxMyObject {
fn from(from: MyObject) -> Self {
Self {
name: from.name,
value: from.value,
lower_bound: from.bound.start,
upper_bound: from.bound.end,
}
}
}
impl From<AuxMyObject> for MyObject {
fn from(from: AuxMyObject) -> Self {
Self {
name: from.name,
value: from.value,
bound: Range {
start: from.lower_bound,
end: from.upper_bound,
},
}
}
}
Lastly, you tell serde to replace your main struct with the auxiliary struct when serializing:
#[derive(Serialize, Deserialize, Debug, PartialEq, Clone)]
#[serde(from = "AuxMyObject", into = "AuxMyObject")]
struct MyObject { … }
Playground

Rust: how to minimize patternmatching when parsing json into complex enum

So, let's say I am expecting a lot of different JSONs of a known format from a network stream. I define structures for them and wrap them with an enum representing all the possibilities:
use serde::Deserialize;
use serde_json;
#[derive(Deserialize, Debug)]
struct FirstPossibleResponse {
first_field: String,
}
#[derive(Deserialize, Debug)]
struct SecondPossibleResponse {
second_field: String,
}
#[derive(Deserialize, Debug)]
enum ResponseFromNetwork {
FirstPossibleResponse(FirstPossibleResponse),
SecondPossibleResponse(SecondPossibleResponse),
}
Then, being a smart folk, I want to provide myself with a short way of parsing these JSONs into my structures, so I am implementing a trait (and here is the problem):
impl From<String> for ResponseFromNetwork {
fn from(r: String) -> Self {
match serde_json::from_slice(&r.as_bytes()) {
Ok(v) => ResponseFromNetwork::FirstPossibleResponse(v),
Err(_) => match serde_json::from_slice(&r.as_bytes()) {
Ok(v) => ResponseFromNetwork::SecondPossibleResponse(v),
Err(_) => unimplemented!("idk"),
},
}
}
}
...To use it later like this:
fn main() {
let data_first = r#"
{
"first_field": "first_value"
}"#;
let data_second = r#"
{
"second_field": "first_value"
}"#;
print!("{:?}", ResponseFromNetwork::from(data_first.to_owned()));
print!("{:?}", ResponseFromNetwork::from(data_second.to_owned()));
}
Rust playground
So, as mentioned earlier, the problem is that - this match tree is the only way I got parsing work, and you can imagine - the more variations of different JSONs I might possibly get over the network - the deeper and nastier the tree grows.
I want to have it in some way like ths, e.g. parse it once and then operate depending on the value:
use serde_json::Result;
impl From<String> for ResponseFromNetwork {
fn from(r: String) -> Self {
let parsed: Result<ResponseFromNetwork> = serde_json::from_slice(r.as_bytes());
match parsed {
Ok(v) => {
match v {print!("And here we should match on invariants or something: {:?}", v);
v
}
Err(e) => unimplemented!("{:?}", e),
}
}
}
But it doesn't really work:
thread 'main' panicked at 'not implemented: Error("unknown variant `first_field`, expected `FirstPossibleResponse` or `SecondPossibleResponse`", line: 3, column: 25)', src/main.rs:28:23
Playground
#[serde(untagged)] is designed for precisely that use case. Just add it in front of the definition of enum ResponseFromNetwork and your code will work the way you want it to:
#[derive(Deserialize, Debug)]
#[serde(untagged)]
enum ResponseFromNetwork {
FirstPossibleResponse(FirstPossibleResponse),
SecondPossibleResponse(SecondPossibleResponse),
}
impl From<String> for ResponseFromNetwork {
fn from(r: String) -> Self {
match serde_json::from_slice(r.as_bytes()) {
Ok(v) => v,
Err(e) => unimplemented!("{:?}", e),
}
}
}
Playground
If the formats of the response JSON strings can be extended (maybe not if they are predefined and unchangable), adding a tag field, say "kind", in each JSON, and annotating each variant struct with #[serde(tag = "kind")] and the enum with #[serde(untagged)] can address the issue.
[playground]
use serde::Deserialize;
use serde_json;
#[derive(Deserialize, Debug)]
#[serde(tag = "kind")]
struct FirstPossibleResponse {
first_field: String,
}
#[derive(Deserialize, Debug)]
#[serde(tag = "kind")]
struct SecondPossibleResponse {
second_field: String,
}
#[derive(Deserialize, Debug)]
#[serde(tag = "kind")]
struct ThirdPossibleResponse {
third_field: String,
}
#[derive(Deserialize, Debug)]
#[serde(untagged)]
enum ResponseFromNetwork {
FirstPossibleResponse(FirstPossibleResponse),
SecondPossibleResponse(SecondPossibleResponse),
ThirdPossibleResponse(ThirdPossibleResponse),
}
impl From<String> for ResponseFromNetwork {
fn from(r: String) -> Self {
match serde_json::from_slice(&r.as_bytes()) {
Ok(v) => v,
Err(_) => unimplemented!("idk"),
}
}
}
fn main() {
let data_first = r#"
{
"kind":"FirstPossibleResponse",
"first_field": "first_value"
}"#;
let data_second = r#"
{
"kind":"SecondPossibleResponse",
"second_field": "second_value"
}"#;
let data_third = r#"
{
"kind":"ThirdPossibleResponse",
"third_field": "third_value"
}"#;
println!("{:?}", ResponseFromNetwork::from(data_first.to_owned()));
println!("{:?}", ResponseFromNetwork::from(data_second.to_owned()));
println!("{:?}", ResponseFromNetwork::from(data_third.to_owned()));
}

How to ignore unknown enum variant while deserializing?

I have several non-exhaustive enums which I need to handle nicely.
When a not yet known variant is detected, I need to simply ignore value and continue processing the others.
I am currently deserializing vectors of data from and I managed to properly obtain vectors of MyStruct for my application.
My application needs to be forward-compatible with new versions of enums and simply ignores unknown variants.
For example, currently:
use serde::{Deserialize};
#[derive(Deserialize, Debug)]
#[non_exhaustive]
pub enum CaseStyle {
Lowercase,
Uppercase,
}
#[derive(Deserialize, Debug)]
#[non_exhaustive]
pub enum Encoding {
Plain,
Base64,
}
#[derive(Deserialize, Debug)]
pub struct MyStruct {
case_style: CaseStyle,
encoding: Encoding,
}
fn main() {
let j = r#"[
{"case_style": "Lowercase","encoding":"Plain"},
{"case_style": "Snakecase","encoding":"Plain"},
{"case_style": "Lowercase","encoding":"Aes"},
{"case_style": "Uppercase","encoding":"Base64"}
]"#;
// Convert the JSON string to vec.
let deserialized: Vec<MyStruct> = serde_json::from_str(&j).unwrap();
// Prints deserialized = [MyStruct { case_style: Lowercase, encoding: Plain }, MyStruct { case_style: Uppercase, encoding: Base64 }]
println!("deserialized = {:?}", deserialized);
}
This example fails because of the 2 unknown variant in json data. How could I just ignore these unknown variants from the deserialization?
You can deserialize a Vec<Option<MyStruct>> by converting all deserialization errors to None and all successes to Some(...). Afterwards, you can remove the Option by flattening them. You could skip the Option but this would require you to write a custom deserializer for Vec.
Based on the serde_with crate:
use serde::Deserialize;
use serde_with::{serde_as, DefaultOnError};
#[derive(Deserialize, Debug)]
#[non_exhaustive]
pub enum CaseStyle {
Lowercase,
Uppercase,
}
#[derive(Deserialize, Debug)]
#[non_exhaustive]
pub enum Encoding {
Plain,
Base64,
}
#[derive(Deserialize, Debug)]
pub struct MyStruct {
case_style: CaseStyle,
encoding: Encoding,
}
fn main() {
let j = r#"[
{"case_style": "Lowercase","encoding":"Plain"},
{"case_style": "Snakecase","encoding":"Plain"},
{"case_style": "Lowercase","encoding":"Aes"},
{"case_style": "Uppercase","encoding":"Base64"}
]"#;
#[serde_as]
#[derive(Deserialize)]
struct W(#[serde_as(as = "Vec<DefaultOnError>")] Vec<Option<MyStruct>>);
// Convert the JSON string to vec.
let deserialized: Vec<MyStruct> = serde_json::from_str::<W>(&j)
.unwrap()
.0
.into_iter()
.flatten()
.collect();
// Prints deserialized = [MyStruct { case_style: Lowercase, encoding: Plain }, MyStruct { case_style: Uppercase, encoding: Base64 }]
println!("deserialized = {:?}", deserialized);
}
use serde::{Deserialize, Deserializer};
use serde_with::{serde_as, DefaultOnError};
#[derive(Deserialize, Debug)]
#[non_exhaustive]
pub enum CaseStyle {
Lowercase,
Uppercase,
}
#[derive(Deserialize, Debug)]
#[non_exhaustive]
pub enum Encoding {
Plain,
Base64,
}
#[derive(Deserialize, Debug)]
pub struct MyStruct {
case_style: CaseStyle,
encoding: Encoding,
}
#[derive(Deserialize, Debug)]
pub struct VecMyStruct {
#[serde(deserialize_with = "skip_on_error")]
items: Vec<MyStruct>,
}
fn skip_on_error<'de, D>(deserializer: D) -> Result<Vec<MyStruct>, D::Error>
where
D: Deserializer<'de>,
{
#[serde_as]
#[derive(Deserialize, Debug)]
struct MayBeT(#[serde_as(as = "DefaultOnError")] Option<MyStruct>);
let values: Vec<MayBeT> = Deserialize::deserialize(deserializer)?;
Ok(values.into_iter().filter_map(|t| t.0).collect())
}
fn main() {
let j = r#"{"items":[
{"case_style": "Lowercase","encoding":"Plain"},
{"case_style": "Snakecase","encoding":"Plain"},
{"case_style": "Lowercase","encoding":"Aes"},
{"case_style": "Uppercase","encoding":"Base64"}
]}"#;
// Convert the JSON string to vec.
let deserialized: VecMyStruct = serde_json::from_str(&j).unwrap();
// Prints deserialized = VecMyStruct { items: [MyStruct { case_style: Lowercase, encoding: Plain }, MyStruct { case_style: Uppercase, encoding: Base64 }] }
println!("deserialized = {:?}", deserialized);
}
I found this way of doing thanks to #jonasbb. The only issue that I now have is to have a more generic "skip_on_error" method, that could take any T: Deserialize<'de> and not only MyStruct. As soon as I add T type, compiler tells me errors while implementing Deserialization for MaybeT.

Rust config crate and polymorphic types

#[derive(Debug, Deserialize)]
struct S3StorageConfig {
url: String,
}
#[derive(Debug, Deserialize)]
struct LocalStorageConfig {
root: std::path::PathBuf,
}
#[derive(Debug, Deserialize)]
struct StorageConfig {
storage_type: String
}
#[derive(Debug, Deserialize)]
pub struct Config {
storages: Vec<StorageConfig>
}
impl Config {
pub fn new(path:Option<std::path::PathBuf>) -> Result<Self, config::ConfigError> {
let mut cfg = config::Config::default();
if let Some(file_path) = path {
cfg.merge(config::File::from(file_path)).unwrap();
}
cfg.merge(config::Environment::with_prefix("datastore"))?;
cfg.try_into()
}
}
Suppose I want to have a config that has
[[storages]]
type: s3
url: ...
and
[[storages]]
type: local
root: ...
And when config does try_into, it is able to find these structs and assign them to the correct structs, by grace of the type field.
What magic would I need to do to make this happen?
Thanks,
So, I'm not 100% sure what you're trying to achieve here but you can serialize/deserialize into the types that you want with serde and using an enum instead.
Ex:
// This enum takes the place of your 'S3StorageConfig' and 'LocalStorageConfig'.
#[derive( Serialize, Deserialize, Debug )]
#[serde( tag = "type" )]
enum Storage {
Cloud{ url: String },
Local{ root: PathBuf },
}
fn main( ) {
let vec = vec![
Storage::Cloud{ url: "www.youtube.com".to_string( ) },
Storage::Local{ root: PathBuf::from( "C:\\Windows\\Fonts" ) },
];
let storage = serde_json::to_string( &vec ).unwrap( );
let vec: Vec<Storage> = serde_json::from_str( &storage ).unwrap( );
println!( "{:#?}", vec );
}
Now you will return a Storage enum variant from your Config class.
You wont need to impl TryInto if this is the direction you decide to take.
impl Config {
pub fn new( ) -> Result<Storage, config::ConfigError> {
// Read in the file here and use 'serde' to deserialize the
// content of the file into the correct enum variant that you
// can now return.
}
}

Resources