I am working in a relatively large codebase where options are represented in JSON as arrays, so None is represented in JSON as [] and Some(thing) as [thing]. (Yes, the codebase also contains Haskell, in case you are wondering.) How can I override the default serde_json behaviour, which is to omit optional fields, to match this?
E.g. a struct:
SomeData {
foo: Some(1),
bar: None
}
should be serialized to JSON as:
{
"foo": [1],
"bar": []
}
Of course, one could theoretically implement custom serialization for each and every optional field in every struct that interacts with the codebase but that would be a huge undertaking, even if it were possible.
There don't seem to be any options in the serde_json serialisation of some and none so I imagine that the solution will be creating a new serializer that inherits almost everything from serde_json apart from the Option serialization and deserialization. Are there any examples of projects that do this? It would also be possible to make a fork but maintaining a fork is never much fun.
Of course, one could theoretically implement custom serialization for each and every optional field in every struct that interacts with the codebase
A custom implementation for each and every field is not necessary. By using serialize_with, you only need one transformation function describing the serialization of any serializable Option<T> as a sequence.
fn serialize_option_as_array<T, S>(value: &Option<T>, serializer: S) -> Result<S::Ok, S::Error>
where
T: Serialize,
S: Serializer,
{
let len = if value.is_some() { 1 } else { 0 };
let mut seq = serializer.serialize_seq(Some(len))?;
for element in value {
seq.serialize_element(element)?;
}
seq.end()
}
Using it in your struct:
use serde_derive::Serialize;
use serde::ser::{Serialize, Serializer, SerializeSeq};
use serde_json;
#[derive(Debug, Serialize)]
struct SomeData {
#[serde(serialize_with = "serialize_option_as_array")]
foo: Option<i32>,
#[serde(serialize_with = "serialize_option_as_array")]
bar: Option<u32>,
}
let data = SomeData {
foo: Some(5),
bar: None,
};
println!("{}", serde_json::to_string(&data)?);
The output:
{"foo":[5],"bar":[]}
Playground
See also:
How to transform fields during serialization using Serde?
Related
I'm attempting to use hcl-rs = 0.7.0 to parse some HCL. I'm just experimenting with arbitrary HCL, so I'm not looking to parse terraform specific code.
I'd like to be able to parse a block like this and get it's label as part of result
nested_block "nested_block_label" {
foo = 123
}
This currently doesn't work, but hopefully it shows my intention. Is something like this possible?
#[test]
fn deserialize_struct_with_label() {
#[derive(Deserialize, PartialEq, Debug)]
struct TestRoot {
nested_block: TestNested,
}
#[derive(Deserialize, PartialEq, Debug)]
struct TestNested {
label: String,
foo: u32,
}
let input = r#"
nested_block "nested_block_label" {
foo = 123
}"#;
let expected = TestRoot{ nested_block: TestNested { label: String::from("nested_block_label"), foo: 123 } };
assert_eq!(expected, from_str::<TestRoot>(input).unwrap());
}
Your problem is that hcl by default seems to interpret
nested_block "nested_block_label" {
foo = 123
}
as the following "serde structure":
"nested_block" -> {
"nested_block_label" -> {
"foo" -> 123
}
}
but for your Rust structs, it would have to be
"nested_block" -> {
"label" -> "nested_block_label"
"foo" -> 123
}
I'm not aware of any attributes that would allow you to bend the former into the latter.
As usual, when faced with this kind of situation, it is often easiest to first deserialize to a generic structure like hcl::Block and then convert to whatever struct you want manually. Disadvantage is that you'd have to do that for every struct separately.
You could, in theory, implement a generic deserialization function that wraps the Deserializer it receives and flattens the two-level structure you get into the one-level structure you want. But implementing Deserializers requires massive boilerplate. Possibly, somebody has done that before, but I'm not aware of any crate that would help you here, either.
As a sort of medium effort solution, you could have a special Labelled wrapper structure that always catches and flattens this intermediate level:
#[derive(Debug, PartialEq)]
struct Labelled<T> {
label: String,
t: T,
}
impl<'de, T: Deserialize<'de>> Deserialize<'de> for Labelled<T> {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
struct V<T>(std::marker::PhantomData<T>);
impl<'de, T: Deserialize<'de>> serde::de::Visitor<'de> for V<T> {
type Value = Labelled<T>;
fn visit_map<A>(self, mut map: A) -> Result<Self::Value, A::Error>
where
A: serde::de::MapAccess<'de>,
{
if let (Some((label, t)), None) =
(map.next_entry()?, map.next_entry::<String, ()>()?)
{
Ok(Labelled { label, t })
} else {
Err(serde::de::Error::invalid_type(
serde::de::Unexpected::Other("Singleton map"),
&self,
))
}
}
}
deserializer.deserialize_map(V::<T>(Default::default()))
}
}
would be used like this:
#[derive(Deserialize, PartialEq, Debug)]
struct TestRoot {
nested_block: Labelled<TestNested>,
}
#[derive(Deserialize, PartialEq, Debug)]
struct TestNested {
foo: u32,
}
Lastly, there may be some trick like adding #[serde(rename = "$hcl::label")]. Other serialization libraries (e.g. quick-xml) have similar and allow marking fields as something special this way. hcl-rs does the same internally, but it's undocumented and I can't figure out from the source whether what you need is possible.
I am delete a key from redis when using rust, this is part of the code looks like:
pub async fn del_redis_key(key: &str,) -> Result<()> {
let config_redis_string = get_config("redisConnectionStr");
let redis_con_string: &str = config_redis_string.as_str();
let redis_client = redis::Client::open(redis_con_string).expect("can create redis client");
let mut redis_conn = get_con(redis_client);
let mut redis_conn_unwrap = redis_conn.unwrap();
let del_result = redis_conn_unwrap.del(key).map_err(RedisCMDError)?;
FromRedisValue::from_redis_value(&del_result).map_err(|e| RedisTypeError(e).into())
}
now I want to output the delete result del_result as a string, I have tried to convert the result as a json string like this:
let result_json = serde_json::to_string(&del_result).unwrap();
println!("{}",result_json);
it seems did not work because the third party redis lib Value did not implement the serialize trait, the Value code looks like:
#[derive(PartialEq, Eq, Clone)]
pub enum Value {
/// A nil response from the server.
Nil,
/// An integer response. Note that there are a few situations
/// in which redis actually returns a string for an integer which
/// is why this library generally treats integers and strings
/// the same for all numeric responses.
Int(i64),
/// An arbitary binary data.
Data(Vec<u8>),
/// A bulk response of more data. This is generally used by redis
/// to express nested structures.
Bulk(Vec<Value>),
/// A status response.
Status(String),
/// A status response which represents the string "OK".
Okay,
}
is it possible to output the delete result as a string so that I could output into log? I am using the redis lib redis = "0.21.3".
Serde can actually derive for remote structures. Check this link for details: https://serde.rs/remote-derive.html
But in your particular example, things get a bit more tricky. Because the Value type is recursive (Bulk(Vec<Value>)) and currently you cannot use #[serde(with = ...)] inside a Vec.
A way out is to define your own serializing function for the whole Vec<Value> type. Here is an example: https://github.com/serde-rs/serde/issues/1211
I've implemented this method for you. It is not quite straightforward though.
use redis::Value;
use serde::Serialize;
#[derive(Serialize)]
#[serde(remote = "Value")]
pub enum ValueRef {
Nil,
Int(i64),
Data(Vec<u8>),
#[serde(with = "redis_value_vec")]
Bulk(Vec<Value>),
Status(String),
Okay,
}
mod redis_value_vec {
use super::ValueRef;
use redis::Value;
use serde::{Serialize, Serializer};
pub fn serialize<S: Serializer>(array: &[Value], serializer: S) -> Result<S::Ok, S::Error> {
#[derive(Serialize)]
struct W<'a>(#[serde(with = "ValueRef")] &'a Value);
serializer.collect_seq(array.iter().map(W))
}
}
fn main() {
#[derive(Serialize)]
struct W<'a>(#[serde(with = "ValueRef")] &'a Value);
let val = Value::Nil;
println!("{}", serde_json::to_string(&W(&val)).unwrap());
}
I suggest you submit a PR to that crate, adding a serde feature, optionally enabling this derive. This is very common.
If all you want to do is print out the result for logging purposes, Value implements Debug. So you can print it out with the debug format specifier:
println!("{:?}", del_result);
Say I have this struct:
use serde::{Serialize, Deserialize};
#[derive(Deserialize)]
struct MyStruct {
field_1: Option<usize>, // should only have field_1 or field_2
field_2: Option<usize>, // should only have field_1 or field_2
other_field: String,
}
How can I deserialize this but only allow one of these fields to exist?
The suggestions in the comments to use an enum are likely your best bet. You don't need to replace your struct with an enum, instead you'd add a separate enum type to represent this constraint, e.g.:
use serde::{Serialize, Deserialize};
#[derive(Deserialize)]
enum OneOf {
F1(usize), F2(usize)
}
#[derive(Deserialize)]
struct MyStruct {
one_of_field: OneOf,
other_field: String,
}
Now MyStruct's one_of_field can be initialized with either an F1 or an F2.
#dimo414's answer is correct about the enum being necessary, but the code sample will not function in the way the question is described. This is caused by a couple factors related to how enums are deserialized. Mainly, it will not enforce mutual exclusion of the two variants and will silently pick the first variant that matches and ignore extraneous fields. Another issue is the enum will be treated as a separate structure within MyStruct (Ex: {"one_of_field":{"F1":123},"other_field":"abc"} in JSON).
Solution
For anyone wanting an easy solution, here it is. However keep in mind that variants of the mutually exclusive type can not contain #[serde(flatten)] fields (more information in the issue section). To accommodate neither field_1 or field_2, Option<MutuallyExclusive> can be used in MyStruct.
/// Enum containing mutually exclusive fields. Variants names will be used as the
/// names of fields unless annotated with `#[serde(rename = "field_name")]`.
#[derive(Deserialize)]
enum MutuallyExclusive {
field_1(usize),
field_2(usize),
}
#[derive(Deserialize)]
/// `deny_unknown_fields` is required. If not included, it will not error when both
/// `field_1` and `field_2` are both present.
#[serde(deny_unknown_fields)]
struct MyStruct {
/// Flatten makes it so the variants of MutuallyExclusive are seen as fields of
/// this struct. Without it, foo would be treated as a separate struct/object held
/// within this struct.
#[serde(flatten)]
foo: MutuallyExclusive,
other_field: String,
}
The Issue
TL;DR: It should be fine to use deny_unknown_fields with flatten in this way so long as types used in MutuallyExclusive do not use flatten.
If you read the serde documentation, you may notice it warns that using deny_unknown_fields in conjunction with flatten is unsupported. This is problematic as it throws the long-term reliability of the above code into question. As of writing this, serde will not produce any errors or warnings about this configuration and is able to handle it as intended.
The pull request adding this warning cited 3 issues when doing so:
#[serde(flatten)] error behavior is different to normal one
deny_unknown_fields incorrectly fails with flattened untagged enum
Structs with nested flattens cannot be deserialized if deny_unknown_fields is set
To be honest, I don't really care about the first one. I personally feel it is a bit overly pedantic. It simply states that the error message is not exactly identical between a type and a wrapper for that type using flatten for errors triggered by deny_unknown_fields. This should not have any effect on the functionality of the code above.
However, the other two errors are relevant. They both relate to nested flatten types within a deny_unknown_fields type. Technically the second issue uses untagged for the second layer of nesting, but it has the same effect as flatten in this context. The main idea is that deny_unknown_fields is unable to handle more than a single level of nesting without causing issues. The use case is in any way at fault, but the way deny_unknown_fields and flattened are handled makes it difficult to implement a workaround.
Alternative
However, if anyone still feels uncomfortable with using the above code, you can use this version instead. It will be a pain to work with if there are a lot of other fields, but sidesteps the warning in the documentation.
#[derive(Debug, Deserialize)]
#[serde(untagged, deny_unknown_fields)]
enum MyStruct {
WithField1 {
field_1: usize,
other_field: String,
},
WithField2 {
field_2: usize,
other_field: String,
},
}
You can deserialize your struct and then verify that all the invariants your type should uphold. You can implement Deserialize for your type to this while also relying on the derive macro to do the heavy lifting.
use serde::{Deserialize, Deserializer};
#[derive(Debug, Deserialize)]
#[serde(remote = "Self")]
struct MyStruct {
field_1: Option<usize>, // should only have field_1 or field_2
field_2: Option<usize>, // should only have field_1 or field_2
other_field: String,
}
impl<'de> Deserialize<'de> for MyStruct {
fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
use serde::de::Error;
let s = Self::deserialize(deserializer)?;
if s.field_1.is_some() && s.field_2.is_some() {
return Err(D::Error::custom("should only have field_1 or field_2"));
}
Ok(s)
}
}
fn main() -> () {
dbg!(serde_json::from_value::<MyStruct>(serde_json::json!({
"field_1": 123,
"other_field": "abc"
})));
dbg!(serde_json::from_value::<MyStruct>(serde_json::json!({
"field_2": 456,
"other_field": "abc"
})));
dbg!(serde_json::from_value::<MyStruct>(serde_json::json!({
"field_1": 123,
"field_2": 456,
"other_field": "abc"
})));
}
Playground
I need a map with the Option values in my configuration. However, serde seems to ignore any pairs with the None value
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
use toml;
#[derive(Debug, Serialize, Deserialize)]
struct Config {
values: HashMap<String, Option<u32>>,
}
fn main() {
let values = [("foo", Some(5)), ("bar", None)]
.iter()
.map(|(name, s)| (name.to_string(), s.clone()))
.collect();
let config = Config { values };
let s = toml::ser::to_string(&config).unwrap();
println!("{}", s);
}
produces
[values]
foo = 5
The same goes for deserializing: I simply cannot represent bar: None in any form,
since the TOML has no notion of None or null or alike.
Are there some tricks to do that?
The closest alternative I have found is to use a special sentinel value (the one you will probably use in Option::unwrap_or), which appears in the TOML file as the real value (e.g. 0), and converts from Option::None on serialization. But on deserialization, the sentinel value converts to Option::None and leaves us with the real Option type.
Serde has a special #[serde(with = module)] attribute to customize the ser/de field behavior, which you can use here. The full working example is here.
I'm using [prost] to generate structs from protobuf. One of those structs is quite simple:
enum Direction {
up = 0;
down = 1;
sideways = 2;
}
This generates code which looks like:
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)]
#[repr(i32)]
#[derive(serde_derive::Deserialize)]
pub enum Direction {
Up = 0,
Down = 1,
Sideways = 2,
}
There's a significant number of JSON files I have to parse into these messages. Those are tens of thousands of lines long, but when this field appears it looks like:
{ "direction": "up" }
So, in short, its deserialized format is a string, serialized is i32.
If I just run that, and try to parse JSON, I get:
thread 'tests::parse_json' panicked at 'Failed to parse: "data/my_data.json": Error("invalid type: string \"up\", expected i32", line: 132, column: 23)
That, of course, makes sense - there's no reflection to guide deserialization from "up" to 0.
Question: How can I set up serde to parse those strings into their matching integer values? I've read the serde docs thoroughly, and it appears I might need to write a custom deserializer for this, although that seems like overkill.
I've tried a few different serde attributes, like:
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)]
#[repr(i32)]
#[derive(serde_derive::Deserialize)]
#[serde(from = "&str")] // This line
pub enum Direction {
Up = 0,
Down = 1,
Sideways = 2,
}
with this function:
impl From<&str> for Direction {
fn from(item: &str) -> Self {
match item {
"up" => Self::Up,
"down" => Self::Down,
"sideways" => Self::Sideways,
_ => panic!("Invalid value for Direction: {}", item),
}
}
}
but, despite what the serde docs tell me, that method doesn't even get called (but compilation succeeds).
I also tried a field attribute on the field which is a Direction:
#[serde(deserialize_with = \"super::super::common::Direction::from\")]
but that of course wants a different signature than just &str: the trait std::convert::From<__D> is not implemented for common::Direction
Do I just have to write a custom deserializer? Seems like a common enough use case that there would be a pattern to use.
Note: this is the opposite problem of that solved by serde_repr. I didn't see a way to put it to work here.
I implemented my own deserializer, thanks to the guide in this answer. There's likely a simpler or more idiomatic approach out there, so if you know one, please share!
Serde attribute, set on the field rather than the Enum:
config.field_attribute(
"direction",
"#[serde(deserialize_with = \"super::super::common::Direction::from_str\")]"
);
Deserializer:
impl Direction {
pub fn from_str<'de, D>(deserializer: D) -> Result<i32, D::Error>
where
D: Deserializer<'de>,
{
let s: &str = Deserialize::deserialize(deserializer)?;
match s.to_lowercase().as_str() {
"up" => Ok(Self::Tx as i32),
"down" => Ok(Self::Down as i32),
"sideways" => Ok(Self::Sideways as i32),
_ => Err(de::Error::unknown_variant(s, &["up", "down", "sideways"])),
}
}
}