This question already has answers here:
How to deserialize a JSON file which contains null values using Serde?
(3 answers)
Closed 1 year ago.
I am using a structure to receive data from a http request body in rust. This is the struct define:
use rocket::serde::Deserialize;
use rocket::serde::Serialize;
#[derive(Debug, PartialEq, Eq, Deserialize, Serialize)]
#[allow(non_snake_case)]
pub struct LearningWordRequest {
pub wordId: i64,
pub word: String,
pub wordStatus: i32
}
but I have to put all of the fields into the body to parse, if I did not do it like that. The server will show:
2021-11-10T15:02:43 [WARN] - `Json < LearningWordRequest >` data guard failed: Parse("{\"word\":\"hood\",\"wordId\":-1}", Error("missing field `wordStatus`", line: 1, column: 27)).
is it possible to make the parse just ignore the field if I did not put this field? Sometimes it is no need to transfer all fields in the http body, this is my server side receive code:
#[post("/v1/add",data = "<record>")]
pub fn add_learning_word(record: Json<LearningWordRequest>) -> content::Json<String> {
let res = ApiResponse {
result: "ok",
..Default::default()
};
let response_json = serde_json::to_string(&res).unwrap();
return content::Json(response_json);
}
Change the struct member to be optional:
#[derive(Debug, PartialEq, Eq, Deserialize, Serialize)]
#[allow(non_snake_case)]
pub struct LearningWordRequest {
pub wordId: i64,
pub word: String,
pub wordStatus: Option<i32>
}
For example:
fn main() {
let present = r#" {
"wordId": 43,
"word": "foo",
"wordStatus": 1337
}"#;
let absent = r#" {
"wordId": 43,
"word": "foo"
}"#;
println!("{:?}", serde_json::from_str::<LearningWordRequest>(present));
println!("{:?}", serde_json::from_str::<LearningWordRequest>(absent));
}
Outputs:
Ok(LearningWordRequest { wordId: 43, word: "foo", wordStatus: Some(1337) })
Ok(LearningWordRequest { wordId: 43, word: "foo", wordStatus: None })
Related
I'm trying to wrap an API using rust, so long no problems but one endpoint is giving me a big pain in the head.
the endpoint returns a valid JSON as the other endpoints from this API.
The problem is when I try to work with the response I get the following error:
error decoding response body: trailing characters at line 1 column 146
Get function:
pub fn get<T: DeserializeOwned>(&self, endpoint: API, request: Option<String>) -> Result<T> {
let mut url: String = format!("{}{}", self.host, String::from(endpoint));
if let Some(request) = request {
if !request.is_empty() {
url.push_str(format!("?{}", request).as_str());
}
}
let client = &self.inner_client;
let response = client.get(url.as_str()).send()?;
self.handler(response)
}
Handler function:
fn handler<T: DeserializeOwned>(&self, response: Response) -> Result<T> {
match response.status() {
StatusCode::OK => {
let l = response.json::<T>()?;
Ok(l)
},
s => {
bail!(format!("Received response: {:?}", s));
}
}
}
Enpoint funcion:
pub fn get_depth<S>(&self, symbol: S) -> Result<OrderBook>
where
S: Into<String>,
{
let mut parameters: BTreeMap<String, String> = BTreeMap::new();
parameters.insert("symbol".into(), symbol.into());
let request = build_request(parameters);
self.client.get(API::Spot(Spot::Depth), Some(request))
}
Models:
#[derive(Debug, Serialize, Deserialize, Clone)]
#[serde(rename_all = "camelCase")]
pub struct OrderBook {
pub last_update_id: u64,
pub asks: Vec<Asks>,
pub bids: Vec<Bids>,
}
#[derive(PartialEq, Debug, Serialize, Deserialize, Clone)]
pub struct Bids {
#[serde(with = "string_or_float")]
pub price: f64,
#[serde(with = "string_or_float")]
pub qty: f64,
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct Asks {
#[serde(with = "string_or_float")]
pub price: f64,
#[serde(with = "string_or_float")]
pub qty: f64,
}
API response:
{
"lastUpdateId": 1595750078,
"bids": [
[
"0.00001661",
"24250.00000000",
[]
],
[
"0.00001660",
"65159.00000000",
[]
]
],
"asks": [
[
"0.00001662",
"26397.00000000",
[]
],
[
"0.00001663",
"54421.00000000",
[]
]
]
}
I was testing this mocking the API response and removing the empty arrays from asks and bids and it worked flawless. I don't know why the [] is a problem and I can figure out how to solve this problem. So really I will appreciate any help.
Your problem boils down to matching your rust struct with the response structure. I'm using the example with String from your playground link.
Response
{
"lastUpdateId":1595750078,
"bids":[
[
"0.00001661",
"24250.00000000",
[
]
],
[
"0.00001660",
"65159.00000000",
[
]
]
],
"asks":[
[
"0.00001662",
"26397.00000000"
],
[
"0.00001663",
"54421.00000000"
]
]
}
Your Structs
#[derive(Debug, Serialize, Deserialize, Clone)]
#[serde(rename_all = "camelCase")]
pub struct OrderBook {
pub last_update_id: u64,
pub asks: Vec<Asks>,
pub bids: Vec<Bids>,
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct Bids {
pub price: String,
pub qty: String,
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct Asks {
pub price: String,
pub qty: String,
}
Now if you look at the bids json, the empty array is not represented in the Bids structure in your code. Hence calling serde_json::from_str::<OrderBook>(json) fails, as it's encountering a new array where nothing is expected anymore.
Solution
Introduce a new (optional) field in Bids, representing the array. I cannot say which structure this would be, as your example only shows the empty representation. pub x: Vec<()> would also work, though I'm certain the API might sent real data if available.
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct Bids {
pub price: String,
pub qty: String,
pub something_else: Vec<()>, // or Vec<AnotherStruct>
}
I am using rust rocket as the http server side. On the server side, I define a rust struct like this to receive the data from client side:
#[derive(Deserialize, Serialize)]
#[allow(non_snake_case)]
pub struct PlayRecordRequest {
pub id: i64,
pub title: String,
pub url: String,
pub mvId: i64,
pub album: MusicAlbum,
pub artist: Vec<Artist>,
}
and recieve in the http controller like this:
#[post("/user/v1/save-play-record", data = "<record>")]
pub fn save_play_record<'r>(
record: Json<PlayRecordRequest>,
) -> content::Json<String> {
info!("save play record,title:{}",record.title);
save_play_record_impl(record);
let res = ApiResponse {
result: "ok".to_string(),
..Default::default()
};
let result_json = serde_json::to_string(&res).unwrap();
return content::Json(result_json);
}
the problem is that when the client side did not have some fields, the code run into error. is it possible to auto fit the field for Deserialize, if the client have the field, Deserialize it normally, if the client did not contains some fields, just ignore it and do not run into error. I read the official document of serde and find the skip annotation. But the annotation just use to mark the field should be ignored, what I want is that the struct could auto fit all field exists or not. is it possible to do like this?
There are two ways to handle this.
First is with options. Options imply that the data may or may not exist. If the data is null or missing it convert to an Option::none value. You can preserve the lack of data on serialization if you add #[serde(skip_serializing_if = "Option::is_none")]
Second option is to apply defaults to the value if the data is missing. Although from your use case this doesn't seem to be ideal.
Here is a code snippet of the two cases that you can run on https://play.rust-lang.org/:
use::serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize, Debug)]
#[allow(non_snake_case)]
pub struct Foo {
#[serde(skip_serializing_if = "Option::is_none")]
pub bar: Option<i64>,
#[serde(default = "default_baz")]
pub baz: i64,
}
fn default_baz()-> i64{
3
}
fn main(){
let data = r#"{"bar": null, "baz": 1}"#;
let v: Foo = serde_json::from_str(data).unwrap();
println!("{:#?}", v);
let s = serde_json::to_string(&v).unwrap();
println!("Don't serialize None values: {:#?}", s);
let data = r#"{"missing_bar": null}"#;
let v: Foo = serde_json::from_str(data).unwrap();
println!("{:#?}", v)
}
I wonder whether there's a way to preserve the original String using serde_json? Consider this example:
#[derive(Debug, Serialize, Deserialize)]
struct User {
#[serde(skip)]
pub raw: String,
pub id: u64,
pub login: String,
}
{
"id": 123,
"login": "johndoe"
}
My structure would end up containing such values:
User {
raw: String::from(r#"{"id": 123,"login": "johndoe"}"#),
id: 1,
login: String::from("johndoe")
}
Currently, I'm doing that by deserializing into Value, then deserializing this value into the User structure and assigning Value to the raw field, but that doesn't seem right, perhaps there's a better way to do so?
This solution uses the RawValue type from serde_json to first get the original input string. Then a new Deserializer is created from that String to deserialize the User type.
This solution can work with readers, by using Box<serde_json::value::RawValue> as an intermediary type and it can also work with struct which borrow from the input, by using &'de serde_json::value::RawValue as the intermediary. You can test it in the solution by (un-)commenting the borrow field.
use std::marker::PhantomData;
#[derive(Debug, serde::Serialize, serde::Deserialize)]
#[serde(remote = "Self")]
struct User<'a> {
#[serde(skip)]
pub raw: String,
pub id: u64,
pub login: String,
// Test for borrowing input data
// pub borrow: &'a str,
#[serde(skip)]
pub ignored: PhantomData<&'a ()>,
}
impl serde::Serialize for User<'_> {
fn serialize<S: serde::Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
Self::serialize(self, serializer)
}
}
impl<'a, 'de> serde::Deserialize<'de> for User<'a>
where
'de: 'a,
{
fn deserialize<D: serde::Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
use serde::de::Error;
// Deserializing a `&'a RawValue` would also work here
// but then you loose support for deserializing from readers
let raw: Box<serde_json::value::RawValue> = Box::deserialize(deserializer)?;
// Use this line instead if you have a struct which borrows from the input
// let raw = <&'de serde_json::value::RawValue>::deserialize(deserializer)?;
let mut raw_value_deserializer = serde_json::Deserializer::from_str(raw.get());
let mut user =
User::deserialize(&mut raw_value_deserializer).map_err(|err| D::Error::custom(err))?;
user.raw = raw.get().to_string();
Ok(user)
}
}
fn main() {
// Test serialization
let u = User {
raw: String::new(),
id: 456,
login: "USERNAME".to_string(),
// Test for borrowing input data
// borrow: "foobar",
ignored: PhantomData,
};
let json = serde_json::to_string(&u).unwrap();
println!("{}", json);
// Test deserialization
let u2: User = serde_json::from_str(&json).unwrap();
println!("{:#?}", u2);
}
Test on the Playground.
I have json documents, that may contain objects which have keys that refer to empty objects, like this
{
"mylist": {
"foo": {},
"bar": {}
}
}
I would like to deserialize them to a Vec of Strings (and serialize it back to the format above later)
pub struct MyStruct {
#[serde(skip_serializing_if = "Option::is_none")]
pub my_list: Option<Vec<String>>; // should contain "foo", "bar"
}
How can I do that with serde?
You need to write your own deserializing method, and use deserialize_with or implement it directly for your type:
use serde::Deserialize; // 1.0.127
use serde::Deserializer;
use serde_json;
use std::collections::HashMap; // 1.0.66
#[derive(Deserialize, Debug)]
pub struct MyStruct {
#[serde(deserialize_with = "deserialize_as_vec", alias = "mylist")]
pub my_list: Vec<String>,
}
#[derive(Deserialize)]
struct DesHelper {}
fn deserialize_as_vec<'de, D>(deserializer: D) -> Result<Vec<String>, D::Error>
where
D: Deserializer<'de>,
{
let data: HashMap<String, DesHelper> = HashMap::<String, DesHelper>::deserialize(deserializer)?;
Ok(data.keys().cloned().collect())
}
fn main() {
let example = r#"
{
"mylist": {
"foo": {},
"bar": {}
}
}"#;
let deserialized: MyStruct = serde_json::from_str(&example).unwrap();
println!("{:?}", &deserialized);
}
Results:
MyStruct { my_list: ["foo", "bar"] }
Playground
Notice the use of the helper struct for the empty parts of them. Code is pretty straight forward, you basically deserialize a map and then just take the keys that is what you need.
I have a group of different messages that come in as JSON and can be distinguished based on a single field, but then each variant has a different collection of secondary fields:
#[derive(Debug, Serialize, Deserialize)]
struct MessageOne {
///op will always be "one"
op: String,
x: f64,
y: f64,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageTwo {
///op will always be "two"
op: String,
a: f64,
b: i64,
}
The different message types are routed to different processing functions (e.g. process_message_one, process_message_two, etc). Is there an elegant or idiomatic way to automatically select the correct message sub-type? Currently I've defined a generic message:
#[derive(Debug, Serialize, Deserialize)]
struct MessageGeneric {
op: String,
}
then parse the incoming JSON into the MessageGeneric, read the op field and then deserialize again, matching on op to select the correct message type. Full example:
#![allow(unused)]
extern crate serde; // 1.0.78
extern crate serde_json; // 1.0.27
#[macro_use]
extern crate serde_derive;
use std::collections::HashMap;
#[derive(Debug, Serialize, Deserialize)]
struct MessageGeneric {
op: String,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageOne {
///op will always be "one"
op: String,
x: f64,
y: f64,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageTwo {
///op will always be "two"
op: String,
a: f64,
b: f64,
}
fn process_message_one(m: &MessageOne) {
println!("Processing a MessageOne: {:?}", m);
}
fn process_message_two(m: &MessageTwo) {
println!("Processing a MessageTwo: {:?}", m);
}
fn main() {
let data = r#"{
"op": "one",
"x": 1.0,
"y": 2.0
}"#;
let z: MessageGeneric = serde_json::from_str(data).unwrap();
match z.op.as_ref() {
"one" => {
let zp: MessageOne = serde_json::from_str(data).unwrap();
process_message_one(&zp);
},
"two" => {
let zp: MessageTwo = serde_json::from_str(data).unwrap();
process_message_two(&zp);
},
_ => println!("Unknown Message Type")
}
}
I've seen Serde's enum representations but it was unclear to me if/how that would be applied in this case. The messages coming in are defined by an external API, so I can't control their content beyond knowing what the variants are.
There is no point to keep "one" or "two" in your structure MessageOne and MessageTwo: if you have constructed this structure you already know if it is message one or message two.
extern crate serde; // 1.0.78
extern crate serde_json; // 1.0.27
#[macro_use]
extern crate serde_derive;
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag = "op")]
enum Message {
#[serde(rename = "one")]
One { x: f64, y: f64 },
#[serde(rename = "two")]
Two { a: f64, b: f64 },
}
fn process_message(message: &Message) {
println!("Processing a : {:?}", message);
}
use serde_json::Error;
fn main() -> Result<(), Error> {
let data = r#"{
"op": "one",
"x": 1.0,
"y": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
let data = r#"{
"op": "two",
"a": 1.0,
"b": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
let data = r#"{
"op": "42",
"i": 1.0,
"j": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
Ok(())
}
Standard Output
Processing a : One { x: 1.0, y: 2.0 }
Processing a : Two { a: 1.0, b: 2.0 }
Standard Error
Error: Error("unknown variant `42`, expected `one` or `two`", line: 2, column: 18)