Deserializing TOML into vector of enum with values - rust

I'm trying to read a TOML file to create a struct that contains a vector of enums with associated values. Here's the sample code:
extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate toml;
use std::fs::File;
use std::io::Read;
#[derive(Debug, Deserialize, PartialEq)]
struct Actor {
name: String,
actions: Vec<Actions>,
}
#[derive(Debug, Deserialize, PartialEq)]
enum Actions {
Wait(usize),
Move { x: usize, y: usize },
}
fn main() {
let input_file = "./sample_actor.toml";
let mut file = File::open(input_file).unwrap();
let mut file_content = String::new();
let _bytes_read = file.read_to_string(&mut file_content).unwrap();
let actor: Actor = toml::from_str(&file_content).unwrap();
println!("Read actor {:?}", actor);
}
Cargo.toml
[dependencies]
serde_derive = "1.0.10"
serde = "1.0.10"
toml = "0.4.2"
sample_actor.toml
name = "actor1"
actions = [move 3 4, wait 20, move 5 6]
I know the file looks "wrong", but I have no idea how I should write the actions in the TOML file such that the crate would be able to recognize them as an enum with X number of values.
The error I get when running this example with cargo run is the following:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { inner: ErrorInner { kind: NumberInvalid, line: Some(1), col: 11, message: "", key: [] } }', /checkout/src/libcore/result.rs:906:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.
I know that I probably need to implement FromStr for my enum to convert a string into my enum, and I briefly know that custom deserializers can be implemented to deserialize in a specific way, but I'm not sure how these pieces fit together.
It seems an equivalent example using serde_json instead of TOML works straight out (using JSON files instead of TOML of course).
JSON version of the code:
extern crate serde;
extern crate serde_json;
#[macro_use]
extern crate serde_derive;
use serde_json::Error;
use std::fs::File;
use std::io::Read;
#[derive(Debug, Serialize, Deserialize)]
enum Foo {
bar(u32),
baz { x: u32, y: u32 },
}
#[derive(Debug, Serialize, Deserialize)]
struct Address {
street: String,
city: String,
nums: Vec<Foo>,
}
fn main() {
/*
let address = Address {
street: "10 Downing Street".to_owned(),
city: "London".to_owned(),
nums : vec![Foo::bar(1), Foo::baz{x : 2, y : 3}],
};
// Serialize it to a JSON string.
let j = serde_json::to_string(&address).unwrap();
// Print, write to a file, or send to an HTTP server.
println!("{}", j);
*/
let input_file = "./sample_address.json";
let mut file = File::open(input_file).unwrap();
let mut file_content = String::new();
let _bytes_read = file.read_to_string(&mut file_content).unwrap();
let address: Address = serde_json::from_str(&file_content).unwrap();
println!("{:?}", address);
}
The JSON data read/written in this example is:
Address { street: "10 Downing Street", city: "London", nums: [bar(1), baz { x: 2, y: 3 }] }
Maybe the TOML crate can't support my use-case?

Serde has lots of options for serializing enums. One that works for your case:
use serde::{Deserialize, Serialize}; // 1.0.117
use toml; // 0.5.7
#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(tag = "type", content = "args")]
enum Actions {
Wait(usize),
Move { x: usize, y: usize },
}
fn main() {
let a_wait = Actions::Wait(5);
println!("{}", toml::to_string(&a_wait).unwrap());
let a_move = Actions::Move { x: 1, y: 1 };
println!("{}", toml::to_string(&a_move).unwrap());
}
type = "Wait"
args = 5
type = "Move"
[args]
x = 1
y = 1

Related

I can not deserialize json String to struct with serde

I have trouble deserializing json to struct.
Here is code that simulates my problem:
use std::sync::mpsc;
use std::sync::mpsc::{Receiver, Sender };
extern crate serde_json;
extern crate serde;
#[macro_use]
extern crate serde_derive;
#[derive(Deserialize, Debug)]
struct Foo<'a> {
a: u32,
b: u32,
c: &'a str,
}
fn main() {
let (msg_tx, msg_rx): (Sender<Foo>, Receiver<Foo>) = mpsc::channel();
{
let js = r#"{"a":33, "b":44, "c": "ss"}"#; // initially I have a json String, here I simulate a problem
let js_string = String::from(js);
let f = test(js_string.as_str());
msg_tx.send(f);
}
}
fn test(js: &str) -> Foo {
let foo: Foo = serde_json::from_str(js).unwrap();
foo
}
Running that code results to the error 'js' does not live long enough.
I know that changing the type of Foo c field to String will resolve the problem, but I would like to know if there is a another solution.
The reason for this error is the way serde crate works in that situation, - it uses inside returned foo variable a reference to original variable, which is js_string, so it goes out of scope just after calling msg_tx.send(f); but f has a reference to it and lives longer than that scope.
I'm still a rust beginner and want to master lifetime concept. I tried to fix my problem with function wrapper to set right lifetime, but failed.
You have to ensure that js_string lives longer than the channel, to ensure this you can create a scope for "working" with the channel:
use std::sync::mpsc;
use std::sync::mpsc::{Receiver, Sender };
extern crate serde_json;
extern crate serde;
#[macro_use]
extern crate serde_derive;
#[derive(Deserialize, Debug)]
struct Foo<'a> {
a: u32,
b: u32,
c: &'a str,
}
fn main() {
let js = r#"{"a":33, "b":44, "c": "ss"}"#;
let js_string = String::from(js);
let f = test(js_string.as_str());
{
let (msg_tx, msg_rx): (Sender<Foo>, Receiver<Foo>) = mpsc::channel();
msg_tx.send(f);
} // channel is dropped here and js_string is no longer borrowed
} // js_string is dropped here
fn test(js: &str) -> Foo {
let foo: Foo = serde_json::from_str(js).unwrap();
foo
}

YAML enum object representation

I have this use case that I will explain in Rust. How do I represent equivalent in YAML
enum MainEnum {
Opt1(T1),
Opt2(T2)
}
struct T1 {
x: u32,
}
struct T2 {
y: bool
}
How do I represent this in YAML? I want YaML to either have the x field or y field depending on which Enum value is being selected in YAML.
This program does not run. Any ideas?
use serde_yaml;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
struct T1 {
x: bool,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
struct T2 {
y: u32,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
enum MainEnum {
Opt1(T1),
Opt2(T2),
}
fn main() {
let config = r#"
---
- Opt1
x: "true"
"#;
let me: MainEnum = serde_yaml::from_str(&config).unwrap();
println!("{:?}", me);
}
runtime error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Scan(ScanError { mark: Marker { index: 17, line: 4, col: 5 }, info: "mapping values are not allowed in this context" })', src/main.rs:27:25
The first issue seems to be the YAML syntax you use. For a map in YAML you need to terminate the parent with a : so
let config = r#"
---
- Opt1:
x: true
"#;
would be actually correct YAML. But since you are asking serde_yaml for one instance of MainEnum, not for a list (a vec, for example) of MainEnum, it does not expect a YAML sequence but a single entry:
fn main() {
let config = r#"
---
Opt1:
x: true
"#;
let me: MainEnum = serde_yaml::from_str(&config).unwrap();
println!("{:?}", me);
}
This parses: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=029279e1e6baa77611c2a36e7d5bc3a4

How to store arbitrary JSON object in sqlite using Diesel

I have an input JSON:
{"key1": "val1", "key2": 1}
and I want to store it in an sqlite DB to later respond to some API request with the exact same value.
This is my migration:
CREATE TABLE my_table (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
arbitrary_json TEXT NOT NULL
);
My Cargo.toml:
[package]
name = "diesel_playground"
version = "0.1.0"
authors = ["User <user#example.com>"]
edition = "2018"
[dependencies]
diesel = { version = "1.4" , features = ["sqlite"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
Using following code:
#[macro_use]
extern crate diesel;
mod schema;
use schema::my_table;
use serde::{Deserialize, Serialize};
use diesel::prelude::*;
use diesel::sqlite::SqliteConnection;
use std::env;
pub fn establish_connection() -> SqliteConnection {
let database_url = env::var("DATABASE_URL").expect("DATABASE_URL must be set");
SqliteConnection::establish(&database_url)
.expect(&format!("Error connecting to {}", database_url))
}
#[derive(Debug, Deserialize, Serialize, Queryable, Identifiable)]
#[table_name = "my_table"]
struct Item {
id: i32,
arbitrary_json: serde_json::Value,
}
#[derive(Debug, Deserialize, Serialize, Insertable, Queryable)]
#[table_name = "my_table"]
struct NewItem {
arbitrary_json: serde_json::Value,
}
fn main() {
let my_json = serde_json::json!({
"key1": "val1",
"key2": 1
});
let new_item = NewItem {
arbitrary_json: my_json,
};
let conn = establish_connection();
diesel::insert_into(my_table::table)
.values(&new_item)
.execute(&conn)
.expect("Error adding new item");
let my_item = my_table::table
.find(1)
.first::<Item>(&conn)
.expect("Error selecting id 1");
assert!(my_item.arbitrary_json == new_item.arbitrary_json);
}
I get the following error:
error[E0277]: the trait bound `serde_json::value::Value: diesel::Expression` is not satisfied
--> src/main.rs:27:41
|
27 | #[derive(Debug, Deserialize, Serialize, Insertable, Queryable)]
| ^^^^^^^^^^ the trait `diesel::Expression` is not implemented for `serde_json::value::Value`
I could create a model with String JSON representation and derive From / Into for API input type, but I don't want to insert .into() everywhere in my code now. A DRY solution would to be to do this as I proposed in the code attached.
In my answer I will keep JSON object in the DB in a string representation (schema: TEXT).
For our unsupported type we need following traits implemented: ToSql, FromSql, AsExpression and FromSqlRow.
Now, since one can not implement a trait for a type coming from an external crate it has to be wrapped into a single element tuple:
struct MyJsonType(serde_json::Value)
Now FromSql trait implementation :
impl FromSql<Text, DB> for MyJsonType {
fn from_sql(
bytes: Option<&<diesel::sqlite::Sqlite as Backend>::RawValue>,
) -> deserialize::Result<Self> {
let t = <String as FromSql<Text, DB>>::from_sql(bytes)?;
Ok(Self(serde_json::from_str(&t)?))
}
}
And ToSql trait implementation:
impl ToSql<Text, DB> for MyJsonType {
fn to_sql<W: Write>(&self, out: &mut Output<W, DB>) -> serialize::Result {
let s = serde_json::to_string(&self.0)?;
<String as ToSql<Text, DB>>::to_sql(&s, out)
}
}
Now the remaining traits can be derived using macros:
#[derive(AsExpression, Debug, Deserialize, Serialize, FromSqlRow)]
#[sql_type = "Text"]
struct MyJsonType(serde_json::Value);
It should be fine to use our new type now:
#[derive(Debug, Deserialize, Serialize, Queryable, Identifiable)]
#[table_name = "my_table"]
struct Item {
id: i32,
arbitrary_json: MyJsonType
}

Can Serde deserialize JSON to one of a set of types depending on the value of a field?

I have a group of different messages that come in as JSON and can be distinguished based on a single field, but then each variant has a different collection of secondary fields:
#[derive(Debug, Serialize, Deserialize)]
struct MessageOne {
///op will always be "one"
op: String,
x: f64,
y: f64,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageTwo {
///op will always be "two"
op: String,
a: f64,
b: i64,
}
The different message types are routed to different processing functions (e.g. process_message_one, process_message_two, etc). Is there an elegant or idiomatic way to automatically select the correct message sub-type? Currently I've defined a generic message:
#[derive(Debug, Serialize, Deserialize)]
struct MessageGeneric {
op: String,
}
then parse the incoming JSON into the MessageGeneric, read the op field and then deserialize again, matching on op to select the correct message type. Full example:
#![allow(unused)]
extern crate serde; // 1.0.78
extern crate serde_json; // 1.0.27
#[macro_use]
extern crate serde_derive;
use std::collections::HashMap;
#[derive(Debug, Serialize, Deserialize)]
struct MessageGeneric {
op: String,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageOne {
///op will always be "one"
op: String,
x: f64,
y: f64,
}
#[derive(Debug, Serialize, Deserialize)]
struct MessageTwo {
///op will always be "two"
op: String,
a: f64,
b: f64,
}
fn process_message_one(m: &MessageOne) {
println!("Processing a MessageOne: {:?}", m);
}
fn process_message_two(m: &MessageTwo) {
println!("Processing a MessageTwo: {:?}", m);
}
fn main() {
let data = r#"{
"op": "one",
"x": 1.0,
"y": 2.0
}"#;
let z: MessageGeneric = serde_json::from_str(data).unwrap();
match z.op.as_ref() {
"one" => {
let zp: MessageOne = serde_json::from_str(data).unwrap();
process_message_one(&zp);
},
"two" => {
let zp: MessageTwo = serde_json::from_str(data).unwrap();
process_message_two(&zp);
},
_ => println!("Unknown Message Type")
}
}
I've seen Serde's enum representations but it was unclear to me if/how that would be applied in this case. The messages coming in are defined by an external API, so I can't control their content beyond knowing what the variants are.
There is no point to keep "one" or "two" in your structure MessageOne and MessageTwo: if you have constructed this structure you already know if it is message one or message two.
extern crate serde; // 1.0.78
extern crate serde_json; // 1.0.27
#[macro_use]
extern crate serde_derive;
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag = "op")]
enum Message {
#[serde(rename = "one")]
One { x: f64, y: f64 },
#[serde(rename = "two")]
Two { a: f64, b: f64 },
}
fn process_message(message: &Message) {
println!("Processing a : {:?}", message);
}
use serde_json::Error;
fn main() -> Result<(), Error> {
let data = r#"{
"op": "one",
"x": 1.0,
"y": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
let data = r#"{
"op": "two",
"a": 1.0,
"b": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
let data = r#"{
"op": "42",
"i": 1.0,
"j": 2.0
}"#;
let message: Message = serde_json::from_str(data)?;
process_message(&message);
Ok(())
}
Standard Output
Processing a : One { x: 1.0, y: 2.0 }
Processing a : Two { a: 1.0, b: 2.0 }
Standard Error
Error: Error("unknown variant `42`, expected `one` or `two`", line: 2, column: 18)

How do I store a result using Serde Zero-copy deserialization of a Futures-enabled Hyper Chunk?

I'm using futures, tokio, hyper, and serde_json to request and deserialize some data that I need to hold until my next request. My initial thought was to make a struct containing the hyper::Chunk and the deserialized data that borrows from the Chunk, but couldn't get the lifetimes right. I tried using the rental crate, but I can't get this to work either. Perhaps I'm using the 'buffer lifetime before declaring the buffer Vec, but maybe I've messed something else up:
#[rental]
pub struct ChunkJson<T: serde::de::Deserialize<'buffer>> {
buffer: Vec<u8>,
json: T
}
Is there some way to make the lifetimes right or should I just use DeserializeOwned and give up on zero-copy?
For more context, the following code works (periodically deserializing JSON from two URLs, retaining the results so we can do something with them both). I'd like to change my X and Y types to use Cow<'a, str> for their fields, changing from DeserializeOwned to Deserialize<'a>. For this to work, I need to store the slice that has been deserialized for each, but I don't know how to do this. I'm looking for examples that use Serde's zero-copy deserialization and retain the result, or some idea for restructuring my code that would work.
#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
extern crate futures;
extern crate tokio_core;
extern crate tokio_periodic;
extern crate hyper;
use std::collections::HashMap;
use std::error::Error;
use futures::future;
use futures::Future;
use futures::stream::Stream;
use hyper::Client;
fn stream_json<'a, T: serde::de::DeserializeOwned + Send + 'a>
(handle: &tokio_core::reactor::Handle,
url: String,
period: u64)
-> Box<Stream<Item = T, Error = Box<Error>> + 'a> {
let client = Client::new(handle);
let timer = tokio_periodic::PeriodicTimer::new(handle).unwrap();
timer
.reset(::std::time::Duration::new(period, 0))
.unwrap();
Box::new(futures::Stream::zip(timer.from_err::<Box<Error>>(), futures::stream::unfold( (), move |_| {
let uri = url.parse::<hyper::Uri>().unwrap();
let get = client.get(uri).from_err::<Box<Error>>().and_then(|res| {
res.body().concat().from_err::<Box<Error>>().and_then(|chunks| {
let p: Result<T, Box<Error>> = serde_json::from_slice::<T>(chunks.as_ref()).map_err(|e| Box::new(e) as Box<Error>);
match p {
Ok(json) => future::ok((json, ())),
Err(err) => future::err(err)
}
})
});
Some(get)
})).map(|x| { x.1 }))
}
#[derive(Serialize, Deserialize, Debug)]
pub struct X {
foo: String,
}
#[derive(Serialize, Deserialize, Debug)]
pub struct Y {
bar: String,
}
fn main() {
let mut core = tokio_core::reactor::Core::new().unwrap();
let handle = core.handle();
let x_stream = stream_json::<HashMap<String, X>>(&handle, "http://localhost/X".to_string(), 2);
let y_stream = stream_json::<HashMap<String, Y>>(&handle, "http://localhost/Y".to_string(), 5);
let mut xy_stream = x_stream.merge(y_stream);
let mut last_x = HashMap::new();
let mut last_y = HashMap::new();
loop {
match core.run(futures::Stream::into_future(xy_stream)) {
Ok((Some(item), stream)) => {
match item {
futures::stream::MergedItem::First(x) => last_x = x,
futures::stream::MergedItem::Second(y) => last_y = y,
futures::stream::MergedItem::Both(x, y) => {
last_x = x;
last_y = y;
}
}
println!("\nx = {:?}", &last_x);
println!("y = {:?}", &last_y);
// Do more stuff with &last_x and &last_y
xy_stream = stream;
}
Ok((None, stream)) => xy_stream = stream,
Err(_) => {
panic!("error");
}
}
}
}
When trying to solve a complicated programming problem, it's very useful to remove as much as you can. Take your code and remove what you can until the problem goes away. Tweak your code a bit and keep removing until you can't any more. Then, turn the problem around and build from the smallest piece and work back to the error. Doing both of these will show you where the problem lies.
First, let's make sure we deserialize correctly:
extern crate serde;
extern crate serde_json;
#[macro_use]
extern crate serde_derive;
use std::borrow::Cow;
#[derive(Debug, Deserialize)]
pub struct Example<'a> {
#[serde(borrow)]
name: Cow<'a, str>,
key: bool,
}
impl<'a> Example<'a> {
fn info(&self) {
println!("{:?}", self);
match self.name {
Cow::Borrowed(_) => println!("Is borrowed"),
Cow::Owned(_) => println!("Is owned"),
}
}
}
fn main() {
let data: Vec<_> = br#"{"key": true, "name": "alice"}"#.to_vec();
let decoded: Example = serde_json::from_slice(&data).expect("Couldn't deserialize");
decoded.info();
}
Here, I forgot to add the #[serde(borrow)] attribute, so I'm glad I did this test!
Next, we can introduce the rental crate:
#[macro_use]
extern crate rental;
rental! {
mod holding {
use super::*;
#[rental]
pub struct VecHolder {
data: Vec<u8>,
parsed: Example<'data>,
}
}
}
fn main() {
let data: Vec<_> = br#"{"key": true, "name": "alice"}"#.to_vec();
let holder = holding::VecHolder::try_new(data, |data| {
serde_json::from_slice(data)
});
let holder = match holder {
Ok(holder) => holder,
Err(_) => panic!("Unable to construct rental"),
};
holder.rent(|example| example.info());
// Make sure we can move the data and it's still valid
let holder2 = { holder };
holder2.rent(|example| example.info());
}
Next we try to create a rental of Chunk:
#[rental]
pub struct ChunkHolder {
data: Chunk,
parsed: Example<'data>,
}
Unfortunately, this fails:
--> src/main.rs:29:1
|
29 | rental! {
| ^
|
= help: message: Field `data` must have an angle-bracketed type parameter or be `String`.
Oops! Checking the docs for rental, we can add #[target_ty_hack="[u8]"] to the data field. This leads to:
error[E0277]: the trait bound `hyper::Chunk: rental::__rental_prelude::StableDeref` is not satisfied
--> src/main.rs:29:1
|
29 | rental! {
| ^ the trait `rental::__rental_prelude::StableDeref` is not implemented for `hyper::Chunk`
|
= note: required by `rental::__rental_prelude::static_assert_stable_deref`
That's annoying; since we can't implement that trait for Chunk, we just need to box Chunk, proving that it has a stable address:
#[rental]
pub struct ChunkHolder {
data: Box<Chunk>,
parsed: Example<'data>,
}
I also looked to see if there is a way to get a Vec<u8> back out of Chunk, but it doesn't appear to exist. That would have been another solution with less allocation and indirection.
At this point, "all" that's left is to integrate this back into the futures code. It's a lot of work for anyone but you to recreate that, but I don't foresee any obvious problems in doing so.

Resources