I can not deserialize json String to struct with serde - rust

I have trouble deserializing json to struct.
Here is code that simulates my problem:
use std::sync::mpsc;
use std::sync::mpsc::{Receiver, Sender };
extern crate serde_json;
extern crate serde;
#[macro_use]
extern crate serde_derive;
#[derive(Deserialize, Debug)]
struct Foo<'a> {
a: u32,
b: u32,
c: &'a str,
}
fn main() {
let (msg_tx, msg_rx): (Sender<Foo>, Receiver<Foo>) = mpsc::channel();
{
let js = r#"{"a":33, "b":44, "c": "ss"}"#; // initially I have a json String, here I simulate a problem
let js_string = String::from(js);
let f = test(js_string.as_str());
msg_tx.send(f);
}
}
fn test(js: &str) -> Foo {
let foo: Foo = serde_json::from_str(js).unwrap();
foo
}
Running that code results to the error 'js' does not live long enough.
I know that changing the type of Foo c field to String will resolve the problem, but I would like to know if there is a another solution.
The reason for this error is the way serde crate works in that situation, - it uses inside returned foo variable a reference to original variable, which is js_string, so it goes out of scope just after calling msg_tx.send(f); but f has a reference to it and lives longer than that scope.
I'm still a rust beginner and want to master lifetime concept. I tried to fix my problem with function wrapper to set right lifetime, but failed.

You have to ensure that js_string lives longer than the channel, to ensure this you can create a scope for "working" with the channel:
use std::sync::mpsc;
use std::sync::mpsc::{Receiver, Sender };
extern crate serde_json;
extern crate serde;
#[macro_use]
extern crate serde_derive;
#[derive(Deserialize, Debug)]
struct Foo<'a> {
a: u32,
b: u32,
c: &'a str,
}
fn main() {
let js = r#"{"a":33, "b":44, "c": "ss"}"#;
let js_string = String::from(js);
let f = test(js_string.as_str());
{
let (msg_tx, msg_rx): (Sender<Foo>, Receiver<Foo>) = mpsc::channel();
msg_tx.send(f);
} // channel is dropped here and js_string is no longer borrowed
} // js_string is dropped here
fn test(js: &str) -> Foo {
let foo: Foo = serde_json::from_str(js).unwrap();
foo
}

Related

Deserialize JSON starting with array to use with restson

I have a struct:
#[derive(Serialize,Deserialize,Debug)]
struct Post {
#[serde(rename(deserialize = "userId"))]
user_id: i32,
id: i32,
title: String,
body: String,
}
I need to deserialize JSON to Vec<Post>:
extern crate restson;
extern crate serde;
extern crate serde_derive;
use std::fs;
use std::path::Path;
use restson::{RestClient, RestPath, Error};
use serde_derive::{Serialize, Deserialize};
const URI: &str ="https://jsonplaceholder.typicode.com/";
impl RestPath<()> for Vec<Post> {
fn get_path(_: ()) -> Result<String, Error> {
Ok(String::from("posts"))
}
}
fn main() {
let mut client = RestClient::new(URI).unwrap();
let posts: Vec<Post> = client.get(()).unwrap();
println!("{:?}", posts.len());
}
I'm totally new, so please help me.

How to deserialize messages within Actix actors?

My intention is receiving events through WebSockets and use them on the closures of main.
This works when the messages are pure text (String), but the idea is deserializing that text into some structs.
In this example I've added only Data, Error and Event, but in other cases, it could be different, so I've used generics to do that, but I'm a little lost. The compiler has suggested several things that I've tried, but I don't know how to "force" that the message is casted into a specific type (Data in this example, but EventManager could be used on other parts, so it should be generic).
I've attached this code, that tries to show my idea, although it doesn't compile:
events.rs:
use actix::*;
use actix_web::ws::{Client, Message, ProtocolError};
use futures::Future;
use serde::de;
use serde_json::from_str;
struct MyActor<T> {
manager: EventManager<T>,
}
impl<T: 'static> Actor for MyActor<T> {
type Context = Context<Self>;
}
impl<T: 'static> StreamHandler<Message, ProtocolError> for MyActor<T> {
fn handle(&mut self, msg: Message, _ctx: &mut Context<Self>) {
match msg {
Message::Text(text) => {
debug!("Received {}", text);
for idx in 0..self.manager.events.len() {
let data =
from_str(&text).expect(&format!("Error when deserializing {:?}", text));
(self.manager.events[idx].handler)(data)
}
}
_ => panic!(),
}
}
}
pub struct Event<T> {
handler: Box<Fn(T) + 'static>,
}
pub struct EventManager<T> {
events: Vec<Event<T>>,
}
impl<T: 'static> EventManager<T>
where
T: serde::Deserialize<'static>,
{
pub fn new() -> Self {
Self { events: vec![] }
}
pub fn capture<F>(&mut self, function: F)
where
F: for<'h> Fn(T) + 'static,
{
let event = Event {
handler: Box::new(function),
};
self.events.push(event);
}
pub fn run(self) {
let runner = System::new("example");
debug!("run");
Arbiter::spawn(
Client::new("example")
.connect()
.map(|(reader, _writer)| {
MyActor::create(|ctx| {
MyActor::add_stream(reader, ctx);
MyActor { manager: self }
});
})
.map_err(|err| {}),
);
runner.run();
}
}
main.rs:
#[macro_use]
extern crate log;
extern crate actix;
extern crate actix_web;
extern crate env_logger;
extern crate futures;
extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;
pub mod events;
use actix::*;
use serde::de;
use serde::de::{Deserialize, Deserializer};
use events::EventManager;
#[derive(Debug, Message, Serialize, Deserialize)]
#[serde(untagged)]
pub enum Data {
Error(Error),
Event(Event),
}
#[derive(Debug, Serialize, Deserialize)]
pub struct Error {
message: String,
code: String,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct Event {
name: String,
content: String,
}
fn main() {
env_logger::init();
let mut client = EventManager::<Data>new();
client.capture(|data| debug!("event: {:?}", data));
client.run();
}
All the code could be see in https://github.com/foochi/how-deserialize-within-actix
There are some fixes to get it compile.
The trick to get it compile is to use Higher-Rank Trait Bounds (HTRB) trait bounds instead of declaring 'static lifetimes.
Follow the compiler suggestion and bind the T: serde::Deserialize<'_> trait:
impl<T> StreamHandler<Message, ProtocolError> for MyActor<T>
where
for<'de> T: serde::Deserialize<'de> + 'static,
Then change also the Deserialize<'static> trait bound associated to theEventManager impl with a HTRB trait bound to get it compatible with the requirement of StreamHandler implementation:
impl<T: 'static> EventManager<T>
where
for<'de> T: serde::Deserialize<'de>,
Finally, if you correct the line the create the client with the right sintax:
let mut client: EventManager<Data> = EventManager::new();
the example code should compile.
Note: For capture the use of a Higher Trait Bound for declaring the Fn requirement is redundant, do simply:
pub fn capture<F>(&mut self, function: F)
where
F: Fn(T) + 'static,

Deserializing TOML into vector of enum with values

I'm trying to read a TOML file to create a struct that contains a vector of enums with associated values. Here's the sample code:
extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate toml;
use std::fs::File;
use std::io::Read;
#[derive(Debug, Deserialize, PartialEq)]
struct Actor {
name: String,
actions: Vec<Actions>,
}
#[derive(Debug, Deserialize, PartialEq)]
enum Actions {
Wait(usize),
Move { x: usize, y: usize },
}
fn main() {
let input_file = "./sample_actor.toml";
let mut file = File::open(input_file).unwrap();
let mut file_content = String::new();
let _bytes_read = file.read_to_string(&mut file_content).unwrap();
let actor: Actor = toml::from_str(&file_content).unwrap();
println!("Read actor {:?}", actor);
}
Cargo.toml
[dependencies]
serde_derive = "1.0.10"
serde = "1.0.10"
toml = "0.4.2"
sample_actor.toml
name = "actor1"
actions = [move 3 4, wait 20, move 5 6]
I know the file looks "wrong", but I have no idea how I should write the actions in the TOML file such that the crate would be able to recognize them as an enum with X number of values.
The error I get when running this example with cargo run is the following:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { inner: ErrorInner { kind: NumberInvalid, line: Some(1), col: 11, message: "", key: [] } }', /checkout/src/libcore/result.rs:906:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.
I know that I probably need to implement FromStr for my enum to convert a string into my enum, and I briefly know that custom deserializers can be implemented to deserialize in a specific way, but I'm not sure how these pieces fit together.
It seems an equivalent example using serde_json instead of TOML works straight out (using JSON files instead of TOML of course).
JSON version of the code:
extern crate serde;
extern crate serde_json;
#[macro_use]
extern crate serde_derive;
use serde_json::Error;
use std::fs::File;
use std::io::Read;
#[derive(Debug, Serialize, Deserialize)]
enum Foo {
bar(u32),
baz { x: u32, y: u32 },
}
#[derive(Debug, Serialize, Deserialize)]
struct Address {
street: String,
city: String,
nums: Vec<Foo>,
}
fn main() {
/*
let address = Address {
street: "10 Downing Street".to_owned(),
city: "London".to_owned(),
nums : vec![Foo::bar(1), Foo::baz{x : 2, y : 3}],
};
// Serialize it to a JSON string.
let j = serde_json::to_string(&address).unwrap();
// Print, write to a file, or send to an HTTP server.
println!("{}", j);
*/
let input_file = "./sample_address.json";
let mut file = File::open(input_file).unwrap();
let mut file_content = String::new();
let _bytes_read = file.read_to_string(&mut file_content).unwrap();
let address: Address = serde_json::from_str(&file_content).unwrap();
println!("{:?}", address);
}
The JSON data read/written in this example is:
Address { street: "10 Downing Street", city: "London", nums: [bar(1), baz { x: 2, y: 3 }] }
Maybe the TOML crate can't support my use-case?
Serde has lots of options for serializing enums. One that works for your case:
use serde::{Deserialize, Serialize}; // 1.0.117
use toml; // 0.5.7
#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(tag = "type", content = "args")]
enum Actions {
Wait(usize),
Move { x: usize, y: usize },
}
fn main() {
let a_wait = Actions::Wait(5);
println!("{}", toml::to_string(&a_wait).unwrap());
let a_move = Actions::Move { x: 1, y: 1 };
println!("{}", toml::to_string(&a_move).unwrap());
}
type = "Wait"
args = 5
type = "Move"
[args]
x = 1
y = 1

How to create and return a C++ struct from Rust FFI?

I'm trying to create and return a C++ struct. I am currently getting a cannot move out of dereference of raw pointer error when I try to compile. Any idea how I can make this work?
#![allow(non_snake_case)]
#![allow(unused_variables)]
extern crate octh;
// https://thefullsnack.com/en/string-ffi-rust.html
use std::ffi::CString;
#[no_mangle]
pub unsafe extern "C" fn Ghelloworld(
shl: *const octh::root::octave::dynamic_library,
relative: bool,
) -> *mut octh::root::octave_dld_function {
let name = CString::new("helloworld").unwrap();
let pname = name.as_ptr() as *const octh::root::std::string;
std::mem::forget(pname);
let doc = CString::new("Hello World Help String").unwrap();
let pdoc = doc.as_ptr() as *const octh::root::std::string;
std::mem::forget(pdoc);
return octh::root::octave_dld_function_create(Some(Fhelloworld), shl, pname, pdoc);
}
pub unsafe extern "C" fn Fhelloworld(
args: *const octh::root::octave_value_list,
nargout: ::std::os::raw::c_int,
) -> octh::root::octave_value_list {
let list: *mut octh::root::octave_value_list = ::std::ptr::null_mut();
octh::root::octave_value_list_new(list);
std::mem::forget(list);
return *list;
}
I'm trying to create and return a C++ struct
You cannot; C++ (like Rust) does not have a stable, defined ABI. There is no way in Rust to specify that a structure has repr(C++), therefore you cannot create such a structure, much less return it.
The only stable ABI is the one presented by C. You can define structs as repr(C) to be able to return them directly:
extern crate libc;
use std::ptr;
#[repr(C)]
pub struct ValueList {
id: libc::int32_t,
}
#[no_mangle]
pub extern "C" fn hello_world() -> ValueList {
let list_ptr = ::std::ptr::null_mut();
// untested, will cause segfault unless list_ptr is set
unsafe { ptr::read(list_ptr) }
}
That method is highly suspicious though; normally you'd see it as
#[no_mangle]
pub extern "C" fn hello_world() -> ValueList {
unsafe {
let mut list = mem::uninitialized();
list_initialize(&mut list);
list
}
}
See also:
Is it possible to use a C++ library from Rust when the library uses templates (generics)?
I encourage you to read my Rust FFI Omnibus.

How do I store a result using Serde Zero-copy deserialization of a Futures-enabled Hyper Chunk?

I'm using futures, tokio, hyper, and serde_json to request and deserialize some data that I need to hold until my next request. My initial thought was to make a struct containing the hyper::Chunk and the deserialized data that borrows from the Chunk, but couldn't get the lifetimes right. I tried using the rental crate, but I can't get this to work either. Perhaps I'm using the 'buffer lifetime before declaring the buffer Vec, but maybe I've messed something else up:
#[rental]
pub struct ChunkJson<T: serde::de::Deserialize<'buffer>> {
buffer: Vec<u8>,
json: T
}
Is there some way to make the lifetimes right or should I just use DeserializeOwned and give up on zero-copy?
For more context, the following code works (periodically deserializing JSON from two URLs, retaining the results so we can do something with them both). I'd like to change my X and Y types to use Cow<'a, str> for their fields, changing from DeserializeOwned to Deserialize<'a>. For this to work, I need to store the slice that has been deserialized for each, but I don't know how to do this. I'm looking for examples that use Serde's zero-copy deserialization and retain the result, or some idea for restructuring my code that would work.
#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
extern crate futures;
extern crate tokio_core;
extern crate tokio_periodic;
extern crate hyper;
use std::collections::HashMap;
use std::error::Error;
use futures::future;
use futures::Future;
use futures::stream::Stream;
use hyper::Client;
fn stream_json<'a, T: serde::de::DeserializeOwned + Send + 'a>
(handle: &tokio_core::reactor::Handle,
url: String,
period: u64)
-> Box<Stream<Item = T, Error = Box<Error>> + 'a> {
let client = Client::new(handle);
let timer = tokio_periodic::PeriodicTimer::new(handle).unwrap();
timer
.reset(::std::time::Duration::new(period, 0))
.unwrap();
Box::new(futures::Stream::zip(timer.from_err::<Box<Error>>(), futures::stream::unfold( (), move |_| {
let uri = url.parse::<hyper::Uri>().unwrap();
let get = client.get(uri).from_err::<Box<Error>>().and_then(|res| {
res.body().concat().from_err::<Box<Error>>().and_then(|chunks| {
let p: Result<T, Box<Error>> = serde_json::from_slice::<T>(chunks.as_ref()).map_err(|e| Box::new(e) as Box<Error>);
match p {
Ok(json) => future::ok((json, ())),
Err(err) => future::err(err)
}
})
});
Some(get)
})).map(|x| { x.1 }))
}
#[derive(Serialize, Deserialize, Debug)]
pub struct X {
foo: String,
}
#[derive(Serialize, Deserialize, Debug)]
pub struct Y {
bar: String,
}
fn main() {
let mut core = tokio_core::reactor::Core::new().unwrap();
let handle = core.handle();
let x_stream = stream_json::<HashMap<String, X>>(&handle, "http://localhost/X".to_string(), 2);
let y_stream = stream_json::<HashMap<String, Y>>(&handle, "http://localhost/Y".to_string(), 5);
let mut xy_stream = x_stream.merge(y_stream);
let mut last_x = HashMap::new();
let mut last_y = HashMap::new();
loop {
match core.run(futures::Stream::into_future(xy_stream)) {
Ok((Some(item), stream)) => {
match item {
futures::stream::MergedItem::First(x) => last_x = x,
futures::stream::MergedItem::Second(y) => last_y = y,
futures::stream::MergedItem::Both(x, y) => {
last_x = x;
last_y = y;
}
}
println!("\nx = {:?}", &last_x);
println!("y = {:?}", &last_y);
// Do more stuff with &last_x and &last_y
xy_stream = stream;
}
Ok((None, stream)) => xy_stream = stream,
Err(_) => {
panic!("error");
}
}
}
}
When trying to solve a complicated programming problem, it's very useful to remove as much as you can. Take your code and remove what you can until the problem goes away. Tweak your code a bit and keep removing until you can't any more. Then, turn the problem around and build from the smallest piece and work back to the error. Doing both of these will show you where the problem lies.
First, let's make sure we deserialize correctly:
extern crate serde;
extern crate serde_json;
#[macro_use]
extern crate serde_derive;
use std::borrow::Cow;
#[derive(Debug, Deserialize)]
pub struct Example<'a> {
#[serde(borrow)]
name: Cow<'a, str>,
key: bool,
}
impl<'a> Example<'a> {
fn info(&self) {
println!("{:?}", self);
match self.name {
Cow::Borrowed(_) => println!("Is borrowed"),
Cow::Owned(_) => println!("Is owned"),
}
}
}
fn main() {
let data: Vec<_> = br#"{"key": true, "name": "alice"}"#.to_vec();
let decoded: Example = serde_json::from_slice(&data).expect("Couldn't deserialize");
decoded.info();
}
Here, I forgot to add the #[serde(borrow)] attribute, so I'm glad I did this test!
Next, we can introduce the rental crate:
#[macro_use]
extern crate rental;
rental! {
mod holding {
use super::*;
#[rental]
pub struct VecHolder {
data: Vec<u8>,
parsed: Example<'data>,
}
}
}
fn main() {
let data: Vec<_> = br#"{"key": true, "name": "alice"}"#.to_vec();
let holder = holding::VecHolder::try_new(data, |data| {
serde_json::from_slice(data)
});
let holder = match holder {
Ok(holder) => holder,
Err(_) => panic!("Unable to construct rental"),
};
holder.rent(|example| example.info());
// Make sure we can move the data and it's still valid
let holder2 = { holder };
holder2.rent(|example| example.info());
}
Next we try to create a rental of Chunk:
#[rental]
pub struct ChunkHolder {
data: Chunk,
parsed: Example<'data>,
}
Unfortunately, this fails:
--> src/main.rs:29:1
|
29 | rental! {
| ^
|
= help: message: Field `data` must have an angle-bracketed type parameter or be `String`.
Oops! Checking the docs for rental, we can add #[target_ty_hack="[u8]"] to the data field. This leads to:
error[E0277]: the trait bound `hyper::Chunk: rental::__rental_prelude::StableDeref` is not satisfied
--> src/main.rs:29:1
|
29 | rental! {
| ^ the trait `rental::__rental_prelude::StableDeref` is not implemented for `hyper::Chunk`
|
= note: required by `rental::__rental_prelude::static_assert_stable_deref`
That's annoying; since we can't implement that trait for Chunk, we just need to box Chunk, proving that it has a stable address:
#[rental]
pub struct ChunkHolder {
data: Box<Chunk>,
parsed: Example<'data>,
}
I also looked to see if there is a way to get a Vec<u8> back out of Chunk, but it doesn't appear to exist. That would have been another solution with less allocation and indirection.
At this point, "all" that's left is to integrate this back into the futures code. It's a lot of work for anyone but you to recreate that, but I don't foresee any obvious problems in doing so.

Resources