How to deserialize a TOML table containing an array of tables - rust

Take the following TOML data:
[[items]]
foo = 10
bar = 100
[[items]]
foo = 12
bar = 144
And the following rust code:
use serde_derive::Deserialize;
use toml::from_str;
use toml::value::Table;
#[derive(Deserialize)]
struct Item {
foo: String,
bar: String
}
fn main() {
let items_string: &str = "[[items]]\nfoo = 10\nbar = 100\n\n[[items]]\nfoo = 12\nbar = 144\n";
let items_table: Table = from_str(items_string).unwrap();
let items: Vec<Item> = items_table["items"].as_array().unwrap().to_vec();
// Uncomment this line to print the table
// println!("{:?}", items_table);
}
As you can see by yourself, the program does not compile, giving this error in return:
expected struct Item, found enum toml::value::Value
I understand its meaning, but I don't know how I could solve this and achieve what I wanted to do in the first place: cast a child array of a parent table into an array of structs and NOT into an array of tables.

You can parse into the pre-defined TOML types such as Table, but these types don't know about types outside of the pre-defined ones. Those types are mostly used when the actual type of the data is unknown, or unimportant.
In your case that means that the TOML Table type doesn't know about your Item type and cannot be made to know about it.
However you can easily parse into a different type:
use serde_derive::Deserialize;
use std::collections::HashMap;
use toml::from_str;
#[derive(Deserialize, Debug)]
struct Item {
foo: u64,
bar: u64,
}
fn main() {
let items_string: &str = "[[items]]\nfoo = 10\nbar = 100\n\n[[items]]\nfoo = 12\nbar = 144\n";
let items_table: HashMap<String, Vec<Item>> = from_str(items_string).unwrap();
let items: &[Item] = &items_table["items"];
println!("{:?}", items_table);
println!("{:?}", items);
}
(Permalink to the playground)

Related

How to deserialize a nested big array

I have a type Foo which contains a big array. With the help of serde_big_array, serde_derive, and serde_with, I can derive serialization and deserializalization.
use serde_big_array;
use serde_big_array::BigArray;
use serde_derive::{Deserialize, Serialize};
pub const SIZE: usize = 30000;
#[derive(Clone, Debug, Serialize, Deserialize)]
struct Foo {
#[serde(with = "BigArray")]
pub vals: [bool; SIZE],
}
This works fine.
However, when use this type in another structure I run into trouble.
#[derive(Clone, Debug, Serialize, Deserialize)]
struct Bar {
field0: Foo,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn deserialization_attempt_test() {
let foo = Foo {
vals: [false; SIZE],
};
let instance: Bar = Bar { field0: foo };
let json = serde_json::to_string(&instance).unwrap();
println!("PMD0");
let s_back = serde_json::from_str::<Bar>(&json).unwrap();
println!("PMD1");
}
}
Running this tests with cargo t deserialization_attempt_test -- --nocapture prints PMD0 but not PMD1. The test fails with the message
thread 'tests::deserialization_attempt' has overflowed its stack
fatal runtime error: stack overflow
How do I implement Serialize and Deserialize for Bar?
The problem seems to be that the stack (when running through cargo test) is too small to handle your test. Note that if you move deserialization_attempt_test into a main function and run it with cargo run it will work. Similarly if you reduce the array size to e.g. 2000 it will also work.
A workaround, although not that satsifying, is to manually set RUST_MIN_STACK when running cargo test. For instance, this will work:
RUST_MIN_STACK=8388608 cargo test
More information about the error can be found here and here. Here is the documentation on RUST_MIN_STACK.
If you wrap your JSON deserializer in a serde_stacker instance, then it works without setting the environment variable that #munksgaard suggests.
#[test]
fn serialization_bug_fix_attempt_test() {
let foo = Foo {
vals: [false; SIZE],
};
let instance: Bar = Bar { field0: foo };
let json = serde_json::to_string(&instance).unwrap();
println!("PMD0");
let mut deserializer: serde_json::Deserializer<serde_json::de::StrRead> =
serde_json::Deserializer::from_str(&json);
let deserializer = serde_stacker::Deserializer::new(&mut deserializer);
let value: Value = Value::deserialize(deserializer).unwrap();
let target: Bar = from_value(value).unwrap();
println!("PMD1");
}

Rust serde returns a value referencing data owned by the current function

use serde::Deserialize;
#[derive(Deserialize, Debug)]
struct User<'a> {
name: &'a str,
}
fn main() {
let out = parse();
println!("{:?}", out);
}
fn parse<'a>() -> Vec<User<'a>> {
let args: Vec<String> = std::env::args().collect();
let json = args[0].to_owned();
let out = serde_json::from_str::<Vec<User>>(&json);
let out = out.unwrap();
return out;
}
And then, I get error,
--> src/main.rs:18:10
|
16 | let out = serde_json::from_str::<Vec<User>>(&json);
| ----- `json` is borrowed here
17 | let out = out.unwrap();
18 | return out;
| ^^^ returns a value referencing data owned by the current function
How do I solve this? Please help.
In cases like this, usually it's best to use String instead of &str. Like this:
#[derive(Deserialize, Debug)]
struct User {
name: String,
}
String is an owned type, so it can live "on its own" so to speak. Whereas &str is just a reference (a pointer) to a piece of data.
What you are trying to do in your parse() function is return a value, containing references to json - a value that will get dropped automatically as the function execution is over. Imagine, for example, what would out contain at this point:
println!("{:?}", out);
It would contain references to a value that has already been dropped.

How to implement an array of object with generic types (but different in actual type)

I am quite new to Rust and I'm strugging in solving a generic type problem in a particular pattern.
Let's say I have a structure as following
#[derive(Debug)]
struct Test<T>
where T: Eq
{
pub value: Option<T>,
pub array: Option<Vec<Test<T>>>
}
and I failed to use it when array contains different actual type.
fn main() {
let a = Test {
value:Some(16),
array:None
};
let b = Test {
value:Some("abc"),
array:None
};
let c = Test {
value:None,
array:Some(vec![a,b])
};
println!("{:?}",c);
}
compiled error:
|
24 | array:Some(vec![a,b])
| ^ expected integer, found `&str`
|
= note: expected struct `Test<{integer}>`
found struct `Test<&str>`
how to change the definition of Test to achieve my goal?
Thanks.
Some edit after adopting the answer:
while I changed to Box:
#[derive(Debug)]
struct Test
{
pub value:Option<Box<dyn Eq>>,
pub array: Option<Vec<Test>>
}
fn main() {
let a = Test {
value:Some(Box::new(16)),
array:None
};
let b = Test {
value:Some(Box::new("123")),
array:None
};
let c = Test {
value:None,
array:Some(vec![a,b])
};
println!("{:?}",c);
}
5 | pub value:Option<Box<dyn Eq>>,
| ^^^^^^ `Eq` cannot be made into an object
Some edit 2
by using Any trait, here compiles:
use std::any::Any;
#[derive(Debug)]
struct Test
{
pub value:Option<Box<dyn Any>>,
pub array: Option<Vec<Test>>
}
fn main() {
let a = Test {
value:Some(Box::new(16)),
array:None
};
let b = Test {
value:Some(Box::new("123")),
array:None
};
let c = Test {
value:None,
array:Some(vec![a,b])
};
println!("{:?}",c);
}
output:
Test { value: None, array: Some([Test { value: Some(Any), array: None }, Test { value: Some(Any), array: None }]) }
This is an error in expection. While Test is generic over T, that doesn't mean that all Test can be used interchangeably, without respect to what T actually is. In your case, a is a Test<{integer}>, while b is a Test<&'static str>. The compiler has inferred what the actual types are, filling in the blank that is the generic type parameter T.
In c, the compiler will figure out that Vec<T> is initialized by a Test<{integer} (because a is a Test<{integer}), and since all T for that Vec have to be the same, the second parameter (b) also has to be a Test<{integer}>.
You can either use an enum in the definition of Test to provide for both cases (&str and u32 for example). Or you use dynamic dispatch, e.g. via a Box<dyn Eq>. However, since you are apparently going to compare values of different types, the enum-approach is probably the only viable, since you can't easily compare completely unknown types for equality.

How can I move structs containing Vec to/from a static array without getting E0507?

I need a static array of structs and the structs contain a Vec. I can manage the lifetimes of the actual values. I get the following error:
: Mar23 ; cargo test
Compiling smalltalk v0.1.0 (/Users/dmason/git/AST-Smalltalk/rust)
error[E0507]: cannot move out of `dispatchTable[_]` as `dispatchTable` is a static item
--> src/minimal.rs:32:44
|
30 | let old = ManuallyDrop::into_inner(dispatchTable[pos]);
| ^^^^^^^^^^^^^^^^^^ move occurs because `dispatchTable[_]` has type `ManuallyDrop<Option<Box<Dispatch>>>`, which does not implement the `Copy` trait
error: aborting due to previous error
Here is a minimal compilable example:
#[derive(Copy, Clone)]
struct MethodMatch {
hash: i64,
method: Option<bool>,
}
#[derive(Clone)]
pub struct Dispatch {
class: i64,
table: Vec<MethodMatch>,
}
const max_classes : usize = 100;
use std::mem::ManuallyDrop;
const no_dispatch : ManuallyDrop<Option<Box<Dispatch>>> = ManuallyDrop::new(None);
static mut dispatchTable : [ManuallyDrop<Option<Box<Dispatch>>>;max_classes] = [no_dispatch;max_classes];
use std::sync::RwLock;
lazy_static! {
static ref dispatchFree : RwLock<usize> = {RwLock::new(0)};
}
pub fn addClass(c : i64, n : usize) {
let mut index = dispatchFree.write().unwrap();
let pos = *index;
*index += 1;
replaceDispatch(pos,c,n);
}
pub fn replaceDispatch(pos : usize, c : i64, n : usize) -> Option<Box<Dispatch>> {
let mut table = Vec::with_capacity(n);
table.resize(n,MethodMatch{hash:0,method:None});
unsafe {
let old = ManuallyDrop::into_inner(dispatchTable[pos]);
dispatchTable[pos]=ManuallyDrop::new(Some(Box::new(Dispatch{class:c,table:table})));
old
}
}
The idea I had was to have replaceDispatch create a new Dispatch option object, and replace the current value in the array with the new one, returning the original, with the idea that the caller will get the Dispatch option value and be able to use and then drop/deallocate the object.
I found that it will compile if I add .clone() right after the identified error point. But then the original value never gets dropped, so (the into_inner is redundant and) I'm creating a memory leak!. Do I have to manually drop it (if I could figure out how)? I thought that's what ManuallyDrop bought me. In theory, if I created a copy of the fields from the Vec into a copy, that would point to the old data, so when that object got dropped, the memory would get freed. But (a) that seems very dirty, (b) it's a bit of ugly, unnecessary code (I have to handle the Some/None cases, look inside the Vec, etc.), and (c) I can't see how I'd even do it!!!!
As the compiler tells you, you cannot move a value out of a place observable by others. But since you have the replacement at the ready, you can use std::mem::replace:
pub fn replaceDispatch(pos: usize, c: i64, n: usize) -> Option<Box<Dispatch>> {
... table handling omitted ...
unsafe {
let old = std::mem::replace(
&mut dispatchTable[pos],
ManuallyDrop::new(Some(Box::new(Dispatch {
class: c,
table: table,
}))),
);
ManuallyDrop::into_inner(old)
}
}
Playground
In fact, since you're using the Option to manage the lifetime of Dispatch, you don't need ManuallyDrop at all, and you also don't need the Box: playground.

Rust macro that counts and generates repetitive struct fields

I want to write a macro that generates varying structs from an integer argument. For example, make_struct!(3) might generate something like this:
pub struct MyStruct3 {
field_0: u32,
field_1: u32,
field_2: u32
}
What's the best way to transform that "3" literal into a number that I can use to generate code? Should I be using macro_rules! or a proc-macro?
You need a procedural attribute macro and quite a bit of pipework. An example implementation is on Github; bear in mind that it is pretty rough around the edges, but works pretty nicely to start with.
The aim is to have the following:
#[derivefields(u32, "field", 3)]
struct MyStruct {
foo: u32
}
transpile to:
struct MyStruct {
pub field_0: u32,
pub field_1: u32,
pub field_2: u32,
foo: u32
}
To do this, first, we're going to establish a couple of things. We're going to need a struct to easily store and retrieve our arguments:
struct MacroInput {
pub field_type: syn::Type,
pub field_name: String,
pub field_count: u64
}
The rest is pipework:
impl Parse for MacroInput {
fn parse(input: ParseStream) -> syn::Result<Self> {
let field_type = input.parse::<syn::Type>()?;
let _comma = input.parse::<syn::token::Comma>()?;
let field_name = input.parse::<syn::LitStr>()?;
let _comma = input.parse::<syn::token::Comma>()?;
let count = input.parse::<syn::LitInt>()?;
Ok(MacroInput {
field_type: field_type,
field_name: field_name.value(),
field_count: count.base10_parse().unwrap()
})
}
}
This defines syn::Parse on our struct and allows us to use syn::parse_macro_input!() to easily parse our arguments.
#[proc_macro_attribute]
pub fn derivefields(attr: TokenStream, item: TokenStream) -> TokenStream {
let input = syn::parse_macro_input!(attr as MacroInput);
let mut found_struct = false; // We actually need a struct
item.into_iter().map(|r| {
match &r {
&proc_macro::TokenTree::Ident(ref ident) if ident.to_string() == "struct" => { // react on keyword "struct" so we don't randomly modify non-structs
found_struct = true;
r
},
&proc_macro::TokenTree::Group(ref group) if group.delimiter() == proc_macro::Delimiter::Brace && found_struct == true => { // Opening brackets for the struct
let mut stream = proc_macro::TokenStream::new();
stream.extend((0..input.field_count).fold(vec![], |mut state:Vec<proc_macro::TokenStream>, i| {
let field_name_str = format!("{}_{}", input.field_name, i);
let field_name = Ident::new(&field_name_str, Span::call_site());
let field_type = input.field_type.clone();
state.push(quote!(pub #field_name: #field_type,
).into());
state
}).into_iter());
stream.extend(group.stream());
proc_macro::TokenTree::Group(
proc_macro::Group::new(
proc_macro::Delimiter::Brace,
stream
)
)
}
_ => r
}
}).collect()
}
The behavior of the modifier creates a new TokenStream and adds our fields first. This is extremely important; assume that the struct provided is struct Foo { bar: u8 }; appending last would cause a parse error due to a missing ,. Prepending allows us to not have to care about this, since a trailing comma in a struct is not a parse error.
Once we have this TokenStream, we successively extend() it with the generated tokens from quote::quote!(); this allows us to not have to build the token fragments ourselves. One gotcha is that the field name needs to be converted to an Ident (it gets quoted otherwise, which isn't something we want).
We then return this modified TokenStream as a TokenTree::Group to signify that this is indeed a block delimited by brackets.
In doing so, we also solved a few problems:
Since structs without named members (pub struct Foo(u32) for example) never actually have an opening bracket, this macro is a no-op for this
It will no-op any item that isn't a struct
It will also no-op structs without a member

Resources