I would like to parse the following part of a .yaml-file:
networks:
foo1: 192.168.1.0/24
bar1: 192.168.2.0/24
foo2: 2001:CAFE:1::/64
bar2: 2001:CAFE:2::/64
... where the key of each property is the labels the name of a network and the value indicates the assigned subnet to it.
The deserialized structs should look like the following:
#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Networks {
networks: Vec<Network>
}
#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Network {
name: String,
subnet: String,
}
I would prefer this notation over
networks:
- name: foo1
subnet: 192.168.1.0/24
- ...
since this would add unnecessary boilerplate. The problem is that I cannot properly parse this list of networks into structs, since to my current knowledge, the property of a struct has to have an equivalent key to it - and since the key-names here are "random" I cannot do that.
The only other workaround I've found is parsing the entries as HashMaps and automatically convert those into tuples of (String, String) (networks would then be a Vec<(String, String)>) via the serde-tuple-vec-map crate (which loses the named parameters).
This seems like something that would be easy to configure, yet I haven't found any solutions elsewhere.
Implementing a custom deserializer and using it with #[serde(deserialize_with = "network_tuples")] networks: Vec<Network> is one way of doing it:
fn network_tuples<'de, D>(des: D) -> Result<Vec<Network>, D::Error>
where
D: serde::Deserializer<'de>,
{
struct Vis(Vec<Network>);
impl<'de> serde::de::Visitor<'de> for Vis {
type Value = Vec<Network>;
fn expecting(&self, _formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
todo!("return nice descriptive error")
}
fn visit_map<A>(mut self, mut map: A) -> Result<Self::Value, A::Error>
where
A: serde::de::MapAccess<'de>,
{
while let Some((name, subnet)) = map.next_entry()? {
self.0.push(Network { name, subnet });
}
Ok(self.0)
}
}
des.deserialize_map(Vis(vec![]))
}
You could do without the visitor by first deserializing to a HashMap<String, String> and then converting, but that has a performance penalty and you'd lose the ability to have two networks with the same name (not sure if you want that though).
Playground
Of course, you could also combine my favorite serde trick #[serde(from = "SomeStructWithVecStringString")] with the serde-tuple-vec-map crate. But I've already written many answers using from, so I'll refer to an example.
Related
How does one add a limit over length of a String (Vec or Array) when deserializing using serde ?
Best guess is:
use serde::Deserialize;
use arrayvec::ArrayString;
#[derive(Deserialize)]
pub struct Foo {
#[serde(try_from = "ArrayString<4096>")] // FAILS!
bar: String,
}
Which does not work because try_from attribute is only available for "containers" and not "fields".
If there is a way to make this solution work, the main concerns are efficiency and security. Is this solution fast (not dramatically slowing down the deserialization process for short String) and safe (does it prevents the memory from being used up by a malicious request), and is it idiomatic ?
As pointed out by #cafce25, by the point your Deserialize impl gets the string, it is already fully parsed from the input stream. This is just how serde works, and thus this restriction will not provide any protection.
You must implement this kind of restriction at the request level, or with your own parser.
If you still wish to implement a restriction like this, it's pretty simple to do with a newtype that delegates to String::deserialize:
use std::ops::Deref;
use serde::de;
use serde::Deserialize;
struct LimitedString<const MAX_LENGTH: usize>(String);
impl<const MAX_LENGTH: usize> Deref for LimitedString<MAX_LENGTH> {
type Target = String;
fn deref(&self) -> &Self::Target {
&self.0
}
}
impl<'de, const MAX_LENGTH: usize> de::Deserialize<'de> for LimitedString<MAX_LENGTH> {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: de::Deserializer<'de>,
{
<String as de::Deserialize>::deserialize(deserializer).and_then(|inner| {
if inner.len() > MAX_LENGTH {
Err(de::Error::invalid_length(inner.len(), &"an integer lower than the maximum"))
} else {
Ok(Self(inner))
}
})
}
}
#[derive(Deserialize)]
pub struct Foo {
bar: LimitedString<4096>,
}
playground
I'm attempting to use hcl-rs = 0.7.0 to parse some HCL. I'm just experimenting with arbitrary HCL, so I'm not looking to parse terraform specific code.
I'd like to be able to parse a block like this and get it's label as part of result
nested_block "nested_block_label" {
foo = 123
}
This currently doesn't work, but hopefully it shows my intention. Is something like this possible?
#[test]
fn deserialize_struct_with_label() {
#[derive(Deserialize, PartialEq, Debug)]
struct TestRoot {
nested_block: TestNested,
}
#[derive(Deserialize, PartialEq, Debug)]
struct TestNested {
label: String,
foo: u32,
}
let input = r#"
nested_block "nested_block_label" {
foo = 123
}"#;
let expected = TestRoot{ nested_block: TestNested { label: String::from("nested_block_label"), foo: 123 } };
assert_eq!(expected, from_str::<TestRoot>(input).unwrap());
}
Your problem is that hcl by default seems to interpret
nested_block "nested_block_label" {
foo = 123
}
as the following "serde structure":
"nested_block" -> {
"nested_block_label" -> {
"foo" -> 123
}
}
but for your Rust structs, it would have to be
"nested_block" -> {
"label" -> "nested_block_label"
"foo" -> 123
}
I'm not aware of any attributes that would allow you to bend the former into the latter.
As usual, when faced with this kind of situation, it is often easiest to first deserialize to a generic structure like hcl::Block and then convert to whatever struct you want manually. Disadvantage is that you'd have to do that for every struct separately.
You could, in theory, implement a generic deserialization function that wraps the Deserializer it receives and flattens the two-level structure you get into the one-level structure you want. But implementing Deserializers requires massive boilerplate. Possibly, somebody has done that before, but I'm not aware of any crate that would help you here, either.
As a sort of medium effort solution, you could have a special Labelled wrapper structure that always catches and flattens this intermediate level:
#[derive(Debug, PartialEq)]
struct Labelled<T> {
label: String,
t: T,
}
impl<'de, T: Deserialize<'de>> Deserialize<'de> for Labelled<T> {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
struct V<T>(std::marker::PhantomData<T>);
impl<'de, T: Deserialize<'de>> serde::de::Visitor<'de> for V<T> {
type Value = Labelled<T>;
fn visit_map<A>(self, mut map: A) -> Result<Self::Value, A::Error>
where
A: serde::de::MapAccess<'de>,
{
if let (Some((label, t)), None) =
(map.next_entry()?, map.next_entry::<String, ()>()?)
{
Ok(Labelled { label, t })
} else {
Err(serde::de::Error::invalid_type(
serde::de::Unexpected::Other("Singleton map"),
&self,
))
}
}
}
deserializer.deserialize_map(V::<T>(Default::default()))
}
}
would be used like this:
#[derive(Deserialize, PartialEq, Debug)]
struct TestRoot {
nested_block: Labelled<TestNested>,
}
#[derive(Deserialize, PartialEq, Debug)]
struct TestNested {
foo: u32,
}
Lastly, there may be some trick like adding #[serde(rename = "$hcl::label")]. Other serialization libraries (e.g. quick-xml) have similar and allow marking fields as something special this way. hcl-rs does the same internally, but it's undocumented and I can't figure out from the source whether what you need is possible.
I have a simple structure that contains a resources field. I would like my resources to always be Urls.
In the actual file that I am trying to deserialize, resources can either be URLs or Path.
Here is my structure:
pub struct Myfile {
pub resources: Vec<Resource>,
}
pub type Resource = Url;
I would like to use serde to:
Try to deserialize each Resource using the implementation from the url crate.
If it fails, try to deserialize each one of them into a Path and then use url::from_*_path() to get a Url.
I am trying to adapt the string or map,map and structure examples but I am struggling to understand where to even start.
Since my end result will by a Url, the examples seem to show that I should be implementing Deserialize for Url. But I still need to current implementation. My Resource is an alias so I can't implement Deserialize for it.
Is there any simple way to deserialize both Paths and Urls into Urls?
PS: I will probably go for the answer proposed but after reading through this post, I tried the following which seems to work too:
#[derive(Clone, Debug)]
pub struct Resource(Url);
impl<'de> Deserialize<'de> for Resource {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where D: Deserializer<'de>
{
let s = String::deserialize(deserializer)?;
if let path = Path::new(&s) {
if path.is_file() {
Ok(Resource(Url::from_file_path(path).unwrap()))
} else {
Ok(Resource(Url::from_directory_path(path).unwrap()))
}
} else {
Url::deserialize(s.into_deserializer()).map(|x| Resource(x))
}
}
}
But is less convenient to work with than regular Urls.
I think the key to doing this with reasonable effort is to realize that serde::Deserialize for Url is also just cooking with water, i.e. just expecting a string and calling Url::parse on it.
So it's time to deploy my favourite serde trick: deserialize to a struct that serde can handle easily:
#[derive(Deserialize)]
pub struct MyfileDeserialize {
pub resources: Vec<String>,
}
Tell serde that it should get the struct you finally want from that easily handlable struct:
#[derive(Deserialize, Debug)]
#[serde(try_from = "MyfileDeserialize")]
pub struct Myfile {
pub resources: Vec<Resource>,
}
Finally, you just need to define how to turn MyfileDeserialize into Myfile.
impl TryFrom<MyfileDeserialize> for Myfile {
type Error = &'static str; // TODO: replace with anyhow or a proper error type
fn try_from(t: MyfileDeserialize) -> Result<Self, &'static str> {
Ok(Self {
resources: t
.resources
.into_iter()
.map(|url| {
if let Ok(url) = Url::parse(&url) {
return Ok(url);
};
if let Ok(url) = Url::from_file_path(url) {
return Ok(url);
};
// try more from_*_path here.
Err("Can't as url or path")
})
.collect::<Result<Vec<_>, _>>()?,
})
}
}
Playground
Edit regarding your PS:
If you are willing to mess around with manually implementing deserializer traits and functions, I suppose you do have the option of completely getting rid of any wrapper structs or mediating TryFrom types: add a #[serde(deserialize_with = …)] to your resources, and in there, first do a <Vec<String>>::deserialize(de)?, and then turn that into a Vec<Url> as usual.
Playground
I have my types defined and then I keep them in something like ArrayVec<[MyType, 16]> (from arrayvec crate) variables (members of a structure). Restson has the RestPath trait, allow us to define a path used to form a URI when performing a REST query.
However, due to the restriction that only local traits can be implemented for arbitrary types (AKA the orphan rule) I can't use it straightforwardly for ArrayVec<[MyType, 16]>.
I somehow overcame the problem by implementing the trait for ~specialization~ instantiation of the following enum:
enum ModelArray<T> {
Array(T)
}
and then decorticate the instance of T using:
type MyArray = ModelArray<ArrayVec<[MyType; 16]>>;
let encapsulated_array: MyArray = client.get(()).unwrap();
let ModelArray::<ArrayVec<[MyType; 16]>>::Array(myarray) = encapsulated_array;
This works as minimal example, but I suffer from the fact I cannot call client.get(()).unwrap() directly to the member of other structure.
I'm surprised full specialization of a generic type isn't treated by Rust as local type and orphan rule still applies. Why?
Are there other nice ways to overcome the limitation and would let me nicely assign result of Restson's get() into members of a structure?
Working code:
extern crate restson;
extern crate arrayvec;
extern crate serde_derive;
use restson::RestPath;
use arrayvec::ArrayVec;
#[derive(Deserialize, Debug)]
struct MyType {
id: u16,
data: u32,
}
struct DataModel {
my_data: ArrayVec<[MyType; 16]>
}
#[derive(Deserialize, Debug)]
#[serde(untagged)]
enum ModelArray<T> {
Array(T)
}
impl RestPath<u16> for MyType {
fn get_path(id: u16) -> Result<String, Error> {
Ok(format!("data/MyType/{}", id))
}
}
impl RestPath<()> for ModelArray<ArrayVec<[MyType; 16]>> {
fn get_path(_: ()) -> Result<String, Error> {
Ok(String::from("data/MyType"))
}
}
use restson::RestClient;
pub fn load_data() {
let mut client = RestClient::new(&format!("http://{}", "localhost:8080")).unwrap();
let element: Type = client.get(24).unwrap();
println!("Room: {:?}", elementh);
type ModelArray = ModelArray<ArrayVec<[MyType; 16]>>;
let encapsulated: ModelArray = client.get(()).unwrap();
let ModelArray::<ArrayVec<[MyType; 16]>>::Array(elements) = encapsulated;
println!("Room: {:?}", elements[0]);
}
On Rust Playground (lack of restson crate wouldn't allow you to build)
Respective complete code: on GitHubGist
I'm building a rogue-like, I've already gotten a data loader working and part of the ECS working (building from scratch). The data is stored in .yml files and is used to describe things in the game (in this instance mobs) and what features those things have, for example:
---
orc:
feature_packs:
- physical
- basic_identifiers_mob
features:
- component: char
initial_value: T
goblin:
feature_packs:
- physical
- basic_identifiers_mob
features:
- component: char
initial_value: t
As you can see, there are two mobs described, a goblin and an orc, they both possess two feature packs (groups of features) and also posses a char feature that's used to describe what they look like to the player.
The initial_value field can be a string, an integer, a floating point, a bool, a range, etc depending on what the component requires, this will indicate the value or possible values the component may have when the component is generated during entity building/creation.
The problem is that I don't know how to, when iterating over the features, select the struct based on the component's name, for example, select the Char struct for the "char" feature.
To better describe what I mean, I've written an example in a language I better understand, Ruby:
data_manager = function_that_loads_data('folder_path')
Entity_Manager.build(:mob, :orc, data_manager)
class Entity_Manager
class << self
attr_accessor :entities, :components
end
def self.build(entity_type, template_name, data_manager)
template = data_manager[entity_type][template_name]
entity_id = generate_unique_id
entities[entity_id] = Entity.new(entity_id, components: template.components.keys)
template.components.each do |component|
components[component.name][entity_id] =
Components.get(component.name).new(component.initial_value) # <= This part, how do I do the equivalent in rust, a function that will return or allow me to get or create a struct based on the value of a string variable
end
end
end
Now serde is the only thing I know that seems to be able to read text data and transform it into data, so to that end
How can I use serde (or a more appropriate non-serde using solution) to take the names of the feature and retrieve the correct struct, all implementing a type?
Incidentally, the one solution I'm trying not to use is a giant match statement.
The repo of my work as it stands is here
Data manager - Loads and manages data loaded into the game
Entity manager - Manages entities and there components (doesn't support bit keys atm)
Entity Builder - Where Entity's will be built using data from the data manager (this is where I'm currently stuck)
Components - a list of simple components
What I'm trying to avoid is doing somthing like this:
pub fn get(comp_name: &String) -> impl Component {
match comp_name.as_ref() {
"kind" => Kind,
"location" => Location,
"name" => Name,
"position" => Position,
"char" => Char,
}
}
because it's not really maintainable, though a macro would help a lot, I'm not very good at those atm and it doesn't even work, rust keeps thinking I'm trying to initialize the types when I just want to return one of several possible types that all will implement Component
EDIT: Becuase it looks like I'm not clear enough:
I'm not trying to load gameplay objects into the game, I'm loading templates
I'm using those templates to then generate the entities that will be exist during gameplay
I can already load the data I want into the game in the following structure:
pub enum InitialValue {
Char(char),
String(String),
Int(i32),
Float(f32),
Bool(bool),
Range(Range<i32>),
Point((i32,i32))
}
impl InitialValue {
pub fn unwrap_char(&self) -> &char {
match &self {
InitialValue::Char(val) => val,
_ => panic!("Stored value does not match unwrap type")
}
}
pub fn unwrap_string(&self) -> &String {
match &self {
InitialValue::String(val) => val,
_ => panic!("Stored value does not match unwrap type")
}
}
pub fn unwrap_int(&self) -> &i32 {
match &self {
InitialValue::Int(val) => val,
_ => panic!("Stored value does not match unwrap type")
}
}
pub fn unwrap_float(&self) -> &f32 {
match &self {
InitialValue::Float(val) => val,
_ => panic!("Stored value does not match unwrap type")
}
}
pub fn unwrap_bool(&self) -> &bool {
match &self {
InitialValue::Bool(val) => val,
_ => panic!("Stored value does not match unwrap type")
}
}
pub fn unwrap_range(&self) -> &Range<i32> {
match &self {
InitialValue::Range(val) => val,
_ => panic!("Stored value does not match unwrap type")
}
}
pub fn unwrap_point(&self) -> &(i32, i32) {
match &self {
InitialValue::Point(val) => val,
_ => panic!("Stored value does not match unwrap type")
}
}
}
#[derive(Debug, Deserialize)]
pub struct Component {
#[serde(rename="component")]
name: String,
#[serde(default)]
initial_value: Option<InitialValue>,
}
#[derive(Debug, Deserialize)]
pub struct Template {
pub feature_packs: Vec<String>,
pub features: Vec<Component>,
}
How do I transform the templates into instances of entities?
Specifcally, How do I for a given Component.name find the component
and then initialize it? OR is my aproach wrong and there's a better
way.
And if I am doing it wrong, How do other games load data in and then use it to generate in
game entities?
Sounds like you want a tagged union, or sum type; Rust knows these as enumerations. Serde even supports using container internal tags. So here's my little experiment:
#[macro_use] extern crate serde_derive;
extern crate serde_yaml;
#[derive(Debug, Serialize, Deserialize)]
#[serde(tag="component")]
enum Feature {
Char { initial_value : char },
Weight { kgs : u32 }
}
fn main() {
let v = vec![
Feature::Char{initial_value:'x'},
Feature::Weight{kgs:12}
];
println!("{}", serde_yaml::to_string(&v).unwrap());
}
This outputs:
---
- component: Char
initial_value: x
- component: Weight
kgs: 12
Probably the next step is to make dedicated structs for the variants.