I'm having trouble serializing the following struct. I have narrowed it down to, that the problem lies within the variable objects, containing trait structs within a HashMap. I'll try to explain my circumstances:
I have the following main struct, which i'm interested in obtaining data from:
#[derive(Serialize)]
pub struct System {
pub objects: HashMap<(u32, u32), Box<dyn CommonObjects>>,
pub paths: Vec<Path>,
pub loops: Vec<Loop>,
pub physics: Physics,
pub run_limit_loops: u32,
pub run_limit_paths: u32,
pub error_tol_loops: f64,
pub error_tol_paths: f64,
pub unknown_flow_outlets: u32,
pub unknown_pumps_id: u32,
pub system: u32,
}
The kind of structs which are contained within objects are as the following:
pub struct Outlet {
// Variables found in all structs contained within 'objects'
pub Q: f64,
pub hL: f64,
pub ob: u32,
pub id: u32,
pub active: bool,
pub system: u32,
pub con_Q: HashMap<u32, f64>,
// Variables found only in this struct.
pub p_dyn: f64,
pub flow_specified: String,
pub submerged: bool,
}
pub struct Pipe {
// Variables found in all structs contained within 'objects'
pub Q: f64,
pub hL: f64,
pub ob: u32,
pub id: u32,
pub active: bool,
pub system: u32,
pub con_Q: HashMap<u32, f64>,
// Variables found only in this struct.
pub Re: f64,
pub f: f64,
}
These struct has some variables which are only used within themselves, and some variables which are common for all structs contained within objects (p_dyn is only used within Outlet, while Q is used for all structs contained within objects) To get these variables, get-function are defined, both within the local struct, and by the trait CommonObject. All i am interested in, is serializing objects in order to get all the variables in a string format, both the common ones, and the ones only appearing locally within a struct, so that i can send the variables to other programs to further visualization.
In this following code the error occurs:
// Where the system struct originates from.
let systems: Vec<System> = system_analyse::analyse_system(data);
// I try to only serialize the objects contained within one of the system structs.
let jsonstringresult = serde_json::to_string(&systems[0].objects);
match jsonstringresult {
Ok(v) => {
println!("{:?}", &v);
v
},
Err(e) => {
// The error message originates from here.
println!("An error occured at serializing: {}", e);
String::new()
}
}
I get the following error:
An error occured at serializing: key must be a string
I have found this thread discussing the issue of serializing dynamic traits, and i've followed the instructions and added the following to the trait:
pub trait CommonObjects: erased_serde::Serialize {
...
}
serialize_trait_object!(CommonObjects);
Which makes the code compile in the first place. I've also found this site getting the same error, but the issue there seems to be with enums. Maybe the problem in this site is related to my problem, but i cant figure out how if so.
I'm open to all sort of feedback and even fundamentally change the structure of the code if so necessary.
A quick way to serialize a HashMap with non-string keys to Json is to use the serde_as macro from the serde_with crate.
use serde_with::serde_as;
#[serde_as]
#[derive(Serialize)]
pub struct System {
#[serde_as(as = "Vec<(_, _)>")]
pub objects: HashMap<(u32, u32), Box<dyn CommonObjects>>,
//...
}
The #[serde_as(as = "Vec<(_, _)>")] encodes the map as a sequence of tuples, representing pairs of keys and values. In Json, this will become an array of 2-element arrays.
Related
I need to serialize a struct from a remote crate and all of the fields in the struct are private. There are getter's implemented in the remote struct to get those values. I am following this guidance and got it to work just fine for primitive types. However, I'm struggling with how to implement this for non-primitive types (ie: String) that the remote struct contains.
Below is a small piece of what I've implemented to frame the issue. The DataWrapper struct simply wraps Data, where Data is the remote struct.
#[derive(Serialize)]
pub struct DataWrapper {
#[serde(with = "DataDef")]
pub data: Data,
}
#[derive(Serialize)]
#[serde(remote = "remote_crate::data::Data")]
pub struct DataDef {
#[serde(getter = "Data::image_id")] // This works
image_id: u16,
#[serde(getter = "Data::description")] // This does not work
description: String,
}
The error I get when compiling this is
#[derive(Serialize)]
^^^^^^^^^ expected struct `std::string::String`, found `&str`
This makes sense, since the getter Data::description returns &str rather than a String. But, I'm not seeing a way in my code to coerce this so the compiler is happy.
If I change DataDef::description to be &str instead of String, then I have to implement lifetimes. But, when I do that, the compiler then says the remote "struct takes 0 lifetime arguments".
Appreciate any tips on how I can serialize this and other non-primitive types.
One approach you could do, so that you have full control of the serialization. Is to have the data wrapper be a copy of the struct fields you need, instead of the entire remote struct. Then you can implement From<remote_crate::data::Data> for DataWrapper and use serde without trying to coerce types.
#[derive(Serialize)]
pub struct Data {
image_id: u16,
description: String,
}
impl From<remote_crate::data::Data> for Data {
fn from(val: remote_crate::data::Data) -> Self {
Self {
image_id: val.image_id,
description: val.description.to_string(),
}
}
}
// Then you could use it like this:
// let my_data: Data = data.into();
I couldn't get it to work with an &str, but if you're OK with an allocation, you can write a custom getter:
mod local {
use super::remote::RemoteData;
use serde::{Deserialize, Serialize};
fn get_owned_description(rd: &RemoteData) -> String {
rd.description().into()
}
#[derive(Serialize)]
#[serde(remote = "RemoteData")]
pub struct DataDef {
#[serde(getter = "get_owned_description")]
description: String,
}
}
mod remote {
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize)]
pub struct RemoteData {
description: String,
}
impl RemoteData {
pub fn description(&self) -> &str {
&self.description
}
}
}
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=e9c7c0b069d7e16b6faac2fa2b840c72
I need to serialize some data into files. For the sake of memory efficiency, I want to use the default compact serializer of MessagePack (MsgPack), as it only serializes field values w/o their names. I also want to be able to make changes to the data structure in future versions, which obviously can't be done w/o also storing some meta/versioning information. I imagine the most efficient way to do it is to simply use some "header" field for that purpose. Here is an example:
pub struct Data {
pub version: u8,
pub items: Vec<Item>,
}
pub struct Item {
pub field_a: i32,
pub field_b: String,
pub field_c: i16, // Added in version 3
}
Can I do something like that in rmp-serde (or maybe some other crate?) - to somehow annotate that a certain struct field should only be taken into account for specific file versions?
You can achieve this by writing a custom deserializer like this:
use serde::de::Error;
use serde::{Deserialize, Deserializer, Serialize};
#[derive(Serialize)]
pub struct Data {
pub version: u8,
pub items: Vec<Item>,
}
#[derive(Serialize)]
pub struct Item {
pub field_a: i32,
pub field_b: String,
pub field_c: i16, // Added in version 3
}
impl<'de> Deserialize<'de> for Data {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
// Inner structs, used for deserializing only
#[derive(Deserialize)]
pub struct InnerData {
version: u8,
items: Vec<InnerItem>,
}
#[derive(Deserialize)]
pub struct InnerItem {
field_a: i32,
field_b: String,
field_c: Option<i16>, // Added in version 3 - note that this field is optional
}
// Deserializer the inner structs
let inner_data = InnerData::deserialize(deserializer)?;
// Get the version so we can add custom logic based on the version later on
let version = inner_data.version;
// Map the InnerData/InnerItem structs to Data/Item using our version based logic
Ok(Data {
version,
items: inner_data
.items
.into_iter()
.map(|item| {
Ok(Item {
field_a: item.field_a,
field_b: item.field_b,
field_c: if version < 3 {
42 // Default value
} else {
// Get the value of field_c
// If it's missing return an error, since it's required since version 3
// Otherwise return the value
item.field_c
.map_or(Err(D::Error::missing_field("field_c")), Ok)?
},
})
})
.collect::<Result<_, _>>()?,
})
}
}
Short explanation how the deserializer works:
We create a "dumb" inner struct which is a copy of your structs but the "new" fields are optional
We deserialize to the new inner structs
We map from our inner to our outer structs using version-based logic
If one of the new fields is missing in a new version we return a D::Error::missing_field error
The usual way to declare a HashMap in a Rust struct is as follows:
struct MyStruct {
query: HashMap<String, String>,
counter: u32,
}
How would I write the above code if I do not know what the HashMap would contain beforehand? I have tried the below code without success.
struct MyStruct {
query: HashMap<K, V>,
counter: u32,
}
You will need to add your generics to your struct declaration as well:
struct MyStruct<K,V> {
query: HashMap<K, V>,
counter: u32,
}
Have a look at Rust Book/Generic Data Types
I am using the Diesel crate to perform some database work. In some tables, two columns of the table should be treated together as a single key.
This pattern is repeated in many places in the database, so it would be nice to avoid a heap of repeated copy-paste code to handle this. However, I can't convince Diesel to automatically produce a type I can use in queries or inserts.
Consider the table
table! {
records (iid) {
iid -> Integer,
id_0 -> BigInt,
id_1 -> BigInt,
data -> Text,
}
}
and the ideal types
#[derive(Debug, Copy, Clone, FromSqlRow)]
pub struct RecordId {
id_0: i64,
id_1: i64,
}
#[derive(Queryable, Debug)]
pub struct Record {
pub iid: i32,
pub id: RecordId,
pub data: String,
}
This code compiles OK, but when I try to use it I get an error, for example:
pub fn find(connection: &SqliteConnection) -> types::Record {
records
.find(1)
.get_result::<types::Record>(connection)
.unwrap()
}
produces:
error[E0277]: the trait bound `(i32, types::RecordId, std::string::String): diesel::Queryable<(diesel::sql_types::Integer, diesel::sql_types::BigInt, diesel::sql_types::BigInt, diesel::sql_types::Text), _>` is not satisfied
--> src/main.rs:76:21
|
76 | records.find(1).get_result::<types::Record>(connection).unwrap()
| ^^^^^^^^^^ the trait `diesel::Queryable<(diesel::sql_types::Integer, diesel::sql_types::BigInt, diesel::sql_types::BigInt, diesel::sql_types::Text), _>` is not implemented for `(i32, types::RecordId, std::string::String)`
|
= help: the following implementations were found:
<(A, B, C) as diesel::Queryable<(SA, SB, SC), __DB>>
<(A, B, C) as diesel::Queryable<diesel::sql_types::Record<(SA, SB, SC)>, diesel::pg::Pg>>
= note: required because of the requirements on the impl of `diesel::Queryable<(diesel::sql_types::Integer, diesel::sql_types::BigInt, diesel::sql_types::BigInt, diesel::sql_types::Text), _>` for `types::Record`
= note: required because of the requirements on the impl of `diesel::query_dsl::LoadQuery<_, types::Record>` for `diesel::query_builder::SelectStatement<types::records::table, diesel::query_builder::select_clause::DefaultSelectClause, diesel::query_builder::distinct_clause::NoDistinctClause, diesel::query_builder::where_clause::WhereClause<diesel::expression::operators::Eq<types::records::columns::iid, diesel::expression::bound::Bound<diesel::sql_types::Integer, i32>>>>`
If I create a version which does not contain the RecordId but has the sub-pieces directly, then there is no error:
pub struct RecordDirect {
pub iid: i32,
pub id_0: i64,
pub id_1: i64,
pub data: String,
}
// ...
pub fn find_direct(connection: &SqliteConnection) -> types::RecordDirect {
records
.find(1)
.get_result::<types::RecordDirect>(connection)
.unwrap()
}
Similarly, I can manually implement the Queryable trait and that works OK too,
#[derive(Debug)]
pub struct RecordManual {
pub iid: i32,
pub id: RecordId,
pub data: String,
}
impl Queryable<records::SqlType, diesel::sqlite::Sqlite> for RecordManual {
type Row = (i32, i64, i64, String);
fn build(row: Self::Row) -> Self {
RecordManual {
iid: row.0,
id: RecordId {
id_0: row.1,
id_1: row.2,
},
data: row.3,
}
}
}
// ...
pub fn find_manual(connection: &SqliteConnection) -> types::RecordManual {
records
.find(1)
.get_result::<types::RecordManual>(connection)
.unwrap()
}
This case is ugly to maintain and I could not work out how to get it working for insertion — manually implementing Insertable seems a little trickier than Queryable.
To make this easier for anyone looking at it to play with, I've created a repository containing an almost compiling small reproducer containing the code blocks from this post. (Normally I'd put it up on the rust-playground, but that doesn't support diesel). You can find that code at https://github.com/mikeando/diesel_custom_type_demo.
Is there a way to make the #[derive(Queryable)] (and #[derive(Insertable)]) work for these kinds of cases?
A minimal reproducer for the failing initial case is:
#[macro_use]
extern crate diesel;
use diesel::prelude::*;
mod types {
use diesel::deserialize::Queryable;
use diesel::sqlite::SqliteConnection;
table! {
records (iid) {
iid -> Integer,
id_0 -> BigInt,
id_1 -> BigInt,
data -> Text,
}
}
#[derive(Debug, Copy, Clone, FromSqlRow)]
pub struct RecordId {
id_0: i64,
id_1: i64,
}
// Using a RecordId in a Record compiles, but
// produces an error when used in an actual query
#[derive(Queryable, Debug)]
pub struct Record {
pub iid: i32,
pub id: RecordId,
pub data: String,
}
}
use types::records::dsl::*;
pub fn find(connection:&SqliteConnection) -> types::Record {
records.find(1).get_result::<types::Record>(connection).unwrap()
}
Is there a way to make the #[derive(Queryable)] (and #[derive(Insertable)]) work for these kinds of cases?
For #[derive(Insertable)] this is simply possible by adding a #[diesel(embedded)] to your id field and a #[derive(Insertable)] on both structs. See the documentation of Insertable for details.
For #[derive(Queryable)] this is not possible because Queryable is supposed to be a plain mapping from query result to struct, with the basic assumption that the "shape" of the output remains the same (at least for the derive).
I just implemented a simple trait to keep the history of a struct property:
fn main() {
let mut weight = Weight::new(2);
weight.set(3);
weight.set(5);
println!("Current weight: {}. History: {:?}", weight.value, weight.history);
}
trait History<T: Copy> {
fn set(&mut self, value: T);
fn history(&self) -> &Vec<T>;
}
impl History<u32> for Weight {
fn set(&mut self, value: u32) {
self.history.push(self.value);
self.value = value;
}
fn history(&self) -> &Vec<u32> {
&self.history
}
}
pub struct Weight {
value: u32,
history: Vec<u32>,
}
impl Weight {
fn new(value: u32) -> Weight {
Weight {
value,
history: Vec::new(),
}
}
}
I don't expect this is possible, but could you add the History trait (or something equivalent) to something which doesn't already have a history property (like u32 or String), effectively tacking on some information about which values the variable has taken?
No. Traits cannot add data members to the existing structures. Actually, only a programmer can do that by modifying the definition of a structure. Wrapper structures or hash-tables are the ways to go.
No, traits can only contain behavior, not data. But you could make a struct.
If you could implement History for u32, you'd have to keep the entire history of every u32 object indefinitely, in case one day someone decided to call .history() on it. (Also, what would happen when you assign one u32 to another? Does its history come with it, or does the new value just get added to the list?)
Instead, you probably want to be able to mark specific u32 objects to keep a history. A wrapper struct, as red75prime's answer suggests, will work:
mod hist {
use std::mem;
pub struct History<T> {
value: T,
history: Vec<T>,
}
impl<T> History<T> {
pub fn new(value: T) -> Self {
History {
value,
history: Vec::new(),
}
}
pub fn set(&mut self, value: T) {
self.history.push(mem::replace(&mut self.value, value));
}
pub fn get(&self) -> T
where
T: Copy,
{
self.value
}
pub fn history(&self) -> &[T] {
&self.history
}
}
}
It's generic, so you can have a History<u32> or History<String> or whatever you want, but the get() method will only be implemented when the wrapped type is Copy.* Your Weight type could just be an alias for History<u32>. Here it is in the playground.
Wrapping this code in a module is a necessary part of maintaining the abstraction. That means you can't write weight.value, you have to call weight.get(). If value were marked pub, you could assign directly to weight.value (bypassing set) and then history would be inaccurate.
As a side note, you almost never want &Vec<T> when you can use &[T], so I changed the signature of history(). Another thing you might consider is returning an iterator over the previous values (perhaps in reverse order) instead of a slice.
* A better way of getting the T out of a History<T> is to implement Deref and write *foo instead of foo.get().