How to make serde deserialize BigInt as u64? - rust

I'm using toml and num-bigint with the serde feature to deserialize the following data:
[[trades]]
action = "Buy"
date_time = 2019-04-15T15:36:00+01:00
fee = [1, [44000000]]
id = "#1"
price = [-1, [20154500000]]
quantity = [1, [200000000]]
But I'm getting this error:
Error: Error { inner: ErrorInner { kind: Custom, line: Some(7), col: 14, at: Some(156), message: "invalid value: integer `20154500000`, expected u32", key: ["trades", "price"] } }
Of course, if I make the price value smaller than u32::MAX, the program compiles fine. But I want to use this high value, because I'm scaling numbers by 1e8 to avoid dealing with floating-point arithmetic.
Is it possible to make serde deserialize BigInts to u64 instead?
use num_bigint::BigInt;
use serde_derive::Deserialize;
use toml::from_str;
use toml::value::Datetime;
#[derive(Debug, Deserialize)]
pub struct Trade {
pub action: String,
pub date_time: Datetime,
pub fee: BigInt,
pub id: Option<String>,
pub price: BigInt,
pub quantity: BigInt,
}
#[derive(Debug, Deserialize)]
pub struct TradeTable {
pub trades: Vec<Trade>,
}
fn main() {
let trades_string: String = String::from("[[trades]]\naction = \"Buy\"\ndate_time = 2019-04-15T15:36:00+01:00\nexchange = \"Degiro\"\nfee = [1, [44000000]]\nid = \"#1\"\nprice = [-1, [20154500000]]\nquantity = [1, [200000000]]");
let trade_table: TradeTable = from_str(&trades_string).unwrap();
let trades: Vec<Trade> = trade_table.trades;
}
Also, here's a link to a Rust Playground. Note that you will need to copy the code to your local machine, because you need the serde feature from num-bigint:
Cargo.toml
[dependencies.num-bigint]
version = "0.2.6"
features = ["serde"]

How did you create this file -- did you make it by serializing a BigInt, or did you write it by hand?
I wrote the data by hand.
Your data is invalid, the following work:
use num_bigint::BigInt;
use std::str::FromStr;
#[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
pub struct Trade {
pub price: BigInt,
}
fn main() {
let price = BigInt::from_str("-201545000000000").unwrap();
let trade = Trade { price };
let j = serde_json::to_string(&trade).unwrap();
println!("{}", j);
let a: Trade = serde_json::from_str(&j).unwrap();
assert_eq!(trade, a);
let j = toml::to_string(&trade).unwrap();
println!("{}", j);
let b: Trade = toml::from_str(&j).unwrap();
assert_eq!(trade, b);
}
You are not supposed to create it by hand.

Related

Get an attribute from a struct in a Vec of structs

I am trying to filter a vector of structs by a struct attribute and then return another attribute. However I'm not quite sure how to elegantly extract the value.
My function will take in the name of a budget and then I want to return the id of that budget by searching through a list of Budgets.
Is there a cleaner way to do this in one pass without allocating a new struct?
// Budget structs
#[derive(Serialize, Deserialize, Debug)]
pub struct Budgets {
pub budgets: Vec<Budget>
}
#[derive(Serialize, Deserialize, Debug)]
pub struct Budget {
pub id: String,
name: String,
last_modified_on: String,
first_month: String,
last_month: String,
date_format: DateFormat,
currency_format: Option<CurrencyFormat>,
accounts: Option<Accounts>,
}
impl Adapter {
fn get_budget_id(&self, budget_name: &str) -> anyhow::Result<String> {
let budget_data = self.get_budgets()?;
let budget = budget_data.data.budgets.into_iter()
.filter(|b| b.name == budget_name);
//.collect::<Budgets>() <-- FromIterator not implemented
//.remove(0);
let b = Budgets {
budgets: Vec::from_iter(budget)
};
if b.budgets.is_empty() {
Err(anyhow::anyhow!("Error, no budget with {} name found", budget_name))
} else {
Ok(b.budgets[0].id.clone())
}
}
}
After some deeper digging into the docs and suggestions on here, this was my final solution:
fn get_budget_id(&self, budget_name: &str) -> anyhow::Result<String> {
let budget_data = self.get_budgets()?;
if let Some(budget) = budget_data.data.budgets.into_iter()
.find(|b| b.name == budget_name) {
Ok(budget.id)
} else {
Err(anyhow::anyhow!("Error, no budget with name {} found", budget_name))
}
}
let budget:Budget = budget_data.data.budgets.into_iter()
.find(|b| b.name == budget_name)
.ok_or(anyhow::anyhow!("Error, no budget with {} name found", budget_name))?;

Deserialize using a function of the tag

An API with this internally tagged field structure, with "topic" being the tag:
{
"topic": "Car"
"name": "BMW"
"HP": 250
}
This can be deserialized with
#[derive(Serialize, Deserialize)]
#[serde(tag = "topic")]
pub enum catalog {
CarEntry(Car),
... (other types)
}
#[derive(Serialize, Deserialize)]
pub struct Car {
pub name: String
pub HP: i32
}
It turns out that instead of reporting the topic as just Car, the API actually sends Car.product1 or Car.product2 etc.
This breaks the deserialization, because the deserializer doesn't know what the type is based on the string. Is there a way to supply a function to chop off the type string so that the correct model is found?
I don't think serde provides a way to mangle the tag before using it (at least I don't see anything relevant). And the generated serializers for tagged enums are relatively complex, with internal caching if the tag isn't the first field, and whatnot, so I wouldn't want to reproduce that in a custom deserializer.
The cheapest (but not necessarily most efficient) shot at this is to deserialize to serde_json::Value first, manually process the tag, and then deserialize the serde_json::Values to whatever struct you want.
Do that in a custom deserializer, and it starts looking reasonable:
impl<'de> Deserialize<'de> for Catalog {
fn deserialize<D>(d: D) -> Result<Self, <D as Deserializer<'de>>::Error>
where
D: Deserializer<'de>,
{
use serde_json::{Map, Value};
#[derive(Deserialize)]
struct Pre {
topic: String,
#[serde(flatten)]
data: Map<String, Value>,
}
let v = Pre::deserialize(d)?;
// Now you can mangle Pre any way you want to get your final structs.
match v.topic.as_bytes() {
[b'C', b'a', b'r', b'.', _rest # ..] => Ok(Catalog::CarEntry(
serde_json::from_value(v.data.into()).map_err(de::Error::custom)?,
)),
[b'B', b'a', b'r', b'.', _rest # ..] => Ok(Catalog::BarEntry(
serde_json::from_value(v.data.into()).map_err(de::Error::custom)?,
)),
_ => return Err(de::Error::unknown_variant(&v.topic, &["Car.…", "Bar.…"])),
}
}
}
Playground
Btw, what do you want to do with the suffix of topic? Throw it away? How do you plan on handling serialization if you do throw it away?
You can directly use enum instead of defining extra struct type.
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag = "topic")]
pub enum Catalog {
Car { name: String, hp: i32 }
}
fn main() {
let car = Catalog::Car { name: String::from("BMW"), hp: 2000 };
// Convert the Car to a JSON string.
let serialized = serde_json::to_string(&car).unwrap();
// Prints serialized = {"topic":"Car","name":"BMW","hp":2000}
println!("serialized = {}", serialized);
// Convert the JSON string back to a Car.
let deserialized: Catalog = serde_json::from_str(&serialized).unwrap();
// Prints deserialized = Car { name: "BMW", hp: 2000 }
println!("deserialized = {:?}", deserialized);
}
Playground
You can use #[serde(rename()] to rename type in output
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug)]
#[serde(tag = "topic")]
pub enum Catalog {
#[serde(rename(serialize = "Car", deserialize = "CarEntry"))]
CarEntry(Car),
}
#[derive(Serialize, Deserialize, Debug)]
pub struct Car {
pub name: String,
pub hp: i32
}
fn main() {
let car = Car { name: String::from("BMW"), hp: 2000 };
let catalog = Catalog::CarEntry(car);
// Convert the Car to a JSON string.
let serialized = serde_json::to_string(&catalog).unwrap();
// Prints serialized = {"topic":"Car","name":"BMW","hp":2000}
println!("serialized = {}", serialized);
// Convert the JSON string back to a Car.
let deserialized: Car = serde_json::from_str(&serialized).unwrap();
// Prints deserialized = Car { name: "BMW", hp: 2000 }
println!("deserialized = {:?}", deserialized);
}
Playground

Can one specify serde's rename_all rule at runtime?

I have a data model that I would like to be deserialized from "camelCase" to the rust standard "snake_case" when reading from a source, X. But I'd like to leave it in "snake_case" when reading or writing to another source, Y.
For example, the following code,
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
struct Data {
foo_bar: String,
hello_word: String,
}
can only be encoded and decoded in camel case. Even if I manually defined my Serialize and Deserialize implementations, I can't define multiple for the same struct. I could define a second struct that's a copy/paste of the other and then derive but that method would get tedious with multiple large structs. What I would really like to do is specify that rename_all attribute at run-time. But I'm not seeing any way to do that in serde's API.
I think the best way sigh is to just write out one struct Data_ per #[serde(rename_all = ...)], then write one additional struct Data that will be the in-memory representation (which won't be serializable, to remove ambiguity), then implement From in both directions for the Data_s and Data so that they're interconvertible.
Thankfully, we can use a macro so that we only have to specify the fields once. (It is incredibly disgusting nonetheless.)
This playground available here.
use serde::{Deserialize, Serialize}; // 1.0.130
use serde_json; // 1.0.69
macro_rules! interconvertible {
($T:ident <-> $U:ident, $($field_name:ident),*) => {
impl From<$T> for $U {
fn from(t: $T) -> Self {
let $T { $($field_name),* } = t;
Self { $($field_name),* }
}
}
impl From<$U> for $T {
fn from(u: $U) -> Self {
let $U { $($field_name),* } = u;
Self { $($field_name),* }
}
}
};
}
macro_rules! create_data_structs {
($($field_name:ident: $field_type:ty),* $(,)?) => {
#[derive(Serialize, Deserialize, Debug)]
#[serde(rename_all = "camelCase")]
struct DataX {
$($field_name: $field_type),*
}
#[derive(Serialize, Deserialize, Debug)]
#[serde(rename_all = "snake_case")]
struct DataY {
$($field_name: $field_type),*
}
#[derive(Debug)]
struct Data {
$($field_name: $field_type),*
}
interconvertible!(DataX <-> Data, $($field_name),*);
interconvertible!(DataY <-> Data, $($field_name),*);
}
}
create_data_structs!(foo_bar: String, hello_world: String);
fn main() -> serde_json::Result<()> {
let x1: DataX = serde_json::from_str(r#"{"fooBar": "a", "helloWorld": "b"}"#)?;
let y1: DataY = serde_json::from_str(r#"{"foo_bar": "a", "hello_world": "b"}"#)?;
println!("{:?}, {:?}", x1, y1);
let x2: Data = x1.into();
let y2: Data = y1.into();
println!("{:?}, {:?}", x2, y2);
let x_string = serde_json::to_string(&DataX::from(x2))?;
let y_string = serde_json::to_string(&DataY::from(y2))?;
println!("{:?}, {:?}", x_string, y_string);
Ok(())
}
The output is:
DataX { foo_bar: "a", hello_world: "b" }, DataY { foo_bar: "a", hello_world: "b" }
[Data { foo_bar: "a", hello_world: "b" }, Data { foo_bar: "a", hello_world: "b" }]
"{\"fooBar\":\"a\",\"helloWorld\":\"b\"}", "{\"foo_bar\":\"a\",\"hello_world\":\"b\"}"
Since I'm only every decoding from source X I can utilize the #[serde(alias = ???)] macro. So my above use case would be
#[derive(Serialize, Deserialize)]
struct Data {
#[serde(alias="fooBar")]
foo_bar: String,
#[serde(alias="helloWorld")]
hello_word: String,
}
It's still a little tedious but better than an intermediate struct. It won't work though if I want to decode or encode to different cases.
(I'm not going to mark this as an answer because it's a work-around for my specific use case. If anyone has a more generic solution feel free to answer.)

"argument requires that `record` is borrowed for `'static`" when using belonging_to associations with tokio-diesel

I am attempting to use Tokio-Diesel to use async/await with my ORM. I am not able to use Diesel's belonging_to associations, as it complains that I need a static lifetime. I've tried async blocks but nothing changes the lifetime error.
Cargo.toml
[package]
name = "tokio_diesel"
version = "0.1.0"
authors = ["servonlewis"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
diesel = { version = "1.4.3", features = ["postgres", "r2d2", "serde_json", "chrono", "uuid"] }
r2d2 = "0.8.8"
tokio = { version = "0.2", features = ["full"] }
tokio-diesel = "0.3.0"
uuid = { version = "0.6.5", features = ["serde", "v4"] }
main.rs
#[macro_use]
extern crate diesel;
use diesel::prelude::*;
use tokio_diesel::*;
table! {
articles (id) {
id -> Integer,
title -> Text,
}
}
table! {
articles_visible_to_groups (id) {
id -> Integer,
article_id -> Integer,
groups_id -> Integer,
}
}
table! {
groups (id) {
id -> Integer,
name -> Varchar,
}
}
joinable!(articles_visible_to_groups -> articles (article_id));
joinable!(articles_visible_to_groups -> groups (groups_id));
allow_tables_to_appear_in_same_query!(articles, articles_visible_to_groups, groups,);
use crate::articles::dsl::*;
use crate::groups::dsl::*;
use diesel::pg::PgConnection;
use diesel::r2d2::{ConnectionManager, Pool};
pub type PostgresPool = Pool<ConnectionManager<PgConnection>>;
pub fn get_pool() -> PostgresPool {
let url = "postgres://actix:actix#localhost:5432/actix".to_string();
let mgr = ConnectionManager::<PgConnection>::new(url);
r2d2::Pool::builder()
.build(mgr)
.expect("could not build connection pool")
}
#[derive(Queryable, Identifiable, Associations, Debug)]
#[table_name = "articles"]
pub struct Article {
pub id: i32,
pub title: String,
}
#[derive(Identifiable, Queryable, Associations, Debug, PartialEq)]
#[belongs_to(Group, foreign_key = "groups_id")]
#[belongs_to(Article)]
#[table_name = "articles_visible_to_groups"]
pub struct ArticleVisibleToGroup {
pub id: i32,
pub article_id: i32,
pub groups_id: i32,
}
#[derive(Queryable, Identifiable, Associations, Debug, PartialEq)]
pub struct Group {
pub id: i32,
pub name: String,
}
pub async fn groups_who_can_see(conn: &PostgresPool, an_id: i32) -> Vec<Group> {
use diesel::pg::expression::dsl::any;
let record = articles
.find(an_id)
.load_async::<Article>(conn)
.await
.unwrap();
let list = ArticleVisibleToGroup::belonging_to(&record)
.select(articles_visible_to_groups::groups_id)
.load_async::<i32>(conn)
.await
.unwrap();
groups
.filter(groups::id.eq_all(any(list)))
.load_async::<Group>(conn)
.await
.unwrap()
}
#[tokio::main]
async fn main() {
println!("Hello, world!");
// pool.get_ref().to_owned(),
let pool = get_pool();
let res = groups_who_can_see(&pool, 1).await;
println!("{:#?}", res);
}
error[E0597]: `record` does not live long enough
--> src/main.rs:80:52
|
80 | let list = ArticleVisibleToGroup::belonging_to(&record)
| ------------------------------------^^^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `record` is borrowed for `'static`
...
91 | }
| - `record` dropped here while still borrowed
Looks like i was able to get around this issue by just not using belonging_to and instead using filter.
Instead of the example above, this one works asyncronously.
pub async fn groups_who_can_see(
conn: &PostgresPool,
an_id: String,
) -> FieldResult<Vec<Group>> {
use diesel::pg::expression::dsl::any;
let my_id = uuid::Uuid::parse_str(&an_id)?;
let list = articles_visible_to_groups
.filter(article_id.eq_all(my_id))
.select(articles_visible_to_groups::groups_id)
.load_async::<uuid::Uuid>(&conn)
.await?;
let data = groups
.filter(groups::id.eq_all(any(list)))
.load_async::<Group>(&conn)
.await;
graphql_translate(data)
}

Is it possible to return either a borrowed or owned type in Rust?

In the following code, how can I return the reference of floor instead of a new object? Is it possible to let the function return either a borrowed reference or an owned value?
extern crate num; // 0.2.0
use num::bigint::BigInt;
fn cal(a: BigInt, b: BigInt, floor: &BigInt) -> BigInt {
let c: BigInt = a - b;
if c.ge(floor) {
c
} else {
floor.clone()
}
}
Since BigInt implements Clone, you can use a std::borrow::Cow:
use num::bigint::BigInt; // 0.2.0
use std::borrow::Cow;
fn cal(a: BigInt, b: BigInt, floor: &BigInt) -> Cow<BigInt> {
let c: BigInt = a - b;
if c.ge(floor) {
Cow::Owned(c)
} else {
Cow::Borrowed(floor)
}
}
Later, you can use Cow::into_owned() to get a owned version of BigInt, or just use it as a reference:
fn main() {
let a = BigInt::from(1);
let b = BigInt::from(2);
let c = &BigInt::from(3);
let result = cal(a, b, c);
let ref_result = &result;
println!("ref result: {}", ref_result);
let owned_result = result.into_owned();
println!("owned result: {}", owned_result);
}

Resources