DeserializationError - failed to fill whole buffer - rust

I'm writing a simple rust app using the Rocket web framework, Postgres database, and Diesel to manage database migrations. The code compiles OK, and other parts of the application run properly, but for some reason, my Expense endpoints don't seem to be working.
When hitting the /expense endpoint for example to get all the Expenses, I get the following error in the log:
Err(
DeserializationError(
Error {
kind: UnexpectedEof,
message: "failed to fill whole buffer"
}
)
)
This error is obviously not very helpful, and doesn't have a lot of detail. Why am I receiving this error, and how can I resolve this problem?
Here are the relevant parts of the code:
Expense Migration
CREATE TABLE expenses (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
description TEXT NULL,
amount DECIMAL NOT NULL,
tax_year INT NOT NULL,
purchase_date TIMESTAMP WITH TIME ZONE NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);
Expense Model
#[derive( Debug, Serialize, AsChangeset, Deserialize, Queryable, Insertable )]
#[table_name = "expenses"]
pub struct Expense {
pub id: Option<i32>,
pub name: String,
pub description: Option<String>,
pub amount: BigDecimal,
pub tax_year: i32,
pub purchase_date: Option<DateTime<Utc>>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
impl Expense {
pub fn get_all(conn: &PgConnection) -> Result<Vec<Expense>, Error> {
expenses::table.order(expenses::id.desc()).load::<Expense>(conn)
}
...
}
Controller
#[get("/", format = "json")]
pub fn get_all(conn: db::Connection) -> Result<ApiResponse, ApiError> {
let result = Expense::get_all(&conn);
match result {
Ok(r) => Ok(success(json!(r))),
Err(e) => Err(db_error(e)),
}
}
Schema
table! {
expenses (id) {
id -> Nullable<Int4>,
name -> Text,
description -> Nullable<Text>,
amount -> Numeric,
tax_year -> Int4,
purchase_date -> Nullable<Timestamptz>,
created_at -> Timestamptz,
updated_at -> Timestamptz,
}
}

In the database migration, you are not specifying a precision for the amount column(DECIMAL), which Diesel is expecting.
Try adding a precision to the amount column as per below, and reapplying the migration.
CREATE TABLE expenses (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
description TEXT NULL,
amount DECIMAL(6,2) NOT NULL,
tax_year INT NOT NULL,
purchase_date TIMESTAMP WITH TIME ZONE NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

Related

rust: Correct use of Decimal type in diesel

I'm learning to use diesel orm library, my database uses DECIMAL(8,2) type, but when I use Decimal in my model, I get an error
I am using Decimal provided by rust_decimal
diesel = { version="1.4.8", features = ["mysql", "r2d2", "chrono", "numeric"] }
rust_decimal = { version ="1.23", features = ["serde-with-str", "db-diesel-mysql"] }
rust_decimal_macros = "1.23"
my mysql table
CREATE TABLE `books` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(20) NOT NULL ,
`price` DECIMAL(8,2) UNSIGNED NOT NULL ,
`user_id` BIGINT UNSIGNED NOT NULL ,
`type` TINYINT(1) UNSIGNED DEFAULT '1' NOT NULL,
`create_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
`update_at` DATETIME on update CURRENT_TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `user_id` (`user_id`),
FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE = InnoDB ;
diesel generated schema
table! {
books (id) {
id -> Unsigned<Bigint>,
name -> Varchar,
price -> Unsigned<Decimal>,
user_id -> Unsigned<Bigint>,
#[sql_name = "type"]
type_ -> Unsigned<Tinyint>,
create_at -> Datetime,
update_at -> Datetime,
}
}
use crate::prelude::*;
use crate::schema::books;
use chrono::NaiveDateTime;
pub use rust_decimal::Decimal;
#[derive(Identifiable, Queryable, Serialize, Deserialize, Debug, Clone)]
#[table_name = "books"]
pub struct Book {
pub id: PK,
pub name: String,
pub price: Decimal,
pub user_id: PK,
pub type_: u8,
pub create_at: NaiveDateTime,
pub update_at: NaiveDateTime,
}
This is the error I get when I run cargo check
error[E0277]: the trait bound `rust_decimal::Decimal: FromSql<diesel::sql_types::Unsigned<diesel::sql_types::Numeric>, Mysql>` is not satisfied
--> src/controller/api/book.rs:19:25
|
19 | Ok(books::table.load::<models::book::Book>(&conn)?)
| ^^^^ the trait `FromSql<diesel::sql_types::Unsigned<diesel::sql_types::Numeric>, Mysql>` is not implemented for `rust_decimal::Decimal`
|
= help: the following implementations were found:
<rust_decimal::Decimal as FromSql<diesel::sql_types::Numeric, Mysql>>
= note: required because of the requirements on the impl of `Queryable<diesel::sql_types::Unsigned<diesel::sql_types::Numeric>, Mysql>` for `rust_decimal::Decimal`
= note: 2 redundant requirements hidden
= note: required because of the requirements on the impl of `Queryable<(diesel::sql_types::Unsigned<BigInt>, diesel::sql_types::Text, diesel::sql_types::Unsigned<diesel::sql_types::Numeric>, diesel::sql_types::Unsigned<BigInt>, diesel::sql_types::Unsigned<TinyInt>, diesel::sql_types::Datetime, diesel::sql_types::Datetime), Mysql>` for `Book`
= note: required because of the requirements on the impl of `LoadQuery<_, Book>` for `books::table`
note: required by a bound in `load`
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/diesel-1.4.8/src/query_dsl/mod.rs:1238:15
|
1238 | Self: LoadQuery<Conn, U>,
| ^^^^^^^^^^^^^^^^^^ required by this bound in `load`
For more information about this error, try `rustc --explain E0277`.
warning: `actix_backend` (bin "server") generated 6 warnings
error: could not compile `actix_backend` due to previous error; 6 warnings emitted
This is the rust version I'm using now
cargo --version
cargo 1.60.0 (d1fd9fe 2022-03-01)
I have also tried using bigdecimal also getting the same error
According to the documentation of diesel::sql_types::Unsigned<T> diesel does not provide builtin support for Unsigned<Decimal>. (There are no specific ToSql/FromSql/AsExpression impls listed on that page, in contrast to for example Unsigned<Integer>.) The same is true for rust_numeric::Decimal (Also only a FromSql/ToSql impl for Numeric/Decimal no one for Unsigned<Decimal>.
That all means neither of the crates support an Unsigned<Decimal> column out of the box. You can provide support for such columns by implementing the corresponding traits on your on your own. That means implementing FromSql/ToSql + deriving AsExpression/FromSqlRow for the corresponding new type wrapper.
That would result in code like this:
use diesel::sql_types::{Unsigned, Decimal};
use diesel::serialize::{self, ToSql};
use diesel::deserialize::{self, FromSql};
use diesel::mysql::Mysql;
#[derive(AsExpression, FromSqlRow)]
#[sql_type = "Unsigned<Decimal>"]
struct DecimalWrapper(rust_decimal::Decimal);
impl FromSql<Unsigned<Decimal>, Mysql> for DecimalWrapper {
fn from_sql(bytes: Option<&[u8]>) -> deserialize::Result<Self> {
<rust_decimal::Decimal as FromSql<Decimal, Mysql>>::from_sql(bytes).map(Self)
}
}
impl ToSql<Unsigned<Decimal>, Mysql> for DecimalWrapper {
fn to_sql<W: Write>(&self, out: &mut serialize::Output<'_, W, DB>) -> serialize::Result {
<_ as ToSql<Decimal, Mysql>>::to_sql(&self.0, out)
}
}

Eager loading of relations

I have three models with these relations:
Artist 1<->n Song 1<->n Play
I want to load Plays and eagerly load its Songs and its Artists.
I did this in PHP framework Eloquent easily. And it takes only 3 DB requests (one for each model).
Is this some how possible with diesel?
I think the Play model should look something like this:
#[derive(Queryable, Serialize, Deserialize)]
pub struct Play {
pub id: u64,
pub song_id: u64,
pub song: Song, // <-- ????
pub date: NaiveDateTime,
pub station_id: u64,
}
And loading something like this:
// loads one model only at the moment
// it should load the related models Song and Artist too
let items = plays.filter(date.between(date_from, date_to)).load(&*conn)?;
The resulting struct should be JSON serialized to be used by the REST API. But I can't find a way to get the needed struct in one simple and efficient way.
One way to achieve this is to join the two other tables. I did not figure out how to avoid enumerating all table fields unfortunately. This is not an elegant solution, but it should work.
fn select_plays_with_song_and_artist(
conn: &Conn,
date_from: NaiveDate,
date_to: NaiveDate,
) -> Result<Vec<(models::Play, models::Song, models::Artist)>> {
plays::table
.filter(plays::date.between(date_from, date_to))
.join(songs::table.join(artists::table))
.select((
(
plays::id,
plays::song_id,
plays::date,
plays::station_id,
),
(
songs::id,
// other relevant fields...
),
(
artist::id,
// other relevant fields...
),
))
.load::<(models::Play, models::Song, models::Artist)>(conn)
}
First of all: If you post a question about diesel online, please always include the relevant part of your schema, otherwise others need to guess how it looks like.
I will assume the following schema for your question:
table! {
plays (id)
id -> BigInt,
song_id -> BigInt,
date -> Timestamp,
station_id -> BigInt,
}
}
table! {
songs(id) {
id -> BigInt,
name -> Text,
}
}
joinable!(plays -> songs (song_id));
Now there is some issue with your Play struct. u64 is not a type supported by diesel. Checkout this documentation about which types are compatible with a BigInt SQL type.
That means I will assume the following structs implementing Queryable:
#[derive(Queryable, Serialize, Deserialize)]
pub struct Play {
pub id: i64,
// pub song_id: u64, leave this out as it is a duplicate with `song` below
pub song: Song,
pub date: NaiveDateTime,
pub station_id: i64,
}
#[derive(Queryable, Serialize, Deserialize)]
pub struct Song {
pub id: i64,
pub name: String,
}
Now to come to your core question: At first, diesel does not have the concept of a single model like known from other ORM's. If you have ever used such a thing, don't try to apply that knowledge to diesel, it won't work. There are a set of traits describing a capability of each struct. As we are talking about querying here I will concentrate on Queryable. Queryable is designed to map the result of a query (which could include zero, one or more tables) to a rust data structure. By default (by using the derive) it does this by mapping fields structural, that means your select clause needs to match the structure deriving Queryable.
Now back to your concrete problem. It means that we must construct a query that returns (i64, (i64, String), NaiveDateTime, i64) (or better the fields with the corresponding SQL types).
For this case this is as easy as using a single join:
plays::table.inner_join(songs::table)
.select((plays::id, songs::all_columns, plays::date, plays::station_id))
The only important thing here for the mapping is the select clause. You can append/prepend whatever other QueryDsl method you like.
I couldn't find anything elegant way of doing this. Here's the snippet, if you want to do it in two steps.
First, get the both models using join on both table.
impl Comment {
pub fn list(conn: &PgConnection, page_id: i32) -> Result<Vec<(Comment, User)>, Error> {
use crate::schema::users;
comment::table
.inner_join(users::table)
.select((comment::all_columns, users::all_columns))
.filter(comment::page_id.eq(page_id))
.load(conn)
}
}
Now, iterate over the result to make it fit into your own api response structure.
#[derive(Serialize)]
pub struct CommentListResponse {
#[serde(flatten)]
pub comment: Comment,
pub user: UserResponse
}
let result = Comment::list(&connection, page.id)?;
let mut comment_list = vec![];
for (comment, user) in result {
let cr = CommentListResponse {
comment,
user: UserResponse::from(user),
};
comment_list.push(cr);
}
Ok(HttpResponse::Ok().json(comment_list))

Rust diesel aggregation query

I have some problems with performing a query using Diesel library.
I have table:
create table tasks
(
id serial not null
constraint tasks_pkey
primary key,
name varchar(128) not null,
rate numeric(10, 2) default 0 not null,
time_spent numeric(10, 2)
);
Model:
#[derive(Debug, Serialize, Deserialize, Identifiable, Queryable)]
#[table_name="tasks"]
pub struct Task {
pub id: i32,
pub name: String,
pub rate: BigDecimal,
pub time_spent: Option<BigDecimal>
}
Schema:
table! {
tasks (id) {
id -> Int4,
name -> Varchar,
rate -> Numeric,
time_spent -> Nullable<Numeric>,
}
}
The target is that I need to get Sum of rate * time_spent. I meam total amount. Who can help me with that?

Pass any Diesel table as an argument

I have Rust project with Diesel implemented and it generated schema.rs file which contains all of my tables:
table! {
users (id) {
id -> Uuid,
name -> Varchar,
}
}
table! {
items (id) {
id -> Uuid,
name -> Varchar,
}
}
How could I pass any table as an argument inside of my utility function?
For instance,
pub trait Search {
fn internal_get_by_id(
diesel_table: diesel::table, // this argument should pass any diesel table
table_id: diesel::table::id, // this argument should pass Uuid from table
conn: &Conn,
id: Uuid,
) -> Fallible<Option<Self>>
where
Self: Sized,
{
diesel_table
.filter(table_id.eq(id))
.first(conn.raw())
.optional()
.map_err(Error::from)
}
fn get_by_id(conn: &Conn, id: Uuid) -> Fallible<Option<Self>>
where
Self: Sized;
}
impl Search for User {
fn get_by_id(conn: &Conn, id: Uuid) -> Fallible<Option<User>> {
Self::internal_get_by_id(users::table, users::id, conn, id)
}
}
impl Search for Item {
fn get_by_id(conn: &Conn, id: Uuid) -> Fallible<Option<Item>> {
Self::internal_get_by_id(items::table, items::id, conn, id)
}
}
First of all: It is generally not a good idea to write code that is generic over multiple tables/columns using Diesel in Rust, especially if you are new to the language and don't have a really good understanding on trait bounds and where clauses yet.
You need to list all trait bounds that are required to allow building this generic query so that everything can be checked at compile time. The following implementation should solve this (not tested, hopefully I did not miss a trait bound)
fn internal_get_by_id<T, C>(
diesel_table: T,
table_id: C,
conn: &Conn,
id: Uuid,
) -> Fallible<Option<Self>>
where
Self: Sized,
T: Table + FilterDsl<dsl::Eq<C, Uuid>>,
C: Column + Expression<SqlType = diesel::sql_types::Uuid>,
dsl::Filter<T, dsl::Eq<C, Uuid>>: LimitDsl,
dsl::Limit<dsl::Filter<T, dsl::Eq<C, Uuid>>>: LoadQuery<Conn, Self>,
Self: Queryable<dsl::SqlTypeOf<dsl::Limit<dsl::Filter<T, dsl::Eq<C, Uuid>>>>, Conn::Backend>,
{
diesel_table
.filter(table_id.eq(id))
.first(conn.raw())
.optional()
.map_err(Error::from)
}

What is the standard pattern to relate three tables (many-to-many relation) within Diesel?

Database - Postgres
I have the following relation:
users <—>> users_organizations <<—> organizations
Schema:
table! {
organizations (id) {
id -> Int4,
name -> Varchar,
}
}
table! {
users (id) {
id -> Int4,
name -> Varchar,
email -> Varchar,
password -> Varchar,
}
}
table! {
users_organizations (id, user_id, organization_id) {
id -> Int4,
user_id -> Int4,
organization_id -> Int4,
}
}
Models:
#[derive(Identifiable, Queryable, Debug, Serialize, Deserialize)]
pub struct Organization {
pub id: i32,
pub name: String,
}
#[derive(Identifiable, Queryable, PartialEq, Debug, Serialize, Deserialize)]
pub struct User {
pub id: i32,
pub name: String,
pub email: String,
pub password: String,
}
#[derive(Identifiable, Queryable, Debug, Associations, Serialize, Deserialize)]
#[belongs_to(User)]
#[belongs_to(Organization)]
#[table_name = "users_organizations"]
pub struct UserOrganization {
pub id: i32,
pub user_id: i32,
pub organization_id: i32,
}
I want to create an organization. To support such relation, I have to manually add ids of user and organization to the users_organizations table. Is there any better approach to implement such relation?
let new_organization = NewOrganization { name: &msg.name };
let organization = insert_into(organizations::table)
.values(&new_organization)
.get_result::(conn)
.map_err(|_| error::ErrorInternalServerError(“Error creating organization”))?;
let new_user_org = NewUserOrganizationIDs {
user_id: msg.user_id,
organization_id: organization.id,
};
insert_into(users_organizations::table)
.values(&new_user_org)
.get_result::<UserOrganization>(conn)
.map_err(|_| error::ErrorInternalServerError("Error creating user-organization data"))
Same question here. In case of selecting all organizations that relate to user (and vice verse) I came up with the following code:
let user = users::table.filter(users::id.eq(&msg.user_id))
.get_result::<User>(conn)
.map_err(|_| error::ErrorNotFound("User doesn't exist"))?;
let user_organizations = UserOrganization::belonging_to(&user)
.get_results::<UserOrganization>(conn)
.map_err(|_| error::ErrorNotFound("User doesn't have any organization"))?;
let mut organization_ids = vec![];
for user_org in &user_organizations {
organization_ids.push(user_org.organization_id);
}
organizations::table.filter(organizations::id.eq_any(organization_ids))
.get_results::<Organization>(conn)
.map_err(|_| error::ErrorNotFound("Organizations don't exist"))
This answer is from the Diesel chat by #SRugina and #weiznich (edited and adapted for the question).
How do I write a many-to-many relationship with Diesel?
I normally combine belonging_to and join, so something like:
UserOrganization::belonging_to(&organizations)
.inner_join(user::table)
Is there anything akin to belonging_to_many?
No, belonging_to_many does not exist because Diesel does not try to hide the database from you. Doing that would cause issues as soon as you want to do complex or non-standard things. Depending on your exact use case, joining all three tables could also be an option.
How do I use inner_join in this scenario?
You have three tables: users, organizations and user_organizations and you want to get all organizations for a specific user.
There are two variants doing that. The first variant is only one query, but may not match your required data layout if you want to do that for all users:
users::table
.inner_join(user_organizations::table.inner_join(organizations::table))
.filter(users::id.eq(user_id))
.select(organizations::all_columns)
.load::<Organization>(conn)?;
The second variant allows grouping the results for each user using the built-in associations API:
let user = users::table
.find(user_id)
.first::<User>(conn)?;
UserOrganization::belonging_to(&user)
.inner_join(organizations::table)
.select(organizations::all_columns)
.load::<Organization>(conn)?;
Inserting requires three separate inserts. We do not try to hide that because in the end it's a user choice how to handle data consistency in case of a failed insert there. Using a transaction is a common choice.

Resources