Eager loading of relations - rust

I have three models with these relations:
Artist 1<->n Song 1<->n Play
I want to load Plays and eagerly load its Songs and its Artists.
I did this in PHP framework Eloquent easily. And it takes only 3 DB requests (one for each model).
Is this some how possible with diesel?
I think the Play model should look something like this:
#[derive(Queryable, Serialize, Deserialize)]
pub struct Play {
pub id: u64,
pub song_id: u64,
pub song: Song, // <-- ????
pub date: NaiveDateTime,
pub station_id: u64,
}
And loading something like this:
// loads one model only at the moment
// it should load the related models Song and Artist too
let items = plays.filter(date.between(date_from, date_to)).load(&*conn)?;
The resulting struct should be JSON serialized to be used by the REST API. But I can't find a way to get the needed struct in one simple and efficient way.

One way to achieve this is to join the two other tables. I did not figure out how to avoid enumerating all table fields unfortunately. This is not an elegant solution, but it should work.
fn select_plays_with_song_and_artist(
conn: &Conn,
date_from: NaiveDate,
date_to: NaiveDate,
) -> Result<Vec<(models::Play, models::Song, models::Artist)>> {
plays::table
.filter(plays::date.between(date_from, date_to))
.join(songs::table.join(artists::table))
.select((
(
plays::id,
plays::song_id,
plays::date,
plays::station_id,
),
(
songs::id,
// other relevant fields...
),
(
artist::id,
// other relevant fields...
),
))
.load::<(models::Play, models::Song, models::Artist)>(conn)
}

First of all: If you post a question about diesel online, please always include the relevant part of your schema, otherwise others need to guess how it looks like.
I will assume the following schema for your question:
table! {
plays (id)
id -> BigInt,
song_id -> BigInt,
date -> Timestamp,
station_id -> BigInt,
}
}
table! {
songs(id) {
id -> BigInt,
name -> Text,
}
}
joinable!(plays -> songs (song_id));
Now there is some issue with your Play struct. u64 is not a type supported by diesel. Checkout this documentation about which types are compatible with a BigInt SQL type.
That means I will assume the following structs implementing Queryable:
#[derive(Queryable, Serialize, Deserialize)]
pub struct Play {
pub id: i64,
// pub song_id: u64, leave this out as it is a duplicate with `song` below
pub song: Song,
pub date: NaiveDateTime,
pub station_id: i64,
}
#[derive(Queryable, Serialize, Deserialize)]
pub struct Song {
pub id: i64,
pub name: String,
}
Now to come to your core question: At first, diesel does not have the concept of a single model like known from other ORM's. If you have ever used such a thing, don't try to apply that knowledge to diesel, it won't work. There are a set of traits describing a capability of each struct. As we are talking about querying here I will concentrate on Queryable. Queryable is designed to map the result of a query (which could include zero, one or more tables) to a rust data structure. By default (by using the derive) it does this by mapping fields structural, that means your select clause needs to match the structure deriving Queryable.
Now back to your concrete problem. It means that we must construct a query that returns (i64, (i64, String), NaiveDateTime, i64) (or better the fields with the corresponding SQL types).
For this case this is as easy as using a single join:
plays::table.inner_join(songs::table)
.select((plays::id, songs::all_columns, plays::date, plays::station_id))
The only important thing here for the mapping is the select clause. You can append/prepend whatever other QueryDsl method you like.

I couldn't find anything elegant way of doing this. Here's the snippet, if you want to do it in two steps.
First, get the both models using join on both table.
impl Comment {
pub fn list(conn: &PgConnection, page_id: i32) -> Result<Vec<(Comment, User)>, Error> {
use crate::schema::users;
comment::table
.inner_join(users::table)
.select((comment::all_columns, users::all_columns))
.filter(comment::page_id.eq(page_id))
.load(conn)
}
}
Now, iterate over the result to make it fit into your own api response structure.
#[derive(Serialize)]
pub struct CommentListResponse {
#[serde(flatten)]
pub comment: Comment,
pub user: UserResponse
}
let result = Comment::list(&connection, page.id)?;
let mut comment_list = vec![];
for (comment, user) in result {
let cr = CommentListResponse {
comment,
user: UserResponse::from(user),
};
comment_list.push(cr);
}
Ok(HttpResponse::Ok().json(comment_list))

Related

How to create similar models using rust, rocket.rs and diesel without duplicating code

I am building a restful api using rocket.rs. I am a Rust beginner and learning rocket.rs. I have two database tables: chair and table. I use Rust Diesel ORM. The chair is represented by the Chair struct and table by the Table struct. A chair can refer to a table having Some(table_id) or to no table if None.
I am trying to create a POST endpoint that uses a chair id as parameter and transforms it into a Chair struct. Also, I want to validate that this chair is associated to a table. If the provided chair id is not associated to a table, then an error is sent. I have an implementation working already, but I feel that this is not a clean solution.
#[derive(Queryable, Serialize)]
pub struct Chair {
pub id: i32,
pub color: String,
pub table_id: Option<i32>
}
#[derive(Queryable, Serialize)]
pub struct ChairWithTable {
pub id: i32,
pub color: String,
pub table: i32
}
impl TryFrom<Chair> for ChairWithTable {
type Error = ChairError;
fn try_from(chair: Chair) -> Result<Self, Self::Error> {
if chair.table_id.is_none() {
Err(ChairError::WithoutTable)
} else {
Ok(ChairWithTable {
id: chair.id,
color: chair.color,
table_id: chair.table_id.unwrap()
})
}
}
}
pub struct Table {
pub id: i32,
// ... whatever other fields. this is not important in the example
}
impl<'a> FromParam<'a> for ChairWithTable {
// implemention details omitted
}
impl<'a> FromParam<'a> for Chair {
// implemention details omitted
}
Here we pretend we can add pieces (could be anything else, not very relevant to the case anyhow) to a chair but only if the chair is bound to a table. Does not make a lot of sense, but I changed the table names so bear with me.
#[post("/chairs/<chair>/pieces", rank = 1)]
async fn pieces(chair: ChairWithTable, conn: DatabaseConnection) -> Result<Json<[/*irrelevent*/]>, Forbidden<Option<String>>> {
// implementation details omitted
}
So what I feel is wrong about my solution, is that there is a lot of duplication between Chair and ChairWithTable. Actually I copied all the fields and removed the Option wrapping the i32. I also had to impl TryFrom<Chair> for ChairWithTable. This is already verbose and it would get much more complicated if I had even just two Option fields on the same model. I could also omit the ChairWithTable struct and only keep Chair struct. But then when I get in my controller method, I have to check if table_id is Some or None every time and throw an error mid way in the controller code instead of using the param guards as validation.
Coming from web dev background in php and TypeScript, my brain would try to solve this with TypeScript's utility types. I would do something like take the Chair struct and make the field required with something that looks like this type ChairWithTabl = Pick<Chair, 'id'|'color'> & Required<Pick<Chair, 'table_id'>>. Another way of solving it would for me to go with a library like Zod that creates types on the fly with type inference.
Think that one solution in pure Rust would be to aim for struct composition like this answer for this question. But as far as I know, it is not possible to use composition on diesel models
How would you recommend to tackle this kind problem in the context of Rust, Rocket.rs and Diesel?

Best way to populate a struct from a similar struct?

I am new to Rust, and am attempting to take a struct returned from a library (referred to as source struct) and convert it into protobuf message using prost. The goal is to take the source struct, map the source struct types to the protobuf message types (or rather, the appropriate types for prost-generated struct, referred to as message struct), and populate the message struct fields using fields from the source struct. The source struct fields are a subset of message struct fields. For example:
pub struct Message {
pub id: i32,
pub email: String,
pub name: String,
}
pub struct Source {
pub email: Email,
pub name: Name,
}
So, I would like to take fields from from Source, map the types to corresponding types in Message, and populate the fields of Message using Source (fields have the same name). Currently, I am manually assigning the values by creating a Message struct and doing something like message_struct.email = source_struct.email.to_string();. Except I have multiple Message structs based on protobuf, some having 20+ fields, so I'm really hoping to find a more efficient way of doing this.
If I understand you correctly you want to generate define new struct based on fields from another. In that case you have to use macros.
https://doc.rust-lang.org/book/ch19-06-macros.html
Also this question (and answer) could be useful for you Is it possible to generate a struct with a macro?
To convert struct values from one to another struct best way is to use From<T> or Into<T> trait.
https://doc.rust-lang.org/std/convert/trait.From.html
This is called FRU (functional record update).
This currently works only for structs with the same type and structs with the same type modulo generic parameters.
The RFC-2528 talks about a generalization that would make this work:
struct Foo {
field1: &'static str,
field2: i32,
}
struct Bar {
field1: f64,
field2: i32,
}
let foo = Foo { field1: "hi", field2: 1 };
let bar = Bar { field1: 3.14, ..foo };
Unfortunately, this has not yet been implemented.
You could make a method to create a Message from a Source like this.
impl Message {
pub fn from_source(source: &Source, id: i32) -> Self {
Message {
id: id,
email: source.email.to_string(),
name: source.name.to_string(),
}
}
}
And then,
let source = // todo
let id = // todo
let message = Message::from_source(&source, id);

How to date format an SQL result using Diesel?

I am using Diesel to query a DB in PostgreSQL using JOINs:
let product_id = 1;
sales::table
.inner_join(product::table)
.select((
product::description,
sales::amount,
sales::date_sale
))
.filter(sales::product_id.eq(product_id))
.load(&diesel::PgConnection)
My model:
pub struct Sales {
pub id: i32,
pub product_id: Option<i32>,
pub amount: Option<BigDecimal>,
pub date_sale: Option<NaiveDateTime>
}
The result is as expected, but I need to give a date format to the field sales::date_sale that in pgadmin I do it with to_char(date_sale, 'dd/mm/YYYY').
Is it possible to use to_char in Diesel or in what way can I modify the data that the Diesel ORM brings me?
In addition to the answers provided by harmic there are two more possibilities to solve this problem.
sql_function!
Diesel provides an interface to easily define query ast nodes for sql functions that are not provided by diesel itself. Users are encouraged to use this functionality to define missing functions on their own. (In fact diesel uses internally the same method to define query ast nodes for sql functions provided out of the box). The defined query ast node is usable in every context where the types of its expression are valid, so it can be used in select and where clauses.
(This is basically the type safe version of the raw sql solution above)
For the given question something like this should work:
#[derive(Queryable)]
pub struct Sales {
pub id: i32,
pub product_id: Option<i32>,
pub amount: Option<BigDecimal>,
pub date_sale: Option<String>,
}
sql_function! {
fn to_char(Nullable<Timestamp>, Text) -> Nullable<Text>;
}
let product_id = 1;
sales::table
.inner_join(product::table)
.select((
product::description,
sales::amount,
to_char(sales::date_sale, "dd/mm/YYYY")
))
.filter(sales::product_id.eq(product_id))
.load(&diesel::PgConnection);
#[derive(Queryable)] + #[diesel(deserialize_as = "Type")]
Diesels Queryable derive provides a way to apply certain type manipulations at load time via a custom attribute. In combination with the chrono solution provided by harmic this gives the following variant:
#[derive(Queryable)]
pub struct Sales {
pub id: i32,
pub product_id: Option<i32>,
pub amount: Option<BigDecimal>,
#[diesel(deserialize_as = "MyChronoTypeLoader")]
pub date_sale: Option<String>
}
struct MyChronoTypeLoader(Option<String>);
impl Into<Option<String>> for MyChronoTypeLoader {
fn into(self) -> String {
self.0
}
}
impl<DB, ST> Queryable<ST, DB> for MyChronoTypeLoader
where
DB: Backend,
Option<NaiveDateTime>: Queryable<ST, DB>,
{
type Row = <Option<NaiveDateTime> as Queryable<ST, DB>>::Row;
fn build(row: Self::Row) -> Self {
MyChronoTypeLoader(Option::<NaiveDateTime>::build(row).map(|d| d.format("%d/%m/%Y").to_string()))
}
}
When using an ORM, data is extracted from the database in the most suitable representation for interpretation by the ORM; after which you would manipulate it in the target domain (rust in this case).
Since date_sale is an Option<NaiveDateTime>, you can use the formatting options provided by chrono:
sale.date_sale.unwrap().format("%d/%m/%Y").to_string()
(Your real code would not use unwrap(), of course!)
Alternatively, if you really need the database to do the formatting, you could use a [sql][2] to insert some raw SQL into your query:
let results = sales::table.select((sales::id, sql("to_char(date_sale, 'dd/mm/YYYY')")))
.load::<(i32, String)>(&conn);
This would be more useful if you were trying to implement some logic that would be more easy or efficient to implement in SQL.
An example would be a filtering condition: suppose you wanted to include rows from your table where the week number was an even number. While you could load the whole table and then filter it in the rust domain, that would not be very efficient. Instead you could do this:
let results = sales::table
.filter(sql("extract(week from date_sale)::smallint % 2=0"))
.load::<Sales>(&conn);

Why do I get a trait not implemented for an optional field in Diesel struct

I have this struct:
#[table_name = "clients"]
#[derive(Serialize, Deserialize, Queryable, Insertable, Identifiable, Associations)]
pub struct Client {
pub id: Option<i64>,
pub name: String,
pub rank: Option<i64>,
}
and the following implementation:
impl Client {
pub fn get(name: String, connection: &PgConnection) -> Option<Self> {
match clients::table
.filter(clients::name.eq(&name))
.limit(1)
.load::<Client>(connection)
{
Ok(clients) => Some(clients[0]),
Err(_) => None,
}
}
}
which gives me the following error:
.load::<Client>(connection) {
^^^^ the trait `diesel::Queryable<diesel::sql_types::BigInt, _>` is not implemented for `std::option::Option<i64>`
Your error message says that you cannot query a BigInt (a 64 bits int) into an Option<i64>. That is because you forgot to say that id is nullable in your table declaration. It must look like:
table! {
clients {
id -> Nullable<BigInt>,
name -> Text,
rank -> Nullable<BigInt>,
}
}
You can see the implementation of Queryable you are looking for in the documentation.
For me the issue was not realising that diesel maps fields based entirely on the order of fields in the struct. It completely ignores field names.
If your struct has fields defined in a different order to the schema.rs file then it will map them incorrectly and cause type errors.
https://docs.diesel.rs/diesel/deserialize/trait.Queryable.html#deriving
When this trait is derived, it will assume that the order of fields on your struct match the order of the fields in the query. This means that field order is significant if you are using #[derive(Queryable)]. Field name has no effect.

What is the standard pattern to relate three tables (many-to-many relation) within Diesel?

Database - Postgres
I have the following relation:
users <—>> users_organizations <<—> organizations
Schema:
table! {
organizations (id) {
id -> Int4,
name -> Varchar,
}
}
table! {
users (id) {
id -> Int4,
name -> Varchar,
email -> Varchar,
password -> Varchar,
}
}
table! {
users_organizations (id, user_id, organization_id) {
id -> Int4,
user_id -> Int4,
organization_id -> Int4,
}
}
Models:
#[derive(Identifiable, Queryable, Debug, Serialize, Deserialize)]
pub struct Organization {
pub id: i32,
pub name: String,
}
#[derive(Identifiable, Queryable, PartialEq, Debug, Serialize, Deserialize)]
pub struct User {
pub id: i32,
pub name: String,
pub email: String,
pub password: String,
}
#[derive(Identifiable, Queryable, Debug, Associations, Serialize, Deserialize)]
#[belongs_to(User)]
#[belongs_to(Organization)]
#[table_name = "users_organizations"]
pub struct UserOrganization {
pub id: i32,
pub user_id: i32,
pub organization_id: i32,
}
I want to create an organization. To support such relation, I have to manually add ids of user and organization to the users_organizations table. Is there any better approach to implement such relation?
let new_organization = NewOrganization { name: &msg.name };
let organization = insert_into(organizations::table)
.values(&new_organization)
.get_result::(conn)
.map_err(|_| error::ErrorInternalServerError(“Error creating organization”))?;
let new_user_org = NewUserOrganizationIDs {
user_id: msg.user_id,
organization_id: organization.id,
};
insert_into(users_organizations::table)
.values(&new_user_org)
.get_result::<UserOrganization>(conn)
.map_err(|_| error::ErrorInternalServerError("Error creating user-organization data"))
Same question here. In case of selecting all organizations that relate to user (and vice verse) I came up with the following code:
let user = users::table.filter(users::id.eq(&msg.user_id))
.get_result::<User>(conn)
.map_err(|_| error::ErrorNotFound("User doesn't exist"))?;
let user_organizations = UserOrganization::belonging_to(&user)
.get_results::<UserOrganization>(conn)
.map_err(|_| error::ErrorNotFound("User doesn't have any organization"))?;
let mut organization_ids = vec![];
for user_org in &user_organizations {
organization_ids.push(user_org.organization_id);
}
organizations::table.filter(organizations::id.eq_any(organization_ids))
.get_results::<Organization>(conn)
.map_err(|_| error::ErrorNotFound("Organizations don't exist"))
This answer is from the Diesel chat by #SRugina and #weiznich (edited and adapted for the question).
How do I write a many-to-many relationship with Diesel?
I normally combine belonging_to and join, so something like:
UserOrganization::belonging_to(&organizations)
.inner_join(user::table)
Is there anything akin to belonging_to_many?
No, belonging_to_many does not exist because Diesel does not try to hide the database from you. Doing that would cause issues as soon as you want to do complex or non-standard things. Depending on your exact use case, joining all three tables could also be an option.
How do I use inner_join in this scenario?
You have three tables: users, organizations and user_organizations and you want to get all organizations for a specific user.
There are two variants doing that. The first variant is only one query, but may not match your required data layout if you want to do that for all users:
users::table
.inner_join(user_organizations::table.inner_join(organizations::table))
.filter(users::id.eq(user_id))
.select(organizations::all_columns)
.load::<Organization>(conn)?;
The second variant allows grouping the results for each user using the built-in associations API:
let user = users::table
.find(user_id)
.first::<User>(conn)?;
UserOrganization::belonging_to(&user)
.inner_join(organizations::table)
.select(organizations::all_columns)
.load::<Organization>(conn)?;
Inserting requires three separate inserts. We do not try to hide that because in the end it's a user choice how to handle data consistency in case of a failed insert there. Using a transaction is a common choice.

Resources