query on many to many relation mongodb database struct - node.js

I have two collections in MongoDB: one saves post data of blog, the other saves comment data of blog with below schemas. How can I use nodejs and mongoose to query all posts with comment belong to it and respond to single page application?. Thanks!
var PostSchema = mongoose.Schema({
created: {
type: Date,
default: Date.now
},
content: {
type: String,
default: '',
trim: true
},
user: {
type: Schema.ObjectId,
ref: 'user'
}
});
var CommentSchema = mongoose.Schema({
created: {
type: Date,
default: Date.now
},
content: {
type: String,
default: '',
trim: true
},
ofpost: {
type: Schema.ObjectId,
ref: 'post' //which post this comment belong to
},
user: {
type: Schema.ObjectId,
ref: 'user'
}
});
var Post = mongoose.model('Post', PostSchema);
var Comment = mongoose.model('Comment', CommentSchema);
//example:the Comment1 and Comment2 belong to Post1
var Post1 = new Post({ content: 'good day', user: 'John' });
var Comment1 = new Comment({content: 'yeah', ofpost: Post1._id, user:'Tom'})
var Comment2 = new Comment({content: 'agree', ofpost: Post1._id, user:'Tina'})

As mongodb is NoSQL type of database and has no JOIN's or any sort of relationship between documents, you have to take care of such.
There are generally two ways to do so:
Caching
Consider storing comments data within blog document. You can have embedded documents without any problem. In reality it leads to some extra caches, like comments count, array of user id's of comments and other stuff that will make your queries indexed and more easy ways to search through collection.
Multiple Queries
If you still need separate collections, then you need to 'simulate' joins. Most efficient ways is to make temporary indexing arrays and multiple queries to different collections. Usually it should be just 2 queries for one Join (many to many), and small iteration to add second query documents to first array of documents.
Here is the flow that is suitable and performs well still, on example:
Two collections, first is posts, and second is comments which has id of post.
Make query to posts.
Iterate through each post and add its id into postIds array, as well make postMap object where key will be id of post and value will be specific post. - this is so called indexing posts.
Make query to comments collection with $in argument with postIds array of post id's. This collection should have indexing on post id field in order to make this query very efficient. As well this query can include sorting by date (additional compound indexing will speedup it).
Iterate through each comment and using postMap add it to comments array of post.
So we have only 2 queries, and one iteration through all comments to embed data into posts O(n). Without second step, adding to posts will be potentially O(p*c) where p - number of posts and c - number of comments. Which is obviously much slower as well on big queries can be potentially slow.
Summary
Second approach is more manageable approach from data point of view, as well is easier on writes, while is more complicated on reads.
Still will require some caching, like number of comments for blog posts.

Related

Proper way of updating average rating for a review system using Mongoose

I'm currently learning some backend stuff using an Udemy course and I have an example website that lets you add campgrounds (campground name, picture, description, etc.) and review them. I'm using the Express framework for Node.js, and Mongoose to access the database.
My campground schema looks like:
const campgroundSchema = new mongoose.Schema({
name: String,
image: String,
description: String,
price: String,
comments: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "Comment"
}
],
rating: {type: Number, default: 0}
});
And my comment/review schema looks like:
const commentSchema = new mongoose.Schema({
text: String,
rating: {
type: Number,
min: 1,
max: 5,
validate: {validator: Number.isInteger}
},
campground: {type: mongoose.Schema.Types.ObjectId, ref: "Campground"}
});
Campgrounds and Comments also have references to a User but I've left that out for simplicity.
I'm looking to know the best practice for updating and displaying the campground average rating.
The method used by the tutorial I'm following is to recalculate the average rating each time a comment is added, changed, or deleted. Here's how it would work for a new comment:
Campground.findById(campgroundId).populate("comments").exec(function(err, campground) {
Comment.create(newComment, function(err, comment) {
campground.comments.push(comment);
campground.rating = calculateRating(campground.comments);
campground.save();
});
});
"calculateRating" iterates through the comment array, gets the total sum, and returns the sum divided by the number of comments.
My gut instinct tells me that there should be a way to make the "rating" field of Campground perform the functionality of the "calculateRating" function, so that I don't have to update the rating every time a comment is added, changed, or removed. I've been poking around documentation for a while now, but since I'm pretty new to Mongoose and databases in general, I'm a bit lost on how to proceed.
In summary: I want to add functionality to my Campground model so that when I access its rating, it automatically accesses each comment referenced in the comments array, sums up their ratings, and returns the average.
My apologies if any of my terminology is incorrect. Any tips on how I would go about achieving this would be very much appreciated!
Love,
Cal
I think what you are trying to do is get a virtual property of the document that gets the average rating but it does not get persisted to the mongo database.
according to mongoosejs :- Virtuals are document properties that you can get and set but that do not get persisted to MongoDB. They are set on the schema.
You can do this:
CampgroundSchema.virtual('averageRating').get(function() {
let ratings = [];
this.comments.forEach((comment) => ratings.push(comment.rating));
return (ratings.reduce((a,b)=>a+b)/ratings.length).toFixed(2);
});
After that on your view engine after finding campgrounds or a campground, all you need to call is ; campground.averageRating;
Read more here : https://mongoosejs.com/docs/guide.html#virtuals
also note that you can not make any type of query on virtual properties.

Mongodb Relationship: Posts and Comments (ref v sub-documents)

I know there are a lot of similar questions, but they're too old and since Mongodb has evolved alot for last 5-6 years I am looking for a good schema design.
Goal: I want to have a post with comments by users.
What I have designed so far is:
Separate post model:
const projectSchema = new mongoose.Schema({
user: { type: mongoose.Schema.Types.ObjectId, required: true, ref: 'User' },
title: { type: String, required: true },
image: { type: String, default: undefined },
description: { type: String, required: true, minLength: 200, maxlength: 500 },
comments: [{
type: mongoose.Schema.Types.ObjectId, ref: 'Comment'
}],
state: { type: Boolean, default: true },
collaborators: { type: Array, default: [] },
likes: { type: Array, default: [] }
})
And a separate comments model:
const commentSchema = new mongoose.Schema({
comment: { type: String, required: true },
project: { type: String, required: true, ref: 'Project' },
user: { type: String, required: true, ref: 'User' }
})
The reason I am going for the relational approach is because if the comments increase to say 10,000 in number, it will increase the size of schema by alot.
This way, no matter how many comments we can populate them using their IDs, also, we will have different collection for comments iself.
Reference : one-to-many
Is this a good approach for my project?
The way I am querying the comments from one single post:
const project = await Project.findById(
new mongoose.Types.ObjectId(req.params.projectId)
).populate({
path: 'comments',
populate: { path: 'user' }
}).lean()
Whether it's a good design depends how many comments per post do you expect, and what query will be performed on your app.
There's a good blog from mongodb.com on how to design your database schema
The common design is:
One to Few (Use embed)
One to Many (Use embed reference)
One to squillions (The usual relational database one-to-many approach)
Summary is:
So, even at this basic level, there is more to think about when designing a MongoDB schema than when designing a comparable relational schema. You need to consider two factors:
Will the entities on the ā€œNā€ side of the One-to-N ever need to stand alone?
What is the cardinality of the relationship: is it one-to-few; one-to-many; or one-to-squillions?
Based on these factors, you can pick one of the three basic One-to-N schema designs:
Embed the N side if the cardinality is one-to-few and there is no need to access the embedded object outside the context of the parent object
Use an array of references to the N-side objects if the cardinality is one-to-many or if the N-side objects should stand alone for any reasons
Use a reference to the One-side in the N-side objects if the cardinality is one-to-squillions
There is also a blog about advanced schema design which is worth the read.
You seems to be using the two-way referencing approach.
The difference between yours and one-to-squillions is you are not only storing post id reference on comment document, but also storing comment ids as reference in post document, while one-to-squillions will only stores project id reference in comment document.
Using your approach will be better if you need to get comment ids of a post. But the disadvantage is you need to run two queries when deleting or creating a comment, one to delete / create comment id from post, and the other one to delete / create the comment document it self. It's also will be slower to find "which post belongs to given comment id".
While using one-to-squillions would gives you worse performance when performing a query to get comments by post id. But you can mitigate this by properly indexing your comment collection.

Best way to structure my mongoose schema: embedded array , populate, subdocument?

Here is my current Schema
Brand:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var BrandSchema = new mongoose.Schema({
name: { type: String, lowercase: true , unique: true, required: true },
photo: { type: String , trim: true},
email: { type: String , lowercase: true},
year: { type: Number},
timestamp: { type : Date, default: Date.now },
description: { type: String},
location: { },
social: {
website: {type: String},
facebook: {type: String },
twitter: {type: String },
instagram: {type: String }
}
});
Style:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var StyleSchema = new mongoose.Schema({
name: { type: String, lowercase: true , required: true},
});
Product
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var ProductSchema = new mongoose.Schema({
name: { type: String, lowercase: true , required: true},
brandId : {type: mongoose.Schema.ObjectId, ref: 'Brand'},
styleId: {type: mongoose.Schema.ObjectId, ref: 'Style'},
year: { type: Number },
avgRating: {type: Number}
});
Post:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var PostSchema = new mongoose.Schema({
rating: { type: Number},
upVote: {type: Number},
brandId : {type: mongoose.Schema.ObjectId, ref: 'Brand'},
comment: {type: String},
productId: {type: mongoose.Schema.ObjectId, ref: 'Style'},
styleId: {type: mongoose.Schema.ObjectId, ref: 'Style'},
photo: {type: String}
});
I'm currently making use of the mongoose populate feature:
exports.productsByBrand = function(req, res){
Product.find({product: req.params.id}).populate('style').exec(function(err, products){
res.send({products:products});
});
};
This works, however, being a noob --- i've started reading about performance issues with the mongoose populate, since it's really just adding an additional query.
For my post , especially, it seems that could be taxing. The intent for the post is to be a live twitter / instagram-like feed. It seems that could be a lot of queries, which could greatly slow my app down.
also, I want to be able to search prodcuts / post / brand by fields at some point.
Should i consider nesting / embedding this data (products nested / embedded in brands)?
What's the most efficient schema design or would my setup be alright -- given what i've specified I want to use it for?
User story:
There will be an Admin User.
The admin will be able to add the Brand with the specific fields in the Brand Schema.
Brands will have associated Products, each Product will have a Style / category.
Search:
Users will be able to search Brands by name and location (i'm looking into doing this with angular filtering / tags).
Users will be able to search Products by fields (name, style, etc).
Users will be able to search Post by Brand Product and Style.
Post:
Users will be able to Post into a feed. When making a Post, they will choose a Brand and a Product to associate the Post with. The Post will display the Brand name, Product name, and Style -- along with newly entered Post fields (photo, comment, and rating).
Other users can click on the Brand name to link to the Brand show page. They can click on the Product name to link to a Product show page.
Product show page:
Will show Product fields from the above Schema -- including associated Style name from Style schema. It will also display Post pertaining to the specific Product.
Brand show page:
Will simply show Brand fields and associated products.
My main worry is the Post, which will have to populate / query for the Brand , Product, and Style within a feed.
Again, I'm contemplating if I should embed the Products within the Brand -- then would I be able to associate the Brand Product and Style with the Post for later queries? Or, possibly $lookup or other aggregate features.
Mongodb itself does not support joins. So, mongoose populate is an attempt at external reference resolution. The thing with mongodb is that you need to design your data so that:
most of you queries need not to refer multiple collections.
after getting data from query, you need not to transform it too much.
Consider the entities involved, and their relations:
Brand is brand. Doesn't depend on anything else.
Every Product belongs to a Brand.
Every Product is associated with a Style.
Every Post is associated with a Product.
Indirectly, every Post is associated to a Brand and Style, via product.
Now about the use cases:
Refer: If you are looking up one entity by id, then fetching 1-2 related entities is not really a big overhead.
List: It is when you have to return a large set of objects and each object needs an additional query to get associated objects. This is a performance issue. This is usually reduced by processing "pages" of result set at a time, say 20 records per request. Lets suppose you query 20 products (using skip and limit). For 20 products you extract two id arrays, one of referred styles, and other of referred brands. You do 2 additional queries using $in:[ids], get brands and styles object and place them in result set. That's 3 queries per page. Users can request next page as they scroll down, and so on.
Search: You want to search for products, but also want to specify brand name and style name. Sadly, product model only holds ids for style and brand. Same issue with searching Posts with brand and product. Popular solution is to maintain a separate "search index", a sort of table, that stores data exactly the way it will be searched for, with all searchable fields (like brand name, style name) at one place. Maintaining such search collections in mongodb manually can be a pain. This is where ElasticSearch comes in. Since you are already using mongoose, you can simply add mongoosastic to your models. ElasticSearch's search capabilities are far greater than a DB Storage engine will offer you.
Extra Speed: There is still some room for speeding things up: Caching. Attach mongoose-redis-cache and have frequent repeated queries served, in-memory from Redis, reducing load on mongodb.
Twitter like Feeds: Now if all Posts are public then listing them up for users in chronological order is a trivial query. However things change when you introduce "social networking" features. Then you need to list "activity feeds" of friends and followers. There's some wisdom about social inboxes and Fan-out lists in mongodb blog.
Moral of the story is that not all use cases have only "db schema query" solutions. Scalability is one of such cases. That's why other tools exist.

Mongoose ā€“ linking objects to each other without duplicating

I have a model "Category". Collection categories contains several objects.
I also a have model "Post". Collection posts may contain a lot of objects with users' posts. "Post" object may relate to 1+ categories. How to link "Post" object to 1+ "Category"-objects without placing "Post"-object inside "Category"-object as subdocument? Certainly, I need to have an option to find all posts related to certain category.
One of the ways I can imagine is to store in "Post"-object obj_id of all categories which it's related to. Smth like this:
var postSchema = mongoose.Schema({
title: String,
description: String,
category: [ObjectId],
created_time: Number,
})
and add category later...
post.category.push(obj_id);
but is it really a mongoose-way? Which way is correct? Thanks.
P.S. I've also read about population methods in mongoose docs, may it be useful in my case? Still not completely clear for me what is this.
Populate is a better tool for this since you are creating a many to many relationship between posts and categories. Subdocuments are appropriate when they belong exclusively to the parent object. You will need to change your postSchema to use a reference:
var postSchema = mongoose.Schema({
title: String,
description: String,
category: [{ type: Schema.Types.ObjectId, ref: 'Category' }],
created_time: Number,
});
You can add categories by pushing documents onto the array:
post.category.push(category1);
post.save(callback);
Then rehydrate them during query using populate:
Post.findOne({ title: 'Test' })
.populate('category')
.exec(function (err, post) {
if (err) return handleError(err);
console.log(post.category);
});

Mongoose: populate() / DBref or data duplication?

I have two collections:
Users
Uploads
Each upload has a User associated with it and I need to know their details when an Upload is viewed. Is it best practice to duplicate this data inside the the Uploads record, or use populate() to pull in these details from the Users collection referenced by _id?
OPTION 1
var UploadSchema = new Schema({
_id: { type: Schema.ObjectId },
_user: { type: Schema.ObjectId, ref: 'users'},
title: { type: String },
});
OPTION 2
var UploadSchema = new Schema({
_id: { type: Schema.ObjectId },
user: {
name: { type: String },
email: { type: String },
avatar: { type: String },
//...etc
},
title: { type: String },
});
With 'Option 2' if any of the data in the Users collection changes I will have to update this across all associated Upload records. With 'Option 1' on the other hand I can just chill out and let populate() ensure the latest User data is always shown.
Is the overhead of using populate() significant? What is the best practice in this common scenario?
If You need to query on your Users, keep users alone. If You need to query on your uploads, keep uploads alone.
Another question you should ask yourself is: Every time i need this data, do I need the embedded objects (and vice-versa)? How many time this data will be updated? How many times this data will be read?
Think about a friendship request:
Each time you need the request you need the user which made the request, then embed the request inside the user document.
You will be able to create an index on the embedded object too, and your search will be mono query / fast / consistent.
Just a link to my previous reply on a similar question:
Mongo DB relations between objects
I think this post will be right for you http://www.mongodb.org/display/DOCS/Schema+Design
Use Cases
Customer / Order / Order Line-Item
Orders should be a collection. customers a collection. line-items should be an array of line-items embedded in the order object.
Blogging system.
Posts should be a collection. post author might be a separate collection, or simply a field within posts if only an email address. comments should be embedded objects within a post for performance.
Schema Design Basics
Kyle Banker, 10gen
http://www.10gen.com/presentation/mongosf2011/schemabasics
Indexing & Query Optimization
Alvin Richards, Senior Director of Enterprise Engineering
http://www.10gen.com/presentation/mongosf-2011/mongodb-indexing-query-optimization
**These 2 videos are the bests on mongoddb ever seen imho*
Populate() is just a query. So the overhead is whatever the query is, which is a find() on your model.
Also, best practice for MongoDB is to embed what you can. It will result in a faster query. It sounds like you'd be duplicating a ton of data though, which puts relations(linking) at a good spot.
"Linking" is just putting an ObjectId in a field from another model.
Here is the Mongo Best Practices http://www.mongodb.org/display/DOCS/Schema+Design#SchemaDesign-SummaryofBestPractices
Linking/DBRefs http://www.mongodb.org/display/DOCS/Database+References#DatabaseReferences-SimpleDirect%2FManualLinking

Resources