Mongoose many to many population with uni-directional references - node.js

I have the following schemas which are used to represent a many-to-many relationship :
var CategorySchema = new Schema({
title: {type: String},
});
mongoose.model('Category', CategorySchema);
var ProductSchema = new Schema({
title: {type: String},
categories: [
{
type: Schema.ObjectId,
ref: 'Category'
}
]
});
mongoose.model('Product', ProductSchema );
When I query the Categories or the Products I want to be able to get in the result all the linked documents.
Populating the categories when querying the Product is straightforward:
Product.find().populate('categories').exec(...)
But how to do this from the Category side? I know I can add an array of ObjectId ref to the Product documents in the CategorySchema. But I'd like to avoid bi-directional referencing (I don't want to maintain it, and have a risk of inconsistency).
EDIT: here is the solution I implemented
/**
* List all Categories
*/
exports.all = function (req, res) {
//Function needed in order to send the http response only once all
//the categories' product has been retrieved and added to the returned JSON document.
function sendResponse(categories) {
res.json(categories);
}
AppCategory.list(function (err, categories) {
if (err) {
errors.serverError();
} else {
_.forEach(categories, function (category, index) {
category.products = [];
Product.byCategory(category._id, function (err, products) {
category.products= category.products.concat(products);
if (index === categories.length - 1) {
sendResponse(categories);
}
});
});
}
});
};
ProductSchema.statics = {
byCategory: function (categoryId, callback) {
this.find({'categories': categoryId})
.sort('-title')
.exec(callback);
}
};

You probably don't want to do that. :-) I would guess a product can be in some reasonably-small number of categories, but a category might have many thousands of products. In that case, trying to do Category.populate('products') is not going to work from an efficiency standpoint. You'll use lots of memory, not be able to do pagination in a straightforward way, load duplicate product data into memory when a product belongs to several categories, etc. Better to load the products in a category by querying directly against the products collection. You can filter by category easily enough a la Product.find({'categories._id': $in: arrayOfCategoryIds}}).

Related

Updating a Subdocument array INSIDE another Subdocument array Mongoose

I am at my wits end with something that is seemingly straightforward:
I need to be able to push new gifts into the Events Array under the specific user. Because each event will have numerous gifts added, I want to keep them all under the user, as they are the one creating the event, and the gifts will live inside of their event where they belong.
The PROBLEM is: when I use the mongoose method 'findByIdAndUpdate', I can only find the main user, and from there, push an event to the events array. What I NEED to be able to do: push gifts to a specific event under that user. I am using mongoose Subdocuments. See my schema below and how I have a subdocument schema (EventSchema) inside of the main user schema, and a subdocument (gift) schema inside the event schema.
SCHEMA:
const Schema = mongoose.Schema;
let giftArr = new Schema({
giftname: String,
giftlink: String,
claimed: Boolean,
claimee: String
})
let eventSchema = new Schema({
eventname: String,
eventowner: String,
date: {
type: Date,
default: Date.now
},
attendees: [
{
attendeename: String
}
],
gift: [giftArr]
})
let userSchema = new Schema({
username: String,
email: { type: String, required: false },
events: [eventSchema]
});
Here are my controllers for my POST & GET routes:
export const insertEventsById = ((req, res) => {
const update = { $push: { events: req.body } }
const id = req.params.userID
Gift.findByIdAndUpdate(id, update, (err, data) => {
if (err) {
console.log(err);
} else {
res.json(data)
console.log(data);
}
})
})
export const getUserById = (req, res) => {
Gift.findById(req.params.userID, (err, user) => {
if(err){
res.send(err)
}
res.json(user)
})
}
To further illustrate, here is my postman GET request for a USER. I can push to the 'events' array (red arrow) as my findByIdAndUpdate method shows above, but when I attempt to go one nested level deeper, into the gift array (green arrow), I cannot find any documentation on that.
I been up and down the mongoose subdocuments and queries pages, and I cannot find a method that will pull specifically the '_id' of the particular event I need. I have even tried the methods on the embedded schemas to specifically look for _id's that way.
Can someone point out where I am going wrong here? Thanks in advance...as always fellow Stacks.

How to model a collection in nodejs+mongodb

Hello I am new to nodejs and mongodb.
I have 3 models:
"user" with fields "name phone"
"Shop" with fields "name, address"
"Member" with fields "shop user status". (shop and user hold the "id" of respective collections).
Now when I create "shops" api to fetch all shop, then I need to add extra field "isShopJoined" which is not part of the model. This extra field will true if user who see that shop is joined it otherwise it will be false.
The problem happens when I share my model with frontend developers like Android/iOS and others, They will not aware of that extra field until they see the API response.
So is it ok if I add extra field in shops listing which is not part of the model? Or do I need to add that extra field in model?
Important note
All the code below has NOT been tested (yet, I'll do it when I can setup a minimal environment) and should be adapted to your project. Keep in mind that I'm no expert when it comes to aggregation with MongoDB, let alone with Mongoose, the code is only here to grasp the general idea and algorithm.
If I understood correctly, you don't have to do anything since the info is stored in the Member collection. But it forces the front-end to do an extra-request (or many extra-requests) to have both the list of Shops and to check (one by one) if the current logged user is a Member of the shop.
Keep in mind that the front-end in general is driven by the data (and so, the API/back-end), not the contrary. The front-end will have to adapt to what you give it.
If you're happy with what you have, you can just keep it that way and it will work, but that might not be very effective.
Assuming this:
import mongoose from "mongoose";
const MemberSchema = new mongoose.Schema({
shopId: {
type: ObjectId,
ref: 'ShopSchema',
required: true
},
userId: {
type: ObjectId,
ref: 'UserSchema',
required: true
},
status: {
type: String,
required: true
}
});
const ShopSchema = new mongoose.Schema({
name: {
type: String,
required: true
},
address: {
//your address model
}
});
const UserSchema = new mongoose.Schema({
name: {
type: String,
required: true
},
phone: {
type: String,
required: true,
},
// Add something like this
shopsJoined: {
type: Array,
default: [],
required: true
}
});
You could tackle this problem via 2 ways:
MongoDB Aggregates
When retrieving (back-end side) the list of shops, if you know the user that made the request, instead of simply returning the list of Shops, you could return an aggregate of Shops and Members resulting in an hybrid document containing both the info of Shops and Models. That way, the front-end have all the info it needs with one back-end request.
Important note
The following code might not work as-is and you'll have to adapt it, I currently have nothing to test it against. Keep in mind I'm not very familiar with aggregates, let alone with Mongoose, but you'll get the general idea by looking the code and comments.
const aggregateShops = async (req, res, next) => {
try {
// $lookup will merge the "Model" and "Shop" documents into one
// $match will return only the results matching the condition
const aggreg = await Model.aggregate({$lookup: {
from: 'members', //the name of the mongodb collection
localField: '_id', //the "Shop" field to match with foreign collection
foreignField: 'shopId', //the "Member" field to match with local collection
as: 'memberInfo' //the field name in which to store the "Member" fields;
}, {
$match: {memberInfo: {userId: myUserId}}
}});
// the result should be an array of object looking like this:
/*{
_id: SHOP_OBJECT_ID,
name: SHOP_NAME,
address: SHOP_ADDRESS,
memberInfo: {
shopId: SHOP_OBJECT_ID,
userId: USER_OBJECT_ID,
status: STATUS_JOINED_OR_NOT
}
}*/
// send back the aggregated result to front-end
} catch (e) {
return next(e);
}
}
Drop the Members collection and store the info elsewhere
Instinctively, I would've gone this way. The idea is to either store an array field shopsJoined in the User model, or a membersJoined array field in the Shops model. That way, the info is retrieved no matter what, since you still have to retrieve the Shops and you already have your User.
// Your PATCH route should look like this
const patchUser = async (req, res, next) => {
try {
// How you chose to proceed here is up to you
// I tend to facilitate front-end work, so get them to send you (via req.body) the shopId to join OR "un-join"
// They should already know what shops are joined or not as they have the User
// For example, req.body.shopId = "+ID" if it's a join, or req.body.shopId = "-ID" if it's an un-join
if (req.body.shopId.startsWith("+")) {
await User.findOneAndUpdate(
{ _id: my_user_id },
{ $push: { shopsJoined: req.body.shopId } }
);
} else if (req.body.shopId.startsWith("-")) {
await User.findOneAndUpdate(
{ _id: my_user_id },
{ $pull: { shopsJoined: req.body.shopId } }
);
} else {
// not formatted correctly, return error
}
// return OK here depending on the framework you use
} catch (e) {
return next(e);
}
};
Of course, the above code is for the User model, but you can do the same thing for the Shop model.
Useful links:
MongoDB aggregation pipelines
Mongoose aggregates
MongoDB $push operator
MongoDB $pull operator
Yes you have to add the field to the model because adding it to the response will be only be a temporary display of the key but what if you need that in the future or in some list filters, so its good to add it to the model.
If you are thinking that front-end will have to be informed so just go it, and also you can set some default values to the "isShopJoined" key let it be flase for the time.

Mongoose relations design

I've recently started using Mongoose with Express.js in a Node.js application and I have a question about a proper way to design my schemas.
I have several schemas that have some relationships, i.e. Location schema has an array of Objects (it's not a JS object in this context), and Object schema has its Location property. I've learned that relationships in Mongoose are resolved using population, but when I implemented this approach I noticed that I have to type a lot of duplicate code, i.e. whenever I want to create a new Object I have to also update the Location's array of Objects and then assign the Location to the Object's property. Wouldn't it be more trivial to just manually assemble all the Objects that has a locationId property equal to the Location that I want to get from the database in a separate query?
I have also considered just storing Objects in an array in a Location document (as subdocuments) but I decided that I want to be able to work with Objects (create, remove, update) separately from Locations (without querying a Location) so this approach doesn't fit my needs I guess. But then population has its drawbacks too in my case, so I guess it's really the best to just go with manually collecting Objects of a specific Location in a separate query by that Location's id.
I would like to hear an opinion of some professional or advanced user of this technology on designing Mongoose schemas so that I and others don't get into trouble later maintaining and scaling our applications.
Here are my current schemas in question:
var locationSchema = new mongoose.Schema({
title: String,
objects: [{ type: String, ref: 'object' }]
});
var objectSchema = new mongoose.Schema({
title: String,
location: { type: String, ref: 'location' }
});
Checkout this example
db/schemas.js:
const Schema = mongoose.Schema;
const ObjectSchema = {
title: Schema.Types.String
}
const LocationSchema = new Schema({
title: Schema.Types.String,
objects: [{type: Schema.Types.ObjectId, ref: 'Object'}]
})
module.exports = {
Object: ObjectSchema,
Location: LocationSchema
};
db/model.js:
const
mongoose = require('mongoose'),
schemas = require('./schemas');
module.exports = model => mongoose.model(model, schemas[model+'Schema']);
usage:
const
model = require('./db/model'),
LocationModel = model('Location');
LocationModel
.findOne({_id: 'some id here'})
.populate('objects')
.exec((err, LocationInstance) => {
console.log(LocationInstance.title, ' objects:', LocationInstance.objects);
});
when You create an object and want to relate to location:
const
model = require('./db/model'),
ObjectModel = model('Object'),
LocationModel = model('Location');
let
ObjectInstance = new ObjectModel({title: 'Something'});
ObjectInstance.save((err, result) => {
LocationModel
.findByIdAndUpdate(
'some id here',
{$push: {objects: ObjectInstance._id}},
(err) => {
console.log('Object:', ObjectInstance.title, ' added to location');
});
});
updating object data:
const
model = require('./db/model'),
ObjectModel = model('Object');
let id = 'id of object';
ObjectModel
.findByIdAndUpdate(
id,
{title: 'Something #2'},
(err) => {
console.log('Object title updated');
});
finding location by object:
const
model = require('./db/model'),
LocationModel = model('Object');
let id = 'id of object';
LocationModel
.findOne({objects: id})
.populate('objects')
.exec((err, LocationInstance) => {
console.log('Location objects:', LocationInstance.objects);
});
nothing special findOne({objects: id}) will search inside location documents that has relation by id in objects array
any other question welcome (:

How should I approach my back end design using the MEAN stack?

This is maybe more a case of looking for advice. However, I will supply sample code as an example of what I want to achieve. I am trying to build my first back end system and I keep running into problems with the design.
My MongoDB database consists of 4 major parts - Profiles, Teams, Drafts and Users. The profile (being the main data which sources everything using IDs) schema has properties to hold arrays with the Teams and Drafts IDs. The idea is that when the profile is served it will populate all those properties with the relevant data by using the IDs.
Example of the Profile schema using Mongoose:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var UserProfileSchema = new Schema({
Name : String,
Email : String,
Admin : Boolean,
Status : Number,
UserID : String,
PrivateProfile: Boolean,
Drafts: [String], //This array holds the draft IDs
Teams:[String] //Holds the team profiles IDs
});
module.exports = mongoose.model('UserProfile', UserProfileSchema);
Example of Team Profile Schema using Mongoose:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var TeamProfileSchema = new Schema({
Name : String,
CaptainID : String,
CaptainName : String,
Members : [String], //Array of user IDs
DateCreated : Boolean,
Reputation : Number
});
module.exports = mongoose.model('TeamProfile', TeamProfileSchema);
Example of a route to find all the Team Profiles the user is the Captain of and fetch all the members associated with that team:
router.route('/teams/captain/:user_id')
.get(function (req, res) {
TeamProfile.find({
CaptainID : req.params.user_id
}, function (err, teams) {
if (err)
res.send(err);
for (var x in teams) {
var membersArray = [];
for (var i in teams[x].Members) {
var ID = teams[x].Members[i];
UserProfile.find({
UserID : ID
}, function (err, profile) {
if (err)
res.send(err);
membersArray.push(profile);
});
}
teams[x].Members = membersArray;
console.log(teams[x].Members);
}
res.json(teams);
});
})
I understand that the route will not work, but how do I pull this off? I used a more vanilla approach only for the purpose to explain what I want to achieve. Any help would be highly appreciated.
I advise you to use combination of denormalization and normalization.
because MongoDB is not a relational database.
denormalization is much faster then normalization. You don't need to use relationship anymore.
You can read this article hope may helpfull to you.
Cheers

Dynamically create collection with Mongoose

I want to give users the ability to create collections in my Node app. I have really only seen example of hard coding in collections with mongoose. Anyone know if its possible to create collections dynamically with mongoose? If so an example would be very helpful.
Basically I want to be able to store data for different 'events' in different collections.
I.E.
Events:
event1,
event2,
...
eventN
Users can create there own custom event and store data in that collection. In the end each event might have hundreds/thousands of rows. I would like to give users the ability to perform CRUD operations on their events. Rather than store in one big collection I would like to store each events data in a different collection.
I don't really have an example of what I have tried as I have only created 'hard coded' collections with mongoose. I am not even sure I can create a new collection in mongoose that is dynamic based on a user request.
var mongoose = require('mongoose');
mongoose.connect('localhost', 'events');
var schema = mongoose.Schema({ name: 'string' });
var Event1 = mongoose.model('Event1', schema);
var event1= new Event1({ name: 'something' });
event1.save(function (err) {
if (err) // ...
console.log('meow');
});
Above works great if I hard code 'Event1' as a collection. Not sure I create a dynamic collection.
var mongoose = require('mongoose');
mongoose.connect('localhost', 'events');
...
var userDefinedEvent = //get this from a client side request
...
var schema = mongoose.Schema({ name: 'string' });
var userDefinedEvent = mongoose.model(userDefinedEvent, schema);
Can you do that?
I believe that this is a terrible idea to implement, but a question deserves an answer. You need to define a schema with a dynamic name that allows information of 'Any' type in it. A function to do this may be a little similar to this function:
var establishedModels = {};
function createModelForName(name) {
if (!(name in establishedModels)) {
var Any = new Schema({ any: Schema.Types.Mixed });
establishedModels[name] = mongoose.model(name, Any);
}
return establishedModels[name];
}
Now you can create models that allow information without any kind of restriction, including the name. I'm going to assume an object defined like this, {name: 'hello', content: {x: 1}}, which is provided by the 'user'. To save this, I can run the following code:
var stuff = {name: 'hello', content: {x: 1}}; // Define info.
var Model = createModelForName(name); // Create the model.
var model = Model(stuff.content); // Create a model instance.
model.save(function (err) { // Save
if (err) {
console.log(err);
}
});
Queries are very similar, fetch the model and then do a query:
var stuff = {name: 'hello', query: {x: {'$gt': 0}}}; // Define info.
var Model = createModelForName(name); // Create the model.
model.find(stuff.query, function (err, entries) {
// Do something with the matched entries.
});
You will have to implement code to protect your queries. You don't want the user to blow up your db.
From mongo docs here: data modeling
In certain situations, you might choose to store information in
several collections rather than in a single collection.
Consider a sample collection logs that stores log documents for
various environment and applications. The logs collection contains
documents of the following form:
{ log: "dev", ts: ..., info: ... } { log: "debug", ts: ..., info: ...}
If the total number of documents is low you may group documents into
collection by type. For logs, consider maintaining distinct log
collections, such as logs.dev and logs.debug. The logs.dev collection
would contain only the documents related to the dev environment.
Generally, having large number of collections has no significant
performance penalty and results in very good performance. Distinct
collections are very important for high-throughput batch processing.
Say I have 20 different events. Each event has 1 million entries... As such if this is all in one collection I will have to filter the collection by event for every CRUD op.
I would suggest you keep all events in the same collection, especially if event names depend on client code and are thus subject to change. Instead, index the name and user reference.
mongoose.Schema({
name: { type: String, index: true },
user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', index: true }
});
Furthermore I think you came at the problem a bit backwards (but I might be mistaken). Are you finding events within the context of a user, or finding users within the context of an event name? I have a feeling it's the former, and you should be partitioning on user reference, not the event name in the first place.
If you do not need to find all events for a user and just need to deal with user and event name together you could go with a compound index:
schema.index({ user: 1, name: 1 });
If you are dealing with millions of documents, make sure to turn off auto index:
schema.set('autoIndex', false);
This post has interesting stuff about naming collections and using a specific schema as well:
How to access a preexisting collection with Mongoose?
You could try the following:
var createDB = function(name) {
var connection = mongoose.createConnection(
'mongodb://localhost:27017/' + name);
connection.on('open', function() {
connection.db.collectionNames(function(error) {
if (error) {
return console.log("error", error)
}
});
});
connection.on('error', function(error) {
return console.log("error", error)
});
}
It is important that you get the collections names with connection.db.collectionNames, otherwise the Database won't be created.
This method works best for me , This example creates dynamic collection for each users , each collection will hold only corresponding users information (login details), first declare the function dynamicModel in separate file : example model.js
/* model.js */
'use strict';
var mongoose = require('mongoose'),
Schema = mongoose.Schema;
function dynamicModel(suffix) {
var addressSchema = new Schema(
{
"name" : {type: String, default: '',trim: true},
  "login_time" : {type: Date},
"location" : {type: String, default: '',trim: true},
}
);
return mongoose.model('user_' + suffix, addressSchema);
}
module.exports = dynamicModel;
In controller File example user.js,first function to create dynamic collection and second function to save data to a particular collection
/* user.js */
var mongoose = require('mongoose'),
function CreateModel(user_name){//function to create collection , user_name argument contains collection name
var Model = require(path.resolve('./model.js'))(user_name);
}
function save_user_info(user_name,data){//function to save user info , data argument contains user info
var UserModel = mongoose.model(user_name) ;
var usermodel = UserModel(data);
usermodel.save(function (err) {
if (err) {
console.log(err);
} else {
console.log("\nSaved");
}
});
}
yes we can do that .I have tried it and its working.
REFERENCE CODE:
app.post("/",function(req,res){
var Cat=req.body.catg;
const link= req.body.link;
const rating=req.body.rating;
Cat=mongoose.model(Cat,schema);
const item=new Cat({
name:link,
age:rating
});
item.save();
res.render("\index");
});
I tried Magesh varan Reference Code ,
and this code works for me
router.post("/auto-create-collection", (req, res) => {
var reqData = req.body; // {"username":"123","password":"321","collectionName":"user_data"}
let userName = reqData.username;
let passWord = reqData.password;
let collectionName = reqData.collectionName;
// create schema
var mySchema = new mongoose.Schema({
userName: String,
passWord: String,
});
// create model
var myModel = mongoose.model(collectionName, mySchema);
const storeData = new myModel({
userName: userName,
passWord: passWord,
});
storeData.save();
res.json(storeData);
});
Create a dynamic.model.ts access from some where to achieve this feature.
import mongoose, { Schema } from "mongoose";
export default function dynamicModelName(collectionName: any) {
var dynamicSchema = new Schema({ any: Schema.Types.Mixed }, { strict: false });
return mongoose.model(collectionName, dynamicSchema);
}
Create dynamic model
import dynamicModelName from "../models/dynamic.model"
var stuff = { name: 'hello', content: { x: 1 } };
var Model = await dynamicModelName('test2')
let response = await new Model(stuff).save();
return res.send(response);
Get the value from the dynamic model
var Model = dynamicModelName('test2');
let response = await Model.find();
return res.send(response);

Resources