Hello I am new to nodejs and mongodb.
I have 3 models:
"user" with fields "name phone"
"Shop" with fields "name, address"
"Member" with fields "shop user status". (shop and user hold the "id" of respective collections).
Now when I create "shops" api to fetch all shop, then I need to add extra field "isShopJoined" which is not part of the model. This extra field will true if user who see that shop is joined it otherwise it will be false.
The problem happens when I share my model with frontend developers like Android/iOS and others, They will not aware of that extra field until they see the API response.
So is it ok if I add extra field in shops listing which is not part of the model? Or do I need to add that extra field in model?
Important note
All the code below has NOT been tested (yet, I'll do it when I can setup a minimal environment) and should be adapted to your project. Keep in mind that I'm no expert when it comes to aggregation with MongoDB, let alone with Mongoose, the code is only here to grasp the general idea and algorithm.
If I understood correctly, you don't have to do anything since the info is stored in the Member collection. But it forces the front-end to do an extra-request (or many extra-requests) to have both the list of Shops and to check (one by one) if the current logged user is a Member of the shop.
Keep in mind that the front-end in general is driven by the data (and so, the API/back-end), not the contrary. The front-end will have to adapt to what you give it.
If you're happy with what you have, you can just keep it that way and it will work, but that might not be very effective.
Assuming this:
import mongoose from "mongoose";
const MemberSchema = new mongoose.Schema({
shopId: {
type: ObjectId,
ref: 'ShopSchema',
required: true
},
userId: {
type: ObjectId,
ref: 'UserSchema',
required: true
},
status: {
type: String,
required: true
}
});
const ShopSchema = new mongoose.Schema({
name: {
type: String,
required: true
},
address: {
//your address model
}
});
const UserSchema = new mongoose.Schema({
name: {
type: String,
required: true
},
phone: {
type: String,
required: true,
},
// Add something like this
shopsJoined: {
type: Array,
default: [],
required: true
}
});
You could tackle this problem via 2 ways:
MongoDB Aggregates
When retrieving (back-end side) the list of shops, if you know the user that made the request, instead of simply returning the list of Shops, you could return an aggregate of Shops and Members resulting in an hybrid document containing both the info of Shops and Models. That way, the front-end have all the info it needs with one back-end request.
Important note
The following code might not work as-is and you'll have to adapt it, I currently have nothing to test it against. Keep in mind I'm not very familiar with aggregates, let alone with Mongoose, but you'll get the general idea by looking the code and comments.
const aggregateShops = async (req, res, next) => {
try {
// $lookup will merge the "Model" and "Shop" documents into one
// $match will return only the results matching the condition
const aggreg = await Model.aggregate({$lookup: {
from: 'members', //the name of the mongodb collection
localField: '_id', //the "Shop" field to match with foreign collection
foreignField: 'shopId', //the "Member" field to match with local collection
as: 'memberInfo' //the field name in which to store the "Member" fields;
}, {
$match: {memberInfo: {userId: myUserId}}
}});
// the result should be an array of object looking like this:
/*{
_id: SHOP_OBJECT_ID,
name: SHOP_NAME,
address: SHOP_ADDRESS,
memberInfo: {
shopId: SHOP_OBJECT_ID,
userId: USER_OBJECT_ID,
status: STATUS_JOINED_OR_NOT
}
}*/
// send back the aggregated result to front-end
} catch (e) {
return next(e);
}
}
Drop the Members collection and store the info elsewhere
Instinctively, I would've gone this way. The idea is to either store an array field shopsJoined in the User model, or a membersJoined array field in the Shops model. That way, the info is retrieved no matter what, since you still have to retrieve the Shops and you already have your User.
// Your PATCH route should look like this
const patchUser = async (req, res, next) => {
try {
// How you chose to proceed here is up to you
// I tend to facilitate front-end work, so get them to send you (via req.body) the shopId to join OR "un-join"
// They should already know what shops are joined or not as they have the User
// For example, req.body.shopId = "+ID" if it's a join, or req.body.shopId = "-ID" if it's an un-join
if (req.body.shopId.startsWith("+")) {
await User.findOneAndUpdate(
{ _id: my_user_id },
{ $push: { shopsJoined: req.body.shopId } }
);
} else if (req.body.shopId.startsWith("-")) {
await User.findOneAndUpdate(
{ _id: my_user_id },
{ $pull: { shopsJoined: req.body.shopId } }
);
} else {
// not formatted correctly, return error
}
// return OK here depending on the framework you use
} catch (e) {
return next(e);
}
};
Of course, the above code is for the User model, but you can do the same thing for the Shop model.
Useful links:
MongoDB aggregation pipelines
Mongoose aggregates
MongoDB $push operator
MongoDB $pull operator
Yes you have to add the field to the model because adding it to the response will be only be a temporary display of the key but what if you need that in the future or in some list filters, so its good to add it to the model.
If you are thinking that front-end will have to be informed so just go it, and also you can set some default values to the "isShopJoined" key let it be flase for the time.
I need to retrieve an object and also get the relations and nested relations.
So, I have the three models below:
User model:
module.exports = {
attributes: {
name: {
type: 'string'
},
pets: {
collection: 'pet',
via: 'owner',
}
}
Pet model:
module.exports = {
attributes: {
name: {
type: 'string'
},
owner: {
model: 'user'
},
vaccines: {
collection: 'vaccine',
via: 'pet',
}
}
Vaccine model:
module.exports = {
attributes: {
name: {
type: 'string'
},
pet: {
model: 'pet'
}
}
Calling User.findOne(name: 'everton').populate('pets').exec(....) I get the user and associated Pets. How can I also get the associated vaccines with each pet? I didn't find references about this in the official documentation.
I've ran into this issue as well, and as far as I know, nested association queries are not built into sails yet (as of this post).
You can use promises to handle the nested population for you, but this can get rather hairy if you are populating many levels.
Something like:
User.findOne(name: 'everton')
.populate('pets')
.then(function(user) {
user.pets.forEach(function (pet) {
//load pet's vaccines
});
});
This has been a widely discussed topic on sails.js and there's actually an open pull request that adds the majority of this feature. Check out https://github.com/balderdashy/waterline/pull/1052
While the answer of Kevin Le is correct it can get a little messy, because you're executing async functions inside a loop. Of course it works, but let's say you want to return the user with all pets and vaccines once it's finished - how do you do that?
There are several ways to solve this problem. One is to use the async library which offers a bunch of util functions to work with async code. The library is already included in sails and you can use it globally by default.
User.findOneByName('TestUser')
.populate('pets')
.then(function (user) {
var pets = user.pets;
// async.each() will perform a for each loop and execute
// a fallback after the last iteration is finished
async.each(pets, function (pet, cb) {
Vaccine.find({pet: pet.id})
.then(function(vaccines){
// I didn't find a way to reuse the attribute name
pet.connectedVaccines = vaccines;
cb();
})
}, function(){
// this callback will be executed once all vaccines are received
return res.json(user);
});
});
There is an alternative approach solving this issue with bluebird promises, which are also part of sails. It's probably more performant than the previous one, because it fetches all vaccines with just one database request. On the other hand it's harder to read...
User.findOneByName('TestUser')
.populate('pets')
.then(function (user) {
var pets = user.pets,
petsIds = [];
// to avoid looping over the async function
// all pet ids get collected...
pets.forEach(function(pet){
petsIds.push(pet.id);
});
// ... to get all vaccines with one db call
var vaccines = Vaccine.find({pet: petsIds})
.then(function(vaccines){
return vaccines;
});
// with bluebird this array...
return [user, vaccines];
})
//... will be passed here as soon as the vaccines are finished loading
.spread(function(user, vaccines){
// for the same output as before the vaccines get attached to
// the according pet object
user.pets.forEach(function(pet){
// as seen above the attribute name can't get used
// to store the data
pet.connectedVaccines = vaccines.filter(function(vaccine){
return vaccine.pet == pet.id;
});
});
// then the user with all nested data can get returned
return res.json(user);
});
I am new to Sequelize and would like to know if anyone knows if there is a way to avoid the code duplication in Sequelize when using the migration functionality?
What I mean is that in the migration you would have something like this:
//migration-xyz.js
module.exports = {
up: function(migration, DataTypes, done) {
migration.createTable('Users',
{
name: DataTypes.STRING,
surname: DataTypes.STRING
});
},
down: function(migration, DataTypes, done) {
// logic for reverting the changes
}
}
But then the model would look something like this:
//user.js
module.exports = function(sequelize, DataTypes) {
var User = sequelize.define('User', {
name: DataTypes.STRING,
surname: DataTypes.STRING
});
return User;
};
Is there a way to get rid of the code duplication?
Thanks for the help! :)
As #WiredPrairie referenced in his/her comment, migrations and the attributes on a model are distinct things. Often we will create a model that has certain attributes and the associated migration to create the corresponding table. Later on more attributes will be added to the model and we will need to create a migration that adds columns for just the new attributes.
A way to remove the duplication would be to use sequelize.sync which creates your database from the model files. This has downsides though and should not be used on complex applications that are being deployed in different environments.
Maybe in the future the sequelize-cli will have a create:migration --model [modelfile] option where it will create the table for the corresponding model. This would have the same "duplication" of code but would make things a little faster.
When you initially ran the migrations via sequelize-cli
it set up the folder structure as:
-config
-config/config.json
-migrations
-models
-models/index.js
If you make sure your model file:
//user.js
module.exports = function(sequelize, DataTypes) {
var User = sequelize.define('User', {
name: DataTypes.STRING,
surname: DataTypes.STRING
});
return User;
};
is in the models directory, it will automatically be picked up when you
require('./models');
So, I've used the following code to import the new models for that revision of the migration:
//migrations-xyz.js
var Models = require('../models');
module.exports = {
up: function (queryInterface, Sequelize) {
/*
Add altering commands here.
Return a promise to correctly handle asynchronicity.
Example:
return queryInterface.createTable('users', { id: Sequelize.INTEGER});
*/
var tables = _.map(Models.sequelize.models, function(def){
var model = Models[def],
options = model.options,
tableName = model.tableName,
attributes = model.tableAttributes;
return queryInterface.createTable(tableName, attributes, options);
});
return Promise.all(tables);
},
down: function (queryInterface, Sequelize) {
/*
Add reverting commands here.
Return a promise to correctly handle asynchronicity.
Example:
return queryInterface.dropTable('users');
*/
return Models.sequelize.drop({logging: true});
}
};
I am looking for an example nodejs app that uses the sequelize ORM.
My main concern is that it seems next to impossible to define your models in separate js files if those models have complex relationships to one another because of require() dependency loops. Maybe people define all their models in one file that is very very long?
I am mainly interested in how the models are defined and use through out the app. I would like to have some validation that what i am doing on my own is the "good" way to do things.
The short story
The trick in this case is not to initialize the model in the file but just to provide the necesary information for its initialization and let a centralized module take care of the models setup and instantiation.
So the steps are:
Have several Model files with data about the model, like fields, relationships and options.
Have a singleton module which loads all those files and setup all the model classes and relationships.
Setup your singleton module at the app.js file.
Get the model classes from the singleton module do not use require on your model files, load the models from the singleton instead.
The longer story
Here is a more detailed description of this solution with the corresponding source code:
http://jeydotc.github.io/blog/2012/10/30/EXPRESS-WITH-SEQUELIZE.html
EDIT: This is a very old answer! (read down for info)
It's old and limited in many ways!
First, as #jinglesthula mentioned in comments (and I experienced it too) - there are problems with requiring those files. It's because require doesn't work the same way as readdirSync!
Second - you are very limited in relations - the code doesn't provide options to those associations so you are UNABLE to create belongsToMany as it needs through property. You can make the most basic assocs.
Third - you are very limited in model relations! If you read closely the code, you will see that relations is an Object instead of an Array, so if you want to make more than one associations of the same type (like having two times belongsTo) - you cannot!
Fourth - You don't need that singleton thingy. Every module in nodejs is singleton by itself, so all this makes is pretty complex for no reason.
You should see Farm's answer! (The link to the article is broken, but I'll fix it with this official sample from sequelize: https://github.com/sequelize/express-example/blob/master/models/index.js - you can browse the whole project to get an idea of what's going on).
p.s.
I'm editing this post as it's so upvoted that people won't even see any new answers (as I did).
Edit: Just changed the link to a copy of the same post, but in a Github Page
SequelizeJS has a article on their website which solves this problem.
Link is broken, but you can find the working sample project here and browse it. See edited answer above to see why this is a better solution.
Extract from article:
models/index.js
The idea of this file is to configure a connection to the database and to collect all Model definitions. Once everything is in place, we will call the method associated on each of the Models. This method can be used to associate the Model with others.
var fs = require('fs')
, path = require('path')
, Sequelize = require('sequelize')
, lodash = require('lodash')
, sequelize = new Sequelize('sequelize_test', 'root', null)
, db = {}
fs.readdirSync(__dirname)
.filter(function(file) {
return (file.indexOf('.') !== 0) && (file !== 'index.js')
})
.forEach(function(file) {
var model = sequelize.import(path.join(__dirname, file))
db[model.name] = model
})
Object.keys(db).forEach(function(modelName) {
if (db[modelName].options.hasOwnProperty('associate')) {
db[modelName].options.associate(db)
}
})
module.exports = lodash.extend({
sequelize: sequelize,
Sequelize: Sequelize
}, db)
I've create a package sequelize-connect to help people deal with this issue. It follows the Sequelize suggested convention here: http://sequelize.readthedocs.org/en/1.7.0/articles/express/
Additionally it also functions a bit more like Mongoose in terms of its interface. It allows you to specify a set of locations where your models are located and also allows you to define a custom matching function to match your model files.
The usage is basically like this:
var orm = require('sequelize-connect');
orm.discover = ["/my/model/path/1", "/path/to/models/2"]; // 1 to n paths can be specified here
orm.connect(db, user, passwd, options); // initialize the sequelize connection and models
Then you can access the models and sequelize like so:
var orm = require('sequelize-connect');
var sequelize = orm.sequelize;
var Sequelize = orm.Sequelize;
var models = orm.models;
var User = models.User;
Hopefully this helps someone out.
I started using Sequelize in Express.js app. Soon enough ran into issues of the nature you're describing. Maybe I did not quite understand Sequelize, but to me doing things more than just selecting from one table wasn't really convenient. And where ordinarily you would use select from two or more tables, or a union in pure SQL, you would have to run separate queries, and with the async nature of Node it's just added complexity.
Therefore I moved away from using Sequelize. Moreover I am switching from using ANY data fetching from DB in the models. In my opinion it is better to abstract getting data completely. And reasons are - imagine that you don't just use MySQL (in my case, I use MySQL and MongoDB side by side), but you can get your data from any data provider and any transport method, e.g. SQL, no-SQL, filesystem, external API, FTP, SSH etc. If you tried to do all of it in the models, you would eventually create complex and hard to understand code that would be hard to upgrade and debug.
Now what you want to do is to have models get data from a layer that knows where and how to get it, but your models only use API methods, e.g. fetch, save, delete etc. And inside this layer you have specific implementations for specific data providers. E.g. you can request certain data from a PHP file on a local machine or from Facebook API or from Amazon AWS or from remote HTML document, etc.
PS some of these ideas were borrowed from Architect by Cloud9: http://events.yandex.ru/talks/300/
I set it up as Farm and the documentation describe.
But I was having the additonal problem that in my instance methods and class methods that I would attach to the models in each function I would need to require the index file to get a hold of other database objects.
Solved it by making them accessible to all models.
var Config = require('../config/config');
var fs = require('fs');
var path = require('path');
var Sequelize = require('sequelize');
var _ = require('lodash');
var sequelize;
var db = {};
var dbName, dbUsername, dbPassword, dbPort, dbHost;
// set above vars
var sequelize = new Sequelize(dbName, dbUsername, dbPassword, {
dialect: 'postgres', protocol: 'postgres', port: dbPort, logging: false, host: dbHost,
define: {
classMethods: {
db: function () {
return db;
},
Sequelize: function () {
return Sequelize;
}
}
}
});
fs.readdirSync(__dirname).filter(function(file) {
return (file.indexOf('.') !== 0) && (file !== 'index.js');
}).forEach(function(file) {
var model = sequelize.import(path.join(__dirname, file));
db[model.name] = model;
});
Object.keys(db).forEach(function(modelName) {
if ('associate' in db[modelName]) {
db[modelName].associate(db);
}
});
module.exports = _.extend({
sequelize: sequelize,
Sequelize: Sequelize
}, db);
And in the model file
var classMethods = {
createFromParams: function (userParams) {
var user = this.build(userParams);
return this.db().PromoCode.find({where: {name: user.promoCode}}).then(function (code) {
user.credits += code.credits;
return user.save();
});
}
};
module.exports = function(sequelize, DataTypes) {
return sequelize.define("User", {
userId: DataTypes.STRING,
}, { tableName: 'users',
classMethods: classMethods
});
};
I only did this for the class methods but you could also do the same thing for instance methods.
I am following the official guide: http://sequelizejs.com/heroku, which has a models folder, set up each module in separate files, and have a index file to import them and set the relationship among them.
Sample model sequelize
'use strict';
const getRole = require('../helpers/getRole')
const library = require('../helpers/library')
const Op = require('sequelize').Op
module.exports = (sequelize, DataTypes) => {
var User = sequelize.define('User', {
AdminId: DataTypes.INTEGER,
name: {
type: DataTypes.STRING,
validate: {
notEmpty: {
args: true,
msg: 'Name must be filled !!'
},
}
},
email: {
type: DataTypes.STRING,
validate: {
notEmpty: {
args: true,
msg: 'Email must be filled !!'
},
isUnique: function(value, next) {
User.findAll({
where:{
email: value,
id: { [Op.ne]: this.id, }
}
})
.then(function(user) {
if (user.length == 0) {
next()
} else {
next('Email already used !!')
}
})
.catch(function(err) {
next(err)
})
}
}
},
password: {
type: DataTypes.STRING,
validate: {
notEmpty: {
args: true,
msg: 'Password must be filled !!'
},
len: {
args: [6, 255],
msg: 'Password at least 6 characters !!'
}
}
},
role: {
type: DataTypes.INTEGER,
validate: {
customValidation: function(value, next) {
if (value == '') {
next('Please choose a role !!')
} else {
next()
}
}
}
},
gender: {
type: DataTypes.INTEGER,
validate: {
notEmpty: {
args: true,
msg: 'Gender must be filled !!'
},
}
},
handphone: {
type: DataTypes.STRING,
validate: {
notEmpty: {
args: true,
msg: 'Mobile no. must be filled !!'
},
}
},
address: DataTypes.TEXT,
photo: DataTypes.STRING,
reset_token: DataTypes.STRING,
reset_expired: DataTypes.DATE,
status: DataTypes.INTEGER
}, {
hooks: {
beforeCreate: (user, options) => {
user.password = library.encrypt(user.password)
},
beforeUpdate: (user, options) => {
user.password = library.encrypt(user.password)
}
}
});
User.prototype.check_password = function (userPassword, callback) {
if (library.comparePassword(userPassword, this.password)) {
callback(true)
}else{
callback(false)
}
}
User.prototype.getRole = function() {
return getRole(this.role)
}
User.associate = function(models) {
User.hasMany(models.Request)
}
return User;
};
You can import models from other files with sequelize.import
http://sequelizejs.com/documentation#models-import
That way you can have one singleton module for sequelize, which then loads all the other models.
Actually this answer is quite similar to user1778770`s answer.
I am looking for an example nodejs app that uses the sequelize ORM.
You might be interested in looking at the PEAN.JS boilerplate solution.
https://github.com/StetSolutions/pean
PEAN.JS is a full-stack JavaScript open-source solution, which provides a solid starting point for PostgreSQL, Node.js, Express, and AngularJS based applications.
The PEAN project is a fork of the MEAN.JS project (not to be confused with MEAN.IO or the generic MEAN stack).
https://github.com/meanjs/mean
PEAN replaces MongoDB and the Mongoose ORM with PostgreSQL and Sequelize. A primary benefit of the MEAN.JS project is the organization it provides to a stack that has many moving pieces.
You can also use a dependency injection which provides an elegant solution to this. Here's one https://github.com/justmoon/reduct
What worked for me is to:
Create a file for each individual model like user.model.js in folder models/user.model.js.
Create index.js in models/index.js and import every model to it.
Define association, run sync method in index.js and export all models.
Create a database.js file that holds information about Sequalize and import it and initialize it in app.js
Example of one models/user.model.js
import { DataTypes } from 'sequelize';
import { sequelize } from '../database.js';
export const User = sequelize.define("user",{
uid:{
type:DataTypes.STRING,
allowNull:false,
unique: true
},
email:{
type:DataTypes.STRING,
allowNull:true
},
firstName:{
type:DataTypes.STRING,
allowNull:true
},
lastName:{
type:DataTypes.STRING,
allowNull:true
},
companyWebsite:{
type:DataTypes.STRING,
allowNull:true
},
domain:{
type:DataTypes.STRING,
allowNull:true
},
hsPortalId:{
type:DataTypes.INTEGER,
allowNull:true
},
integrations:{
type:DataTypes.STRING
},
brandedKeywords : {
type:DataTypes.STRING
},
companyName: {
type:DataTypes.STRING
},
companyStreet:{
type:DataTypes.STRING
},
companyZip:{
type:DataTypes.STRING
},
companyCountry:{
type:DataTypes.STRING
},
vatId:{
type:DataTypes.STRING
},
brand:{
type:DataTypes.STRING
},
markets:{
type:DataTypes.JSON
},
niche : {
type:DataTypes.JSON
}
},{schema:"api"})
Example of models/index.js
import { Billing } from './billing.model.js';
import { Competitor } from './competitors.model.js';
import { DemoAccount } from './demo.model.js';
import { Notification } from './notification.model.js';
import { Product } from './products.model.js';
import { Reseller } from './resellers.model.js';
import {Reseller_User} from './reseller_user.model.js'
import { Tag } from './tags.model.js';
import {User} from './user.model.js'
Reseller.belongsToMany(User, { through: Reseller_User });
User.belongsToMany(Reseller, { through: Reseller_User });
// this will create a UserId column on your Product table
// https://www.youtube.com/watch?v=HJGWu0cZUe8 40min
User.hasMany(Product,{onDelete: 'CASCADE',})
Product.belongsTo(User)
User.hasOne(DemoAccount,{onDelete: 'CASCADE',})
DemoAccount.belongsTo(User)
User.hasMany(Billing,{onDelete: 'CASCADE',})
Billing.belongsTo(User)
User.hasMany(Tag,{onDelete: 'CASCADE',})
Tag.belongsTo(User)
User.hasMany(Competitor,{onDelete: 'CASCADE'})
Competitor.belongsTo(User)
User.hasMany(Notification,{onDelete: 'CASCADE'})
Notification.belongsTo(User)
User.sync().then(
() => console.log("Sync complete")
);
Reseller.sync().then(
() => console.log("Sync complete")
);
Reseller_User.sync().then(
() => console.log("Sync complete")
);
Product.sync().then(
() => console.log("Product Sync complete")
);
Competitor.sync().then(
() => console.log("Competitor Sync complete")
);
Notification.sync().then(
() => console.log("Competitor Sync complete")
);
Billing.sync().then(
() => console.log("Billing Sync complete")
);
Tag.sync().then(
() => console.log("Tag Sync complete")
);
DemoAccount.sync()
export { User, Reseller, Product, Competitor, Notification, DemoAccount, Billing, Tag };
// DemoAccount.sync({force:true}).then(
// () => console.log("Sync complete")
// );