Maybe there is not a definitive answer here but I would like to know where to handle data validation when dealing with express.js and mongoose. Which of the following is the best practice (I currently use a combination and it's starting to feel very clumsy):
the Model (mongoose)
the Controller / Route (express)
Some older posts I have read are:
this;
this;
and, this;
but conflicting answers just add to the confusion. Maybe it simply isn't clear cut, in which case is one a better option?
When using mongoose I would push most of my validation logic to the mongoose model/schema. You can use mongoose-validator which is just a wrapper around node-validator for simple model validation. If you need validation against other models or more complex logic in the validation you can write your own custom mongoose pre validate or post validate hook (see mongoose middleware).
An additional benefit you gain when using mongoose to validate your model is that mongoose adds an error property to your model which can be accessed via model.errors[property]. This property can be used for validation error messages on the web or for a service client.
When writing more/very complex software tying the validation to the model may become a problem. But I'd deal with this problem when it arises. Since JavaScript has functions as first class citizens your validation functions still can be reused even in these complex situations.
The mongoose validator is a great place to start on a model level, but if you need to have controller specific validation, I use the following code in a utils.js file:
var async = require('async')
exports.validator = function (reqProps, props, mongoEnt, next) {
var propsErr = [];
var mongoErr = {};
async.parallel([function (cb) {
reqProps.forEach(function (rp) {
if (!props[rp])propsErr.push(rp);
})
cb()
}, function (cb) {
if (mongoEnt != null) {
var test = new mongoEnt(props);
test.validate(function (err) {
mongoErr = err;
cb();
});
} else {
mongoErr = null;
cb();
}
}], function (err, result) {
if (propsErr.length != 0) {
return next(new Error('The following props were not included: ' + propsErr));
} else if(mongoErr != null) {
return next(new Error('The following prop was not included: ' +mongoErr.errors[Object.keys(mongoErr.errors).pop()].path));
} else {
return next(null);
}
})
}
This allows me to both validate using the mongoose validator and check for the additional props that I include in the reqProps property in one line of code. Though this is only checking for required properties, you could easily extend it for your own validation scheme.
An example of usage for this code:
var Person = mongoose.model('Person')
exports.addUSCitizen = function(props, next){
utils.validator(['ssn'], props, Person, function (err) {
if(err) return next(err);
//do something using props.ssn
})
}
Related
I'm trying to update counts on a pre hook. The issue is that for some unknown reason the findOneAndUpdate hook doesn't have access to the document, as far as I can tell.
I would like to do this:
source.pre('findOneAndUpdate', function (next) {
console.log('------------->>>>>> findOneAndUpdate: ');
this.objects = this.objects || [];
this.people = this.people || [];
this.events = this.events || [];
this.objectCount = this.objects.length;
this.peopleCount = this.people.length;
this.eventCount = this.events.length;
next();
});
But for some reason the this in the hook isn't the document, its a Query object which seems about useless.
What am I missing? How do I use a pre hook to update counts on a findOneAndUpdate?
You can do smthng like that ->
source.pre('findOneAndUpdate', function (next) {
console.log('------------->>>>>> findOneAndUpdate: ');
this._update.$set.objects = [];
this._update.$set.people = [];
this._update.$set.events = [];
next();
});
pay attention to _update.$set because in the context "this" will be a query. So you can easily add anything you want!
The documentation states:
Query middleware differs from document middleware in a subtle but important way: in document middleware, this refers to the document being updated. In query middleware, mongoose doesn't necessarily have a reference to the document being updated, so this refers to the query object rather than the document being updated.
An update action generally updates a document that only exists in the database (it tells the MongoDB server: "find document X and set property X to value Z"), so the full document isn't available to Mongoose and, hence, you can't update the counts (which requires access to at least the arrays whose length you want to determine).
As an aside: why do you need separate *Count properties in your schema anyway? If you want to query for arrays matching a certain size, you can use the $size operator on the arrays directly.
If you really do need the count properties, then for each update, you need to track the number of changes you made to each of the arrays (in terms of the number of items added/deleted) and use the $inc operator to adjust the counts.
I had a similar issue when I used the updateOne method and was also going to use the updateOne pre hook to make intermittent update before saving to the database. couldn't find a way for it to work. I ended up using the findOneAndUpdate pre hook and doing the updateOne in it.
schema.pre('findOneAndUpdate', async function(next){
const schema = this;
const { newUpdate } = schema.getUpdate();
const queryConditions = schema._condition
if(newUpdate){
//some mutation magic
await schema.updateOne(queryConditions, {newUpdate:"modified data"});
next()
}
next()
})
Another solution is to use the official MongoDB documentation on middleware. They explain why "this" does not refer to the document itself. You may try something in that sense:
source.pre('findOneAndUpdate', async function(next) {
const docToUpdate = await this.model.findOne(this.getFilter());
//modify the appropriate objects
docToUpdate.save(function(err) {
if(!err) {
console.log("Document Updated");
}
});
console.log(docToUpdate);
// The document that `findOneAndUpdate()` will modify
next();
});
This worked for me
SCHEMA.pre('findOneAndUpdate', function(next){
this._update.yourNestedElement
next();
});
schema.pre(['findOneAndUpdate'], async function(next) {
try {
const type = this.get('type')
const query = this.getQuery()
const doc = await this.findOne(query)
if (type) {
this.set('type', doc.type)
}
next()
} catch (e) {
next(new BaseError(e))
}
})
mongoose Documentation:
You cannot access the document being updated in pre('updateOne') or pre('findOneAndUpdate') query middleware. If you need to access the
document that will be updated, you need to execute an explicit query
for the document.
schema.pre('findOneAndUpdate', async function() {
const docToUpdate = await this.model.findOne(this.getQuery());
console.log(docToUpdate); // The document that `findOneAndUpdate()` will modify
});
Is it possible to process a db.model.find() query inside of function context and retrieve a result without using callbacks and promises with mongoose library?
I need to get assured, if some user exists in process of running controller, so, I can't minimize current scope to callback due to large amount of same operations (for example, communication with database). Also I'm trying to realize MVC model in my project, so, I want to keep the helper libs (modules) in separated files. That's why I don't want to use any callbacks or promises - they will much times complicate everything even more then things already do.
For example, how should I rewrite the following code to be executed successfully (if it's actually possible) (you can ignore login model and controller - they are written to represent complicacy if to rewrite that code using callbacks):
user.js lib
var db = require('./lib/db');
class User{
constructor(id){ //get user by id
var result = db.models.user.findOne({_id: id}); //unsupported syntax in real :(
if(!result || result._id != _id)
return false;
else{
this.userInfo = result;
return result;
}
}
}
module.exports = User;
login model
var user = require('./lib/user')
var model = {};
model.checkUserLogged(function(req){
if(!req.user.id || req.user.id == undefined)
return false;
if(!(this.user = new user(req.user.id)))
return false;
else
return true;
});
module.exports = model;
login controller
var proxy = require('express').router();
proxy.all('/login', function(req, res){
var model = require('./models/login');
if(!model.checkUserLogged()){
console.log('User is not logged in!');
res.render('unlogged', model);
}else{
console.log('User exists in database!');
res.render('logged_in', model);
}
});
Generator functions/yields, async/await (es2017), and everything et cetera can be used just to solve the problem without nesting.
Thx in advance.
There are two points wrong:
Mongoose methods can't be called synchronously (Anyway a call to a DB done synchronously is not a good idea at all).
Nor async/await nor generators can be used in the constructor of an ES6 Class. It is explained in this answer.
If you don't want nested code an easy option could be to use async/await (currently available in Node.js using a flag, not recommended for production). Since Mongoose methods return promises they can be used with async/await.
But as I said you can not do that in the constructor, so it has to be somewhere else.
As an example you could do something like this:
var proxy = require('express').router();
var db = require('./lib/db');
proxy.all('/login', async function(req, res){
const result = await db.models.user.findOne({_id: req.user.id}).exec();
if (!result) {
console.log('User is not logged in!');
return res.render('unlogged');
}
res.render('logged_in');
});
Old question, but I want to share a method for handling this that I didn't see in my first couple searches.
I want to get data from a model, run some logic and return the results from that logic. I need a promise wrapper around my call to the model.
Below is a slightly abstracted function that takes a model to run a mongoose/mongo query on, and a couple params to help it do some logic. It then returns the value that is expected in the promise or rejects.
export function promiseFunction(aField: string, aValue, model: Model<ADocument, {}>): Promise<aType> {
return new Promise<string>((resolve, reject) => {
model.findOne({[aField]: aValue}, (err, theDocument) => {
if(err){
reject(err.toString());
} else {
if(theDocument.someCheck === true){
return(theDocument.matchingTypeField)
} else {
reject("there was an error of some type")
}
}
});
})
}
i feel a bit embarrassed, can you please kindly explain parts of the code?
For example, I have no idea, what is this part? where can I read more about it?
function parsePostStory(data) {
return {
name : data.name
}
}
What is req.body? Is it json req body?
Why do we declare empty array and why do we return it? Just for the clarity?
Is Story.create just a mongoose method?
The rest of the code is here:
router.post('/stories', function(req, res) {
var validation = validatePostStory(req.body);
if(validation.length > 0) {
return res.badRequestError(validation);
}
var story = parsePostStory(req.body);
Story.create(story, function(err, story) {
if(err) {
console.log(err.message);
return res.internalServerError();
} res.send(story);
});
});
function validatePostStory(data) {
var array = [];
if (!data.name || typeof data.name !== 'String') {
return array.push('name');
}
return array;
}
function parsePostStory(data) {
return {
name : data.name
}
}
Sorry once more for that kind of a question and thanks a ton.
I'm assuming you know how the request-response cycle works with HTTP requests and the client-server interactions with it. If not, Wikipedia Request-Response and Client-Server (Two link limit, otherwise I would have posted them as links)
A request sends a lot of information to the server. If you console.log the request in NodeJS, you will see that it contains a lot of information that isn't entirely relevant to what you need.
You're using Express as your web framework. In this case, req.body is the information that you are sending to the server from the client. Using req.body will make sure that you're not using the extra information passed in to the server from the client. Req.body is your code that you want. (Note: Req.body isn't natively supported by Express v4, you'll have to use something like body-parser) See Express docs for more details
Now, let's break up this code a bit. You essentially have 3 separate functions. Let's take a look at validatePostStory.
function validatePostStory(data) {
var array = [];
if (!data.name || typeof data.name !== 'String') {
return array.push('name');
}
return array;
}
This function is a validation function. It takes one argument - an object and returns an array. Effectively, what this is doing is checking if the name is a string or not - if not, return an array that has a length of 1. The following conditional checks length and returns a 400 if greater than 0
if(validation.length > 0) {
return res.badRequestError(validation);
}
I'm not entirely sure why this needs to be a separate function. Looks like you can probably just do this instead.
if (!req.body.name || typeof req.body.name !== 'String') {
return res.badRequestError(validation);
}
The following function function essentially converts the data so that mongodb/mongoose can store it in the proper format
function parsePostStory(data) {
return {
name : data.name
}
}
It's the same as saying:
var story = {name: req.body.name}
I would assume Story.create is a custom mongoose method yes.
I'm trying to avoid using callbacks when making mongodb queries. I'm using mongoskin to make calls like so:
req.db.collection('users').find().toArray(function (err, doc) {
res.json(doc);
});
In many cases I need to make multiple queries so I want to use Node.js promise library but I'm not sure how to wrap these functions as promises. Most of the examples I see are trivial for things like readFile, I'm guessing in this case I would need to wrap toArray somehow? Can this be done or would have to be something implemented by mongoskin?
An example could be any set of callbacks, find/insert, find/find/insert, find/update:
req.db.collection('users').find().toArray(function (err, doc) {
if (doc) {
req.db.collection('users').find().toArray(function (err, doc) {
// etc...
});
}
else {
// err
}
});
You can promisify the entire module like so with bluebird:
var Promise = require("bluebird");
var mongoskin = require("mongoskin");
Object.keys(mongoskin).forEach(function(key) {
var value = mongoskin[key];
if (typeof value === "function") {
Promise.promisifyAll(value);
Promise.promisifyAll(value.prototype);
}
});
Promise.promisifyAll(mongoskin);
This only needs to be done in one place for one time in your application, not anywhere in your application code.
After that you just use methods normally except with the Async suffix and don't pass callbacks:
req.db.collection('users').find().toArrayAsync()
.then(function(doc) {
if (doc) {
return req.db.collection('users').find().toArrayAsync();
}
})
.then(function(doc) {
if (doc) {
return req.db.collection('users').find().toArrayAsync();
}
})
.then(function(doc) {
if (doc) {
return req.db.collection('users').find().toArrayAsync();
}
});
So again, if you call a function like
foo(a, b, c, function(err, result) {
if (err) return console.log(err);
//Code
});
The promise-returning version is called like:
fooAsync(a, b, c).then(...)
(Uncaught errors are automatically logged so you don't need to check for them if you are only going to log it)
Just stumbled here with the same question and didn't love "promisfying" mongoskin so did a bit more digging and found monk. It's built on top of mongoskin, tidies up the API and returns
promises for all async calls. Probably worth a peek to anyone else who lands here.
Esailija's answer may work, but its not super efficient since you have to run db.collection on every single db call. I don't know exactly how expensive that is, but looking at the code in mongoskin, its non-trivial. Not only that, but it's globally modifying prototypes, which isn't very safe.
The way I do this with fibers futures is:
wrap the collection methods for each collection
on receiving the result, for methods that return a Cursor wrap the toArray method, call it and return the resulting future (for methods that don't return a cursor, you don't need to do anything else).
use the future as normal
like this:
var Future = require("fibers/future")
// note: when i originally wrote this answer fibers/futures didn't have a good/intuitive wrapping function; but as of 2014-08-18, it does have one
function futureWrap() {
// function
if(arguments.length === 1) {
var fn = arguments[0]
var object = undefined
// object, methodName
} else {
var object = arguments[0]
var fn = object[arguments[1]]
}
return function() {
var args = Array.prototype.slice.call(arguments)
var future = new Future
args.push(future.resolver())
var me = this
if(object) me = object
fn.apply(me, args)
return future
}
}
var methodsYouWantToHave = ['findOne', 'find', 'update', 'insert', 'remove', 'findAndModify']
var methods = {}
methodsYouWantToHave.forEach(function(method) {
internalMethods[method] = futureWrap(this.collection, method)
}.bind(this))
// use them
var document = methods.findOne({_id: 'a3jf938fj98j'}, {}).wait()
var documents = futureWrap(methods.find({x: 'whatever'}, {}).wait(), 'toArray')().wait()
If you don't want to use fibers, I'd recommend using the async-future module, which has a good wrap function built in too.
I've been reading about node.js recently (like many others). I find interesting for some use cases, but am a bit struggling to understand the inner workings, specifically the interaction between closure functions and the process flow of the code.
Let's say I have a function which accepts a key-value array. The function must check that the values follow certain data-quality guidelines (for example some keys must have a value, other keys must have numbers as values etc) before storing the data somewhere (for the purpose of this question let's assume data validation has to be done in the application itself).
In "regular" developments models I'd write something like this:
resultName = validateName(data.name)
resultAddress = validateAddress(data.address)
resultID = validateID(data.id)
if (resultName && resultAddress && resultID) {
store(data)
else {
sendErrorToUser(data)
}
Get the results of the validations, and either explain the error(s) to the user or store data and return some kind of confirmation. The flow is very clear.
The way I understand node.js, the way to do this would be to delegate the validations to a different function (to avoid waiting for each validation to finish), and supply two callback functions to the functions which validate the chunks of data:
* a callback to call when validation is successful
* a callback to call when validation fails
It's easy to now return to the user with a "please wait" message, but I have to wait for all validations to clear (or fail) before storing the data or explaining the problem to the user. As a simple way to figure out if all the validations are done I thought of using a variable that counts the number of functions that called the callback, and emitting a "validation complete" event to store the validated data (or get back to the user with any errors). Or, alternatively, emit an event after each validation is complete and in that event's code check if all validations are complete before emitting the "store" / "error" events.
My question is -- am I approaching this correctly? Or is there a more suitable way to do these kinds of things with node.js (or similar event-based systems).
Thank you!
Alon
Are your validations asynchronous? If they are not you can use the code you posted, the "regular" one.
If the validations are asynchronous (checking uniqueness of an email for instance), you need to provide callbacks:
var validateUniqueEmail = function (data, callback) {
db.find({email: data.email}, function (err, result) {
callback(err, result === null);
})
};
var validateAndStore = function (data, callback) {
asyncValidation(data, function (err, is_valid) {
if (err) {
callback(err, null);
} else if (!is_valid) {
callback('Email is not unique', null);
} else {
db.store(data, callback);
}
});
}
The code above can be simplified a lot by using some validator or ORM modules already existing
example: mongolia validator module.
Let's go. Basically, what you want to do is something along the lines of :
var validate(data, cb){
var allOk = true;
for(var key in data){
allOk = allOk && validate[key](data.key); // validator depends on the key
}
if (allOk) cb(null, data); else cb(new Error "bleh");
}
This could be done the following way (note how we pass the failed keys as the first (error) argument to the callback):
var validate(data, cb){
var status = {true:[], false:[]},
total = Object.keys(data).length,
done = 0;
for (var key in data)
(function(key){
validate[key](data[key], function(ok){
status[ok].push(key);
if (++done == total){
status[false].length ? cb(status[false]) : cb(null);
}
});
})(key);
}
Which you can use this way :
validate(data, function(failures){
if (failures){
// tell the user the input does not validate on the keys stored in failures
} else {
// all ok
store(data);
}
});
Correct me if I'm wrong, but I think what you're asking is how to handle the response from multiple asynchronous calls.
Here's how I do it (using your validation example):
var result = {};
function isAllDataAvailable() {
return result.name !== undefined
&& result.address !== undefined
&& result.id !== undefined;
}
function callback(error) {
if (error) {
showError(error);
// terminate here (?)
return;
}
if (isAllDataAvailable()) {
showOutput();
}
}
validateName(data, callback);
validateAddress(data, callback);
validateEmail(data, callback);
The key here is the result object, which starts out as empty. As each field gets validated, it gets added to the result object (by the validation functions, which I've left out in the above snippet). I've used a single callback method, but you could have multiple, say callbackName, callbackAddress, etc. The validation results are processed only if and when the result object has been fully populated, which is checked in isAllDataAvailable.
Hope this helps.
Consider using: https://github.com/asaf/nodejs-model
It will make your life much easier when dealing with validators.