Synchronous mongoose request - node.js

Is it possible to process a db.model.find() query inside of function context and retrieve a result without using callbacks and promises with mongoose library?
I need to get assured, if some user exists in process of running controller, so, I can't minimize current scope to callback due to large amount of same operations (for example, communication with database). Also I'm trying to realize MVC model in my project, so, I want to keep the helper libs (modules) in separated files. That's why I don't want to use any callbacks or promises - they will much times complicate everything even more then things already do.
For example, how should I rewrite the following code to be executed successfully (if it's actually possible) (you can ignore login model and controller - they are written to represent complicacy if to rewrite that code using callbacks):
user.js lib
var db = require('./lib/db');
class User{
constructor(id){ //get user by id
var result = db.models.user.findOne({_id: id}); //unsupported syntax in real :(
if(!result || result._id != _id)
return false;
else{
this.userInfo = result;
return result;
}
}
}
module.exports = User;
login model
var user = require('./lib/user')
var model = {};
model.checkUserLogged(function(req){
if(!req.user.id || req.user.id == undefined)
return false;
if(!(this.user = new user(req.user.id)))
return false;
else
return true;
});
module.exports = model;
login controller
var proxy = require('express').router();
proxy.all('/login', function(req, res){
var model = require('./models/login');
if(!model.checkUserLogged()){
console.log('User is not logged in!');
res.render('unlogged', model);
}else{
console.log('User exists in database!');
res.render('logged_in', model);
}
});
Generator functions/yields, async/await (es2017), and everything et cetera can be used just to solve the problem without nesting.
Thx in advance.

There are two points wrong:
Mongoose methods can't be called synchronously (Anyway a call to a DB done synchronously is not a good idea at all).
Nor async/await nor generators can be used in the constructor of an ES6 Class. It is explained in this answer.
If you don't want nested code an easy option could be to use async/await (currently available in Node.js using a flag, not recommended for production). Since Mongoose methods return promises they can be used with async/await.
But as I said you can not do that in the constructor, so it has to be somewhere else.
As an example you could do something like this:
var proxy = require('express').router();
var db = require('./lib/db');
proxy.all('/login', async function(req, res){
const result = await db.models.user.findOne({_id: req.user.id}).exec();
if (!result) {
console.log('User is not logged in!');
return res.render('unlogged');
}
res.render('logged_in');
});

Old question, but I want to share a method for handling this that I didn't see in my first couple searches.
I want to get data from a model, run some logic and return the results from that logic. I need a promise wrapper around my call to the model.
Below is a slightly abstracted function that takes a model to run a mongoose/mongo query on, and a couple params to help it do some logic. It then returns the value that is expected in the promise or rejects.
export function promiseFunction(aField: string, aValue, model: Model<ADocument, {}>): Promise<aType> {
return new Promise<string>((resolve, reject) => {
model.findOne({[aField]: aValue}, (err, theDocument) => {
if(err){
reject(err.toString());
} else {
if(theDocument.someCheck === true){
return(theDocument.matchingTypeField)
} else {
reject("there was an error of some type")
}
}
});
})
}

Related

How do I achieve a synchronous requirement using asynchronous NodeJS

I am adding user validation an data modification page on a node.js application.
In a synchronous universe, in a single function I would:
Lookup the original record in the database
Lookup the user in LDAP to see if they are the owner or admin
Do the logic and write the record.
In an asynchronous universe that won't work. To solve it I've built a series of hand-off functions:
router.post('/writeRecord', jsonParser, function(req, res) {
post = req.post;
var smdb = new AWS.DynamoDB.DocumentClient();
var params = { ... }
smdb.query(params, function(err,data){
if( err == null ) writeRecordStep2(post,data);
}
});
function writeRecord2( ru, post, data ){
var conn = new LDAP();
conn.search(
'ou=groups,o=amazon.com',
{ ... },
function(err,resp){
if( err == null ){
writeRecordStep3( ru, post, data, ldap1 )
}
}
}
function writeRecord3( ru, post, data ){
var conn = new LDAP();
conn.search(
'ou=groups,o=amazon.com',
{ ... },
function(err,resp){
if( err == null ){
writeRecordStep4( ru, post, data, ldap1, ldap2 )
}
}
}
function writeRecordStep4( ru, post, data, ldap1, ldap2 ){
// Do stuff with collected data
}
Additionally, because the LDAP and Dynamo logic are in their own source documents, these functions are scattered tragically around the code.
This strikes me as inefficient, as well as inelegant. I'm eager to find a more natural asynchronous pattern to achieve the same result.
Any promise library should sort your issue out. My preferred choice is bluebird. In summary they help you in performing blocking operations.
If you haven't heard about bluebird then just use it. It converts all function of a module and return promise which is then-able. Simply put, it promisifies all functions.
Here is the mechanism:
Module1.someFunction() \\do your job and finally pass the return object to next call
.then() \\Use that object which is return from the first call, do your job and return the updated value
.then() \\same goes on
.catch() \\do your job when any error occurs.
Hope you understand. Here is an example:
var readFile = Promise.promisify(require("fs").readFile);
readFile("myfile.js",
"utf8").then(function(contents) {
return eval(contents);
}).then(function(result) {
console.log("The result of evaluating
myfile.js", result);
}).catch(SyntaxError, function(e) {
console.log("File had syntax error", e);
//Catch any other error
}).catch(function(e) {
console.log("Error reading file", e);
});
I could not tell from your pseudo-code exactly which async operations depend upon results from with other ones and knowing that is key to the most efficient way to code a series of asynchronous operations. If two operations do not depend upon one another, they can run in parallel which generally gets to an end result faster. I also can't tell exactly what data needs to be passed on to later parts of the async requests (too much pseudo-code and not enough real code to show us what you're really attempting to do).
So, without that level of detail, I'll show you two ways to approach this. The first runs each operation sequentially. Run the first async operation, when it's done, run the next one and accumulates all the results into an object that is passed along to the next link in the chain. This is general purpose since all async operations have access to all the prior results.
This makes use of promises built into the AWS.DynamboDB interface and makes our own promise for conn.search() (though if I knew more about that interface, it may already have a promise interface).
Here's the sequential version:
// promisify the search method
const util = require('util');
LDAP.prototype.searchAsync = util.promisify(LDAP.prototype.search);
// utility function that does a search and adds the result to the object passed in
// returns a promise that resolves to the object
function ldapSearch(data, key) {
var conn = new LDAP();
return conn.searchAsync('ou=groups,o=amazon.com', { ... }).then(results => {
// put our results onto the passed in object
data[key] = results;
// resolve with the original object (so we can collect data here in a promise chain)
return data;
});
}
router.post('/writeRecord', jsonParser, function(req, res) {
let post = req.post;
let smdb = new AWS.DynamoDB.DocumentClient();
let params = { ... }
// The latest AWS interface gets a promise with the .promise() method
smdb.query(params).promise().then(dbresult => {
return ldapSearch({post, dbresult}, "ldap1");
}).then(result => {
// result.dbresult
// result.ldap1
return ldapSearch(result, "ldap2")
}).then(result => {
// result.dbresult
// result.ldap1
// result.ldap2
// doSomething with all the collected data here
}).catch(err => {
console.log(err);
res.status(500).send("Internal Error");
});
});
And, here's a parallel version that runs all three async operations at once and then waits for all three of the to be done and then has all the results at once:
// if the three async operations you show can be done in parallel
// first promisify things
const util = require('util');
LDAP.prototype.searchAsync = util.promisify(LDAP.prototype.search);
function ldapSearch(params) {
var conn = new LDAP();
return conn.searchAsync('ou=groups,o=amazon.com', { ... });
}
router.post('/writeRecord', jsonParser, function(req, res) {
let post = req.post;
let smdb = new AWS.DynamoDB.DocumentClient();
let params = { ... }
Promise.all([
ldapSearch(...),
ldapSearch(...),
smdb.query(params).promise()
]).then(([ldap1Result, ldap2Result, queryResult]) => {
// process ldap1Result, ldap2Result and queryResult here
}).catch(err => {
console.log(err);
res.status(500).send("Internal Error");
});
});
Keep in mind that due to the pseudo-code nature of the code in your question, this is also pseudo-code where implementation details (exactly what parameters you're searching for, what response you're sending, etc...) have to be filled in. This should be illustrative of promise chaining to serialize operations and the use of Promise.all() for parallelizing operations and promisifying a method that didn't have promises built in.

Using ES6 Generator functions with SailsJS

I love generators in nodejs. They help nodejs look more like server-side code. I'm trying to use generators with my Sails app. This is my controller and it works when I visit 'get /hi':
/**
* FooController
*
* #description :: Server-side logic for managing foos
* #help :: See http://sailsjs.org/#!/documentation/concepts/Controllers
*/
module.exports = {
hi: function (req, res){
return res.send("Hi there!");
}
};
However when I change that hi action to a generator function...
module.exports = {
hi: function* (req, res){
return res.send("Hi there!");
}
};
this action never gets to return the response. Like it's waiting for something. How does one utilize ES6 generators within SailsJS controllers and in-fact all of Sails?
You can use it, in fact it'll be great if we all use this style, it adds a lot of readability and flexibility in your code (no more callback-hell), but that is not the way to use it.
As #Cohars stated, sails expect a function as controller actions, you can't pass a generator like in Koa, but that does not prevent you from using them, the thing is that a generator by itself is very useless, you need a function that calls it and iterates it and i believe koa does this for you at framework level, but you have some options available to you at library level, like co or yortus/asyncawait/ which is what i use because node-fibers implemented as functions are way more flexible than es6 generators (though they are almost the same) and i'll show you why in a sec.
So here is how you use it:
First npm install co and require it in your controller, or add it as a global in config/bootstrap.js since you will be using it a lot. Once you are done with it, you can use it in your controllers.
module.exports = {
hi: function(req, res){
co(function* (){
// And you can use it with your models calls like this
let user = yield User.findOne(req.param('id'))
return res.send("Hi there" + user.name + "!");
})
}
};
That's it
Now if you rather use async/await, its pretty similar, it goes like this:
module.exports = {
hi: function(req, res){
async(function(){
// Since await is a function, you can access properties
// of the returned values like this which is nice to
// access object properties or array indices
let userName = await(User.findOne(req.param('id'))).name
return res.send("Hi there" + userName + "!");
})();
}
};
There is a caveat though, if you call other methods of you controller via this, remember that they will refer to the generator fn or the passed regular fn if you use async/await, so be sure to save its context, or since you are already using es6 syntax, you can use fat arrows with async/await, unfortunately not with co, since there is not fat arrow version of generators (..yet?)
So it'll be like this:
module.exports = {
hi: function(req, res){
async(() => {
let params = this._parseParams(req);
let userName = await(User.findOne(params.id)).name
return res.send("Hi there" + userName + "!");
})();
},
_parseParams: function(req){
let params = req.allParams()
// Do some parsing here...
return params
}
};
I've been using the second method for months now, and works perfectly, i've tried with co too and works as well, i just liked more the async/await module (and is supposed to be a little bit faster) and its a perfect match if you are using coffeescript since your sintax will be like do async => await User.find()
UPDATE:
I've created a wrapper that you can use if you use yortus async/await or check the source code and modify it to work with co or something else if you wish.
Here is the link https://www.npmjs.com/package/async-handler, is in alpha but i'm using on my own apps and works as expected, if not submit an issue.
With that the las example would look like this:
module.exports = {
hi: asyncHandler((req, res)->{
let params = this._parseParams(req);
let userName = await(User.findOne(params.id)).name
return res.send("Hi there" + userName + "!");
}),
_parseParams: function(req){
let params = req.allParams()
// Do some parsing here...
return params
}
};
With this handler you get the extra benefit of having async/promise errors to propagate correctly on sails if they are not caught by a try/catch
Sails expects a regular function here, not a generator. Maybe you could take a look at co, not sure it would really help with Sails though. If you really want to use generators, you should probably try Koa, which has several frameworks based on it
The way I am doing it is like this:
const Promise = require('bluebird');
module.exports = {
hi: Promise.coroutine(function* (req, res) {
let id = req.params('id'),
userName;
try {
userName = yield User.findOne(id).name;
}
catch (e) {
sails.log.error(e);
return res.serverError(e.message);
}
return res.ok(`Hi there ${userName}!`);
})
}
Works great. You just need to ensure any functions you call from your controllers return promises.
Put this code in your bootstrap.js, and every thing work like a charm!
var _ = require('lodash');
var coExpress = require('co-express');
sails.modules.loadControllers(function (err, modules) {
if (err) {
return callback(err);
}
sails.controllers = _.merge(sails.controllers, modules);
// hacking every action of all controllers
_.each(sails.controllers, function(controller, controllerId) {
_.each(controller, function(action, actionId) {
actionId = actionId.toLowerCase();
console.log('hacking route:', controllerId, actionId);
// co.wrap,generator => callback
action = coExpress(action);
sails.hooks.controllers.middleware[controllerId][actionId] = action;
});
});
// reload routes
sails.router.load(function () {
// reload blueprints
sails.hooks.blueprints.initialize(function () {
sails.hooks.blueprints.extendControllerMiddleware();
sails.hooks.blueprints.bindShadowRoutes();
callback();
});
});
});

Node.js promises with mongoskin

I'm trying to avoid using callbacks when making mongodb queries. I'm using mongoskin to make calls like so:
req.db.collection('users').find().toArray(function (err, doc) {
res.json(doc);
});
In many cases I need to make multiple queries so I want to use Node.js promise library but I'm not sure how to wrap these functions as promises. Most of the examples I see are trivial for things like readFile, I'm guessing in this case I would need to wrap toArray somehow? Can this be done or would have to be something implemented by mongoskin?
An example could be any set of callbacks, find/insert, find/find/insert, find/update:
req.db.collection('users').find().toArray(function (err, doc) {
if (doc) {
req.db.collection('users').find().toArray(function (err, doc) {
// etc...
});
}
else {
// err
}
});
You can promisify the entire module like so with bluebird:
var Promise = require("bluebird");
var mongoskin = require("mongoskin");
Object.keys(mongoskin).forEach(function(key) {
var value = mongoskin[key];
if (typeof value === "function") {
Promise.promisifyAll(value);
Promise.promisifyAll(value.prototype);
}
});
Promise.promisifyAll(mongoskin);
This only needs to be done in one place for one time in your application, not anywhere in your application code.
After that you just use methods normally except with the Async suffix and don't pass callbacks:
req.db.collection('users').find().toArrayAsync()
.then(function(doc) {
if (doc) {
return req.db.collection('users').find().toArrayAsync();
}
})
.then(function(doc) {
if (doc) {
return req.db.collection('users').find().toArrayAsync();
}
})
.then(function(doc) {
if (doc) {
return req.db.collection('users').find().toArrayAsync();
}
});
So again, if you call a function like
foo(a, b, c, function(err, result) {
if (err) return console.log(err);
//Code
});
The promise-returning version is called like:
fooAsync(a, b, c).then(...)
(Uncaught errors are automatically logged so you don't need to check for them if you are only going to log it)
Just stumbled here with the same question and didn't love "promisfying" mongoskin so did a bit more digging and found monk. It's built on top of mongoskin, tidies up the API and returns
promises for all async calls. Probably worth a peek to anyone else who lands here.
Esailija's answer may work, but its not super efficient since you have to run db.collection on every single db call. I don't know exactly how expensive that is, but looking at the code in mongoskin, its non-trivial. Not only that, but it's globally modifying prototypes, which isn't very safe.
The way I do this with fibers futures is:
wrap the collection methods for each collection
on receiving the result, for methods that return a Cursor wrap the toArray method, call it and return the resulting future (for methods that don't return a cursor, you don't need to do anything else).
use the future as normal
like this:
var Future = require("fibers/future")
// note: when i originally wrote this answer fibers/futures didn't have a good/intuitive wrapping function; but as of 2014-08-18, it does have one
function futureWrap() {
// function
if(arguments.length === 1) {
var fn = arguments[0]
var object = undefined
// object, methodName
} else {
var object = arguments[0]
var fn = object[arguments[1]]
}
return function() {
var args = Array.prototype.slice.call(arguments)
var future = new Future
args.push(future.resolver())
var me = this
if(object) me = object
fn.apply(me, args)
return future
}
}
var methodsYouWantToHave = ['findOne', 'find', 'update', 'insert', 'remove', 'findAndModify']
var methods = {}
methodsYouWantToHave.forEach(function(method) {
internalMethods[method] = futureWrap(this.collection, method)
}.bind(this))
// use them
var document = methods.findOne({_id: 'a3jf938fj98j'}, {}).wait()
var documents = futureWrap(methods.find({x: 'whatever'}, {}).wait(), 'toArray')().wait()
If you don't want to use fibers, I'd recommend using the async-future module, which has a good wrap function built in too.

mongoDB promise gets returned too early [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 6 years ago.
Improve this question
I just started trying to implement Promises on my Node.js app. Right now i'm checking if a user and password exists then using mongodb to look them up if user isn't found it sets promise.reject() but it is returning the promise too early and it's still in the pending state. If anyone can help or give me ideas on how to refactor it would be much appreciated.
https://gist.github.com/joshbedo/8957056
It is the expected behavior it returns the promise while it is still in the pending state.
You should use then() to check for promise resolution.
Another thing is that you should not use the traditional mongo interface when starting to work with promises, but rather promisify all method of the Collection prototype, so you can return the Promise created by the mongo find method with its chain of thenable. Otherwise I don't see the point of using promises. The way you do does not reduce the amount of code you have to write.
By the way, IMHO you should use bluebird rather than Q, as it is at the moment the only promise library which is not extremely slow.
Example :
in db.js
var mongoClient = require('mongodb').MongoClient;
var collection = require('mongodb').Collection;
var Promise = require('bluebird');
// We promisify everything. Bluebird add ***Async function which return promises.
Promise.promisifyAll(collection.prototype);
Promise.promisifyAll(mongoClient);
//In mongodb cursor is not built from protoype I use to promisify it each time. Not necessary if you don't use cursors.
collection.prototype._find = collection.prototype.find;
collection.prototype.find = function() {
var cursor = this._find.apply(this, arguments);
cursor.toArrayAsync = Promise.promisify(cursor.toArray, cursor);
return cursor;
};
//then you connect to the DB and exports your collections...
elsewhere, taking your example :
this.loginUser = function(user) {
var userid = user.userid,
var password = (user.password) ?
crypto.createHash("md5").update(user.password).digest("hex"):
undefined
var rememberme = user.rememberme;
if(userid && password) {
// we directly return the promise from mongodb, that we chain with a then
return db.users.findOneAsync({ email: userid, password: password }) // return a promise that we chain
.then(function(user) { // that we chain
if(user) {
var logincode = uuid.v4(),
var token = jwt.encode({email: userid, password: password}, secret);
if(rememberme) {
res.cookie("clogincode", logincode, { magAge: 900000 } );
}
return user; // return in a then callback will just resolve the promise
} else {
throw new Error('User not found'); // throwing in a then callback will just reject the promise
}
}); // end of the then, then return a promise that we return (with the return keyword above)
} else {
return Promise.reject("Username or Password was not entered"); // this is the only case where we have to actually create a promise ourself
}
}

Best place to handle data validation with mongoose and express

Maybe there is not a definitive answer here but I would like to know where to handle data validation when dealing with express.js and mongoose. Which of the following is the best practice (I currently use a combination and it's starting to feel very clumsy):
the Model (mongoose)
the Controller / Route (express)
Some older posts I have read are:
this;
this;
and, this;
but conflicting answers just add to the confusion. Maybe it simply isn't clear cut, in which case is one a better option?
When using mongoose I would push most of my validation logic to the mongoose model/schema. You can use mongoose-validator which is just a wrapper around node-validator for simple model validation. If you need validation against other models or more complex logic in the validation you can write your own custom mongoose pre validate or post validate hook (see mongoose middleware).
An additional benefit you gain when using mongoose to validate your model is that mongoose adds an error property to your model which can be accessed via model.errors[property]. This property can be used for validation error messages on the web or for a service client.
When writing more/very complex software tying the validation to the model may become a problem. But I'd deal with this problem when it arises. Since JavaScript has functions as first class citizens your validation functions still can be reused even in these complex situations.
The mongoose validator is a great place to start on a model level, but if you need to have controller specific validation, I use the following code in a utils.js file:
var async = require('async')
exports.validator = function (reqProps, props, mongoEnt, next) {
var propsErr = [];
var mongoErr = {};
async.parallel([function (cb) {
reqProps.forEach(function (rp) {
if (!props[rp])propsErr.push(rp);
})
cb()
}, function (cb) {
if (mongoEnt != null) {
var test = new mongoEnt(props);
test.validate(function (err) {
mongoErr = err;
cb();
});
} else {
mongoErr = null;
cb();
}
}], function (err, result) {
if (propsErr.length != 0) {
return next(new Error('The following props were not included: ' + propsErr));
} else if(mongoErr != null) {
return next(new Error('The following prop was not included: ' +mongoErr.errors[Object.keys(mongoErr.errors).pop()].path));
} else {
return next(null);
}
})
}
This allows me to both validate using the mongoose validator and check for the additional props that I include in the reqProps property in one line of code. Though this is only checking for required properties, you could easily extend it for your own validation scheme.
An example of usage for this code:
var Person = mongoose.model('Person')
exports.addUSCitizen = function(props, next){
utils.validator(['ssn'], props, Person, function (err) {
if(err) return next(err);
//do something using props.ssn
})
}

Resources