I'm currently running a stack that consists of Express and MongoClient with Mocha and Chai for testing. I'm working on writing test cases for my endpoint and am getting a random error that pops up from time to time. Below is a snippet of one of the suits I'm writing:
describe('Recipes with populated database', () => {
before((done) => {
var recipe1 = {"search_name": "mikes_mac_and_cheese", "text_friendly_name": "Mikes Mac and Cheese","ingredients": [{"name": "elbow_noodles","text_friendly_name": "elbow noodles","quantity": 12,"measurement": "oz"},{"name": "cheddar_cheese","text_friendly_name": "cheddar cheese","quantity": 6,"measurement": "oz"},{"name": "gouda_cheese","text_friendly_name": "gouda cheese","quantity": 6,"measurement": "oz"},{"name": "milk","text_friendly_name": "milk","quantity": 2,"measurement": "oz"}],"steps": ["Bring water to a boil","Cook noodels until al dente.","Add the milk and cheeses and melt down.","Stir constantly to ensure even coating and serve."],"course": ["dinner","lunch","side"],"prep_time": {"minutes": 15,"hours": 0},"cook_time":{"minutes": 25,"hours": 1},"cuisine": "italian","submitted_by": "User1","searchable": true};
db.collectionExists('recipes').then((exists) => {
if (exists) {
db.getDb().dropCollection('recipes', (err, results) => {
if (err)
{
throw err;
}
});
}
db.getDb().createCollection('recipes', (err, results) => {
if (err)
{
throw err;
}
});
db.getDb().collection('recipes').insertOne(recipe1, (err, result) => {
done();
});
});
});
The collectionExists() method simply takes in a name and returns a promise that is resolved to a true/false value. I've already done some debugging and it is working just fine. Where I am getting a problem is when I hit the section of the code where I call createCollection. I get an error about how the collection already exists thus leading to my tests failing. This appears to be happening on every third time I'm running my tests as well.
The purpose of all this is to ensure that my database collection called recipes is completely empty before I start testing so I'm not stuck with old data or in an uncontrolled environment.
You have a race condition between .createCollection and .insertOne. In other words, they start at the same time and go in parallel. There is no way to tell which will be done first.
The way .insert works in MongoDB is that if the collection is missing and you try inserting - it's going to create a collection. So if .insertOne is executed first - the collection is created and that is why you're getting the already exists error in an attempt to createCollection.
Due to the async nature of DB calls you'd have to place the subsequent calls inside the callback of a prev. one. This way there will be no parallel execution:
before((done) => {
var recipe1 = {/**/};
db.collectionExists('recipes')
.then((exists) => {
if (exists) {
// Drop the collection if it exists.
db.getDb().dropCollection('recipes', (err) => {
if (err) {
// If there's an error we want to pass it up to before.
done(err);
return;
}
// Just insert a first document into a non-existent collection.
// It's going to be created.
// Notice the done callback.
db.getDb().collection('recipes').insertOne(recipe1, done);
});
}
// If there were no such collection - simply insert the first doc to create it.
// Note that I'm passing before's done callback inside.
db.getDb().collection('recipes').insertOne(recipe1, done);
})
// We don't want to lose the error from this promise always.
.catch(err => done(err));
});
But. Actually, there is no need to drop and re-create a collection each time you run the tests. You can simply .remove all the objects in the before block. So probably the right solution would be:
before((done) => {
var recipe1 = {/**/};
const recipes = db.getDb().collection('recipes');
// Simply wipe out the data
recipes.remove({}, err => {
if (err) {
done(err);
return;
}
recipes.insertOne(recipe1, done);
});
});
Related
I have an quite simple application the idea is that someone has unique code which value are stored in one mongo collection in other we are keeping some data which we need to return if the key was found in first collection.
As probably you have noticed I'm using NodeJS with MongoDB and Mongoose, Express.
I have a problem with method bellow:
exports.getCompanyByKey = function(req, res) {
console.log(req.params.keyvalue);
var query = Company.where({keyValue : req.params.keyvalue});
query.findOne(function(err, company){
if(err){
res.send(err);
}else{
SampleData.findOne({}, function(err, sample_data){
if(err)
res.send(err);
res.json(sample_data);
});
}
});
};
The problem is that it will always return the data beause it's not throwing an error but empty array - so is there any other good and proper way as it should be don to throw 404 error without statement such as if(length<0) res.status(404).send('Error message).
I simply want to minimalize amount of if statements.
Maybe there is some other way to write implementation od error handling for mongoose which in general instead returning empty array will give us error code with message?
It's not exactly clear what you're asking, but if you want to make an error condition out of something that is not normally an error, then an if statement (or some other test like that) is required to test for that specific condition.
You could make your own function for querying that turns an empty response into an error and you could "hide" the if condition in that function if you want, but it's still an if condition that tests for your specific condition.
So, to return a 404 if the array is empty, you would just add an if statement (as you already appear to know):
exports.getCompanyByKey = function(req, res) {
console.log(req.params.keyvalue);
var query = Company.where({keyValue : req.params.keyvalue});
query.findOne(function(err, company){
if(err){
res.status(500).send(err);
} else {
SampleData.findOne({}, function(err, sample_data){
if(err) {
res.status(500).send(err);
} else {
if (sample_data.length) {
res.json(sample_data);
} else {
res.status(404).send("no data");
}
}
});
}
});
};
FYI, you also need to make sure you are properly setting a status code when there's an error and that you are never sending multiple responses to the same request (even when there's an error). I've also fixed several cases of those issues in your code.
This could likely be written cleaner and responses consolidated by using the promise interface to your database and send an error in one .catch().
For example, you could simplify your code by creating a utility function for .findOne() that detects and sends an error response automatically:
function findOne(res, db, q, cb) {
db.findOne(q, function(err, data) {
if (err) {
res.status(500).send(err);
cb(err);
} else if (!q.length) {
res.status(404).send("no data");
cb(new Error("no data"));
} else {
cb(null, data);
}
});
}
Then, your function could be simplified to this:
exports.getCompanyByKey = function(req, res) {
var query = Company.where({keyValue : req.params.keyvalue});
query.findOne(function(err, company){
if(err){
res.status(500).send(err);
} else {
findOne(res, SampleData, {}, function(err, sample_data) {
// any error response has already been sent
if (!err) {
res.json(sample_data);
}
});
}
});
};
Again, this would be better to use your Db's promise interface.
I need to fetch two different MongoDB collections (db.stats and db.tables ) for the same request req.
Now, in the code below, I am nesting the queries within the callback function.
router.post('/', (req, res) => {
let season = String(req.body.year);
let resultData, resultTable;
db.stats.findOne({Year: season}, function (err, data) {
if (data) {
resultData = getResult(data);
db.tables.findOne({Year: season}, function (err, data) {
if (data) {
resultTable = getTable(data);
res.render('index.html', {
data:{
result : resultData,
message: "Working"}
});
} else {
console.log("Error in Tables");
}
});
} else {
console.log("Error in Stats");
}
});
});
This code works, but there a few things that don't seem right. So my question is:
How do I avoid this nested structure? Because it not only looks ugly but also, while I am processing these requests the client side is unresponsive and that is bad.
What you have right now is known as the callback hell in JavaScript. This is where Promises comes in handy.
Here's what you can do:
router.post('/', (req, res) => {
let season = String(req.body.year);
var queries = [
db.stats.findOne({ Year: season }),
db.tables.findOne({ Year: season })
];
Promise.all(queries)
.then(results => {
if (!results[0]) {
console.log("Error in Stats");
return; // bad response. a better way is to return status 500 here
} else if (!results[1]) {
console.log("Error in Tables");
return; // bad response. a better way is to return status 500 here
}
let resultData = getResult(results[0]);
let resultTable = getTable(results[1]);
res.render('index.html', { data: {
result : resultData,
message: "Working"
} });
})
.catch(err => {
console.log("Error in getting queries", err);
// bad response. a better way is to return status 500 here
});
});
It looks like you are using Mongoose as your ODM to access your mongo database. When you don't pass in a function as the second parameter, the value returned by the function call (e.g. db.stats.findOne({ Year: season })) will be a Promise. We will put all of these unresolved Promises in an array and call Promise.all to resolve them. By using Promise.all, you are waiting until all of your database queries get executed before moving on to render your index.html view. In this case, the results of your database function calls will be stored in the results array in the order of your queries array.
Also, I would recommend doing something like res.status(500).send("A descriptive error message here") whenever there is an error on the server side in addition to the console.log calls.
The above will solve your nested structure problem, but latter problem will still be there (i.e. client side is unresponsive when processing these requests). In order to solve this, you need to first identify your bottleneck. What function calls are taking up most of the time? Since you are using findOne, I do not think that will be the bottleneck unless the connection between your server and the database has latency issues.
I am going to assume that the POST request is not done through AJAX since you have res.render in it, so this problem shouldn't be caused by any client-sided code. I suspect that either one of getResult or getTable (or both) is taking up quite a significant amount of time, considering the fact that it causes the client side to be unresponsive. What's the size of the data when you query your database? If the size of it is so huge that it takes a significant amount of time to process, I would recommend changing the way how the request is made. You can use AJAX on the front-end to make a POST request to the back-end, which will then return the response as a JSON object. That way, the page on the browser would not need to reload, and you'll get a better user experience.
mongodb driver return a promise if you dont send a callback so you can use async await
router.post('/', async(req, res) => {
let season = String(req.body.year);
let resultData, resultTable;
try {
const [data1,data2] = await Promise.all([
db.stats.findOne({Year: season}),
db.tables.findOne({Year: season})
]);
if (data1 && data2) {
resultData = getResult(data1);
resultTable = getTable(data2);
return res.render('index.html', {
data: {
result: resultData,
message: "Working"
}
});
}
res.send('error');
console.log("Error");
} catch (err) {
res.send('error');
console.log("Error");
}
});
Accordingly with nodejs sqlite library (https://github.com/mapbox/node-sqlite3) you can serialize a sql transaction with db.serialize function. All executions in this function will be executed in the specified order.
But with a for loop how can i get the information about the transaction will be success or fail?
function transaction(callback) {
db.serialize(function() {
db.run("BEGIN");
for(...) {
db.run("INSERT", function(err) {
//this code is not serialized with the parent serialize function (this is my problem)
//if one run throw error i must resolve the callback with the error code
if (err)
callback("error");
});
}
db.run("END");
callback("ok");
}
}
Before start transaction you can define some vars e.g. var results = [];. On complete each query inside callback you can collect error or null as results.push(err) and check results.length. When results.length equals to length of for then all queries executed and now check results.every((i) => i === null) value. If true then do commit else rollback.
Much better to control flow is use async module.
'use strict'
var async = require('async');
...
db.run('begin', function(err){
if (err)
return console.log(err);
let run = (sql, callback) => db.run(sql, callback);
async.eachSeries(sqls, run, function(err){
db.run((err) ? 'rollback' : 'commit', ...)
})
})
If you add begin to sqls then code will be more simple.
Let's say I have some sort of game. I have a buyItem function like this:
buyItem: function (req, res) {
// query the users balance
// deduct user balance
// buy the item
}
If I spam that route until the user balance is deducted (the 2nd query) the user's balance is still positive.
What I have tried:
buyItem: function (req, res) {
if(req.session.user.busy) return false;
req.session.user.busy = true;
// query the users balance
// deduct user balance
// buy the item
}
The problem is req.session.user.busy will be undefined for the first ~5 requests. So that doesn't work either.
How do we handle such situations? I'm using the Sails.JS framework if that is important.
Update 2
Sails 1.0 now has full transaction support, via the .getDatastore() method. Example:
// Get a reference to the default datastore, and start a transaction.
await sails.getDatastore().transaction(async (db, proceed)=> {
// Now that we have a connection instance in `db`, pass it to Waterline
// methods using `.usingConnection()` to make them part of the transaction:
await BankAccount.update({ balance: 5000 }).usingConnection(db);
// If an error is thrown, the transaction will be rolled back.
// Or, you can catch errors yourself and call `proceed(err)`.
// To commit the transaction, call `proceed()`
return proceed();
// You can also return a result with `proceed(null, result)`.
});
Update
As several commenters have noted, the code below doesn't work when connection pooling is enabled. At the time that this was originally posted, not all of the adapters pooled by default, but at this point it should be assumed that they do, so that each individual method call (.query(), .findOne(), etc.) could be on a different connection, and operating outside of the transaction. The next major version of Waterline will have transaction support, but until then, the only way to ensure that your queries are transactional is to use the raw database driver package (e.g. pg or mysql).
It sounds like what you need is a transaction. Sails doesn't support transactions at the framework level yet (it's on the roadmap) but if you're using a database that supports them (like Postgres or MySQL), you can use the .query() method of your model to access the underlying adapter and run native commands. Here's an example:
buyItem: function(req, res) {
try {
// Start the transaction
User.query("BEGIN", function(err) {
if (err) {throw new Error(err);}
// Find the user
User.findOne(req.param("userId").exec(function(err, user) {
if (err) {throw new Error(err);}
// Update the user balance
user.balance = user.balance - req.param("itemCost");
// Save the user
user.save(function(err) {
if (err) {throw new Error(err);}
// Commit the transaction
User.query("COMMIT", function(err) {
if (err) {throw new Error(err);}
// Display the updated user
res.json(user);
});
});
});
});
}
// If there are any problems, roll back the transaction
catch(e) {
User.query("ROLLBACK", function(err) {
// The rollback failed--Catastrophic error!
if (err) {return res.serverError(err);}
// Return the error that resulted in the rollback
return res.serverError(e);
});
}
}
I haven't tested this out. But as long as your not using multiple instances or clusters, you should just be able to store the status in memory. Because node is single threaded there shouldn't be any problems with atomicity.
var inProgress = {};
function buyItem(req, res) {
if (inProgress[req.session.user.id]) {
// send error response
return;
}
inProgress[req.session.user.id] = true;
// or whatever the function is..
req.session.user.subtractBalance(10.00, function(err, success) {
delete inProgress[req.session.user.id];
// send success response
});
}
I am little bit confused with my code it's not worked synchronusly as it's should be. I use everyauth to do authentication.
registerUser(function(newUserAttrs) {
var login = newUserAttrs[this.loginKey()];
user.CreateNewUser(newUserAttrs.login, newUserAttrs.password, newUserAttrs.email, function(res, err) {
if(!err) {
return usersByLogin[login] = newUserAttrs;
}
else {
throw err;
}
});
})
in another file I have write this code
exports.CreateNewUser = function(login, pass, mail, callback) {
var sql = "insert into `user`(login,mail,pass) values(?,?,?)";
client.query(sql, [login, mail, pass], function(err, results, fields) {
if(!err) {
callback(results);
console.log('test')
}
else {
callback(results, err);
}
});
};
This code are working fine. I have tested him. the only problem is they are working synchronosly (as normal). Can someone explain me what thing I have done in wrong way that make it async. I want to get it done in sync way.
The current code give me error (it's make a entry in database and produce error on browser)
Error: Step registerUser of `password` is promising: userOrErrors ; however, the step returns nothing. Fix the step by returning the expected values OR by returning a Promise that promises said values.
at Object.exec (E:\proj\Node\node_modules\everyauth\lib\step.js:68:11)
at E:\proj\Node\node_modules\everyauth\lib\stepSequence.js:26:38
at [object Object].callback (E:\proj\Node\node_modules\everyauth\lib\promise.js:13:12)
at RouteTriggeredSequence._bind (E:\proj\Node\node_modules\everyauth\lib\stepSequence.js:25:20)
at RouteTriggeredSequence.start (E:\proj\Node\node_modules\everyauth\lib\stepSequence.js:52:33)
at RouteTriggeredSequence.routeHandler (E:\proj\Node\node_modules\everyauth\lib\routeTriggeredSequence.js:13:13)
at Object.<anonymous> (native)
at nextMiddleware (E:\proj\Node\node_modules\connect\lib\middleware\router.js:175:25)
at param (E:\proj\Node\node_modules\connect\lib\middleware\router.js:183:16)
at pass (E:\proj\Node\node_modules\connect\lib\middleware\router.js:191:10)
Thanks
The two pieces of code you present are asynchronous and not synchronous!
With everyauth, to be able to handle asynchronous user creation you should use a Promise. So your code will be something like :
registerUser(function(newUserAttrs) {
var promise = this.Promise();
var login = newUserAttrs[this.loginKey()];
user.CreateNewUser(newUserAttrs.login, newUserAttrs.password, newUserAttrs.email, function(res, err) {
if(!err) {
return promise.fulfill(newUserAttrs);
}
else {
promise.fulfill(user);
}
});
})
Without promise you couldn't be sure that your new user has been added in your database. But if it doesn't matter you could have something like that:
registerUser(function(newUserAttrs) {
var login = newUserAttrs[this.loginKey()];
user.CreateNewUser(newUserAttrs.login, newUserAttrs.password, newUserAttrs.email, function(res, err) {
if (err) console.log(err);
});
return newUserAttrs;
})
Because you are doing a database query, this code has to be asynchronous. The anonymous function you pass to client.query will not be called until the database query is complete, so your callback gets called asynchronously.
You will need to treat this all as asynchronous, so for instance you'll have to trigger some other callback instead of returning the user object/throwing.