My NodeJS application hangs on the “find” query of the MongoDB when accessed by the 200 or more users simultaneously.
To demonstrate/reproduce the issue I have built following small POC which has one route and it connects to the MongoDB and fetches the data. When I create around 500 simultaneous request using LoadUIweb, sometimes, the “findone” function of collection never returns.
var express = require('express');
var mongodb = require('mongodb');
var config = require('../Config/Config');
var logger = require('../Config/LogManager');
var app = express();
var MONGODB_URI = config.PCMMongoDB;
var db;
var coll;
// Initialize connection once
mongodb.MongoClient.connect(MONGODB_URI, function(err, database) {
if(err) throw err;
db = database;
coll = db.collection('FacilitySettings');
app.listen(3000);
console.log('Listening on port 3000');
});
// Reuse database/collection object
app.get('/', function(req, res) {
logger.loggerinfo.info("Got the request");
//var result = new Array();
coll.find({}, function(err, docs) {
logger.loggerinfo.info("Got the Response", err);
docs.each(function(err, doc) {
if(err){
logger.loggerinfo.info("Got the error while iterating", err);
res.end();
}
if(doc) {
logger.loggerinfo.info("Iterated the Response");
res.write(JSON.stringify(doc) + "\n");
//result.push(doc);
}
else {
logger.loggerinfo.info("Returned the Response");
res.end();
}
});
});
});
The “FacilitySettings” collection has around 100 documents.
When the server hangs, it prints “Got the Request” and “Got the Response” but it never prints “Iterated the Response” and “returned the Response”. It doesn’t seem that the Nodes server itself is hanged because it accepts the new request. It seems that the result from the MongoDB is never returned.
When I check the last executed queries on the MongoDB, it seems that the query never reaches the MongoDB.
I am using following versions:
MongoDB: 3.2.3,
Nodes: 5.2.0,
MongoDB driver: 2.1.7
Has anyone encountered this issue before? What are the tools available, for MongoDB or for Nodes, to trace the cause? Who is the culprit here? Is it MongoDB or is it Node’s MongoDB native driver?
Posting this just incase anyone else comes across same issue.
I updated the MongoDB driver (version 2.1.13) and issue seems to be resolved now. Then I took a diif between the version I was using, v 2.1.7 and 2.1.13 and I found that there were few changes made to cursor class, noticeable around the exhaust condition. I am not sure if this is a relevant change or there is something else that has been fixed which resolved my case.
Related
I am running a quick little nodejs script to find documents in one collection and insert them into another collection but on the same DB. I came up with this guy, but it has no way to close because I think its running open or async?
I have tried placing the db.close() in various places and tried mongoClient.close(). No luck which had me thinking about trying to force a timeout for the async call. Added a connection Time out but it did not have the desired behaviour.
var MongoClient = require('mongodb').MongoClient
, assert = require('assert');
const async = require("async");
// Connection URL
var url = 'mongodb://localhost:27017/sourceDB';
// Use connect method to connect to the Server
MongoClient.connect(url,{connectTimeoutMS: "5"}, (err, db) => {
db.collection('source.collection', function(err, col) {
assert.equal(null, err);
col.find().forEach(function (data) {
console.log(data);
db.collection('destination.collection').insertOne(data, function(err, res) {
assert.equal(null, err);
});
console.log("Moved");
});
});
});
The script does well and picks up the collection and inserts, but the connection remains open.
It is not recommended to explicitly close the connection as shown by this SO thread.
Rather, allow the client library to manage the connection for you.
I have been working with the Aerospike Node.js client for a while now, and I've noticed that if I run the server for more than two hours, if I make a connection call to an Aerospike database I get the following error:
ERROR(23159) [connect.cc:69] [Connect] - Connecting to Cluster Failed
Before this happens, execution goes smoothly and without a hitch. I'm using Aerospike coupled with Express to store and retrieve user content. This is one of the endpoints I have written
app.get("/api/content/:id",function(req,res){
aero.client(aeroSec.generateConfig()).connect(function(err, db){
if(err.code != aero.status.AEROSPIKE_OK) return res.status(403).send();
var options = {};
options.filters = [filter.equal("secindex", req.params.id)];
var query = db.query("ns","posts",options);
var data;
var count = 0;
var s = query.execute();
s.on("data",function(rec){
data = rec;
});
s.on("error",function(err){
data = err;
});
s.on("end",function(){
db.close();
if(data == undefined) res.status(404).send("NOTHING FOUND");
else res.status(200).send(data);
});
});
});
In the interest of full disclosure, I am not querying an index that doesn't exist. A search in the Aerospike forums suggested that I comment out db.close() out of my endpoints, which only slows down the problem and doesn't eliminate it. The error started happening when I connected to a Aerospike database hosted on another server, so I decided to query a local server instead only to find the same error after two hours of normal operation.
Does anyone know what to do? I really like Aerospike as a database, but if these problems persist I will have no other choice but to go to another NoSQL database.
The code snippet shows that application creates as many client objects as number of requests received. The common usage is, a single client object is created and this is shared for all communications to Aerospike server. Another clarification is, client.connect() is a synchronous API call. And there is a sample web application in node.js using express and Aerospike. Here is the link. Please refer here for more clarifications.
The above code can be refactored like this
var aerospike = require('aerospike');
var express = require('express');
var app = express();
var aerospikeConfig = {
// add a node in the cluster.
hosts: [{addr: "192.168.0.1", port: 3000 }]
}
var client = aerospike.client(aerospikeConfig);
// client.connect() is a synchronous API call.
client.connect(function(err, client) {
if(err.code != aerospike.status.AEROSPIKE_OK) {
// application should exit or re-attempt the connect call.
// If connect call fails, connection with Aerospike server is
// is not established. Subsequent aerospike API calls will fail.
return err;
}
});
app.get("/api/content/:id",function(req,res){
var options = {};
options.filters = [filter.equal("secindex", req.params.id)];
var query = client.query("ns","posts",options);
var data;
var count = 0;
var s = query.execute();
s.on("data",function(rec){
data = rec;
});
s.on("error",function(err){
data = err;
});
s.on("end",function(){
client.close();
if(data == undefined) res.status(404).send("NOTHING FOUND");
else res.status(200).send(data);
});
});
Thanks
I'm attempting to deploy a very simple app to Heroku. The code for the application can be found on Ray Wenderlich's site, here: http://www.raywenderlich.com/61078/write-simple-node-jsmongodb-web-service-ios-app I keep getting the same error whenever I try to have Heroku compile the code...
var mongoHost = "mongodb://username:password#ds041140.mongolab.com:41140/heroku_app23491233";
var mongoPort = 41140;
var collectionDriver;
var mongoClient = new MongoClient(new Server(mongoHost, mongoPort)); //B
mongoClient.open(function(err, mongoClient) { //C
if (!mongoClient) {
console.error("Error! Exiting... Must start MongoDB first");
process.exit(1); //D
}
var db = mongoClient.db("heroku_app23491233"); // E
collectionDriver = new CollectionDriver(db); //F
});
When I type heroku logs, the error I get comes from if (!mongoClient) above...
app[web.1]: Error! Exiting... Must start MongoDB first
I'm sure the problem lies somewhere in my attempt to connect to the MongoLab database. I've copied the URI from MongoLab and I've created a user with the proper credentials.
I can connect to localhost just fine with very similar code, so I'm not sure what is going wrong in this example.
Thank you.
Based on the docs, my best guess is that it's because the Server constructor expects the first argument to contain only the host name (in the case of your MongoLab, ds041140.mongolab.com). However, I think you can pass your connection string into MongoClient.connect:
// Make sure to replace username and password with the proper values.
var mongoHost = "mongodb://username:password#ds041140.mongolab.com:41140/heroku_app23491233";
MongoClient.connect(mongoHost, function(err, db) {
if (err) return console.log(err);
db.collection('mongoclient_test').update({a:1}, {b:1}, {upsert:true}, function(err, result) {
db.close();
if (err) return console.log(err);
console.log('Okay', result);
});
});
Documentation Page For MongoClient
Hopefully that helps!
I'm using node js, express and postgresql as backend.
This is the approach I used to make a rest API:
exports.schema = function (inputs, res) {
var query = knex('schema')
.orderBy('sch_title', 'asc')
.select();
query.exec(function (err, schemas) {
if(err){
var response = {
message: 'Something went wrong when trying to fetch schemas',
thrownErr: err
};
console.error(response);
res.send(500, response);
}
if(schemas.length === 0){
var message = 'No schemas was found';
console.error(message);
res.send(400, message);
return;
}
res.send(200, schemas);
});
};
It works but after a while postgres logs an error and it's no longer working:
sorry, too man clients already
Do I need a close each request somehow? Could not find any about this in the express docs. What can be wrong?
This error only occurs on production server. Not on developing machine.
Update
The app only brakes in one 'module'. The rest of the app works fine. So it's only some queries that gives the error.
Just keep one connection open for your whole app. The docs shows an example how to do this.
This code goes in your app.js...
var Knex = require('knex');
Knex.knex = Knex.initialize({
client: 'pg',
connection: {
// your connection config
}
});
And when you want to query in your controllers/middlewares...
var knex = require('knex').knex;
exports.schema = function (req, res) {
var query = knex('schema')
.orderBy('sch_title', 'asc')
.select();
// more code...
};
If you place Knex.initialize inside an app.use or app.VERB, it gets called repeatedly for each request thus you'll end up connecting to PG multiple times.
For most cases, you don't need to do an open+query+close for every HTTP request.
I've been looking for a way to do various operations on a mongo database, depending on which route a user connects to on my website. So doing a html-post to www.mysite.com/data would add info to a mongo DB, and doing a html-get at the same url would get data from the same database. I managed to solve that, but everytime I turn on my server with the website I get 5 connections registred at the mongo database. Why is this, and is it bad?
My code:
I'm runing this code in mongo.js:
var mongodb = require('mongodb');
module.exports.init = function (callback) {
var server = new mongodb.Server("127.0.0.1", 27017, {});
new mongodb.Db('test', server, {w: 1}).open(function (error, client) {
//export the client and maybe some collections as a shortcut
module.exports.client = client;
module.exports.myCollection = new mongodb.Collection(client, 'myCollection');
callback(error);
});
};
I initialize everything running (app.js):
/express set-up/
var mongo = require('./mongo.js');
/.../
mongo.init(function (error) {
if (error)
throw error;
app.listen(3000, function(){
console.log("Express server listening on port %d in %s mode",3000,
app.settings.env);
});
});
And, for example, the post looks like (still app.js):
app.post('/App', function(req, res) {
users = users.concat(req.body);
res.redirect('/App');
//Add user to MongoDB
mongo.myCollection.insert({name: req.body.name, car: req.body.car, website: req.body.website, phone: req.body.phone}, {safe:true}, function(err, objects) {
if (err)
console.warn(err.message);
});
Pretty sure my redirect isn't working as I want it too here, but that's another issue.
Anny suggestions on why I get five connects every time I start the server?
The five connections are because that is the default poolSize (5). You can adjust it, as outlined in the server options docs - it also mentions the default.