I have a script that I use to save all my models (for ex to reindex). I am getting socket closing errors after a couple of hundred saves (Ihave 900 total). I recently upgraded to Mongoose 4.2.3 from 3.x.x and started seeing these errors. I am not sure what else to go on.
Errors:
{ [MongoError: server ds0133252-a0.mongolab.com:133252 sockets closed]
name: 'MongoError',
message: 'server ds051252-a0.mongolab.com:51252 sockets closed' }
{ [MongoError: server ds0133252-a0.mongolab.com:133252 sockets closed]
name: 'MongoError',
....
The script is pretty basic:
var mongoose = require(mongoose),
Product = require('../models/product'),
config = require('config');
mongoose.connect(config.db.mongo.connection, config.db.mongo.options);
Product.find(function(err, products) {
products.forEach(function(p) {
p.markModified('description');
p.save(function(e, product) {
if(e) console.log(e);
console.log(product.id);
});
});
});
The model is pretty complex but hasn't changed in a while. I have disabled the "save" middleware with same errors so it should be pretty standard.
Suggestions?
You can use an async flow control library like async to use an async iterator that lets you limit the number of concurrent save operations.
In this case, async.eachLimit would be a good fit (doc link is to each, scroll down to see the eachLimit variant). For example, to limit the iteration to no more than 5 concurrent saves:
Product.find(function(err, products) {
async.eachLimit(products, 5, function(p, callback) {
p.markModified('description');
p.save(function(e, product) {
if(e) console.log(e);
console.log(product.id);
callback(err);
});
});
});
Note that the callback parameter of eachLimit must be called when the save completes so that the library knows that particular iteration is complete.
Related
When I start my server, an error was shown like this:
Error in DB connection : Error: queryTxt ETIMEOUT yky-api-ajgvf.mongodb.net
I tried many times to work properly in order to fix this problem. I couldn't even deploy my project on heroku properly yet. I love nodejs so much but now I just want to give up continuously learning Nodejs for now.
Someone help me to solve my problem if possible. I'm the one who is learning Node.Js. It's been around 4 or 5 months. Before I decided to leran PHP, but now I'm still trying to learn nodejs.
I'm a newbie. :)
This is my config database code in my small project.
const mongoose = require('mongoose')
const log = console.log
mongoose.connect('mongodb+srv://chanlay:chanlay123#yky-api-ajgvf.mongodb.net/yky-blog-api?retryWrites=true&w=majority', {
useNewUrlParser: true
}, (err) => {
if (!err) {
log('MongoDB Connection Succeeded.')
} else {
log('Error in DB connection : ' + err)
}
});
require('../models/posts')
mongoose.connect() can look like this (docs reference):
mongoose.connect(uri, options, function(error) {
// Check error in initial connection. There is no 2nd param to the callback.
});
// Or using promises
mongoose.connect(uri, options).then(
() => { /** ready to use. The `mongoose.connect()` promise resolves to mongoose instance. */ },
err => { /** handle initial connection error */ }
);
Before demonstrating further, first of all, this block of yours is wrong:
if (!err) {
log('MongoDB Connection Succeeded.')
}
error handling doesn't work that way, this condition will never happen, as when there's no error, there is no reason to call error callback function which checks this condition.
Instead use .then() block, as a successful connection returns a Promise:
mongoose.connect(uri, options).then(
() => { console.log('MongoDB Connection Succeeded.') },
err => { console.log(err) }
);
// OR use .catch() after .then()
mongoose.connect(uri, options).then(() => console.log('MongoDB Connection Succeeded.')).catch( err => console.log(err))
Your answer: set socketTimeoutMS to 60000 or 90000.
If still didn't work, add connectTimeoutMS=90000 and reconnectTries=1000 to options too.
Declare a new variable named options (the name is arbitrary, but better follow the convention), and you put your useNewUrlParser: true also there (reference):
const options = {
useNewUrlParser: true,
socketTimeoutMS: 60000 //or 90000
}
Tip: You can also specify driver options in your connection string as parameters in the query string portion of the URI (reference):
{main-url}?socketTimeoutMS=90000, that is:
const uri = mongodb+srv://chanlay:chanlay123#yky-api-ajgvf.mongodb.net/yky-blog-api?retryWrites=true&w=majority?socketTimeoutMS=90000
but stick with options variable as it's cleaner, more readable, and easier to maintain (otherwise you would put everything in a string which makes it harder for you to read and debug).
Please let me know by leaving to my answer a comment either as a feedback or reporting the persistent error.
My stack is node, express and the pg module. I really try to understand by the documentation and some outdated tutorials. I dont know when and how to disconnect and to end a client.
For some routes I decided to use a pool. This is my code
const pool = new pg.Pool({
user: 'pooluser',host: 'localhost',database: 'mydb',password: 'pooluser',port: 5432});
pool.on('error', (err, client) => {
console.log('error ', err); process.exit(-1);
});
app.get('/', (req, res)=>{
pool.connect()
.then(client => {
return client.query('select ....')
.then(resolved => {
client.release();
console.log(resolved.rows);
})
.catch(e => {
client.release();
console.log('error', e);
})
pool.end();
})
});
In the routes of the CMS, I use client instead of pool that has different db privileges than the pool.
const client = new pg.Client({
user: 'clientuser',host: 'localhost',database: 'mydb',password: 'clientuser',port: 5432});
client.connect();
const signup = (user) => {
return new Promise((resolved, rejeted)=>{
getUser(user.email)
.then(getUserRes => {
if (!getUserRes) {
return resolved(false);
}
client.query('insert into user(username, password) values ($1,$2)',[user.username,user.password])
.then(queryRes => {
client.end();
resolved(true);
})
.catch(queryError => {
client.end();
rejeted('username already used');
});
})
.catch(getUserError => {
return rejeted('error');
});
})
};
const getUser = (username) => {
return new Promise((resolved, rejeted)=>{
client.query('select username from user WHERE username= $1',[username])
.then(res => {
client.end();
if (res.rows.length == 0) {
return resolved(true);
}
resolved(false);
})
.catch(e => {
client.end();
console.error('error ', e);
});
})
}
In this case if I get a username already used and try to re-post with another username, the query of the getUser never starts and the page hangs. If I remove the client.end(); from both functions, it will work.
I am confused, so please advice on how and when to disconnect and to completely end a pool or a client. Any hint or explanation or tutorial will be appreciated.
Thank you
First, from the pg documentation*:
const { Pool } = require('pg')
const pool = new Pool()
// the pool with emit an error on behalf of any idle clients
// it contains if a backend error or network partition happens
pool.on('error', (err, client) => {
console.error('Unexpected error on idle client', err) // your callback here
process.exit(-1)
})
// promise - checkout a client
pool.connect()
.then(client => {
return client.query('SELECT * FROM users WHERE id = $1', [1]) // your query string here
.then(res => {
client.release()
console.log(res.rows[0]) // your callback here
})
.catch(e => {
client.release()
console.log(err.stack) // your callback here
})
})
This code/construct is suficient/made to get your pool working, providing the your thing here things. If you shut down your application, the connection will hang normaly, since the pool is created well, exactly not to hang, even if it does provides a manual way of hanging,
see last section of the article.
Also look at the previous red section which says "You must always return the client..." to accept
the mandatory client.release() instruction
before accesing argument.
you scope/closure client within your callbacks.
Then, from the pg.client documentation*:
Plain text query with a promise
const { Client } = require('pg').Client
const client = new Client()
client.connect()
client.query('SELECT NOW()') // your query string here
.then(result => console.log(result)) // your callback here
.catch(e => console.error(e.stack)) // your callback here
.then(() => client.end())
seems to me the clearest syntax:
you end the client whatever the results.
you access the result before ending the client.
you don´t scope/closure the client within your callbacks
It is this sort of oposition between the two syntaxes that may be confusing at first sight, but there is no magic in there, it is implementation construction syntax.
Focus on your callbacks and queries, not on those constructs, just pick up the most elegant for your eyes and feed it with your code.
*I added the comments // your xxx here for clarity
You shouldn't disconnect the pool on every query, connection pool is supposed to be used to have "hot" connections.
I usually have a global connection on startup and the pool connection close on (if) application stop; you just have to release the connection from pool every time the query ends, as you already do, and use the same pool also in the signup function.
Sometimes I need to preserve connections, I use a wrapper to the query function that checks if the connection is active or not before perform the query, but it's just an optimization.
In case you don't want to manage open/close connections/pool or release, you could try https://github.com/vitaly-t/pg-promise, it manage all that stuff silently and it works well.
The documentation over node-postgres's github says:
pro tip: unless you need to run a transaction (which requires a single client for multiple queries) or you have some other edge case like streaming rows or using a cursor you should almost always just use pool.query. Its easy, it does the right thing ™️, and wont ever forget to return clients back to the pool after the query is done.
So for non-transactional query, calling below code is enough.
var pool = new Pool()
pool.query('select username from user WHERE username= $1',[username], function(err, res) {
console.log(res.rows[0].username)
})
By using pool.query, the library will take care of releasing the client after the query is done.
Its quite simple, a client-connection (single connection) opens up, query with it, once you are done you end it.
The pool concept is different, in the case of mysql : you have to .release() the connection back to the pool once you are done with it, but it seems that with pg is a different story:
From an issue on the github repo : Cannot use a pool after calling end on the pool #1635
"Cannot use a pool after calling end on the pool"
You can't reuse a pool after it has been closed (i.e. after calling
the .end() function). You would need to recreate the pool and discard
the old one.
The simplest way to deal with pooling in a Lambda is to not do it at
all. Have your database interactions create their own connections and
close them when they're done. You can't maintain a pool across
freeze/thaw cycles anyway as the underlying TCP sockets would be
closed.
If opening/closing the connections becomes a performance issue then
look into setting up an external pool like pgbouncer.
So I would say that your best option is to not end the pool, unless you are shutting down the server
I'm looking for a solution to waiting for an event to happen before sending a HTTP response.
Use Case
The idea is I call a function in one of my routes: zwave.connect("/dev/ttyACM5"); This function return immediately.
But there exists 2 events that notice about if it succeed or fail to connect the device:
zwave.on('driver ready', function(){...});
zwave.on('driver failed', function(){...});
In my route, I would like to know if the device succeed or fail to connect before sending the HTTP response.
My "solution"
When an event happen, I save the event in a database:
zwave.on('driver ready', function(){
//In the database, save the fact the event happened, here it's event "CONNECTED"
});
In my route, execute the connect function and wait for the event to
appear in the database:
router.get('/', function(request, response, next) {
zwave.connect("/dev/ttyACM5");
waitForEvent("CONNECTED", 5, null, function(){
response.redirect(/connected);
});
});
// The function use to wait for the event
waitForEvent: function(eventType, nbCallMax, nbCall, callback){
if(nbCall == null) nbCall = 1;
if(nbCallMax == null) nbCallMax = 1;
// Looking for event to happen (return true if event happened, false otherwise
event = findEventInDataBase(eventType);
if(event){
waitForEvent(eventType, nbCallMax, nbCall, callback);
}else{
setTimeout(waitForEvent(eventType, callback, nbCallMax, (nbCall+1)), 1500);
}
}
I don't think it is a good practice because it iterates calls over the database.
So what are your opinions/suggestions about it?
I've gone ahead and added the asynchronous and control-flow tags to your question because at the core of it, that is what you're asking about. (As an aside, if you're not using ES6 you should be able to translate the code below back to ES5.)
TL;DR
There are a lot of ways to handle async control flow in JavaScript (see also: What is the best control flow module for node.js?). You are looking for a structured way to handle it—likely Promises or the Reactive Extensions for JavaScript (a.k.a RxJS).
Example using a Promise
From MDN:
The Promise object is used for asynchronous computations. A Promise represents a value which may be available now, or in the future, or never.
The async computation in your case is the computation of a boolean value describing the success or failure to connect to the device. To do so, you can wrap the call to connect in a Promise object like so:
const p = new Promise((resolve) => {
// This assumes that the events are mutually exclusive
zwave.connect('/dev/ttyACM5');
zwave.on('driver ready', () => resolve(true));
zwave.on('driver failed', () => resolve(false));
});
Once you have a Promise representing the state of the connection, you can attach functions to its "future" value:
// Inside your route file
const p = /* ... */;
router.get('/', function(request, response, next) {
p.then(successful => {
if (successful) {
response.redirect('/connected');
}
else {
response.redirect('/failure');
}
});
});
You can learn more about Promises on MDN, or by reading one of many other resources on the topic (e.g. You're Missing the Point of Promises).
Have you tried this? From the look of it, your zwave probably have already implemented an EventEmmiter, you just need to attach a listener to it
router.get('/', function(request, response, next) {
zwave.connect("/dev/ttyACM5");
zwave.once('driver ready', function(){
response.redirect(/connected);
});
});
There is a npm sync module also. which is used for synchronize the process of executing the query.
When you want to run parallel queries in synchronous way then node restrict to do that because it never wait for response. and sync module is much perfect for that kind of solution.
Sample code
/*require sync module*/
var Sync = require('sync');
app.get('/',function(req,res,next){
story.find().exec(function(err,data){
var sync_function_data = find_user.sync(null, {name: "sanjeev"});
res.send({story:data,user:sync_function_data});
});
});
/*****sync function defined here *******/
function find_user(req_json, callback) {
process.nextTick(function () {
users.find(req_json,function (err,data)
{
if (!err) {
callback(null, data);
} else {
callback(null, err);
}
});
});
}
reference link: https://www.npmjs.com/package/sync
I am using Promised-Mongo to connect MongoDB with Promises from NodeJS backend code. It worked fine, until I enabled MongoDB's client access control. When I run this code, I get "could not authenticate" message":
var pmongo = require('promised-mongo').compatible();
var db = pmongo('myusername:mypassword#localhost/mydb', ['candidates']);
db.candidates.save(req.body)
.then(function () {
// never reached here
})
.catch(function (e) {
// it reached here, where e.message says "could not authenticate"
});
Pure MongoDB code (i.e. no Promises...) works fine:
var mongodb = require('mongodb');
var uri = 'mongodb://myusername:mypassword#localhost/mydb';
mongodb.MongoClient.connect(uri, function (err, db) {
if (err) {
// never reached here
}
var candidates = db.collection('candidates');
candidates.insert(req.body, function(err, result) {
if (err) {
// never reached here
}
res.send('{result: success}');
});
});
Any idea?
Per several issues in the github repository (see here and here) it looks like using this library with authentication is totally broken. Per the second link, most people seem to be wrapping the official library with a promise via something like promisify, bluebird, or a thin custom wrapper.
I've asked some general questions around this topic before (node and blocking). This time the question is a little more specific.
Let's say I've got a node/express app which has a handle that is accepting HTTP requests (doesn't matter, say they're simple GETs).
And it has a separate handler which reads messages off of a RabbitMQ queue, as they arrive, and then does a read from Mongo (Mongo is on a different machine), followed by a write.
If Mongo was "very" busy, would/could that cause the HTTP handler to appear unavailable?
I'm using the Mongo native driver. I would think any blocking that is occurring while the Mongo driver waits for a response from the server would have Node happily accepting and handling HTTP requests, but I don't know for sure.
In a related scenario, swap-out a busy Mongo for a handler that reads a Rabbit message and PUTs a record into a "very" busy ElasticSearch. Will that cause issues with the HTTP handler?
I'd go straight to testing it, but that's a little tricky and gets expensive testing every time I'm not sure what the theory is. So I thought I'd ask.
Here's a (simplified) example of the code:
// HTTP handler...
app.post('/eventcapture/event', (req: express.Request, res: express.Response) => {
var evt: eventDS.IEvent = ('TypeID' in req.body) ? req.body : JSON.parse(req.body);
//create an id
evt._id = uuid.v4();
bus.Publish(evt)
.then((success) => {
res.jsonp(200, { success: true });
})
.catch((failReason:Error) => {
console.error('[ERROR] - Failure writing event: %s,%s', failReason.name, failReason.message);
logError(failReason, evt);
res.jsonp(500, { success: false, reason: failReason });
});
});
// We generically define additional handlers in an array, and then kick them off with a loop.
// Here we have one handler which reads an event, goes to mongo to get additional data which
// it adds into the event before publishing it back out. And a second handler which will catch
// these "augmented" events and push them into Mongo
var processes = [
{
enabled: true,
name: 'augmenter',
inType: 'EventCapture:RawEvent',
handler: (event: eventDS.IEvent) => {
console.log('[LOG] - augment event: %s', event._id);
Profile.FindOne({ _id: event.User.ProfileID })
.then((profile) => {
if (profile) {
console.log('[LOG] - found Profile: %s', profile._id);
event.User.Email = profile.PersonalDetail.Email;
//other values also...
//change the TypeID for publishing
event.TypeID = 'EventCapture:AugmentedEvent';
return event;
}
else throw new Error(util.format('unable to find profile: %s', event.User.ProfileID));
})
.then((augmentedEvent) => bus.Publish(augmentedEvent)) //publish the event back out
.catch((failReason:Error) => {
console.error('[ERROR] - failure publishing augmented event: %s, %s, %s', event._id, failReason.name, failReason.message);
logError(failReason, event);
});
}
},
{
enabled: true,
name: 'mongo',
inType: 'EventCapture:AugmentedEvent',
handler: (event: eventDS.IEvent) => {
console.log('[LOG] - push to mongo: %s', event.User.ProfileID);
Event.Save(event, { safe: true })
.then((success) => console.log('[LOG] - pushed to mongo: %s', event._id))
.catch((failReason:Error) => {
console.error('[ERROR] - failure pushing to mongo: %s, %s', event._id, failReason);
logError(failReason, event);
});
}
}
];
processes.forEach((process, idx, allProcesses) => {
if (process.enabled) {
bus.Subscribe(process.name, process.inType, process.handler);
}
});
No. This is the awesomeness of async programming. Node can do other things while it waits for mongodb to get back to it. You can assume that popular node modules like mongodb write things in an async fashion.
Here's a video that goes into a lot of detail about the event loop: http://vimeo.com/96425312?utm_source=nodeweekly&utm_medium=email
At the end of the day, things like the mongo driver are written using node's low level io and network libraries. These libraries enforce async flow. The author of a package would have to go out of her way to make it sync.