I am using arangojs v6.14.1 and arangodb version 3.6.4.. I also have the nodejs express app which is intended to serve client requests.
I am experiencing an issue while executing concurrent requests. The database connection hangup when I concurrently process client requests:
What the app will do when it receives a request?
Open a database connection -
db = new Database(dbConfig.url);
db.useDatabase(dbConfig.name);
db.useBasicAuth(dbConfig.username, dbConfig.password);
There are multiple middleware functions that need to access to perform various functions and accessibility checks. And for each middleware I tried to
open a new database connection ->
perform the action ->
close the connection ->
return the result to next middleware.
This works fine with single request at a time. But if I tried to call two APIs at same time, I am getting CONNECTIONRESET error. And also throwing socket hangup error.
I tried to commend out the close connection method and it started working fine for a while. But when I increased the number of connections, it again showing the same error as "CONNECTIONRESET".
I have searched the documentation of arangojs regarding the connection manipulation. But I haven't found any information regarding the same.
Any help would be deeply appreciated.
This might be too late for you, but I had a similar issue. After talking with ArangoDB support, it turned out to be an issue with sockets.
Looking at the output from netstat -aplnt I was seeing a lot of CLOSE_WAIT and TIME_WAIT messages, indicating that there were A LOT of connections that were not being closed properly.
The solution was to enable persistent connections (see arangojs config for keep-alive and maxSockets) and then re-using a single connection whenever possible. This was more than just a settings change, requiring some code modifications as well.
I have the following code in my app.js which runs on server start (npm start)
mongo.mongoConnect('connection_string', 'users').then((x) => {
console.log('Database connection successful');
app.listen(5000, () => console.log('Server started on port 5000'));
})
.catch(err => {
console.error(err.stack);
process.exit(1);
});
process.on('SIGINT', mongo.mongoDisconnect).on('SIGTERM', mongo.mongoDisconnect);
As you can see I open up SIGINT and SIGTERM for closing my connections upon process.exit
I've been reading a lot about how to deal with database connections in mongo and know that I should just invoke it once and have it across my application.
Does that mean that even after save() method when saving data to mongo followed by POST request, I should not be closing my connection? If I close it, how am I going to invoke it again since the connections happens on app start?
I'm asking it since in PHP I had the practice to always open and close my connection after querying MySql database.
Likewise, does it mean that the connection will close only on server shutdown in other words it will always be present since I do not want to shut down my node.js backend instance?
It is formally correct to open a connection, run a query, and then close the connection, but it is not a good practice, because opening a connection is an "expensive" operation and connections can be reused, which is much more efficient. The main restriction on an open connection is that it can only be used by 1 thread at a time. (More accurately, once a request is sent on a connection, no other requests can be sent on that connection until the response to that request is received.)
If your application is short lived or inherently single threaded, as may be the case when running as a "serverless" function, it may be acceptable to open and close a connection on each request.
While in theory it might be acceptable to open a single connection at the start of the program, keep a global reference to that connection, and reuse it, in practice there are common ways in which a connection becomes unusable that you would have to account for, and handling all the possibilities requires complex code. It gets even more complicated when, as is possible with MongoDB replica sets, you are actually connecting to more than one server and want to retry a command on a second server if the first one fails to respond.
That is why the standard and "best" practice is to use a "connection pool" to manage your database connections. A pool opens a set of network connections to the database, verifies and maintains their health, and dynamically assigns virtual database connections to actual network connections as needed. The pool is implemented in a library that will have received a lot of real world testing and is extremely likely to be better than anything you would write yourself. Connection pools have configuration options that would let you set any behavior you want, including opening a new connection for each request and closing it when done, but offer a wide range of performance enhancing capabilities, such a reusing connections and avoiding the overhead of creating them for each request.
This is why for MongoDB, the standard Node.js client already implements a connection pool. I do not know what mongo.mongoConnect in your code refers to; you said in the title that you are using mongoose but it uses connect, not mongoConnect to connect to the database. In general you should either be using the standard client or a JavaScript ORM library like mongoose. Either of them will take care of the connection management issues for you.
Refer to the documentation for the client/library you use for exactly the right way to use it. In general, you would initialize some kind of client object and store it globally before entering your main application handler. Then you would use this object to handle your database operations, and the object will transparently manage the underlying connections via the pool implementation. In this kind of setup, you would only close the connection when exiting the program, and usually the library takes care of that for you automatically, so you really never need to close the connection.
Thus, when using a MongoDB connection pool in NodeJS, you write your program basically the same way you would as if you just opened a connection at startup and then kept reusing it. The libraries take care of isolating you from all the problems that can arise from actually doing this. You do not need to, and in fact should not, close the connection after a database operation when using standard MongoDB NodeJS libraries.
Note that other connection pool implementations exist that do require you to close the connection. What you do with those pools is reserve (or "check out" or "open") a connection, use it, perhaps for multiple operations, and the release (or "check in" or "close") the connection when you are done. This is probably what you were doing in PHP. It is important to read and follow the documentation for the connection pool library you are using to make sure you are using it correctly.
This may not be the exact answer you are looking for, but it is not a good idea to open a new connection for every request and then close it. It is an overhead because it takes some time (even in milliseconds) to create a new connection.
Instead, you should create a pool of connections and use it in your app.
It's a good idea to close your mongo connection when your process dies or is stopped, but you should not need to close your mongoose connection after every successful query.
If you are instantiating a new mongo connection before each query you shouldn't need to be doing that either. You should just need to do that once when booting up your server.
you have two approaches
1) reopen a connection on every call using middle wares
2) you have to save your's query in node sometime later on execute all it onces
I want my application server(Hapi/Express) to start the mongodb process before proceeding with server.start(). A good way to do this is via Promises so that the mongod return code can be captured in .then and checked for start success/failure.
I posted a similar question # Nodejs exec mongodb command in Bluebird Promise, that prompted me to ask this one here.
You seem to not understand the basics of processes.
so that the mongod return code can be captured in .then
mongod will not return any code until it exits (it's called "exit code" for a reason). I assume that you want your mongodb running, so this means no code for you.
Starting a database server from the application is absolutely the wrong way to do it. Database and application should be started separately (by OS' startup manager or whatever). If you install mongodb from a package, the auto-startup should be handled for you (via installing proper init script).
Application should only know a connection string (and if it can't connect to the database at this string, show some pretty error message).
Very simple and stupid question I came up with today when was playing around with sails and mongo db (adapter of the waterline odm). I just shut down the mongo while my sails server still has been running and saw nothing, neither in logs of sails or in the browser. How I could handle such situation?
If sails.js attempts a connection to the DB you will most certainly get a message in your console if the Mongo server is shut down, but if no request is made that requires a DB connection, then you will not see any error because no error as of yet exists.
It sounds like you might require a process that will monitor your DB and check to make sure it is still running? There are Sail.js options where you could create a CRON job to check the connection. Or you could use other application monitoring services like New Relic to monitor your DB.
User case:
My nodejs server start with a configuration wizard that allow user to change the port and scheme. Even more, update the express routes
Question:
Is it possible to apply the such kind of configuration changes on the fly? restart the server can definitely bring all the changes online but i'm not sure how to trigger it from code.
Changing core configuration on the fly is rarely practiced. Node.js and most http frameworks do not support it neither at this point.
Modifying configuration and then restarting the server is completley valid solution and I suggest you to use it.
To restart server programatically you have to execute logics outside of the node.js, so that this process can continue once node.js process is killed. Granted you are running node.js server on Linux, the Bash script sounds like the best tool available for you.
Implementation will look something like this:
Client presses a switch somewhere on your site powered by node.js
Node.js then executes some JavaScript code which instructs your OS to execute some bash script, lets say it is script.sh
script.sh restarts node.js
Done
If any of the steps is difficult, ask about it. Though step 1 is something you are likely handling yourself already.
I know this question was asked a long time ago but since I ran into this problem I will share what I ended up doing.
For my problem I needed to restart the server since the user is allowed to change the port on their website. What I ended up doing is wrapping the whole server creation (https.createServer/server.listen) into a function called startServer(port). I would call this function at the end of the file with a default port. The user would change port by accessing endpoint /changePort?port=3000. That endpoint would call another function called restartServer(server,res,port) which would then call the startServer(port) with the new port then redirect user to that new site with the new port.
Much better than restarting the whole nodejs process.