When I connect to Rexster graph server with Grex should I keep the database connection open?
var grex = require('grex');
var client = grex.createClient();
client.connect({ graph: 'graph'}, function(err, client) {
if (err) { console.error(err); }
...
});
I think I should because nodejs is single threaded so there's no chance of different requests trying to use the one connection at the same time.
Yes, you should. There 's no reason to have the overhead of connecting on every request. There will not be any issue of "mangling", as your code will be run in a single thread anyway.
Furthermore, you could even have a pool of connections waiting to serve your requests in case you have a heavy usage application. Some adapters do it for you automatically, for example, MongoClient has a default pool of 5 connections.
Related
We are using Mongoose, Nodejs, Serverless, and AWS Lambda. For making use of the same connection instead of opening and closing the connection each time whenever required, I have created a connection pool of size 10 (Which seems to be sufficient for our use-case right now).
But the thing is, when I see the Cloudwatch logs for Lambda, it's not the same connection that is being used.
Every time a new Lambda is called, a new connection is created, while the subsequent calls to that Lambda use the same connection that was opened in the first call.
Resulting in an increase in the number of connections open at a time. At MongoDB Atlas, I can see the number of open connections is way much.
Below is the code I am using for creating a connection if there is no cached connection available. In case it is available, the cached one will be used and a new connection will not be created.
let cached_db;
exports.createConnection = async () => {
if(cached_db == null){
return await mongoose.connect(
connection_uri,
{ 'useUnifiedTopology': true ,
'useNewUrlParser': true,
'useFindAndModify': false ,
'useCreateIndex': true,
'socketTimeoutMS': 60000,
'connectTimeoutMS': 60000,
'poolSize': 10
}
).then(conn => {
cached_db = conn;
return conn;
}).catch((err) => {
console.error('Something went wrong', err);
throw err;
});
} else {
console.log("Cached db in use.");
return cached_db;
}
}
Can the same connection be used across Lambdas? Is there a way to do it?
You should define the client to the MongoDB server outside the AWS Lambda handler function. Don't define a new MongoClient object each time you invoke your function. Doing so causes the driver to create a new database connection with each function call. This can be expensive and can result in your application exceeding database connection limits.
As an alternative, do the following:
Create the MongoClient object once.
Store the object so your function can reuse the MongoClient across function invocations.
Step 1
Isolate the call to the MongoClient.connect() function into its own module so that the connections can be reused across functions. Let's create a file mongo-client.js for that:
mongo-client.js:
const { MongoClient } = require('mongodb');
// Export a module-scoped MongoClient promise. By doing this in a separate
// module, the client can be shared across functions.
const client = new MongoClient(process.env.MONGODB_URI);
module.exports = client.connect();
Step 2
Import the new module and use it in function handlers to connect to database.
some-file.js:
const clientPromise = require('./mongodb-client');
// Handler
module.exports.handler = async function(event, context) {
// Get the MongoClient by calling await on the connection promise. Because
// this is a promise, it will only resolve once.
const client = await clientPromise;
// Use the connection to return the name of the connected database for example.
return client.db().databaseName;
}
Pool Size
Connection pool size is a cache of database connections maintained so these connections can be reused when future requests to the database are required. Connection pools are used to enhance the performance of executing commands on a database.
Note: maxPoolSize and poolSize are the same, except they relate to whether you are using the useUnifiedTopology: true setting.
If you are using useUnifiedTopology: true, maxPoolSize is the spec-compliant setting to manage how large connection pools can be.
But if you are using useUnifiedTopology: false (or omits it), poolSize is the same thing but from before we had the unified topology.
Note: Each connection consumes about 1MB of RAM.
Value of the Pool Size
The connection pool is on a per-mongod/mongos basis, so when connecting to a 3-member replica there will be three connection pools (one per mongod), each with a maxPoolSize. Additionally, there is a required monitoring connection for each node as well, so you end up with (maxPoolSize+1)*number_of_nodes TCP connections.
In my opinion, if you don't care about CPU and RAM, you should use all available connections (why not if we already have them, right?).
For example: You have Atlas free cluster with 3 replica sets, that supports maximum number of 500 connections, and you have only one application that connects to it, give all connections to that one application. In order to set the value of poolSize, you can use above calculation of connections:
poolSize = (maximum_connections/number_of_nodes) - 1
poolSize = (500/3) - 1
poolSize = 165
If you would have 2 applications that will connect to that same cluster, give each application half of connections.
If you have limited RAM memory, check how much you can spear and calculate poolSize based on that (as I said in the note, you can assume that one connection will consume about 1MB of RAM).
Resources
For more info, check this official MongoDB Docs.
For connection pool, check this and this.
I found from this blog that Lambda may use same connection if restore the same snapshot and creates new connection if new snapshot generation.
So Lambda can't give assurance that to use same connection if we use outside the handle function.
So in my opinion best approach to optimise number of connection to Mongodb is to close connection before lambda complete so your other service can use free connection.
Use below method to close connection after database interaction finishes.
createConnection.close()
I have an Express App which connects to a MongoDB server at startup and serves requests on-demand (I don't disconnect - it's a single threaded server so no pooling - fairly simple stuff)
Problem is that it's possible the MongoDB server will be unavailable for periods of time (it's not on-site) and whilst the Express App doesn't crash, it seems that any requests made to the server will run indefinately until the connection is restored!
I'd like to limit that (e.g. throw an error back after a period of time) but I can't seem to make that happen...
I'm using connect options "{server: {auto_reconnect: true}}" which seems to ensure that once the MongoDB server reappears, requests complete (without it, requests made during downtime seem to run forever...) - and I don't have access to the client code so I can't fix it there...
I'd assumed a combination of 'connectTimeoutMS' or 'socketTimeoutMS' would allow me to terminate requests when MongoDB is unavailable for longer periods, but I just can't get those to work (I've tried them as connect options, passing them in the URI etc. etc.)
Any attempt to open a Collection and Find/Insert/Update just 'hangs' until the MongoDB reappears - I've left it over 30 mins and everything was just sitting these (and completed AOK when the network was restored!)
What's the best way around this? Should I open a connection specifically for each request (not really a performance issue - it's not a high volume app) or is there something else I'm missing?
Updated to add the connect code
var myDB
var mongodb = require('mongodb')
var uri = // some env vars and stuff
mongodb.MongoClient.connect(uri, {server: {auto_reconnect: true}}, function (err, db) {
myDB = db
})
myDB is then used elsewhere to open collections - and the handle from that is used to find/insert etc.
If the connection to the DB is interrupted, myDB.collection() calls (or calls to find/insert on their handles) will simply hang until the connection is restored - nothing I've tried will cause them to 'time out' sooner!?
I assume that you are using mongoose as a driver.
You'd catch the error by this.
var db = require('domain').create();
db.on('error', function(err) {
console.log('DB got a problem');
});
db.run(function() {
mongoose.connect(config, options);
});
or you can directly access
mongoose.connection.readyState
to check the statement of your DB.
Connection ready state
0 = disconnected
1 = connected
2 = connecting
3 = disconnecting
Each state change emits its associated event name.
http://mongoosejs.com/docs/api.html
I was wondering if anyone can help me understand what the proper way of maintaining multiple connections to multiple postgres servers via https://github.com/brianc/node-postgres is.
Obviously when running a node server for long duration we want to make sure we keep everything clean with no leaks and so I am wondering what the proper pattern is.
Please remember that my Node server will need to connect to 7-8 Postgres servers.
https://github.com/brianc/node-postgres supports the idea of pools. I am wondering: do I just connect to all servers on initial Node server set up and maintain open connections and each function can ask for a pool when it needs to talk to a server?
In other words, am I supposed to call pg.connect every time I make a server query? (minus the var pg and var connectionString which could be global)
Can't I just have a single connection be on and ready?
var pg = require('pg');
var connectionString = "pg://brian:1234#localhost/postgres"
pg.connect(connectionString, function(err, client, done) {
client.query('SELECT name FROM users WHERE email = $1', ['brian#example.com'], function(err, result) {
assert.equal('brianc', result.rows[0].name);
done();
});
});
Code snippets are greatly appreciated.
I am new to node, postgresql, and to the whole web development business. I am currently writing a simple app which connects to a postgres database and display the content of a table in a web view. The app will be hosted in OpenShift.
My main entry is in server.js:
var pg = require('pg');
pg.connect(connection_string, function(err, client) {
// handle error
// save client: app.client = client;
});
Now, to handle the GET / request:
function handle_request(req, res){
app.client.query('...', function(err, result){
if (err) throw err; // Will handle error later, crash for now
res.render( ... ); // Render the web view with the result
});
}
My app seems to work: the table is rendered in the web view correctly, and it works for multiple connections (different web clients from different devices). However, if there is no request for a couple of minutes, then subsequent request will crash the app with time out information. Here is the stack information:
/home/hai/myapp/server.js:98
if (err) throw err;
^
Error: This socket is closed.
at Socket._write (net.js:474:19)
at Socket.write (net.js:466:15)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/connection.js:109:15)
at [object Object].submit (/home/hai/myapp/node_modules/pg/lib/query.js:99:16)
at [object Object]._pulseQueryQueue (/home/hai/myapp/node_modules/pg/lib/client.js:166:24)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/client.js:193:8)
at /home/hai/myapp/server.js:97:17
at callbacks (/home/hai/myapp/node_modules/express/lib/router/index.js:160:37)
at param (/home/hai/myapp/node_modules/express/lib/router/index.js:134:11)
at pass (/home/hai/myapp/node_modules/express/lib/router/index.js:141:5)
Is there a way to keep the connection from timed out (better)? Or to reconnect on demand (best)? I have tried to redesign my app by not connecting to the database in the beginning, but upon the GET / request. This solution works only for the first request, then crashed on the second. Any insight is appreciated.
Have you looked into the postgres keepalive setting values? It sends packets to keep idle connections from timing out.
http://www.postgresql.org/docs/9.1/static/runtime-config-connection.html
I also found this similar question:
How to use tcp_keepalives settings in Postgresql?
You could also perform really minor queries from the db at a set interval. However, this method is definitely more hacked.
Edit: You could also try initiating the client like this:
var client = new pg.Client(conString);
Before you make your queries, you can check if the client is still connected. I believe you can use:
if(client.connection._events != null)
client.connect();
faced the same problem.. telling the client to close connection upon the end event
query.on('end', function() {
client.end();
});
did the trick for me...
You can also change the default idle timeout of 30 seconds to whatever value you need. E.g.
pg.defaults.poolIdleTimeout = 600000; // 10 mins
I'm using the parameter keepAlive in true and it works.
This is my configuration and it is solved.
const client_pg = new Client({
connectionString,
keepAlive: true,
keepAliveInitialDelayMillis: 10000
});
In production, I have a game which uses connection-local variables to hold game state. However I notice that if I idle for a certain time on the connection, it disconnects and reconnects which loses the current state. During my tests on a local host, I never noticed this behavior. Is this the norm behavior for socket connections or is something else causing the connections to drop.
If it is a normal behavior how is this typically handled? Should connection values be stored globally so they can be restored should a user drop/reconnect?
Your problem is around socket timeouts. If there's no activity on a certain socket, socket.io will close it automatically.
An easy (and hackish) fix is to send a heartbeat to the connected client to create activity and stop the socket from timing out.
Server:
function sendHeartbeat(){
setTimeout(sendHeartbeat, 8000);
io.sockets.emit('ping', { beat : 1 });
}
io.sockets.on('connection', function (socket) {
socket.on('pong', function(data){
console.log("Pong received from client");
});
}
setTimeout(sendHeartbeat, 8000);
Client:
socket.on('ping', function(data){
socket.emit('pong', {beat: 1});
});
More Information:
You can get more information on configuring socket.io here.
EDIT: Mark commented that if the user does lose the connection (connection drops on his end because of internet troubles), you should be able to restore the user to his last state.
To do that, the best way would be to use a already widely used method for storing user data, cookies and sessions.
An extremely well done tutorial on how to do this located here. Although he uses express to set cookies, you can do this using anything (I do it using rails). Using this method, you can store the user data in a cookie and fetch it during the handshake. From there you can just access the data using socket.handshake.data.
What you need to do is create or identify the session per (re-) connection. You may reduce the number of reconnections per Moox's answer above but it is still not failsafe - e.g. a user loses wifi connection for a bit, etc. In other words - maintain user metadata per session and not per socket, and expect occasional disconnects and reconnects.