We are trying to optimize our database requests and also reduce the number of database connections in our Node.js servers to Postgres database.
Do Redis database drivers and Redis database connections perform better than relational database drivers/connections in Node.js? I hear tales of needing more database connections for a relational database. My experience with Redis is that just a few connections will suffice, for a given Node.js server process, no matter the load on the server.
With async I/O, can connections be reused, or perhaps the same DB connection used for multiple queries/requests in parallel? Does it differ between database vendors?
Does anyone have recommendation as far as the best possible database connection library for Node.js + Postgres in terms of performance
Related
We have architecture problem on our project. This project requires sharding, as soon as we need almost unlimited scalability for the part of services.
Сurrently we use Node.js + MongoDb (Mongoose) and MySQL (TypeORM). Data is separated by databases through the simple 'DB Locator'. So node process needs connections to a lot of DBs (up to 1000).
Requests example:
HTTP request from client with Shop ID;
Get DB IP address/credentials in 'DB Locator' service by Shop ID;
Create connection to specific database with shop data;
Perform db queries.
We tried to implement it in two ways:
Create connection for each request, close it on response.
Problems:
we can't use connection after response (it's the main problem, because sometimes we need some asynchronous actions);
it works slower;
Keep all connections opened.
Problems:
reach simultaneous connections limit or some another limits;
memory leaks.
Which way is better? How to avoid described problems? Maybe there is a better solution?
Solution #1 perfectly worked for us on php as it runs single process on request and easily drops connections on process end. As we know, Express is pure JS code running in v8 and is not process based.
It would be great to close non-used connections automatically but can't find options to do that.
The short answer: stop using of MongoDB with Mongoose 😏
Longer answer:
MongoDB is document-oriented DBMS. The main usage case is when you have some not pretty structured data that you have to store, but you don't need to use too much. There is lazy indexing, dynamic typing and many more things that not allow you to use it as RDBMS, but it is great as a storage of logs or any serialized data.
The worth part here is Mongoose. This is the library that makes you feel like your trashbox is wonderful world with relations, virtual fields and many things that should not to be in DODBMS. Also, there is a lot of legacy code from previous versions that also make some troubles with connections management.
You already use TypeORM that may works instead Mongoose. With some restrictions, for sure.
It works exactly same way as MySQL connection management.
Here is some more data: https://github.com/typeorm/typeorm/blob/master/docs/mongodb.md#defining-entities-and-columns
In this case you may use you TypeORM Repository as transparent client that will init connections and close it or keep it alive on demand.
We have a project on Node.js that is based on restify and we are using RethinkDB as a database. The problem is that RethinkDB should be accessed from different parts of code (from route handlers, middlewares), but not for all requests. I am wondering what is the best way to connect to RethinkDB in this case?
I see next options:
have one long connection that is stored somewhere (approach we use now),
connect to RethinkDB on each HTTP request, which potentially some of the connections being never used,
connect in each part individually, with potentially several connections per HTTP request, but without useless connections.
I ask this question because I am not sure how well Rethink handle well short/long connections and how expensive they are. For instance MongoDB prefers long connections, but all examples in RethinkDB docs uses one connection per HTTP request.
I recommend a connection pool or one connection per query. Especially if you use feature like changefeeds, which is recommened to be on its own connection.
When you use a single connection for everything, you have to also handle re-connection when the connection timeout/broken. I think it's easier to just use a connection per query, or shared a connection on a request/response.
Just ensure to close your connection after using it, otherwise you will leak connections and new connection cannot be created.
Some driver goes further and doesn't require you to think of connection anymore such as: https://github.com/neumino/rethinkdbdash
Or Elixir RethinkDB: https://github.com/hamiltop/rethinkdb-elixir/issues/32 has an issue to create connection pool.
RethinkDB has an issue related connection pool: https://github.com/rethinkdb/rethinkdb/issues/281
That's probably what community is heading too.
Are Redis connection pools necessary with Node.js asynchronous I/O?
Most of the Redis libraries I see allow you to create client connections but there aren't many connection pool modules so I assume it's not as important.
The one thing that confuses me is that Redis has a default of 16 different/segmented databases in one Redis instance.
So if you create a connection pool, which database of the 16 are you connected to? Can you connect to all 16 at once with the same connection pool?
Is there a Node.js Redis library that creates a connection pool with 1 client per database, depending on how many databases you are using?
You've asked too many questions in one post.
Trying to answer them;
Are Redis connection pools necessary with Node.js asynchronous I/O?
Duplicate of Node.js Redis Connection Pooling
So if you create a connection pool, which database of the 16 are you connected to?
By default you're always connected to database 0. Databases in redis are numbered if you're thinking why 0. They cannot be renamed to a string.
Can you connect to all 16 at once with the same connection pool?
Connection pools are not necessary
Is there a Node.js Redis library that creates a connection pool with 1 client per database, depending on how many databases you are using?
After searching i find two :
node-redis-pool
redis-connection-pool
What is the benefit of using Redis as socket.io memory store, does it need additional resources. I'm using MongoDB as the database, can i use MongoDB as memory store for Socket.io, or do i replace MongoDB with Redis as database? What would be more efficient for building a real-time web app and providing maximum concurrent connections?
can i use MongoDB as memory store for Socket.io
Yes, you can try mong.socket.io
do i replace MongoDB with Redis as database?
Redis and MongoDB are different kind of databases, while mongodb is document oriented redis is key/value oriented (we can even say that redis is a data-structure server).
What would be more efficient for building a real-time web app and providing maximum concurrent connections?
Redis will be definitely faster than mongo on that matter, it supports pub/sub out of the box (while mong.socket.io uses a collection to simulate pub/sub) but you must know that all your data stored in redis must live in memory (here the only data that will be stored in redis will be additionnal socket.io informations).
in my application I have a default database and other database I have to connect to in function of client's requests , since with mongoose in node as far as I understood: there is a pool of connections application wide, if I change database, it is changed for all the subsequent requests, I think it could cause some problems, what is the best way to switch Database with mongoose?
Mongoose 3.7.1 (unstable) supports switching databases.
Otherwise you'll need to create separate connection instances for each database.