I have a public static Connection getConnection() to get the DB connection and I have declared the connection object as static.
public static Connection con;
My doubt is as there could be multiple connections by concurrent users should I have the con as a local variable with in getConnection() or static would be fine?
Please point out what issues I would run into for concurrent users if the con object is static.
When you're declaring a Connection as static, only one user at a time is able to connect to a database. This way you are creating an unnecessary bottleneck. Another question is: how do you maintain opening and closing a connection?
Your second proposition, creating a local variable, seems to eliminate the bottlenecking. But opening a connection each time you need to access a database is quite expensive.
So actually the answer is you should use a Connection Pool. An established connection may be leased from a pool and used without an additional overhead.
Related
I have a Sequelize instance and it is exported in a file to be accessed when doing DB operations.
const sequelize = new Sequelize('database', 'username', null, {
dialect: 'mysql'
});
module.exports = sequelize;
So the instance is created when the expressjs server starts and never destroys. I wonder if this is the correct way to do, or should I call new Sequelize every time I use the DB operation?
I think it should be kept alive because that's how DB pooling could take effect. Right?
The bottom line is - yes, it should stay alive. There is no performance hit whatsoever if you keep the instance alive. Because it will be the Sequelize instance (and by extension the ORM) which will handle the future connections. This also includes (as you noted) pooling.
The connections
When it comes to the pooling configuration itself, it get's a little tricky though. Depending on your configuration, the pool has some amount of "space" to work with - the limit to create connections, the idle duration after which connections are removed etc. I can certainly imagine a situation when keeping a connection alive is simply not needed - for example an internal system for a company which is not used overnight.
The Sequelize ORM gives you a good set of options to choose from when configuring your connection pool. In general, you do want to reuse connections as establishing new ones is quite expensive - not just because of the network (e.g. authorization, maybe proxy etc.) but also because of memory allocation which happens when you create a database connection (which is why reconnecting on every request is not a good idea..).
However, it all comes down to which database engine you use (and how busy your system is); MySQL can for example cache connections. When a connection is closed, it is returned to the thread cache rather than discarded (for some period of time). When a new connection opens, MySQL will look into the thread cache rather than try and establish a new connection.
You might want to go through these:
https://stackoverflow.com/a/4041136/8775880 (Good explanation on how pooling works)
https://stackoverflow.com/a/11659275/8775880 (Good explanation on how expensive it is to keep connections open)
Continuing the conversation from
this question:
Two-part question here:
Can a TcpNetClientConnectionFactory have multiple connections to its upstream
server, if the host and port are the same?
If so, how can I programmatically build a new connection for that connection
factory? I see the buildNewConnection method, but it is protected.
The first connection is automatically built as soon as the first Message
passes through the factory. What we need to do is notice when following Messages
have a different ip_connectionId, stand up a new connection, and route those
Messages to that new connection. Obviously, Messages with the original
ip_connectionId would still be routed to the original connection.
Not sure whether it would be better to create multiple connections off of one
connection factory, or create a whole new connection factory, sending message
handler, and receiving channel adapter for each new connection.
If the inbound connection factory is a TcpNetServerConnectionFactory, you can simply use a ThreadAffinityClientConnectionFactory because each inbound connection gets its own thread.
You would call getConnection(). This will bind the connection to the thread (and you can obtain the connection id from it), but you don't really need to map the header in this direction because of the thread affinity, you would only have to map on the return path.
Bear in mind, though, if the ThreadAffinityClientConnectionFactory detects that a connection has been closed, it will create a new one. So, you might want to call getConnection() in your mapper on each call. However, there would still be a race condition, so you might also need to listen for TcpConnectionCloseEvents and TcpConnectionOpenEvents.
If you use NIO on the inbound, or otherwise hand off the work to other threads via an executor, this won't work.
In that case you would need your own wrapping connection factory - you could use the ThreadAffinityClientConnectionFactory as a model, but instead of storing the connections in a ThreadLocal, you'd store them in a map. But you'd still need a ThreadLocal (set upstream on each call) to tell the factory which connection to hand out when the adapter asks for one.
There's a trick you need to be aware of, however.
There is a property singleUse on the connection factory. This serves 2 purposes;
first, it tells the factory to create a new connection each time getConnection() is called instead of a single, shared, connection
second, it tells the inbound adapter to close the connection after the reply is received
So the trick is you need singleUse=true on the real factory (so it gives you a new connection each time getConnection() is called), but singleUse=false on the wrapping factory so the adapters don't close the connection.
I suggest you look at the ThreadAffinityClientConnectionFactory and CachingClientConnectionFactory connection factory to see how they work.
We should probably consider splitting this into two booleans; we could probably also make some improvements to avoid the need for a thread local by adding something like getConnection(String connectionId) to the client factory contract and have the factory look up the connection internally; but that will require work in the adapters.
I'll capture an issue for this and see if we can get something in 5.2.
Rather a long answer, but I hope it makes sense.
I have a Node.js script and a PostgreSQL database, and I'll be using a library that maintains a pool of connections to the database.
Say I have a script that queries the database multiple times (not a transaction) at different parts of the script, how do I tell if I should acquire a single connection/client and reuse it throughout*, or acquire a new client from the pool for each query? (Both works but which has better performance?)
*task in the pg-promise library, connect in the node-postgres library.
...
// Acquire connection from pool.
(Database query)
(Non-database-related code)
(Database query)
// Release connection to pool.
...
or
...
// Acquire connection from pool.
(Database query)
// Release connection to pool.
(Non-database-related code)
// Acquire connection from pool.
(Database query)
// Release connection to pool.
...
I am not sure, how the pool you are using works, but normally they should reuse the connections (don't disconnect after use), so you do not need to be concerned with caching connections.
You can use node-postgres module that will make you task easier.
And about your question when to use pool here is the brief answer.
PostgreSQL server can only handle 1 query at a time per connection.
That means if you have 1 global new pg.Client() connected to your
backend your entire app is bottleknecked based on how fast postgres
can respond to queries. It literally will line everything up, queuing
each query. Yeah, it's async and so that's alright...but wouldn't you
rather multiply your throughput by 10x? Use pg.connect set the
pg.defaults.poolSize to something sane (we do 25-100, not sure the
right number yet).
new pg.Client is for when you know what you're doing. When you need a
single long lived client for some reason or need to very carefully
control the life-cycle. A good example of this is when using
LISTEN/NOTIFY. The listening client needs to be around and connected
and not shared so it can properly handle NOTIFY messages. Other
example would be when opening up a 1-off client to kill some hung
stuff or in command line scripts.
here is the link of that module.
Hopefully this will help.
https://github.com/brianc/node-postgres
You can see the documentation over there and about the pooling. Thanks :)
And about closing the pool it provides the callback done which can be called when you want to close that pool.
I'm using the Node native client 1.4 in my application and I found something in the document a little bit confusing:
A Connection Pool is a cache of database connections maintained by the driver so that connections can be re-used when new connections to the database are required. To reduce the number of connection pools created by your application, we recommend calling MongoClient.connect once and reusing the database variable returned by the callback:
Several questions come in mind when reading this:
Does it mean the db object also maintains the fail over feature provided by replica set? Which I thought should be the work of MongoClient (not sure about this but the C# driver document does say MongoClient maintains replica set stuff)
If I'm reusing the db object, when should I invoke the db.close() function? I saw the db.close() in every example. But shouldn't we keep it open if we want to reuse it?
EDIT:
As it's a topic about reusing, I'd also want to know how we can share the db in different functions/objects?
As the project grows bigger, I don't want to nest all the functions/objects in one big closure, but I also don't want to pass it to all the functions/objects.
What's a more elegant way to share it among the application?
The concept of "connection pooling" for database connections has been around for some time. It really is a common sense approach as when you consider it, establishing a connection to a database every time you wish to issue a query is very costly and you don't want to be doing that with the additional overhead involved.
So the general principle is there that you have an object handle ( db reference in this case ) that essentially goes and checks for which "pooled" connection it can use, and possibly if the current "pool" is fully utilized then and create another ( or a few others ) connection up to the pool limit in order to service the request.
The MongoClient class itself is just a constructor or "factory" type class whose purpose is to establish the connections and indeed the connection pool and return a handle to the database for later usage. So it is actually the connections created here that are managed for things such as replica set fail-over or possibly choosing another router instance from the available instances and generally handling the connections.
As such, the general practice in "long lived" applications is that "handle" is either globally available or able to be retrieved from an instance manager to give access to the available connections. This avoids the need to "establish" a new connection elsewhere in your code, which has already been stated as a costly operation.
You mention the "example" code which is often present through many such driver implementation manuals often or always calling db.close. But these are just examples and not intended as long running applications, and as such those examples tend to be "cycle complete" in that they show all of the "initialization", the "usage" of various methods, and finally the "cleanup" as the application exits.
Good application or ODM type implementations will typically have a way to setup connections, share the pool and then gracefully cleanup when the application finally exits. You might write your code just like "manual page" examples for small scripts, but for a larger long running application you are probably going to implement code to "clean up" your connections as your actual application exits.
I have a severe problem with my database connection in my web application. Since I use a single database connection for the whole application from singleton Database class, if i try concurrent db operations (two users) the database rollsback the transactions.
This is my static method used:
All threads/servlets call static Database.doSomething(...) methods, which in turn call the the below method.
private static /* synchronized*/ Connection getConnection(final boolean autoCommit) throws SQLException {
if (con == null) {
con = new MyRegistrationBean().getConnection();
}
con.setAutoCommit(true); //TODO
return con;
}
What's the recommended way to manage this db connection/s I have, so that I don't incurr in the same problem.
Keeping a Connection open forever is a very bad idea. It doesn't have an endless lifetime, your application may crash whenever the DB times out the connection and closes it. Best practice is to acquire and close Connection, Statement and ResultSet in the shortest possible scope to avoid resource leaks and potential application crashes caused by the leaks and timeouts.
Since connecting the DB is an expensive task, you should consider using a connection pool to improve connecting performance. A decent applicationserver/servletcontainer usually already provides a connection pool feature in flavor of a JNDI DataSource. Consult its documentation for details how to create it. In case of for example Tomcat you can find it here.
Even when using a connection pool, you still have to write proper JDBC code: acquire and close all the resources in the shortest possible scope. The connection pool will on its turn worry about actually closing the connection or just releasing it back to pool for further reuse.
You may get some more insights out of this article how to do the JDBC basics the proper way. As a completely different alternative, learn EJB and JPA. It will abstract away all the JDBC boilerplate for you into oneliners.
Hope this helps.
See also:
Is it safe to use a static java.sql.Connection instance in a multithreaded system?
Am I Using JDBC Connection Pooling?
How should I connect to JDBC database / datasource in a servlet based application?
When is it necessary or convenient to use Spring or EJB3 or all of them together?
I've not much experience with PostgreSql, but all the web applications I've worked on have used a single connection per set of actions on a page, closing it and disposing it when finished.
This allows the server to pool connections and stops problems such as the one that you are experiencing.
Singleton should be the JNDI pool connection itself; Database class with getConnection(), query methods et al should NOT be singleton, but can be static if you prefer.
In this way the pool exists indefinitely, available to all users, while query blocks use dataSource.getConnection() to draw a connection from the pool; exec the query, and then close statement, result set, and connection (to return it to the pool).
Also, JNDI lookup is quite expensive, so it makes sense to use a singleton in this case.