I have a Next.js app with a very simple database using lowDb. The idea was to have one module with a simple get and set functions, and inside a module, a db is set as a top level constant:
const dbAdapter = new JSONFile(dbFile)
const db = new Low(dbAdapter)
let randomId = generateRandomId()
console.log(`DB: Using database in ${dbFile}, instance ${randomId}`)
/// A promise we call at the beginning of get and set functions to ensure db is initialized
let initPromise = new Promise<undefined>(async function(resolve, reject){
console.log(`DB: init, reading database ...`)
// Read data from JSON file, this will set db.data content
await db.read()
// If file.json doesn't exist, db.data will be null
// Set default data, node >= 15.x
db.data ||= {
tracks: {},
}
resolve(undefined)
})
export async function getTracks() {
await initPromise
return db.data['tracks']
}
...
The getTracks() is then called inside getServerSideProps function of multiple routes to provide actual data to both SSR and client rendered page.
The problem seems to be that as I use the database in multiple pages, Next.js is instantiating this module multiple times, leading to each page having its own separate database (so whatever one page saves, the other one doesn't see, as each instance keeps it's own in-memory cache, and doesn't re-read the db file on every read).
Here is what the log shows:
wait - compiling / (client and server)...
event - compiled client and server successfully in 100 ms (387 modules)
DB: Using database in tracks.json, instance tpzoj
DB: init, reading database ...
Then I navigate to another page, and this happens - a module is built again for another page's bundle, and another instance of db is created:
wait - compiling /edit (client and server)...
event - compiled client and server successfully in 115 ms (394 modules)
DB: Using database in tracks.json, instance gjndv
DB: init, reading database ...
Is there a way to force Next.js to only build a certain module once (instead of bundling it into each page separately)?
This seems to be related: https://github.com/vercel/next.js/issues/10933 - basically the takeaway seems to be "avoid in-process memory cache", and use filesystem or redis instead :(.
Or, in my case wait for db.read() inside every getTracks(), to force re-read of a db on every request :(. Is there a better way?
Related
I am running a nodeJS application using forever npm module.
Node application also connects to Redis DB for cache check. Quite often the API stops working with the following error on the forever log.
{ ReplyError: Ready check failed: ERR max number of clients reached
at parseError (/home/myapp/core/node_modules/redis/node_modules/redis-parser/lib/parser.js:193:12)
at parseType (/home/myapp/core/node_modules/redis/node_modules/redis-parser/lib/parser.js:303:14)
at JavascriptRedisParser.execute (/home/myapp/ecore/node_modules/redis/node_modules/redis-parser/lib/parser.js:563:20) command: 'INFO', code: 'ERR' }
when I execute the client list command on the redis server it shows too many open connections. I have also set the timeout = 3600 in my Redis configuration.
I do not have any unclosed Redis connection object on my application code.
This happens once or twice in a week depending on the application load, as a stop gap solution I am restarting the node server( it works ).
What could be the permanent solution in this case?
I have figured out why. This has nothing to do with Redis. Increasing the OS file descriptor limit was just a temporary solution. I was using Redis in a web application and the connection was created for every new request.
When the server was restarted occasionally, all the held-up connections by the express server were released.
I solved this by creating a global connection object and re-using the same. The new connection is created only when necessary.
You could do so by creating a global connection object, make a connection once, and make sure it is connected before every time you use that. Check if there is an already coded solution depending on your programming language. In my case it was perl with dancer framework and I used a module called Dancer2::Plugin::Redis
redis_plugin
Returns a Dancer2::Plugin::Redis instance. You can use redis_plugin to
pass the plugin instance to 3rd party modules (backend api) so you can
access the existing Redis connection there. You will need to access
the actual methods of the the plugin instance.
In case if you are not running a web-server and you are running a worker process or any background job process, you could do this simple helper function to re-use the connection.
perl example
sub get_redis_connection {
my $redis = Redis->new(server => "www.example.com:6372" , debug => 0);
$redis->auth('abcdefghijklmnop');
return $redis;
}
...
## when required
unless($redisclient->ping) {
warn "creating new redis connection";
$redisclient = get_redis_connection();
}
I was running into this issue in my chat app because I was creating a new Redis instance each time something connected rather than just creating it once.
// THE WRONG WAY
export const getRedisPubSub = () => new RedisPubSub({
subscriber: new Redis(REDIS_CONNECTION_CONFIG),
publisher: new Redis(REDIS_CONNECTION_CONFIG),
});
and where I wanted to use the connection I was calling
// THE WRONG WAY
getNewRedisPubsub();
I fixed it by just creating the connection once when my app loaded.
export const redisPubSub = new RedisPubSub({
subscriber: new Redis(REDIS_CONNECTION_CONFIG),
publisher: new Redis(REDIS_CONNECTION_CONFIG),
});
and then I passed the one-time initialized redisPubSub object to my createServer function.
It was this article here that helped me see my error: https://docs.upstash.com/troubleshooting/max_concurrent_connections
What are the conventions around when and where to create datastore client objects?
datastore = new Datastore({});
In the docs a new Datastore instance seems to be created in every single file. Would there be any benefit in creating a singleton that initialises the Datastore connection and returns the same instance to each part of the application that requires it?
It depends on the underlying code if new Datastore({}) actually creates a new instance or returns a singleton, you'd have to check that.
What you could do is move the creation of the datastore instance to a seperate file and require that instance in every file you need access to datastore. Since dependencies you require are cached you will always get the same instance.
Pseudo code:
datastore.js
const datastore = new Datastore({});
module.exports = datastore;
foo.js
const datastore = require('./datastore');
// do something with datastore
In reply to your follow-up question.
If you look at the source code of the nodejs/Datastore module you will see the same pattern:
src/index.js
* #example <caption>Import the client library</caption>
* const Datastore = require('#google-cloud/datastore');
// ...
module.exports = Datastore;
No matter where you require the client library:
const Datastore = require('#google-cloud/datastore');
It will always return the same instance. Datastore will handle scaling and connections (pooling) for you.
In conclusion: There's no functional difference between requiring the client library in each file or wrapping it in a seperate file and require that in the files where you need a connection.
Personally, I prefer wrapping the connection in a seperate file and require that in my data access files. Benefits for this are:
* You abstract away the actual implementation. If you ever need to change datastore or the way you connect to it it will only ever be in one place.
* In case you need to supply connection parameters (like a password) you only have to do that once. It saves you from writing the same code over and over again.
I'm running a standalone script to manipulate some persisted data in my Loopback application. I need to do this on production servers while the production app is running. I have it working like this:
Filepath is ./scripts/my-script.js and looks like this:
'use strict';
const app = require('../server/server');
const Account = app.models.Account;
Account.find()
.then(accounts => {
// do data stuff with accounts
})
.then(() => process.exit())
.catch(error => {
console.error(error.stack);
process.exit(1);
});
Then I run node ./scripts/my-script.js.
The problem is the entire application including the web server, boot scripts, etc, runs for the duration of the script, which means I'd have two instances of the app running. This happens because I get reference to the Account model like this:
const app = require('../server/server');
const Account = app.models.Account;
When I try:
const loopback = require('loopback');
const Account = loopback.getModel('Account');
It fails with:
./node_modules/loopback/lib/registry.js:304
throw new Error('Model not found: ' + modelName);
^
Error: Model not found: Account
How can I run this script (or perhaps use a different approach) to manipulate persisted data without running a second instance of the application?
What's your end goal here? You could use the loopback-connector-remote to hit the same app instance through the REST API.
You could also perform operations on the same db through the second app instance.
A third option would be to create a model of class Model (instead of PersistedModel), define its datasource as null, and instead of running a second nodejs process, you could execute the script by hitting the associated endpoint. I would just make sure you use a good security policy for that specific endpoint.
I am using mongodb-native-driver in express.js app. I have around 6 collections in the database, so I have created 6 js files with each having a collection as a javascript object (e.g function collection(){}) and the prototypes functions handling all the manipulation on those collections. I thought this would be a good architecture.
But the problem I am having is how to connect to the database? Should I create a connection in each of this files and use them? I think that would be an overkill as the connect in mongodb-native-driver creates a pool of connections and having several of them would not be justified.
So how do I create a single connection pool and use it in all the collections.js files? I want to have the connection like its implemented in mongoose. Let me know if any of my thought process in architecture of the app is wrong.
Using Mongoose would solve these problems, but I have read in several places thats it slower than native-driver and also I would prefer a schema-less models.
Edit: I created a module out of models. Each collection was in a file and it took the database as an argument. Now in the index.js file I called the database connection and kept a variable db after I got the database from the connection. (I used the auto-reconnect feature to make sure that the connection wasn't lost). In the same index.js file I exported each of the collections like this
exports.model1 = require('./model1').(db)
exprorts.model2 = require('./model2').(db)
This ensured that the database part was handled in just one module and the app would just call function that each model.js file exported like save(), fincdbyid() etc (whatever you do in the function is upto you to implement).
how to connect to the database?
In order to connect using the MongoDB native driver you need to do something like the following:
var util = require('util');
var mongodb = require('mongodb');
var client = mongodb.MongoClient;
var auth = {
user: 'username',
pass: 'password',
host: 'hostname',
port: 1337,
name: 'databaseName'
};
var uri = util.format('mongodb://%s:%s#%s:%d/%s',
auth.user, auth.pass, auth.host, auth.port, auth.name);
/** Connect to the Mongo database at the URI using the client */
client.connect(uri, { auto_reconnect: true }, function (err, database) {
if (err) throw err;
else if (!database) console.log('Unknown error connecting to database');
else {
console.log('Connected to MongoDB database server at:');
console.log('\n\t%s\n', uri);
// Create or access collections, etc here using the database object
}
});
A basic connection is setup like this. This is all I can give you going on just the basic description of what you want. Post up some code you've got so far to get more specific help.
Should I create a connection in each of this files and use them?
No.
So how do I create a single connection pool and use it in all the collections.js files?
You can create a single file with code like the above, lets call it dbmanager.js connecting to the database. Export functions like createUser, deleteUser, etc. which operate on your database, then export functionality like so:
module.exports = {
createUser: function () { ; },
deleteUser: function () { ; }
};
which you could then require from another file like so:
var dbman = require('./dbmanager');
dbman.createUser(userData); // using connection established in `dbmanager.js`
EDIT: Because we're dealing with JavaScript and a single thread, the native driver indeed automatically handles connection pooling for you. You can look for this in the StackOverflow links below for more confirmation of this. The OP does state this in the question as well. This means that client.connect should be called only once by an instance of your server. After the database object is successfully retrieved from a call to client.connect, that database object should be reused throughout the entire instance of your app. This is easily accomplished by using the module pattern that Node.JS provides.
My suggestion is to create a module or set of modules which serves as a single point of contact for interacting with the database. In my apps I usually have a single module which depends on the native driver, calling require('mongodb'). All other modules in my app will not directly access the database, but instead all manipulations must be coordinated by this database module.
This encapsulates all of the code dealing with the native driver into a single module or set of modules. The OP seems to think there is a problem with the simple code example I've posted, describing a problem with a "single large closure" in my example. This is all pretty basic stuff, so I'm adding clarification as to the basic architecture at work here, but I still do not feel the need to change any code.
The OP also seems to think that multiple connections could possibly be made here. This is not possible with this setup. If you created a module like I suggest above then the first time require('./dbmanager') is called it will execute the code in the file dbmanager.js and return the module.exports object. The exports object is cached and is also returned on each subsequent call to require('./dbmanager'), however, the code in dbmanager.js will only be executed the first require.
If you don't want to create a module like this then the other option would be to export only the database passed to the callback for client.connect and use it directly in different places throughout your app. I recommend against this however, regardless of the OPs concerns.
Similar, possibly duplicate Stackoverflow questions, among others:
How to manage mongodb connections in nodejs webapp
Node.JS and MongoDB, reusing the DB object
Node.JS - What is the right way to deal with MongoDB connections
As accepted answer says - you should create only one connection for all incoming requests and reuse it, but answer is missing solution, that will create and cache connection. I wrote express middleware to achieve this - express-mongo-db. At first sight this task is trivial, and most people use this kind of code:
var db;
function createConnection(req, res, next) {
if (db) { req.db = db; next(); }
client.connect(uri, { auto_reconnect: true }, function (err, database) {
req.db = db = databse;
next();
});
}
app.use(createConnection);
But this code lead you to connection-leak, when multiple request arrives at the same time, and db is undefined. express-mongo-db solving this by holding incoming clients and calling connect only once, when module is required (not when first request arrives).
Hope you find it useful.
I just thought I would add in my own method of MongoDB connection for others interested or having problems with different methods
This method assumes you don't need authentication(I use this on localhost)
Authentication is still easy to implement
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var client = new MongoClient(new Server('localhost',27017,{
socketOptions: {connectTimeoutMS: 500},
poolSize:5,
auto_reconnect:true
}, {
numberOfRetries:3,
retryMilliseconds: 500
}));
client.open(function(err, client) {
if(err) {
console.log("Connection Failed Via Client Object.");
} else {
var db = client.db("theDbName");
if(db) {
console.log("Connected Via Client Object . . .");
db.logout(function(err,result) {
if(!err) {
console.log("Logged out successfully");
}
client.close();
console.log("Connection closed");
});
}
}
});
Credit goes to Brad Davley which goes over this method in his book (page 231-232)
Note: Please read the edited portion of this post before answering, it might save you time and answers one of my questions.
The problem I'm having is pretty simple but I'm pretty new to this overall and I'm having issues figuring out how to implement a mongodb database connection properly in a node/express app.
I'm using express 3.x and am basing my app layout around this project supplied by the author of Express:
https://github.com/expressjs/express/tree/d8caf209e38a214cb90b11ed59fd15b717b3f9bc/examples/blog (now removed from repo)
I have no interest in making a blog however the way the app is structured appears to be quite nice. The routes are separated and everything is organized nicely.
My problem is I might have 5-6 different route js files and each route js file might have anywhere between 1 and 15 routes; of those routes 1 or 15 might want to access the db.
So my problem is it seems like a really terrible idea to do a db.open(...) every single time I want to query the db. I should mention at this point I'm using the native mongo-db driver (npm install mongodb).
I would also need to include a file like this:
http://pastebin.com/VzFsPyax
...in all of those route files and all of my model files. Then I'm also dealing with dozens upon dozens of open connections.
Is there a way I can structure my app in such a way where I only make 1 connection and it stays open for the duration of the session (having a new one made every request would be bad too)?
If so, how can I do this? If you know the answer please post a code sample using tj's blog app (the one linked earlier in this post) structure as a base guide. Basically have a way where the routes and models can use the db freely while being in separate files than the db open code.
Thanks.
EDIT
I made some progress on solving one of my issues. If you look at tj's blog example he initializes his routes in the app.js like so:
require('./routes/site')(app);
require('./routes/post')(app);
And in the routes js file it starts like this:
module.exports = function(app){
I stumbled on a project earlier today where I saw someone pass 2 variables in the modules.exports call -> function(app, db). Then figured wow could it be that easy, do I need to just adjust my routes to be (app, db) too? Yeah, it seems so.
So now part 1 of the problem is solved. I don't have to require a mongo.js file with the connection boilerplate in every route file. At the same time it's flexible enough where I can decide to pick and choose which route files pass a db reference. This is standard and has no downside right?
Part 2 of the problem (the important one unfortunately) still exists though.
How can I bypass having to do a db.open(...) around every query I make and ideally only make a connection once per session?
Other solution is to pass database to the router via request, like this:
app.js
var db = openDatabase();
var app = express();
app.all('*', function(request, response, next)
{
request.database = db;
next();
});
app.get('/api/user/:id', Users.getByID);
users.js
var Users =
{
getByID: function(request, response)
{
request.database.collection('users').findOne(...)
response.send(user);
}
};
module.exports = Users;
I made a very simple module hub for this case that replaces the use of a global space.
In app.js you can create db connection once:
var hub = require('hub');
hub.db = new Db('foobar', new Server('10.0.2.15', 27017, {}), {native_parser: false});
And use it from any other files:
var hub = require('hub');
// hub.db - here link to db connection
This method uses a feature of 'require'. Module is only loaded for the first time and all the other calls gets a reference to an already loaded instance.
UPDATE
That's what I mean:
In main file like app.js we create Db connection, open it and store into hub:
app.js:
var hub = require('hub');
hub.mongodb = require('mongodb');
hub.mongodbClient = new hub.mongodb.Db('foobar', new hub.mongodb.Server('10.0.2.15', 27017, {}), {native_parser: false});
hub.mongodbClient.open(function(error) {
console.log('opened');
});
Now in any other file (message for example) we have access to opened connection and can simple use it:
message.js:
var hub = require('hub');
var collection = new hub.mongodb.Collection(hub.mongodbClient, 'message');
module.exports.count = function(cb) {
collection.count({}, function(err, count) {
cb(err, count);
});
};
Really silly. In the documentation it seems like db.open requires to be wrapped around whatever is using it, but in reality you can use it without a callback.
So the answer is to just do a db.open() in your database connection module, app.js file or where ever you decide to setup your db server/connection.
As long as you pass a reference to the db in the files using it, you'll have access to an "opened" db connection ready to be queried.