I am writing a small application. You could think of it like this:
Server
-- Security
-- Microservices
-- Others
The server routes to each of those. Each of those routes uses a different MongoDB database (same server).
I don't find the way to make a single, reusable connection to MongoDB server, that could be reused each route.
For example:
//server.js
//import modules etc
async function connectDB(){
const uri = process.env.MONGOURI;
const client = new MongoClient(uri, {
useNewUrlParser: true,
useUnifiedTopology: true,
});
return (await client.connect());
}
But I can not pass the returned client to any route, as the routes take no parameters:
//server.js
import security from "./security";
async function run() {
// security
app.use("security", security) //cant pass mongo client
If I write the client in the routes themselves:
//security.js
router.post("/register", async (req, res) => {
const database = await connectDB()
const users = database.collection("users");
I will have to run connectDB() to each route.
Question
Is there a way to create a connection in the server.js that will be reused in other routes?
Related
My overall goal is to apply the basic logic of the package "rate-limiter-flexible" for my Express Node.js application. I've gotten the basic functionality with the "Memory" version which does not require any communication with a database (IP address counts stored in Node server memory). Now I'm trying to get the functionality to work with the MongoDB rate limiter modules (using an instance of rateLimiterMongo object from the package).
The problem I'm encountering is that my rate limiter middleware is throwing an error intermittently... The only pattern I can find is that the error occurs more frequently if there are > ~10 seconds between requests to the Node app. This is the error which occurs every 5-10 requests:
[RateLimiter Error]: Rejection object:
TypeError: Cannot read properties of null (reading 'points')
at RateLimiterMongo._getRateLimiterRes (C:\root\node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:124:33)
at RateLimiterMongo._afterConsume (C:\root\node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:51:22)
at C:\root\node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:205:16
at processTicksAndRejections (node:internal/process/task_queues:96:5)
So far, I have tried:
Disabling buffering with Mongoose (was a recommendation from the package docs) -- did not work
Changing from MongoDB Atlas free tier to a locally hosted MongoDB instance -- this resolved all occurrences of the error, but I need to be able to use the cloud service
Here is a minimal reproduction of the error I'm facing when connecting to a MongoDB Atlas free tier cluster (via MONGO_DB_URL):
// Node packages:
require('dotenv').config();
const { RateLimiterMongo } = require('rate-limiter-flexible');
const mongoose = require('mongoose');
const express = require('express');
const app = express();
// open a Mongoose connection and save it:
const dbUrl = process.env.MONGO_DB_URL;
const connectDB = async function () {
await mongoose
.connect(dbUrl, {
// options
})
.catch(error => {
console.log("DB not connected!");
// handle error here (initial connection)
});
};
connectDB();
const mongoConn = mongoose.connection;
// options and passing to the RateLimiterMongo constructor:
const opts = {
storeClient: mongoConn,
points: 3, // Number of points
duration: 1, // Per second(s)
};
const rateLimiterMongo = new RateLimiterMongo(opts);
// create the middleware for the express app:
const rateLimiterMiddleware = (req, res, next) => {
rateLimiterMongo.consume(req.ip, 1)
.then((rateLimiterRes) => {
console.log("[RateLimiter Success]: RateLimiterRes object:\n", rateLimiterRes);
next();
// Allowed
})
.catch((rej) => {
console.log("[RateLimiter Error]: Rejection object:\n", rej);
res.status(500).send("RateLimiter error(s)...");
// Blocked
});
};
// Express app code:
app.use(rateLimiterMiddleware);
app.get('/', (req, res) => {
res.status(200).send("Valid Route!");
});
app.listen(3000, () => {
console.log(`Serving on port 3000!`);
});
Thanks all for any help you can provide with this. It may just be a side effect of using the MongoDBAtlas free tier...
Most likely, you use mongoose v6.x, which changes how connection established. It returns promise, which should be awaited, before connection can be used to make queries. More context in migration to 6 guide.
You should await connectDB() call and only then create middleware. In other words, your Express app should wait until connection to MongoDB Atlas is established. You can read through comments in related closed issue on github.
I have single repository of project and have multiple databases for different clients.
Following types of database naming and URL to connectivity that I am using to connect database based on access URL:
Client's Database
shreyas_db (http://shreyas.locahost:3001)
ajay_db (http://ajay.locahost:3001)
vijay_db (http://vijay.locahost:3001)
Please guide how to implement this structure in NodeJs with Express.
Thanks
Here is the solution I figure out:
Add a router
app.use('/api', await APIRouter())
connect to different DBS in router middleware.
Get the subdomain(If you are using Nginx, you may found req.subdomains or req.host does not return what you expected, try to use req.headers.referer instead), use res.locals to save DB, so you can get it from every API call.
export const APIRouter = async () => {
const router = express.Router()
const client = await connectDatabase()
router.use(async (req, res, next) => {
//const host = req.headers.referer?.replace("https://","");
//const subdomain = host ? host.substring(0, host.indexOf('.')) : 'main'
const subdomain = req.subdomains.length ? req.subdomains.length[0] : 'main'
const db = client.db(subdomain)
res.locals.db = {
clients: db.collection<Client>('clients'),
}
next()
}
)
return router
};
In my node.js server conf file, I set app.use('/user', user.js), which maps /user route to a user.js file.
Then I create subrouting in my user.js file to handle my get or post requests.
My question is: what's the responsibility of module.exports=router at the end of this file?
If I remove it, routing stops working, so I don't understand if it is here to tell my server conf file that there are sub paths in user.js?
var express = require('express');
var user = require('../../models/user');
var db = require('../../models/index');
var router = express.Router();
router.get('/addUser',function (req, res, next) {
db.user.create(req.body)
.then(user => res.json({
data: user,
}))
.catch(error => res.json({
error: true,
data: [],
error: error
}));
});
module.exports = router;
When you do
var user = require('../../models/user');
the user object would get whatever is being exported from the module in user.js. So in user.js, the module.exports=router is mapping a router and all logic that's required to map /user (along with the right callbacks etc...)
If you remove it, your require statement can't acquire an exported object from that module, which is why it would fail. Your user object will be nullified effectively.
Check out here for more info: https://www.tutorialsteacher.com/nodejs/nodejs-module-exports
A router cannot listen(PORT) for requests on its own. The router is useful when you have lots of routes. It's useful for separating your app into multiple modules.
const app = express()
app.listen(port)
app is listening for the requests(not the Router) while your user.js is just a separate js file with some codes.
In module.export ,module is a variable that represents the current module and export is an object. Anything you assign to the module.exports will be expose as a module.
copied: Module in Node.js is a simple or complex functionality organized in single or multiple JavaScript files which can be reused throughout the Node.js application.
Once you do module.export = Router Now you have a newly created module. In nodeJs, you need to require a module before use it. const user = require('./user.js') will do that process. Once you require the node module into your app, you need to tell it to execute by app.use('/' , user)
Or you can do something like below too
in your user.js file,
var user = require('../../models/user');
var db = require('../../models/index');
module.export = (app) =>{
app.get('/addUser',function (req, res, next) {
db.user.create(req.body)
.then(user => res.json({
data: user,
}))
.catch(error => res.json({
error: true,
data: [],
error: error
}));
});
}
in your main index.js,
const app = express()
require('./user.js')(app)
I am using express along with sequelize to create a REST API. I want to keep just one instance of my database and use it in all the routes I may create. My first thought was to create the instance as soon as the app runs. Something like this.
const express = require('express')
const databaseService = require( './services/database')
const config = require( './config')
const userRoute = require( './routes/user')
const app = express()
databaseService.connect(config.db_options).then(
connectionInstance => {
app.use('user', userRoute(connectionInstance))
app.listen(3000)
}
)
I have not seen something like that so I believe it's not a good approach.
Any ideas ?
A strategy that I use extensively in several production applications to great success is to define the database connection logic in a single file similar to the following:
database.js
const dbClient = require('some-db-client')
let db
module.exports.connectDB = async () => {
const client = await dbClient.connect(<DB_URL>)
db = client.db()
}
module.exports.getDB = () => {
return db
}
Obviously, the connection logic would need to be more complex then what I have defined above, but you can get the idea that you would connect to your database using whichever database client library that you choose and save a reference to that single instance of your database. Then, wherever you need to interface with your database, you can call the getDB function. But first, you would need to make a call to your connectDB function from your server application entry file.
server.js
async function run() {
const db = require('./database')
await db.connectDB()
const app = require('express')()
const routes = require('./routes')
app.use('/', routes)
app.listen(3000, () => {
console.log('App server listening to port 3000')
})
}
You would make the call to initialize your database connection in the server application entry file before initializing express. Afterwards, any place in your application that needs access to the database can simple call the getDB() function and use the returned database object as necessary from there. This does not require overly complicated connection open and close logic and saves on the overhead of constantly opening and closing connections whenever you want to access the database by maintaining just a single database connection that can be used anywhere.
I have a express router such as:
var express = require('express')
, router = express.Router()
, bodyParser = require('body-parser')
, mongoclient = require('mongodb').MongoClient
, dbconfig = require('../assets/config.db.json');
router.use( bodyParser.json() );
router.use(bodyParser.urlencoded({
extended: true
}));
router.post('/api/search', (req, res, next) => {
var term = req.body.searchTerm;
mongoclient.connect(dbconfig.url, (err, db) => {
db.accessDatabase(term, (err, result) => {
db.close();
if (err) res.json({result: err});
res.json(result);
});
});
});
module.exports = router;
I have have read that if there is a lot of overhead to make a connection to my db each REST call then I need to make a persistent connection for someone using this REST API. What is the proper way to do this within my router? Right now each time a post request is received it will open a connection, access the db and then close the connection.
EDIT: for clarity I used db.accessDatabase(), but this is not the actual syntax being used within my code.
You open do MongoClient.connect once when your app boots up and reuse the db object. It's not a singleton connection pool each .connect creates a new connection pool.
From node-mongodb-native
Personally, i do it like this
var params = {/*config, etc... required to bootstrap models */}
require('./app/models')(params)
.then(afterModels)
.then(startServer)
function afterModels(theModels) {
params.models = theModels
let providers = require('./app/providers')(params)
params.providers = providers
// boot strap routes
require('./app/routes')(params)
// controllers can use params which has models (db object,e tc.. inside)
}
function startServer() {}