I have a Node.js project dealing with multiple customers. For several reasons, mainly separation of concerns, security, and partial transferability, I have to put every customer's data into a separate Mongo database. All databases store the same object types and thus have the same set of collections with identical structure. I have already found many technical solutions to solve that issue, but my question is more concerned with the best practices handling this situation. A simplified view into my code (omitting the boilerplate code to create a server, module structure, etc.):
const mongoose = require('mongoose');
// Customer codes act as DB names, actually comes from config file or DB
let customers = ['c1', 'c2'];
// keeps all mongoose connections
let conns = {};
// create a connection(pool) for each DB
for (let i in customers) {
let dbname = customers[i];
conns[dbname] = mongoose.createConnection(`mongodb://${host}:${port}/${dbname}`);
}
// a simple Schema for blog articles
const ArticleSchema = new mongoose.Schema({
title: String,
text: String
});
// keeps all Article models
let Article = {};
// create the identical models for the different DBs
// remember: the dbname is the unique customer's code
for (let dbname in conns) {
let conn = conns[dbname];
Article[dbname] = conn.model('Article', ArticleSchema);
}
const app = require('express');
// returns all articles of a certain customer
app.get('/customers/:customerid/articles', function(req, res) {
// get the customer's code from the URL
let dbname = req.params.customerid;
// Query the database
Article[dbname].find({}, function(err, articles) {
if (err) return res.status(500).send('DB error');
res.status(200).send(articles);
});
});
This code works. Nonetheless, I wonder, if there ist a best practice I am not aware of to handle this kind of requirement. Especially, it feels strange to keep the connections and models in maps and to access the objects with the dbname in square brackets. Please keep in mind that this is an extremely simplified version. In the real version the code is distributed across several modules handling different object types etc.
Note: A simple mongoose.connection.useDb(dbName) to switch to a different database doesn't work since the model has to be registered with respect to the connection, which itself must be bound to a database (as far as I understand it).
Update: Handling every customer with a single node process and setting up a proxy switching to those processes with respect to the customer - as suggested in the question mentioned by Paul - is currently not an option due to the administrative effort necessary to set things up in our current environment. We have to launch the service quickly right now.
Related
I'm just starting learning node.js, express and mongoose.
In some tutorial I see some project structure like they have different folder for controller, different for services , different for data access layer.
Now my question is what is the difference in services and data access layer file, what we keep there? And where we keep data access layer file in my project structure?
Also what exactly the task of controller and routes ?
The routes files are where you place the application endpoints. Those will refer to the specific application methods you define. This is called routing.
app.post('/users', userController);
For clarification, in the example above we're calling the POST HTTP method for the /users route and redirecting the request to the userController method.
The controller layer is more specific. He is responsible for parsing the HTTP request data and sending it to the service layer.
For example:
async function userController(request, response) {
const { name, age } = request.body;
const serviceRequestBody = { name, age };
const serviceResponse = await userService(serviceRequestBody);
return response.json(serviceResponse);
}
The service layer is normally where you place the project rules (domain specific rules). For example: calculating the birth year based on the user age.
async function createUserService(userData) {
const birthYear = new Date().getFullYear() - userData.age;
const userFormatedData = {...userData, birthYear }; // the three dots means that we're getting all the information inside userData and placing it inside the new variable.
const dbResult = await userRepository(userFormatedData);
return dbResult;
}
The data access layer (can also be called "repository") is responsible for getting, posting, updating or deleting the information from the database.
async function userRepository(userInfo) {
const dbResult = await db.post(userInfo);
return dbResult;
}
For the project structure, its more up to you. I like to structure my projects like this:
-src
|-modules
| | -user // domain specific entities. If you have other entities, they will be inside another folder with the same structure as this one
| |-domain
| |-controllers
| |-repositories
| |-routes
| |-services
|-shared // can be used across any module
|-utils
|-providers
-package.json
-configFile.json
PS: Those abstractions may vary over time as you scale your application. It's the engineer's responsibility to figure out a better structure according to the case he's facing.
If you want to learn more about software engineering, search for Domain Driven Design (DDD), which is set of architecture rules that composes a good and scalable project.
I recently used nestjs, But I realized its overcomplicated, I mean look the following code:
#post('/products')
getAllProducts(#Body('title') title, #Body('price') price, #Body('description') description) { }
It makes function parameters much dirty, Also there could be more decorators above function like #Header, #Params, etc.. Which reduces readability in my opinion.
Same code in nodejs
const { title: title, price: price, description: description } = req.body
nodejs is much more readable...
Then I researched why developers use nestjs, Reason was Modularity. Why we don't implement this on our own...
See below:
See my directory sutructure
In app.js I just kicked the app:
const express = require('express');
const app = express();
// express config
require('./startup/config')(app);
// handling routes
require('./startup/routes')(app);
// db setup
require('./startup/db')(app);
In startup folder I did the basic work like mongoose configuration and connection to db etc..
However, In startup/routes, I just kicked the module as follow:
const shopModule = require('../shop/shop.module');
module.exports = app => {
app.use('/', shopModule);
};
In shop module, I just kicked the routes as follow:
const router = require('express').Router();
const productsRouter = require('./products/index');
const cartRouter = require('./cart/index');
// Products
router.use('/products', productsRouter)
// Cart
router.use('/cart', cartRouter)
module.exports = router;
Now in cart/index.js, I handled the routes related to cart and same for products as follow (I will just show cart):
const router = require('express').Router();
const { getCart } = require('./cart.controller');
router.get('/', getCart);
module.exports = router;
In controller, basically we will do validation stuff etc or extracting data.. Then controller will kick service for database work..
const { userCart } = require('./cart.service');
exports.getCart = (req, res, next) => {
const cart = userCart();
return res.status(200).json(cart);
};
And finally in cart service:
exports.userCart = _ => {
// ... go to database and fetch cart
return [{ prodId: 123, quantity: 2 }];
};
And cart.model.js is responsible for DB schema,
I know the question was too long, but I wanted to explain my question.
I am not saying nestjs should not be used, I am just saying, what about the following structure as it follows the same pattern as angular or nestjs, Right?
With your first point about making the code more readable, why not do something like
#Post('/products')
getAllProducts(#Body() body: any) {}
instead of calling for each part of the body individually, then you can deconstruct the body as you showed with
const {title: title, price: price, description: description} = body;
No need to specify each part of the body that you need as a new parameter, just grab the object itself. The same also goes for #Header(), #Param(), and #Query().
As for how you are setting up your express app, you are completely correct that you can do that; however, if you are working on an open source project, or collaborating with other developers there is nothing that says they have to follow the same format and it could eventually lead to a messy code base. Nest enforces these patterns, similar to how Angular does. Sure, it is still possible to write terrible code, but with an opinionated architecture it does make it more difficult.
NestJS also treats Typescript as a first class citizen, which in my opinion helps get rid of a lot of development problems as well. Plus you get some really cool packages to play with like class-validator and class-transformer to help with validation and transformation of your requests via pipes. You can write an Express server in Typescript, but it isn't necessary, and you can write a NestJS server in JavaScript, but I think you lose a lot of good functionality if you do.
The last point that I don't think has been touched too much it that Nest handles a lot of black boxing for your code through it's modules. If you define a service for a module, it is only available for that module. If you need it in another module you have options, but it helps cut down on the cross-contamination of code and keeps a separations of concerns ideology in mind.
In my opinion, the fact the NestJS gives us specified files for things like authentication (guards), validation and transformation (pipes and interceptors), and error handling (exception filters) makes it easier for anyone to pick up a NestJS server and have a quick understanding of how the request will flow through the server, even if by just looking at the file structure. Personally I know if I see an AuthModule or a guards folder I'm going to be dealing with authentication to some extent.
To answer your final question: there is nothing wrong with how you are showing the express example, especially as you are using Inversion of Control by passing the app into the router to make it work that way, you can absolutely write an Express server that way. Nest just makes you do it that way. (I mean, you could write an entire server using just AppController, AppService and AppModule, but that's really an anti-pattern)
In the end, definitely use what you're comfortable with, but there is a reason Nest has become so popular recently.
it's dirty because you write it in the wrong way
why not use it like this
#Get('/')
getAllProducts(#Body() product: Product) {}
and then destructure it
const {title, price, description} = product
I am new to Node JS technology. I have 3 basic doubts.
In my nodeJS application, I connected to mongodb using mongoose. But I did not mention any collection name. But data is getting saved when I sent data from Form as expected. I want to know that in which collection it will be stored by default. How to see the stored data.
how to mention specific collection name using mongoose if we want to save data in a particular collection.
3.Generally If we want to use any middleware in our app, we connect that
middleware using app.use() right? but in mongoose case, we do not add that
to app.use(). but still we can use the mongoose functionality.
could anyone please tell how it is possible.
Thanks a lot in advance.
How Mongoose interacts with MongoDB is described here.
It has this example:
var schema = new mongoose.Schema({ name: 'string', size: 'string' });
var Tank = mongoose.model('Tank', schema);
and mentions that
The first argument is the singular name of the collection your model is for. Mongoose automatically looks for the plural version of your model name. Thus, for the example above, the model Tank is for the tanks collection in the database.
model() takes a third argument where you can rename the collection:
var Tank = mongoose.model('Tank', schema, 'collectionname');
The collection gets made when model() is called.
app.use() is used for Express middleware. Mongoose isn't really that, which is why you're not using app.use() in this case.
This should probably be broken into multiple questions and you should probably show some code. That said, I'll take a crack at it.
Collection names are defined when you model your schema. So let's say you have:
const UserSchema = new Schema({
name: String
});
And then you later will tell mongoose to model it:
mongoose.model('User', UserSchema);
You'll have a collection called "users" in the database you're connecting to in your mongoose.connect() call.
Regarding middleware, Express middleware is specifically functions that you want to fire during the request/response cycle. You can still call code (e.g. mongoose) outside that cycle, and generally you'll connect to the database when the application starts and then read or save to it in either middleware or in your route handlers. For example, you might have a route like:
const User = mongoose.model('User');
app.get('/users', (req, res, next) => {
User.find({}, (err, users) => {
if (err) return next(err);
res.send(users);
});
});
In that case, you've got a route handler that calls mongoose through the User model previously defined.
Is there a way to use multiple databases with a single connection to mongodb? I've found this:
https://mongodb.github.io/node-mongodb-native/api-generated/mongoclient.html#open
but as best I can tell those docs are old as there does not appear to be an open method on the MongoClient? Do you actually need to establish multiple connections?
Thanks!
Found it:
http://mongodb.github.io/node-mongodb-native/2.0/api/Db.html#db
Here is their example
var MongoClient = require('mongodb').MongoClient,
test = require('assert');
MongoClient.connect('mongodb://localhost:27017/test', function(err, db) {
test.equal(null, err);
// Reference a different database sharing the same connections
// for the data transfer
var secondDb = db.db("integration_tests_2");
...
It is synchronous. Seems strange to me this method doesn't have the word "use" in it. Also seems strange it belongs to the db class. db.db('other_db').. a bit obscure. Did some tests, seems to work, so I'll mark this as the answer for anyone else that ends up here.
I'm building a server using SailsJS (a framework built on top of Express) and I need to keep an object in memory between requests. I would like to do this because loading it to/ from a database is taking way too long. Any ideas how I could do this?
Here's my code:
var params = req.params.all();
Network.findOne({ id: params.id }, function(err, network) {
if(network) {
var synapticNetwork = synaptic.Network.fromJSON(network.jsonValue);
if(synapticNetwork) { ...
Specifically, the fromJSON() function takes way too long and I would rather keep the synapticNetwork object in memory while the server is running (aka. load it when the server starts and just save periodically).
There are plenty libraries out there for caching purposes, one of which is node-cache as you've mentioned. All of them share similar api :
var cache = require('memory-cache');
// now just use the cache
cache.put('foo', 'bar');
console.log(cache.get('foo'))
You can also implement your own module and just require it wherever you need:
var cache = {};
module.exports = {
put: function(key, item) {
cache[key] = item;
},
get: function(key) {
return cache[key];
}
}
There are a lot of potential solutions. The first and most obvious one is using some session middleware for express. Most web frameworks should have some sort of session solution.
https://github.com/expressjs/session
The next option would be to use a caching utility like what Vsevolod suggested. It accomplishes pretty much the same thing as session, except if the data needs to be tied to a user/session then you'll have to store some kind of identifier in the session and use that to retrieve from the cache. Which I think is a bit redundant if that's your use-case.
There are also utilities that will expand your session middle-ware and persist objects in session to a database or other kinds of data stores, so that session information isn't lost even after server restarts. You still get the speed of an in-memory store, but backed by a database in case the in-memory store gets blown away.
Another option is to use Redis. You still have to serialize/deserialize your objects, but Redis is an in-memory data store and is super quick to write to and read from.