NestJs Design Problem: How can I avoid creating a Nodejs Instance for each team? - node.js

I made a CRM app using NestJs with Nodejs. I designed it in a way that each team has its own database because every teams data is difference and has no relation with other teams and also it made the process of back up much easier.
However, Now that I want to deploy my service I noticed that for each team I must create a separate nodejs Instance which makes ram usage very high. Imagine just for 10 teams I may need around ~500MB ram which will hurt me economically even in short run.
Solutions
I used TypeORM in NestJs so the first thought I had was to find a way to have multiple databases (not multiple connections) having them sharing same schema but dynamicly use one of them based on request's scope and details. Which seems the best solution so I can avoid creating another NodeJs instance and in same time I now have seperate database for each team.
I read nestJs and TypeORM documents but didn't found any way to accomplish that. So my other solution was to just use one database for everone and add something like team_id column to each table to make a filter data for each team.
Is it a good way?
Is there any other solutions to use one nestJs instance but with same schema for multiple databases?

I recommend to use one database.
The database can have a table saving all of the teams and other tables will have a new team_id column as you think.
One database for each team has disadvantages.
Multiple DB Connections
Since you need to use same Entities for all of the databases for the teams, you cannot use Single Database Connection. According to every incoming API request, the server will have to switch db connections.
DB Configuration in TypeORM
For multiple databases, the configuration will be looking like below:
imports: [
...,
TypeOrmModule.forRoot({
name
type
host
port
username
password
...
}),
TypeOrmModule.forRoot({
name
type
host
port
username
password
}),
...
]
If you need to add a new team, you have to update your code base for adding a new db for the team and have to redeploy your application. (maybe you will create a new database and perform migration too?)
Backup
I agree with you that it's better to backup a single team with multiple databases. But how about when you want to backup all teams? In most of cases, I believe it will need to backup all teams, not just a specific team.
Teams Management
Where do you save a team's information? How to know what team has what db?
Maybe you saved teams somewhere(in a separated db?). To know which database connection should be used in each request, it needs to make a new query?
Cost
If there are 100 teams, you are gonna make 100 databases? Also each application has development and production environment. In some cases, there can be more environments like staging. 2 envs will double the number of dbs.
Conclusion
Of course there will be a way to automate some of the items in the above list and it's still possible to use multipe databases in NestJS + TypeORM for your project but it looks not a good way and not a worth effort for your project.
I have seen some big multi-tenant applications (like grafana) and they weren't using multiple databases strategy.

I don't know how you are storing users, but since you are speaking about teams I suppose you have a place where users are stored and assigned to a team, could it be a table in a login common database?
A solution could be to bind each team to it's own database; once a user login (accessing data from common login database) you read the team which it belongs and the database for its data, then you can access CRM data from the database bound to the team the user belongs.

Related

Multiple web services sharing single database using Sequelize

I'm trying to make a service architecture which includes two Node.js apps which shares the same database. The overall service architecture looks like below (simplified version)
I'm planning to use Sequelize as an ORM to access the database. As far as I know, if a service uses Sequelize, it needs model to get the structure of data tables. In my case, api and service will access the same database, which means they should share the same Sequelize model.
So here is the question: where should I locate the common Sequelize relevant files? It seems I have two choices:
put them on the upper common location (assuming the project structure is monorepo) so that each apps can use the single same files
maintain copies of files in each apps' project folders. In this case, each apps will be independent(Let's say I want to dockerize each apps) but in case the Sequelize files modified, the same action should be done for the other.
I'm not sure how I understood is correct. Is my question valid? If so, what is the better choice and practice? I appreciate for your answers in advance.
There is no correct answer, it depends on the specific situation, but sharing a database between multiple microservices is a bad design.
Sharing a database means tight coupling at the data level. The direct consequence is that when a service modifies the database table structure, such as deleting the name field of the user table, it may break the APIs of other services and all use the sequelize user model. All services need to update the model definition and modify the implementation code of the API.
If all of your services are maintained by a team, I suggest you choose the first solution, which costs less and is easier to maintain. If your services are maintained by different teams, the two solutions are actually similar, because as long as the table structure is modified, the application layer model needs to be modified or verified whether it still works well.
Therefore, I recommend following the best practices of microservice architecture, first splitting the database vertically according to the business model, and building application APIs on top of it.
Core principles of microservices:
loose coupling
high cohesion

mongoDB multiple instances or multiple databases

I chose to use docker with node-js and mongoDB to make a game, and set all of my game sections (battles, chat, resources, etc) into different servers so each section will have its own process, because for example my resource server will be running a lot as it will do all the calculation of the resources of each user every second and will handle requests from other servers, like if the user have enough resources to build a building or how much resources the player will lose if attacked etc.
For my alpha and beta versions i'm planning on running one server, that will run all the sections of my game, but the way i'm doing it now, every section is having its own mongodb instance, so i have resourcesDB, mainDB (users info, login info and stuff), all the instances won't have many collections, for example resourcesDB have only 2 collections, resource collections, which each user will have 1 document with stuff related to resources, and logs collection that will store all user's usage (building upgrades, battles lost etc.)
I have read about multi instances on the same servers, and they all said that its not the best way to do it, as it will have an impact on my performance, in the future i am planning on separate all of my sections into different servers so each section will have its own server and DB server if needed.
Should I maybe build 1 instance and separate it on the database level? that means that I will have to connect to the same instance from multiple servers, resource server will update all the user's resources count every second, combine that with battle server that will update the state of troops in DB and more for each war that is going on, i will also have chat database that will be updating with each private message or chat message, will it cause any issues? or it will cause problems only if I connect to a single database and separate it on the collections level?
Is there a reason why i should not continue with creating multiple instances? i read that there might be a problem with config files if using multiple instances on same server, but i assume docker handle that part.
UPDATE - i have found an excellent answer here: https://dba.stackexchange.com/questions/156811/mongodb-in-micro-services-structure/156984#156984
Using microservice architecture for your app is good idea but I don't see a good reason to separate the database. MongoDB can handle it for you (Sharding). Also MongoDB not block write to collection A when you write to collection B. if you will separate your app to multiple database it will be hard to do backups and to maintain the app.
You can read here about concurrency options for MongoDB and see how MongoDB handle collection level locking.
https://docs.mongodb.com/manual/faq/concurrency/

How can I switch between a live and a production database without duplicating code?

Here is my situation. I have an extensive REST based API that connects to a MongoDB database using Mongoose. The API is written as a standard "MEAN" stack application.
Currently, when a developer queries the API they're always connecting to the live production database. What I want to do is have an exact duplicate database as a "staging" database, where new data will be added first, vetted over a period of time, and then move to the live database. Then I want developers to be able to query either one simply by modifying their query.
I started looking into this with the Mongoose documentation, and it appears as though the models are tied to the DB connection, and if I want to have multiple connections I also have to have multiple models, one for each connection. This would be a nightmare of WET code and not the path I want to take.
What I want to do is not touch any of my code at all and simply have a switch that changes to the proper database for a given query. So my question is, how can I achieve this? Is it possible? The documentation seems to imply it is not.
Rather than trying to maintain connections two environments in the same code base have you considered setting up stage version of your application? Which database it connects to could be set through an environment variable or some other configuration option.
The developers would still then only have to make a change to query one or the other and you could migrate data from the stage database to production/live database once you have finished your vetting process.

Two nodejs applications, one mongodb database

Good day! I have created an application using nodejs + mongoose and now I want to make something like a superuser application. I need my admin panel application to connect to the same database. So, i have a question.
Should i store the same Schema file in both applications to have an ability to use my Schema methods? In other words, what is the best way to create one more API using the same db?
Thank you!
If I'm not mistaken, why not create another service that only interacts with the database? That way, the systems will refer to the same schema/DB regardless of which application you want to connect to it. So the superuser application and the normal application will just query the DB microservice that interacts the database.
Pro: source of truth for the schema for all applications and the DB queries will just be API calls
Con: additional overhead in creating your ecosystem
If you are using the same DB from two different applications, you will want to make sure those schemas are the same between the two. If one changes its inputs, the other might need to change its display (or risk not expecting all that information). Keep all this in mind during your release process.
I would suggest making the schemas an external library to both, or have the admin panel require the current app. You'll avoid getting two sets out of sync and know to look at one place for the schema definitions.

Can you have a DbContext that is associated with multiple databases?

I have a User database that houses all of the user information plus permissions to applications, etc. If I have a general database as described and then other databases for each Web Application, can I link up databases to make Relationships between the two databases using Fluent API or Code First? There are not so elegant ways to do this, but I wanted to ask the question first before getting involved with a custom solution.
For example: 1 DbContext, DbSets for each table in the 2 databases. Ability to relate entities between databases with Fluent API.
Thanks in advance.
The answer is no. The context is related to a single database. There is even no easy way to hack this because the context still can create queries only for a single database so if you want to have access to multiple databases you need either a context for each database (no cross context queries or relations exists) or you need to expose all tables from other databases as views or aliases in the database used by the context.

Resources