we are developing a big node app with express and MongoDB. We are trying to get the best performance, because we will have multiple clients (maybe 100+) running on the same server.
We were thinking in a one-to-n APP, one instance, one database and multiple clients accessing their domains.
I want to know what is the best settings for this scenario (one server, multiple clients) to performance and development
One instance, one database (clients data would be identified by a company ObjectId on the entry and clients would access a domain or subroute)
One instance, multiple tables (or databases, what is the best?)
Multiple instances, multiple tables
Any other ideas?
On the first setting, the developers will always worry about the current company and this can bring limitations to the app
On the second setting, the concern will continue but the company will not interfere on the database entries (more clean model)
On the third setting (maybe the best for development) only one company will be treated and brings a lot of possibilities, but may bring performance issues (all instances will run on a single server)
Other settings I have not thought of can be better.
Notes:
We are using the mongoose library
I have some experience with WordPress and i like the way themes and plugins are created for it. We are trying to achieve a level of performance similar to Wordpress with PHP (several Wordpress running on a server efficiently)
sorry about bad english
You don't need to manage multiple instance as you can create a company collection and in that collection you can store every single company and then you just need to create a reference of all these values in users.Please make sure that you have made unique index on company collection.It is really easy handle such scenarios in RDBMS(mysql).
And one more thing you can also run multiple mongod client on same instance by just changing the port and if you are looking for that sort of solution then you can do that as well.
Please note following things before using mongo:-
Please use mongo only if you have over TB's of data because that doesn't make any sense to use mongodb for some mb's or gb's of data.
Use of indexes is must in mongo if you want maximum performance.
Mongo stores all the indexes in main memory and if the indexes size is more then memory that it start swapping of indexes which is really costly and hence please make sure that you have different servers for your application and your db.
I still says it would be better to use RDBMS if you don't have TB's of data to deal with.
Why this approach:-
Let me give you a scenario.
You have 100 companies and with in 100 companies you have 1000 users for each of the company. i.e. you have 1L records in your user collection.Now i want to delete a single user or i want to update a user or i want to fetch a user from a single company then i don't need to traverse my complete database as i can make a index on my user collection using user-id and company id(compound index) or even i can make a simple filter query on company id.
For index please read this
https://docs.mongodb.com/manual/core/index-compound/
And btw we are not saving company id as an object instead i am saving only the value of _id from company collection.
Related
I would like to share my thoughts with you and try to get some advice. I would like to define my application with the best architecture as possible. Any comment would be highly appreciated. Here we go...
My technologies: NestJs(Node), neo4j/arangodb(graph DB), Nginx for proxy(Micro-services Approach).
My business case: SaaS application. Many customers with many users, one database per customer and the same code (just one instance) of our codebase.
we have a set of data models which will be same for all customer but a relation between them will differ. As per my research GraphDB is the best match for such operations. so I'm planning to create separate Instance/Database for each customer otherwise too many relations will make harder to scale.
Problem: From my point of view the problem can be seen with two different approach.
I need to allow multiple users to connect to different databases at the same time with the same code (just one installation). In Nestjs App how can I change the database configuration on each API request. Shall I save DB URI in a table, based on user/customer type it will fetch DB URI? then other concerns like does it affect on latency time, if any request failed then is there any possibility that request can fetch data from wrong DB?
How can we create sub-graphs in neo4j/arangodb so we can fetch sub-graph based on the customer.
On the other hand, I found a couple of interesting links:
https://neo4j.com/developer/multi-tenancy-worked-example/
https://www.arangodb.com/enterprise-server/oneshard/
https://dzone.com/articles/multitenant-graph-applications
Someone could provide me aditional info?
Thanks for your time
Best regards
With ArangoDB, a solution that works is:
Use a single database for all customers
Use Foxx microservices in that database to provide access to all data
Enforce a tenantId value on every call to Foxx
Use dedicated collections for each tenant in that database
Set up a web server (e.g. Node.js) in front of ArangoDB that serves data to all tenants
Only allow connections to Foxx from that front end web server
Each tenant will need a few collections, depending on your solution, try to keep that number as low as possible.
This model works pretty well, and you're able to migrate customers between instances / regions as their data is portable, because it's in collections.
I created couchdb with multiple dbs for use in my ionic 3 app. Also upon integrating it with pouchdb for client side syncing i created seperate pouchdbs for each one of the dbs. Total 5 pouchdbs. My question
whether it is good idea storing multiple pouchdbs on client side owing to the no. of http connections that would be created by syncing the pouchdbs. Or shall I put all Couchdb databases into one database and use type fields to separate the docs. Then only one pouchdb need to be created and synced on client.
Also using pouchdb-authenticaion plugin, authentication data is valid for only the database on which signup/login methods were called. Accessing other databases returns unauthenticated.
I would say, if your pouchdbs are syncing in realtime, that should be less expensive to reduce their amount to one and distinguish records by type.
But it should not be that costly, but still very convinient to set up multiple changes feed per each ItemStore (e.g. TodoStore, CommentStore, etc) with corresponding filter function passing only docs of the matching type into the store it belongs to. It can also be achieved by filtering on the basis of design_docs (I'm not sure if it saves anything, at least in the browser)
One change feed distributing docs to store would be probably the cheapest solution. But I suppose the filter function can't be changes after the change feed was established, so it must know about all the stores (i.e. doc types) beforehand
I have a PouchDB app that manages users.
Users have a local PouchDB instance that replicates with a single CouchDB database. Pretty simple.
This is where things get a bit complicated. I am introducing the concept of "groups" to my design. Groups will be different CouchDB databases but locally, they should be a part of the user database.
I was reading a bit about "fancy replication" in the pouchDB site and this seems to be the solution I am after.
Now, my question is, how do I do it? More specifically, How do I replicate from multiple remote databases into a single local one? Some code examples will be super.
From my diagram below, you will notice that I need to essentially add databases dynamically based on the groups the user is in. A critique of my design will also be appreciated.
Should the flow be something like this:
Retrieve all user docs from his/her DB into localUserDB
var groupDB = new PouchDB('remote-group-url');
groupDB.replicate.to(localUserDB);
(any performance issues with multiple pouchdb instances 0_0?)
Locally, when the user makes a change related to a specific group, we determine the corresponding database and replicate by doing something like:
localUserDB.replicate.to(groupDB) (Do I need filtered replication?)
Replicate from many remote databases to your local one:
remoteDB1.replicate.to(localDB);
remoteDB2.replicate.to(localDB);
remoteDB3.replicate.to(localDB);
// etc.
Then do a filtered replication from your local database to the remote database that is supposed to receive changes:
localDB.replicate.to(remoteDB1, {
filter: function (doc) {
return doc.shouldBeReplicated;
}
});
Why filtered replication? Because your local database contains documents from many sources, and you don't want to replicate everything back to the one remote database.
Why a filter function? Since you are replicating from the local database, there's no performance gain from using design docs, views, etc. Just pass in a filter function; it's simpler. :)
Hope that helps!
Edit: okay, it sounds like the names of the groups that the user belongs to are actually included in the first database, which is what you mean by "iterate over." No, you probably shouldn't do this. :) You are trying to circumvent CouchDB's built-in authentication/privilege system.
Instead you should use CouchDB's built-in roles, apply those roles to the user, and then use a "database per role" scheme to ensure users only have access to their proper group DBs. Users can always query the _users API to see what roles they belong to. Simple!
For more details, read the pouchdb-authentication README.
I am in the process of designing document storage for CouchDB and would really appreciate some feedback. These documents are to represent "assets".
These databases will also be synced locally to the browser via pouchdb.
Requirements:
Each user can have many assets
Users can share assets with others by providing them with a URI such as (xyz.com/some_id). Once users click this URI, they are considered to have been "joined" and are now part of a group.
Group users can share assets of their own with other members of the group.
My design
Each user will have his/her own database to store assets - let's call it "user". Each user DB will be prefixed with the his/her unique ID.
Shared assets will be stored in a separate database - let's call it "group". shared assets are DUPLICATED here and have an additional field for userId (to indicate creator).
Group database is prefixed with a unique ID just like a user database is prefixed with one too.
The reason for storing group assets in a separate database is because when pouchdb runs locally, it only knows about the current user and his/her shared assets. It does not know about other users and will should not query these "other" users' databases.
Any input would be GREATLY appreciated.
Seems like a great design. Another alternative would be to just have one database per group ("role"), and then replicate from a user's group(s) into their local PouchDB.
That might get hairy, though, when it comes time to replicate back to the server, because you're going to have to filter the documents as they leave the user's local database, depending on which group-database they belong to. Still, you're going to have to do that on the server side anyway with your current design.
Either way is fine, honestly. The only downside of your current approach is that documents are duplicated on the server side (once per user-db and once per group-db). On the other hand, your client code becomes dead-simple, because you don't have to do any filtered replication. If you have enough space on your server not to worry about it, then I would definitely go with your approach. :)
say i have two collections in mongodb,one for users which contains users basic info,and one for apps which contains applications.now if users are allowed to add apps,and the next time when they login, the web should get the user added app for them.how should i construct this kind of database in mongodb.
users:{_id:ObjectId(),username:'username',password:'password'}
apps:{_id:ObjectId(),appname:'',developer:,description:};
and how should i get the user added app for them??? should i add something like addedAppId like:
users:{_id:ObjectId(),username:'username',password:'password',addedApppId:[]}
to indicate which app they have added,and then get the apps using addedAppId???
Yep, there's nothing wrong with the users collection keeping track of which apps a user has added like you've indicated.
This is known as linking in MongoDB. In a relational system you'd probably create a separate table, added_apps, which had a user ID, app ID and any other relevant information. But since you can't join, keeping this information in the users collection is entirely appropriate.
From the docs:
A key question when designing a MongoDB schema is when to embed and when to link. Embedding is the nesting of objects and arrays inside a BSON document. Links are references between documents.
There are no joins in MongoDB – distributed joins would be difficult on a 1,000 server cluster. Embedding is a bit like "prejoined" data. Operations within a document are easy for the server to handle; these operations can be fairly rich. Links in contrast must be processed client-side by the application; the application does this by issuing a follow-up query.
(this is the extra bit you'd need to do, fetch app information from the user's stored AppId.)
Generally, for "contains" relationships between entities, embedding should be be chosen. Use linking when not using linking would result in duplication of data.
...
Many to many relationships are generally done by linking.