I would like to share my thoughts with you and try to get some advice. I would like to define my application with the best architecture as possible. Any comment would be highly appreciated. Here we go...
My technologies: NestJs(Node), neo4j/arangodb(graph DB), Nginx for proxy(Micro-services Approach).
My business case: SaaS application. Many customers with many users, one database per customer and the same code (just one instance) of our codebase.
we have a set of data models which will be same for all customer but a relation between them will differ. As per my research GraphDB is the best match for such operations. so I'm planning to create separate Instance/Database for each customer otherwise too many relations will make harder to scale.
Problem: From my point of view the problem can be seen with two different approach.
I need to allow multiple users to connect to different databases at the same time with the same code (just one installation). In Nestjs App how can I change the database configuration on each API request. Shall I save DB URI in a table, based on user/customer type it will fetch DB URI? then other concerns like does it affect on latency time, if any request failed then is there any possibility that request can fetch data from wrong DB?
How can we create sub-graphs in neo4j/arangodb so we can fetch sub-graph based on the customer.
On the other hand, I found a couple of interesting links:
https://neo4j.com/developer/multi-tenancy-worked-example/
https://www.arangodb.com/enterprise-server/oneshard/
https://dzone.com/articles/multitenant-graph-applications
Someone could provide me aditional info?
Thanks for your time
Best regards
With ArangoDB, a solution that works is:
Use a single database for all customers
Use Foxx microservices in that database to provide access to all data
Enforce a tenantId value on every call to Foxx
Use dedicated collections for each tenant in that database
Set up a web server (e.g. Node.js) in front of ArangoDB that serves data to all tenants
Only allow connections to Foxx from that front end web server
Each tenant will need a few collections, depending on your solution, try to keep that number as low as possible.
This model works pretty well, and you're able to migrate customers between instances / regions as their data is portable, because it's in collections.
Related
I am building a platform, where there is a master DB which holds all the user data, this db is used when user tries to login and after successful login application should fetch data from respective user DB. We are using MongoDB, mongoose driver and epxpress framework. Please guide
To answer your question:
You can use createConnection() to create multiple connections to different databases.
However, unless you'll only have a handful of users & databases (maybe 10, maybe more), this will very likely not scale very well, since mongoose is not made for this kind of usage. I'd rather advise to either combine the databases into a single one or if you'll have a lot of data or security concerns, create a multi-tenant system hosting an app-instance connected to each database.
I'm developing a mobile app using PouchDB (client-side) and CouchDB (server-side).
I need to secure docs in order to allow users to read/write his own documents only.
I did a filter for this, something like:
function(doc, req) {
return doc.owner == req.userCtx.name || doc.sharedWith == req.userCtx.name;
}
and it works well, but only if the request from client includes the filter:
/somedatabase/_alldocs?filter=filter/secure
I need CouchDB to use the filter in every request, with or without client explicitation, for obvious security reasons. Is this even possible? Otherwise which is the correct approch to handle these security issues?
There is a similar question here but the answer is not applicable in my case since I need to share docs between users and replicate them between all databases is not a valid option.
So I don't know if you have looked at this wiki but it lists few options available. Some of them are outdated tho.
Per user database
Probably the most popular solution. As you said, you need to share documents with other users. This could be done by :
Copy document to other users when sharing. You could have a deamon that listen to _changes feed and update the author file in other users database.
Build a web service to access shared documents (very similar to proxy solution)
Smart Proxy
Build a smart proxy in front of your database and do some business logic to fetch the documents. This gives you more control on your data flow but it will surely be slower.
Note
The validate_doc_read server function could interest you but it has never been part of CouchDB's releases(due to the listed limitations).
Uhm, probably it isn't. The app that we are developing need to share documents with different users. any doc could be shared with a different group of users
we are developing a big node app with express and MongoDB. We are trying to get the best performance, because we will have multiple clients (maybe 100+) running on the same server.
We were thinking in a one-to-n APP, one instance, one database and multiple clients accessing their domains.
I want to know what is the best settings for this scenario (one server, multiple clients) to performance and development
One instance, one database (clients data would be identified by a company ObjectId on the entry and clients would access a domain or subroute)
One instance, multiple tables (or databases, what is the best?)
Multiple instances, multiple tables
Any other ideas?
On the first setting, the developers will always worry about the current company and this can bring limitations to the app
On the second setting, the concern will continue but the company will not interfere on the database entries (more clean model)
On the third setting (maybe the best for development) only one company will be treated and brings a lot of possibilities, but may bring performance issues (all instances will run on a single server)
Other settings I have not thought of can be better.
Notes:
We are using the mongoose library
I have some experience with WordPress and i like the way themes and plugins are created for it. We are trying to achieve a level of performance similar to Wordpress with PHP (several Wordpress running on a server efficiently)
sorry about bad english
You don't need to manage multiple instance as you can create a company collection and in that collection you can store every single company and then you just need to create a reference of all these values in users.Please make sure that you have made unique index on company collection.It is really easy handle such scenarios in RDBMS(mysql).
And one more thing you can also run multiple mongod client on same instance by just changing the port and if you are looking for that sort of solution then you can do that as well.
Please note following things before using mongo:-
Please use mongo only if you have over TB's of data because that doesn't make any sense to use mongodb for some mb's or gb's of data.
Use of indexes is must in mongo if you want maximum performance.
Mongo stores all the indexes in main memory and if the indexes size is more then memory that it start swapping of indexes which is really costly and hence please make sure that you have different servers for your application and your db.
I still says it would be better to use RDBMS if you don't have TB's of data to deal with.
Why this approach:-
Let me give you a scenario.
You have 100 companies and with in 100 companies you have 1000 users for each of the company. i.e. you have 1L records in your user collection.Now i want to delete a single user or i want to update a user or i want to fetch a user from a single company then i don't need to traverse my complete database as i can make a index on my user collection using user-id and company id(compound index) or even i can make a simple filter query on company id.
For index please read this
https://docs.mongodb.com/manual/core/index-compound/
And btw we are not saving company id as an object instead i am saving only the value of _id from company collection.
I'm working on a simple game using SignalR 2 and MVC 5. I need to track number of "deaths" and "kills" for each user. These stats need to be read/write from multiple game instances (user can have multiple concurrent sessions) across multiple servers.
Is adding fields to ApplicationUser : IdentityUser a reasonable solution? I'm planning on using the built-in authentication system because I like how easy it is to support Facebook and other OAuth providers.
How should I update these stats in an optimized manor that reduces multi-user/threads/server issues and is highly scalable? The stats themselves are simple, and probably only update once every few seconds per user, but I'd like a design that can support millions of users across multiple servers.
For example, I know I could add this code inside an MVC controller to update the stats:
var um = HttpContext.GetOwinContext().GetUserManager<ApplicationUserManager>();
var user = um.FindById(User.Identity.GetUserId());
user.Deaths++;
um.Update(user);
However, that doesn't seem very safe/transactional. If another process/connection/server is updating that user at the same time, bad things are likely.
In a pure SQL design I'd probably have a stored procedure that runs in a SQL transaction to get current counter, and increment it. Not sure how to translate that to a good SignalR design that takes advantage of all that the various API layers have to offer (OWIN, MVC, ASP.NET, etc). Ideally something I can easily add Redis to down the road, if direct SQL access becomes an issue.
I have a PouchDB app that manages users.
Users have a local PouchDB instance that replicates with a single CouchDB database. Pretty simple.
This is where things get a bit complicated. I am introducing the concept of "groups" to my design. Groups will be different CouchDB databases but locally, they should be a part of the user database.
I was reading a bit about "fancy replication" in the pouchDB site and this seems to be the solution I am after.
Now, my question is, how do I do it? More specifically, How do I replicate from multiple remote databases into a single local one? Some code examples will be super.
From my diagram below, you will notice that I need to essentially add databases dynamically based on the groups the user is in. A critique of my design will also be appreciated.
Should the flow be something like this:
Retrieve all user docs from his/her DB into localUserDB
var groupDB = new PouchDB('remote-group-url');
groupDB.replicate.to(localUserDB);
(any performance issues with multiple pouchdb instances 0_0?)
Locally, when the user makes a change related to a specific group, we determine the corresponding database and replicate by doing something like:
localUserDB.replicate.to(groupDB) (Do I need filtered replication?)
Replicate from many remote databases to your local one:
remoteDB1.replicate.to(localDB);
remoteDB2.replicate.to(localDB);
remoteDB3.replicate.to(localDB);
// etc.
Then do a filtered replication from your local database to the remote database that is supposed to receive changes:
localDB.replicate.to(remoteDB1, {
filter: function (doc) {
return doc.shouldBeReplicated;
}
});
Why filtered replication? Because your local database contains documents from many sources, and you don't want to replicate everything back to the one remote database.
Why a filter function? Since you are replicating from the local database, there's no performance gain from using design docs, views, etc. Just pass in a filter function; it's simpler. :)
Hope that helps!
Edit: okay, it sounds like the names of the groups that the user belongs to are actually included in the first database, which is what you mean by "iterate over." No, you probably shouldn't do this. :) You are trying to circumvent CouchDB's built-in authentication/privilege system.
Instead you should use CouchDB's built-in roles, apply those roles to the user, and then use a "database per role" scheme to ensure users only have access to their proper group DBs. Users can always query the _users API to see what roles they belong to. Simple!
For more details, read the pouchdb-authentication README.