I have two sets of data in the same collection in cosmos, one are 'posts' and the other are 'users', they are linked by the posts users create.
Currently my structure is as follows;
// user document
{
id: 123,
postIds: ['id1','id2']
}
// post document
{
id: 'id1',
ownerId: 123
}
{
id: 'id2',
ownerId: 123
}
My main issue with this setup is the fungible nature of it, code has to enforce the link and if there's a bug data will very easily be lost with no clear way to recover it.
I'm also concerned about performance, if a user has 10,000 posts that's 10,000 lookups I'll have to do to resolve all the posts..
Is this the correct method for modelling entity relationships?
As said by David, it's a long discussion but it is a very common one so, since I have on hour or so of "free" time, I'm more than glad to try to answer it, once for all, hopefully.
WHY NORMALIZE?
First thing I notice in your post: you are looking for some level of referential integrity (https://en.wikipedia.org/wiki/Referential_integrity) which is something that is needed when you decompose a bigger object into its constituent pieces. Also called normalization.
While this is normally done in a relational database, it is now also becoming popular in non-relational database since it helps a lot to avoid data duplication which usually creates more problem than what it solves.
https://docs.mongodb.com/manual/core/data-model-design/#normalized-data-models
But do you really need it? Since you have chosen to use JSON document database, you should leverage the fact that it's able to store the entire document and then just store the document ALONG WITH all the owner data: name, surname, or all the other data you have about the user who created the document. Yes, I’m saying that you may want to evaluate not to have post and user, but just posts, with user info inside it.This may be actually very correct, as you will be sure to get the EXACT data for the user existing at the moment of post creation. Say for example I create a post and I have biography "X". I then update my biography to "Y" and create a new post. The two post will have different author biographies and this is just right, as they have exactly captured reality.
Of course you may want to also display a biography in an author page. In this case you'll have a problem. Which one you'll use? Probably the last one.
If all authors, in order to exist in your system, MUST have blog post published, that may well be enough. But maybe you want to have an author write its biography and being listed in your system, even before he writes a blog post.
In such case you need to NORMALIZE the model and create a new document type, just for authors. If this is your case, then, you also need to figure out how to handler the situation described before. When the author will update its own biography, will you just update the author document, or create a new one? If you create a new one, so that you can keep track of all changes, will you also update all the previous post so that they will reference the new document, or not?
As you can see the answer is complex, and REALLY depends on what kind of information you want to capture from the real world.
So, first of all, figure out if you really need to keep posts and users separated.
CONSISTENCY
Let’s assume that you really want to have posts and users kept in separate documents, and thus you normalize your model. In this case, keep in mind that Cosmos DB (but NoSQL in general) databases DO NOT OFFER any kind of native support to enforce referential integrity, so you are pretty much on your own. Indexes can help, of course, so you may want to index the ownerId property, so that before deleting an author, for example, you can efficiently check if there are any blog post done by him/her that will remain orphans otherwise.
Another option is to manually create and keep updated ANOTHER document that, for each author, keeps track of the blog posts he/she has written. With this approach you can just look at this document to understand which blog posts belong to an author. You can try to keep this document automatically updated using triggers, or do it in your application. Just keep in mind, that when you normalize, in a NoSQL database, keep data consistent is YOUR responsibility. This is exactly the opposite of a relational database, where your responsibility is to keep data consistent when you de-normalize it.
PERFORMANCES
Performance COULD be an issue, but you don't usually model in order to support performances in first place. You model in order to make sure your model can represent and store the information you need from the real world and then you optimize it in order to have decent performance with the database you have chose to use. As different database will have different constraints, the model will then be adapted to deal with that constraints. This is nothing more and nothing less that the good old “logical” vs “physical” modeling discussion.
In Cosmos DB case, you should not have queries that go cross-partition as they are more expensive.
Unfortunately partitioning is something you chose once and for all, so you really need to have clear in your mind what are the most common use case you want to support at best. If the majority of your queries are done on per author basis, I would partition per author.
Now, while this may seems a clever choice, it will be only if you have A LOT of authors. If you have only one, for example, all data and queries will go into just one partition, limiting A LOT your performance. Remember, in fact, that Cosmos DB RU are split among all the available partitions: with 10.000 RU, for example, you usually get 5 partitions, which means that all your values will be spread across 5 partitions. Each partition will have a top limit of 2000 RU. If all your queries use just one partition, your real maximum performance is that 2000 and not 10000 RUs.
I really hope this help you to start to figure out the answer. And I really hope this help to foster and grow a discussion (how to model for a document database) that I think it is really due and mature now.
I'm just putting my first steps in apps with MongoDB, and I'm not quite sure if what I'm doing is good practice or not.
I'm trying to do relationships between two collections. Let's say those will be User and User_settings collections. What I was going to do is adding field named user_id, that would contain obviously ID of document in User collection. And here come my concerns:
Should I use for it string format, or ObjectId format? I'm not sure it feels quite right to use ObjectId format. And that's because I experienced alot of problems while using this format in nodeJS apps. I had to convert objectId into string, in order to receive result I was looking for. Also, SailsJS is not allowing me to create new record with ObjectId fields. So my question is, can I use multiple ObjectIds in one record? I know I should do some googling, and I did. I'm just looking for 1-0 answer.
Thank you so much!
I'm building a small product system to try and get me into Node.js, Mongo and the like.
I was drawn to Mongo for the possibilities of storing products of different types in the same collection instead of having multiple tables, eav or any of the other MySQL-methods that didn't quite fit.
I was reading through tutorials and at first fell over writing a Provider for the ProductModel, but then someone suggested i should use Mongoose.
However, the issue with Mongoose seems to be that i need a set schema for each model. My first idea was that a product should have a title and a type attribute. Then type would decide the overall product structure. So for example a dvd product might have a director while a t-shirt product would not have a director but would have different sizes.
Is it possible to implement this kind of structure in Mongoose, and how? It's my understanding that if i provide a schema for each produt type i won't be able to query them simultaneously? Does different schemas mean different collection? - I couldn't quite figure that out.
Any help and guidance is appreciated, would like to be able to use a framework instead of writing the DB handling myself (although that what was i planning to do initially, but if someone else allready did it better than me :))
Mongoose has a schemaless type which can be very useful in your case, but might not be the best solution.
ProductSchema = new Schema({any: {}});
If you change anything on this object you need to notify Mongoose about it with obj.markModified('field-name') before saving.
My suggestion is to use your own wrapper over node-mongodb-native, which I'm running on production for 7~8 months without problems.
There's another lib called mongous that you can check it out.
I'm building an application on node.js that has users and products in a many-to-many relationship (one user has many products and the same product might belong to multiple users). Each user has also location info.
Mostly I need to do a lot of writes on the user first visit (a few writes on the following visits) and then I need to match users that, for instance, have the most number of products in common and return that same products in common. I may also want to match users by location (or sort them by matching location)
I'm using postgres right now but I think I would be better of doing mongo in the long run. Problem is that I never worked on NOSQL DB (no fears ;) )
The question is, is the following "schema" suited for the described above?
[user]{
_id
name
age
[location]{
streep
town
country
}
}
[products]{
_id
name
color
[users]{
user_id_1
user_id_2
user_id_3
}
}
I think, because of the requirements, I'm better of this way than with embeding. Am I right? Do you think I should store the products_id in the user document?
Thanks!!
Your data seems quite relational to me. I would not see a great advantage for MongoDB or NoSQL solutions. They work well for document-based solutions that aren't relational.
I would get some data if you're having problems with scaling or performance. Don't assume a solution until you know what the root cause is. It could be node.js - who knows? Some people don't care much for it.
I know this question has been asked a million times, but I can't seem to find one that really gives me a good understanding of how relationships work in Kohana's ORM Module.
I have a database with 5 tables:
approved_submissions
-submission_id
-contents
favorites
-user_id
-submission_id
ratings
-user_id
-submission_id
-rating
users
-user_id
votes
-user_id
-submission_id
-vote
Right now, favorites,ratings, and votes have a Primary Key that consists of every column in the table, so as to prevent a user favoriting the same submission_id multiple times, a user voting on the same submission_id multiple times etc. I also believe these fields are set up using foreign keys that reference approved_submissions and users so as to prevent invalid data existing in the respective fields.
Using the DB module, I can access and update these tables no problem. I really feel as though ORM may offer a more powerful and accessible way to accomplish the same things using less code.
Can you demonstrate how I might update a user voting on a submission_id? A user removing a favorite submission_id? A user changing their rating on a particular submission_id?
Also, do I need to make changes to my database structure or is it okay the way it is?
You're probably looking for has_many_through relationships.
So to add a new submission, you'd do something like
$user->add('submissions', $submission);
and to remove
$user->remove('submissions', $submission);
You may want to consider restructuring your database table and key names so you don't end up doing a lot of configuration.