Document Role =
{ "_id" = "12345",
Name = "Developer"
},
{ "_id" = "67890",
Name = "Manager"
}
Document Employee =
{ "_id" = "00000",
"Name"= "Jack",
"Roles"= [{_id:"12345"},{_id:"67890"}]
}
I want to select one Role and list all the users having the same role
How to do that?
I want to get some thing like.
{ "_id" = "12345",
Name = "Developer"
Employees = [{"_id":"00000"}]
}
Is it possible to use populate to achieve this?
Mongoose .populate() and other methods you might find are not "join magic" for MongoDB. What they in fact all do is execute "additional" query(ies) operations on the database and "merge" the results "under the hood" for your as opposed to you doing the work yourself.
So your best option as long as you can deal with it is to use "embedding" which keeps the "related" information in the document for which you are "pairing" it to, such as for "Roles":
{
"_id": "0000",
"name": "Developer",
"employees": [{ "_id": "12345", "name": "Jack" }]
}
Which is simple, but of course comes at it's own cost and dealing with the "embedded" entries and how you use it according to "updating" or "reading" as is appropriate. It's a single "read" operation, but "updates" may be more costly due to the need to update the embedded information in multiple places, and multiple documents.
If you can "live" with "referencing" and the cost it incurs then you can always do this:
var rolesSchema = Schema({
"name": String,
"emloyees": [{ "type": Schema.Types.ObjectId, "ref": "Employee" }]
});
var employeesSchema = Schema({
"name": String,
"roles": [{ "type": Schema.Types.ObjectId, "ref": "Role" }]
});
var Role = mongoose.model('Role',rolesSchema);
var Employee = mongoose.model('Employee',employeeSchema);
Role.find({ "_id": "12345"}).populate("employees").exec(function(err,docs) {
// populated "joined" results in here
})
What this does behind the scenes is effectively (basic JavaScript representation and "at best") :
var roles = db.role.find({ "_id": "12345" }).map(function(doc) {
doc.employees = doc.employees.map(function(employee) {
return db.employees.find({ "_id": { "$in": doc.employees } }).toArray();
})
})
Mongoose works on the concept of using the "schema" definition to "know" which collection to execute the "other query" on and then return the "joined" results to you. But it is not a single query but multiple hits to the database.
Other schemes might "keep" the referenced collection information in the document itself, as opposed to relying on the "model code" to get that information. But the same principle applies where you need to make another call to the database and perform some type of "merge" in the API provided.
So it all falls down to your choice. Either you "embed" the data and live with that cost, or you "reference" the data and live with the network "cost" that is associated with multiple database hits.
The key point here is "nothing is free", and not even the way that SQL RDBMS perform "joins" which also has a "cost" of it's own and is a lot of the reasoning why NoSQL solutions like MongoDB do it this way and "do not support joins" in a native fashion for the "cost" involved in distributed data systems.
The main lesson here is to "do what suits you and your application", and not just choose the "coolest thing right now", but basically expect what you get from choosing different storage solutions. They all have their own purposes. Horses for Courses as the saying goes.
Related
Summary:
I am building my first large scale full stack application(MERN stack) that is trying to mimic a large scale clothing store. Each article of clothing has many 'tags' that represent its features, top/bottom/accessory/shoes/ect, and subcategories, for example on top there is shirt/outerwear/sweatshirt/etc, and sub-sub-categories within it, for example on shirt there is blouse/t-shirt/etc. Each article has tags for primary colors, hemline, pockets, technical features, the list goes on.
Main question:
how should I best organize the data in mongodb with mongoose schemas in order for it to be quickly searchable when I plan on having 50,000 or more articles? And genuinely curious, how do large clothing retailers typically design databases to be easily searchable by customers when items have so many identifying features?
Things I have tried or thought of:
On the mongoDB website there is a recommendation to use a tree structure with child references. here is the link: https://docs.mongodb.com/manual/tutorial/model-tree-structures-with-child-references/ I like this idea but I read here: https://developer.mongodb.com/article/mongodb-schema-design-best-practices/ that when storing over a few thousand pieces of data, using object ID references is no longer sufficient, and could create issues because of datalimits.
Further, each clothing item would fall into many different parts of the tree. For example it could be a blouse so it would be in the blouse "leaf" of the tree, and then if its blue, it would be in the blue "leaf" of the tree, and if it is sustainably sourced, it would fall into that "leaf" of the tree as well. Considering this, a tree like data structure seems not the right way to go. It would be storing the same ObjectID in many different leaves.
My other idea was to store the article information (description, price, and picture) seperate from the tagging/hierarchical information. Then each tagging object would have a ObjectID reference to the item. This way I could take advantage of the propogate method of mongoose if I wanted to collect that information.
I also created part of the large tree structure as a proof of concept for a design idea I had, and this is only for the front end right now, but this also creates bad searches cause they would look like taxonomy[0].options[0].options[0].options[0].title to get to 'blouse'. Which from my classes doesnt seem like a good way to make the code readable. This is only a snippet of a long long branching object. I was going to try to make this a mongoose schema. But its a lot of work and I wanna make sure that I do it well.
const taxonomy = [
{
title: 'Category',
selected: false,
options: [
{
title: 'top',
selected: false,
options: [
{
title: 'Shirt',
selected: false,
options: [
{
title: 'Blouse',
selected: false,
},
{
title: 'polo',
selected: false,
},
{
title: 'button down',
selected: false,
},
],
},
{
title: 'T-Shirt',
selected: false,
},
{
title: 'Sweater',
selected: false,
},
{
title: 'Sweatshirt and hoodie',
selected: false,
},
],
},
Moving forward:
I am not looking for a perfect answer, but I am sure that someone has tackled this issue before (all big businesses that sell lots of categorized products have) If someone could just point me in the right direction, for example, give me some terms to google, some articles to read, or some videos to watch, that would be great.
thank you for any direction you can provide.
MongoDB is a document based database. Each record in a collection is a document, and every document should be self-contained (it should contain all information that you need inside it).
The best practice would be to create one collection for each logical whole that you can think of. This is the best practice when you have documents with a lot of data, because it is scalable.
For example, you should create Collections for: Products, Subproducts, Categories, Items, Providers, Discounts...
Now, when you creating Schemas, instead of creating nested structure, you can just store a reference of one collection document as a property of another collection document.
NOTE: The maximum document size is 16 megabytes.
BAD PRACTICE
Let us first see what would be the bad practice. Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [{
"sub_product_name": "Subpoduct_name_1",
"sub_product_description": "Description",
"items": [{
"item_name": "item_name_1",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
}]
},
{
"item_name": "item_name_2",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
},
{
"discount_name": "Discount_2",
"percentage": 50
}]
},
]
},
...
]
}
Here product document has sub_products property which is an array of sub_products. Each sub_product has items, and each item has discounts. As you can see, because of this nested structure, the maximum document size would be quickly exceeded.
GOOD PRACTICE
Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [
'sub_product_1_id',
'sub_product_2_id',
'sub_product_3_id',
'sub_product_4_id',
'sub_product_5_id',
...
]
}
Subproduct = {
"id": "sub_product_1_id",
"sub_product_name": "Subroduct_name",
"sub_product_description": "Description",
"items": [
'item_1_id',
'item_2_id',
'item_3_id',
'item_4_id',
'item_5_id',
...
]
}
Item = {
"id": "item_1_id",
"item_name": "item_name_1",
"item_desciption": "Description",
"items": [
'discount_1_id',
'discount_2_id',
'discount_3_id',
'discount_4_id',
'discount_5_id',
...
]
}
Discount = {
"id": "discount_1_id",
"discount_name": "Discount_1",
"percentage": 25
}
Now, you have collection for each logical whole and you are just storing a reference of one collection document as a property of another collection document.
Now you can use one of the best features of the Mongoose that is called population. If you store a reference of one collection document as a property of another collection document, when performing querying of the database, Mongoose will replace references with the actual documents.
Currently I am working on a mobile app. Basically people can post their photos and the followers can like the photos like Instagram. I use mongodb as the database. Like instagram, there might be a lot of likes for a single photos. So using a document for a single "like" with index seems not reasonable because it will waste a lot of memory. However, I'd like a user add a like quickly. So my question is how to model the "like"? Basically the data model is much similar to instagram but using Mongodb.
No matter how you structure your overall document there are basically two things you need. That is basically a property for a "count" and a "list" of those who have already posted their "like" in order to ensure there are no duplicates submitted. Here's a basic structure:
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3")
"photo": "imagename.png",
"likeCount": 0
"likes": []
}
Whatever the case, there is a unique "_id" for your "photo post" and whatever information you want, but then the other fields as mentioned. The "likes" property here is an array, and that is going to hold the unique "_id" values from the "user" objects in your system. So every "user" has their own unique identifier somewhere, either in local storage or OpenId or something, but a unique identifier. I'll stick with ObjectId for the example.
When someone submits a "like" to a post, you want to issue the following update statement:
db.photos.update(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
"likes": { "$ne": ObjectId("54bb2244a3a0f26f885be2a4") }
},
{
"$inc": { "likeCount": 1 },
"$push": { "likes": ObjectId("54bb2244a3a0f26f885be2a4") }
}
)
Now the $inc operation there will increase the value of "likeCount" by the number specified, so increase by 1. The $push operation adds the unique identifier for the user to the array in the document for future reference.
The main important thing here is to keep a record of those users who voted and what is happening in the "query" part of the statement. Apart from selecting the document to update by it's own unique "_id", the other important thing is to check that "likes" array to make sure the current voting user is not in there already.
The same is true for the reverse case or "removing" the "like":
db.photos.update(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
"likes": ObjectId("54bb2244a3a0f26f885be2a4")
},
{
"$inc": { "likeCount": -1 },
"$pull": { "likes": ObjectId("54bb2244a3a0f26f885be2a4") }
}
)
The main important thing here is the query conditions being used to make sure that no document is touched if all conditions are not met. So the count does not increase if the user had already voted or decrease if their vote was not actually present anymore at the time of the update.
Of course it is not practical to read an array with a couple of hundred entries in a document back in any other part of your application. But MongoDB has a very standard way to handle that as well:
db.photos.find(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
},
{
"photo": 1
"likeCount": 1,
"likes": {
"$elemMatch": { "$eq": ObjectId("54bb2244a3a0f26f885be2a4") }
}
}
)
This usage of $elemMatch in projection will only return the current user if they are present or just a blank array where they are not. This allows the rest of your application logic to be aware if the current user has already placed a vote or not.
That is the basic technique and may work for you as is, but you should be aware that embedded arrays should not be infinitely extended, and there is also a hard 16MB limit on BSON documents. So the concept is sound, but just cannot be used on it's own if you are expecting 1000's of "like votes" on your content. There is a concept known as "bucketing" which is discussed in some detail in this example for Hybrid Schema design that allows one solution to storing a high volume of "likes". You can look at that to use along with the basic concepts here as a way to do this at volume.
I have something like:
Schema Subdocument
name: String
data: Mixed
Schema Stuff
documents: [Subdocument]
Now, in my API there are two endpoints, one for the Subdocument and another for Stuff. When I want to get a Subdocument I need to contain the data field, but when I want to get Stuff, I want to show the name of those subdocuments, but I don't want to show the data field because is quite large and it won't be used.
So, to keep things clear, data is not private. It's just that I don't want it to be shown when I get it from Stuff
I tried by doing:
Stuff.findById(id)
.populate("documents")
.populate("-documents.data")
but that doesn't work... I'm getting the Stuffwith the Subdocumentcontaining the name and data. I feel like i'm missing to tell mongoose when I call populate("-documents.data") that documents is an array and I want to exclude the data field for each element in this array.
edit: Sorry the Schema I provided was not for my case. In my case it was not embedded, but a reference, like so:
Schema Subdocument
name: String
data: Mixed
Schema Stuff
documents: [{
type: Schema.Types.ObjectId,
ref: 'Subdocument'
}]
Assuming subDocument is not embedded and using as "ref" as you say populate is working but data part is not included:
Stuff.findById(id).populate( { "path" : "documents", "select" : "-data" })
Your documents have an "embedded" schema, so populate is not used here, it is used only for "referenced" schemas where the other objects are in another collection.
Fortunately with "embedded" there is an easy way using projection:
Stuff.findById(id,{ "documents.name": 1 },function(err,results) {
})
With results like
{ "documents": [{ "name": "this" },{ "name": "that" }] }
Or with .aggregate() and the $map operator:
Stuff.aggregate(
[
{ "$match": { "_id": ObjectID(id) } },
{ "$project": {
"documents": {
"$map": {
"$input": "$documents",
"as": "el",
"in": "$$el.name"
}
}
}}
],function(err,results) {
}
)
That will just tranform into an array of "only" the name "values", which is different to the last form.
{ "documents": ["this", "that"] }
Note, if using .aggregate() you need to properly cast the ObjectId as the autocasting from mongoose schema types does not work in aggregation pipeline stages.
Pardon my ignorance if it is a very basic question.
ABC collection in MongoDB has the following schema.
{
"metadata": {
"store": 5051,
"catg": "XYZ",
},
"category": {
"name": "XYZ",
"id": "CL141778",
}
}
I need to query (where "metadata.catg" == "category.name")
What is the best way to do it without using mongoose dbref ?
In MongoDB you can not compare two fields values with each other in a query. What I expect is that you need to redesign your schema. You need to look at how you query (and update) as supposed to thinking MongoDB is yet another RDBMs.
Show us the real schema, some example documents and a few query types that you need to run and I can see if I can help.
In a contact management app, each user will have his own database. When users wish to share certain categories of contacts with others, a backend will initiate a replication. Each contact is its own document, but also has various children documents such as notes and appointments.
Here is an example...
Contact:
{
"_id": 123,
"type": "contact",
"owner": "jimmy",
"category": "customer",
"name": "Bob Jones",
"email": "bob#example.com"
}
Note:
{
"_id": 456,
"type": "note",
"owner": "jimmy",
"contact_id": 123,
"timestamp": 1383919278,
"content": "This is a note about Bob Jones"
}
So let's say Jimmy wants to share his only his customers with sales manager Kevin, while his personal contacts remain private. When the note passes through the replication filter, is it possible to access the linked contact's category field?
Or do I have to duplicate the category field in every single child of a contact? I would prefer not to have to do this, as each contact may have many children which I would have to update manually every time the category changes.
Here is some pseudo-code for the filter function:
function(doc, req)
{
if(doc.type == “contact”) {
if(doc.category == req.query.category) {
return true;
}
}
else if(doc.contact_id) {
if(doc.contact.category == req.query.category) {
return true;
}
}
return false;
}
If this is possible, please describe how to do it. Thanks!
There are some other options.
There's a not-so-well-known JOIN trick in CouchDB. Instead of using replication, however, you'll have to share the results of a MapReduce View -- unfortunately you can use a view as a filter for replication. If you're using Cloudant (disclaimer: I'm employed by Cloudant) you can use chained-MapReduce to output the result to another database that you could then replication from...
Additionally, I think this SO post/answer on document structures and this join trick could be helpful: Modeling relationships on CouchDB between documents?
No, this is not possible. Each document must be consistent so it has no any explicit relations with others documents. Having contact_id value as reference is just an agreement from your side - CouchDB isn't aware about this.
You need to literally have category document be nested within contact one to do such trick e.g. have single document to process by filter function. This is good solution from point when you need to have consistent state of contact document.