Currently I am working on a mobile app. Basically people can post their photos and the followers can like the photos like Instagram. I use mongodb as the database. Like instagram, there might be a lot of likes for a single photos. So using a document for a single "like" with index seems not reasonable because it will waste a lot of memory. However, I'd like a user add a like quickly. So my question is how to model the "like"? Basically the data model is much similar to instagram but using Mongodb.
No matter how you structure your overall document there are basically two things you need. That is basically a property for a "count" and a "list" of those who have already posted their "like" in order to ensure there are no duplicates submitted. Here's a basic structure:
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3")
"photo": "imagename.png",
"likeCount": 0
"likes": []
}
Whatever the case, there is a unique "_id" for your "photo post" and whatever information you want, but then the other fields as mentioned. The "likes" property here is an array, and that is going to hold the unique "_id" values from the "user" objects in your system. So every "user" has their own unique identifier somewhere, either in local storage or OpenId or something, but a unique identifier. I'll stick with ObjectId for the example.
When someone submits a "like" to a post, you want to issue the following update statement:
db.photos.update(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
"likes": { "$ne": ObjectId("54bb2244a3a0f26f885be2a4") }
},
{
"$inc": { "likeCount": 1 },
"$push": { "likes": ObjectId("54bb2244a3a0f26f885be2a4") }
}
)
Now the $inc operation there will increase the value of "likeCount" by the number specified, so increase by 1. The $push operation adds the unique identifier for the user to the array in the document for future reference.
The main important thing here is to keep a record of those users who voted and what is happening in the "query" part of the statement. Apart from selecting the document to update by it's own unique "_id", the other important thing is to check that "likes" array to make sure the current voting user is not in there already.
The same is true for the reverse case or "removing" the "like":
db.photos.update(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
"likes": ObjectId("54bb2244a3a0f26f885be2a4")
},
{
"$inc": { "likeCount": -1 },
"$pull": { "likes": ObjectId("54bb2244a3a0f26f885be2a4") }
}
)
The main important thing here is the query conditions being used to make sure that no document is touched if all conditions are not met. So the count does not increase if the user had already voted or decrease if their vote was not actually present anymore at the time of the update.
Of course it is not practical to read an array with a couple of hundred entries in a document back in any other part of your application. But MongoDB has a very standard way to handle that as well:
db.photos.find(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
},
{
"photo": 1
"likeCount": 1,
"likes": {
"$elemMatch": { "$eq": ObjectId("54bb2244a3a0f26f885be2a4") }
}
}
)
This usage of $elemMatch in projection will only return the current user if they are present or just a blank array where they are not. This allows the rest of your application logic to be aware if the current user has already placed a vote or not.
That is the basic technique and may work for you as is, but you should be aware that embedded arrays should not be infinitely extended, and there is also a hard 16MB limit on BSON documents. So the concept is sound, but just cannot be used on it's own if you are expecting 1000's of "like votes" on your content. There is a concept known as "bucketing" which is discussed in some detail in this example for Hybrid Schema design that allows one solution to storing a high volume of "likes". You can look at that to use along with the basic concepts here as a way to do this at volume.
Related
Summary:
I am building my first large scale full stack application(MERN stack) that is trying to mimic a large scale clothing store. Each article of clothing has many 'tags' that represent its features, top/bottom/accessory/shoes/ect, and subcategories, for example on top there is shirt/outerwear/sweatshirt/etc, and sub-sub-categories within it, for example on shirt there is blouse/t-shirt/etc. Each article has tags for primary colors, hemline, pockets, technical features, the list goes on.
Main question:
how should I best organize the data in mongodb with mongoose schemas in order for it to be quickly searchable when I plan on having 50,000 or more articles? And genuinely curious, how do large clothing retailers typically design databases to be easily searchable by customers when items have so many identifying features?
Things I have tried or thought of:
On the mongoDB website there is a recommendation to use a tree structure with child references. here is the link: https://docs.mongodb.com/manual/tutorial/model-tree-structures-with-child-references/ I like this idea but I read here: https://developer.mongodb.com/article/mongodb-schema-design-best-practices/ that when storing over a few thousand pieces of data, using object ID references is no longer sufficient, and could create issues because of datalimits.
Further, each clothing item would fall into many different parts of the tree. For example it could be a blouse so it would be in the blouse "leaf" of the tree, and then if its blue, it would be in the blue "leaf" of the tree, and if it is sustainably sourced, it would fall into that "leaf" of the tree as well. Considering this, a tree like data structure seems not the right way to go. It would be storing the same ObjectID in many different leaves.
My other idea was to store the article information (description, price, and picture) seperate from the tagging/hierarchical information. Then each tagging object would have a ObjectID reference to the item. This way I could take advantage of the propogate method of mongoose if I wanted to collect that information.
I also created part of the large tree structure as a proof of concept for a design idea I had, and this is only for the front end right now, but this also creates bad searches cause they would look like taxonomy[0].options[0].options[0].options[0].title to get to 'blouse'. Which from my classes doesnt seem like a good way to make the code readable. This is only a snippet of a long long branching object. I was going to try to make this a mongoose schema. But its a lot of work and I wanna make sure that I do it well.
const taxonomy = [
{
title: 'Category',
selected: false,
options: [
{
title: 'top',
selected: false,
options: [
{
title: 'Shirt',
selected: false,
options: [
{
title: 'Blouse',
selected: false,
},
{
title: 'polo',
selected: false,
},
{
title: 'button down',
selected: false,
},
],
},
{
title: 'T-Shirt',
selected: false,
},
{
title: 'Sweater',
selected: false,
},
{
title: 'Sweatshirt and hoodie',
selected: false,
},
],
},
Moving forward:
I am not looking for a perfect answer, but I am sure that someone has tackled this issue before (all big businesses that sell lots of categorized products have) If someone could just point me in the right direction, for example, give me some terms to google, some articles to read, or some videos to watch, that would be great.
thank you for any direction you can provide.
MongoDB is a document based database. Each record in a collection is a document, and every document should be self-contained (it should contain all information that you need inside it).
The best practice would be to create one collection for each logical whole that you can think of. This is the best practice when you have documents with a lot of data, because it is scalable.
For example, you should create Collections for: Products, Subproducts, Categories, Items, Providers, Discounts...
Now, when you creating Schemas, instead of creating nested structure, you can just store a reference of one collection document as a property of another collection document.
NOTE: The maximum document size is 16 megabytes.
BAD PRACTICE
Let us first see what would be the bad practice. Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [{
"sub_product_name": "Subpoduct_name_1",
"sub_product_description": "Description",
"items": [{
"item_name": "item_name_1",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
}]
},
{
"item_name": "item_name_2",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
},
{
"discount_name": "Discount_2",
"percentage": 50
}]
},
]
},
...
]
}
Here product document has sub_products property which is an array of sub_products. Each sub_product has items, and each item has discounts. As you can see, because of this nested structure, the maximum document size would be quickly exceeded.
GOOD PRACTICE
Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [
'sub_product_1_id',
'sub_product_2_id',
'sub_product_3_id',
'sub_product_4_id',
'sub_product_5_id',
...
]
}
Subproduct = {
"id": "sub_product_1_id",
"sub_product_name": "Subroduct_name",
"sub_product_description": "Description",
"items": [
'item_1_id',
'item_2_id',
'item_3_id',
'item_4_id',
'item_5_id',
...
]
}
Item = {
"id": "item_1_id",
"item_name": "item_name_1",
"item_desciption": "Description",
"items": [
'discount_1_id',
'discount_2_id',
'discount_3_id',
'discount_4_id',
'discount_5_id',
...
]
}
Discount = {
"id": "discount_1_id",
"discount_name": "Discount_1",
"percentage": 25
}
Now, you have collection for each logical whole and you are just storing a reference of one collection document as a property of another collection document.
Now you can use one of the best features of the Mongoose that is called population. If you store a reference of one collection document as a property of another collection document, when performing querying of the database, Mongoose will replace references with the actual documents.
I need advice from experienced NoSQL engineers on how I should structure my data.
I want to model my SQL data structure to NoSQL for Google Cloud Firestore.
I have no prior experience with NoSQL databases but I am proficient with traditional SQL.
I use Node.js for writing queries.
So far, I converted three tables to JSON documents with example data:
{
"session": {
"userId": 99992222,
"token": "jwttoken1191891j1kj1khjjk1hjk1kj1",
"created": "timestamp"
}
}
{
"user": {
"id": 99992222,
"username": "userName",
"avatarUrl": "https://url-xxxx.com",
"lastLogin": "2019-11-23 13:59:48.884549",
"created": "2019-11-23 13:59:48.884549",
"modified": "2019-11-23 13:59:48.884549",
"visits": 1,
"profile": true,
"basketDetail": { // I get this data from a third party API
"response": {
"product_count": 2,
"products": [
{
"product_id": 111,
"usageInMinutes_recent": 0,
"usageInMinutes": 0,
"usageInMinutes_windows": 0,
"usageInMinutes_mac": 0,
"usageInMinutes_linux": 0
},
{
"product_id": 222, // no recent usage here
"usageInMinutes": 0,
"usageInMinutes_windows": 0,
"usageInMinutes_mac": 0,
"usageInMinutes_linux": 0
}
]
}
}
}
}
{
"visitor": {
"id": 999922221,
"created": "2019-11-23 13:59:48.884549"
}
}
My questions:
session.userId, user.id, visitor.id can all signify the same user. What is the Firestore equivalent to foreign keys in SQL? How would I connect/join these three collections in a query?
What do I do about the nested object basketDetail? Is it fine where it is or should I define its own collection?
I anticipate queries
occasionally add all the recent usage.
frequently check if a user owns a specific product_id
frequently replace the whole baskedDetail object with new data.
occasionally update one specific product_id.
How would I connect collections user with basketDetail in a query if I separated it?
Thanks for the advice!
session.userId, user.id, visitor.id can all signify the same user. What is the Firestore equivalent to foreign keys in SQL? How would I connect/join these three collections in a query?
Unfortunately, there is not JOIN clause in Firestore. Queries in Firestore are shallow, can only get elements from the collection that the query is run against. There is no way you can get documents from two collections in a single query unless you are using collection group query, but it's not the case since the collections in your project have different names.
If you have three collections, then three separate queries are required. There is no way you can achieve that in a single go.
What do I do about the nested object basketDetail? Is it fine where it is or should I define its own collection?
There are some limits when it comes to how much data you can put into a document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. So if you think that nested object basketDetail can stay within this limitation then you can use that schema, otherwise, add it to a subcollection. Besides that, all those operations are permitted in Firestore. If you'll have hard times implementing them, post another question so we can take a look at it.
How would I connect collections user with basketDetail in a query if I separated it?
You cannot connect/join two collections. If you separate basketDetail in a subcollection, then two queries are required.
Here's my query as it stands:
"query":{
"fuzzy":{
"author":{
"value":query,
"fuzziness":2
},
"career_title":{
"value":query,
"fuzziness":2
}
}
}
This is part of a callback in Node.js. Query (which is being plugged in as a value to compare against) is set earlier in the function.
What I need it to be able to do is to check both the author and the career_title of a document, fuzzily, and return any documents that match in either field. The above statement never returns anything, and whenever I try to access the object it should create, it says it's undefined. I understand that I could write two queries, one to check each field, then sort the results by score, but I feel like searching every object for one field twice will be slower than searching every object for two fields once.
https://www.elastic.co/guide/en/elasticsearch/guide/current/fuzzy-match-query.html
If you see here, in a multi match query you can specify the fuzziness...
{
"query": {
"multi_match": {
"fields": [ "text", "title" ],
"query": "SURPRIZE ME!",
"fuzziness": "AUTO"
}
}
}
Somewhat like this.. Hope this helps.
I'm learning Mongodb/Mongoose/Express and have come across a fairly complex query (relative to my current level of understanding anyway) that I'm not sure how best to approach. I have a collection - to keep it simple let's call it entities - with an embedded actions array:
name: String
actions: [{
name: String
date: Date
}]
What I'd like to do is to return an array of documents with each containing the most recent action (or most recent to a specified date), and the next action (based on the same date).
Would this be possible with one find() query, or would I need to break this down into multiple queries and merge the results to generate one result array? I'm looking for the most efficient route possible.
Provided that your "actions" are inserted with the "most recent" being the last entry in the list, and usually this will be the case unless you are specifically updating items and changing dates, then all you really want to do is "project" the last item of the array. This is what the $slice projection operation is for:
Model.find({},{ "actions": { "$slice": -1 } },function(err,docs) {
// contains an array with the last item
});
If indeed you are "updating" array items and changing dates, but you want to query for the most recent on a regular basis, then you are probably best off keeping the array ordered. You can do this with a few modifiers such as:
Model.update(
{
"_id": ObjectId("541f7bbb699e6dd5a7caf2d6"),
},
{
"$push": { "actions": { "$each": [], "$sort": { "date": 1 } } }
},
function(err,numAffected) {
}
);
Which is actually more of a trick that you can do with the $sort modifier to simply sort the existing array elements without adding or removing. In versions prior to 2.6 you need the $slice "update" modifier in here as well, but this could be set to a value larger than the expected array elements if you did not actually want to restrict the possible size, but that is probably a good idea.
Unfortunately, if you were "updating" via a $set statement, then you cannot do this "sorting" in a single update statement, as MongoDB will not allow both types of operations on the array at once. But if you can live with that, then this is a way to keep the array ordered so the first query form works.
If it just seems too hard to keep an array ordered by date, then you can in fact retrieve the largest value my means of the .aggregate() method. This allows greater manipulation of the documents than is available to basic queries, at a little more cost:
Model.aggregate([
// Unwind the array to de-normalize as documents
{ "$unwind": "$actions" },
// Sort the contents per document _id and inner date
{ "$sort": { "_id": 1, "actions.date": 1 } },
// Group back with the "last" element only
{ "$group": {
"_id": "$_id",
"name": { "$last": "$name" },
"actions": { "$last": "$actions" }
}}
],
function(err,docs) {
})
And that will "pull apart" the array using the $unwind operator, then process with a next stage to $sort the contents by "date". In the $group pipeline stage the "_id" means to use the original document key to "collect" on, and the $last operator picks the field values from the "last" document ( de-normalized ) on that grouping boundary.
So there are various things that you can do, but of course the best way is to keep your array ordered and use the basic projection operators to simply get the last item in the list.
In a contact management app, each user will have his own database. When users wish to share certain categories of contacts with others, a backend will initiate a replication. Each contact is its own document, but also has various children documents such as notes and appointments.
Here is an example...
Contact:
{
"_id": 123,
"type": "contact",
"owner": "jimmy",
"category": "customer",
"name": "Bob Jones",
"email": "bob#example.com"
}
Note:
{
"_id": 456,
"type": "note",
"owner": "jimmy",
"contact_id": 123,
"timestamp": 1383919278,
"content": "This is a note about Bob Jones"
}
So let's say Jimmy wants to share his only his customers with sales manager Kevin, while his personal contacts remain private. When the note passes through the replication filter, is it possible to access the linked contact's category field?
Or do I have to duplicate the category field in every single child of a contact? I would prefer not to have to do this, as each contact may have many children which I would have to update manually every time the category changes.
Here is some pseudo-code for the filter function:
function(doc, req)
{
if(doc.type == “contact”) {
if(doc.category == req.query.category) {
return true;
}
}
else if(doc.contact_id) {
if(doc.contact.category == req.query.category) {
return true;
}
}
return false;
}
If this is possible, please describe how to do it. Thanks!
There are some other options.
There's a not-so-well-known JOIN trick in CouchDB. Instead of using replication, however, you'll have to share the results of a MapReduce View -- unfortunately you can use a view as a filter for replication. If you're using Cloudant (disclaimer: I'm employed by Cloudant) you can use chained-MapReduce to output the result to another database that you could then replication from...
Additionally, I think this SO post/answer on document structures and this join trick could be helpful: Modeling relationships on CouchDB between documents?
No, this is not possible. Each document must be consistent so it has no any explicit relations with others documents. Having contact_id value as reference is just an agreement from your side - CouchDB isn't aware about this.
You need to literally have category document be nested within contact one to do such trick e.g. have single document to process by filter function. This is good solution from point when you need to have consistent state of contact document.