I am currently building a Mongo, Node, Express web app where deals are rendered on the screen in a list if they are "active" . However, I would like to programtically change the "active" status of a deal to (see below for dealSchema) at the "dealEndingDate". Think of it like a deal expiring. Is there some sort of listener I can add to constantly troll the database and update the documents... I welcome all suggestions. Thank you!
{
"_id": {
"$oid": "5a63c974aa17eb49c5260815"
},
"dealHeadline": "Test",
"active": true,
"dealBeginningTime": {
"$date": "2018-01-20T06:00:00.000Z"},
"dealEndingTime": {
"$date": "2018-01-22T23:45:00.000Z"}
}
A cronjob can be a solution for you .
An alternative way of doing this is to use expires, i.e. dealEndingTime: { type: Date, expires: '60s' }. Please note that this will remove the entire document 60 seconds after the set date. If you still want to keep the document, you can set up a separate Deal collection, where all your deals are stored. Then you can reference the deals for each document by using ObjectId: deals: [{_id: { type: Schema.Types.ObjectId, ref: 'deal' } Then simply checking the length of the array will reveal if there are any active deals for that document.
I hope this is useful to you!
Related
Please what are the constraints in generating an itemId. I generate unique itemId for each item in the form, but the API keeps telling me invalid ID.
https://developers.google.com/forms/api/reference/rest/v1/forms#Item
Please I need help with this
{
"includeFormInResponse": false,
"requests": [
{
"createItem": {
"item": {
"itemId": "4e637fjc",
"description": "First Name",
"questionItem": {
"question": {
"textQuestion": {
"paragraph": false
},
"required": true
}
}
},
"location": {
"index": 0
}
}
},
{
"createItem": {
"item": {
"itemId": "njyf3izr",
"description": "Middle Name",
"questionItem": {
"question": {
"textQuestion": {
"paragraph": false
},
"required": true
}
}
},
"location": {
"index": 1
}
}
},
}
]
When I had tested Google Forms API before, unless I'm mistaken, I had thought that the rule of item ID might be required to be 00000000 to 7fffffff as the hex value. By the way, for example, 0 is used as 00000000.
When I saw your showing request body, you are trying to use 4e637fjc and njyf3izr as the item ID. In the case of these values, the values are not hex values. I thought that by this, an error like Invalid ID occurred.
But, I think that actually, this is not published in the official document. So, I would like to tell this.
Added:
About your following reply,
Do you mean something like this, with Javascript. crypto.randomBytes(256).toString('hex').slice(0, 8)
From your tag, when you want to use Google Apps Script or Node.js, how about the following sample script? Unfortunately, Google Apps Script cannot directly use "crypto". So, I proposed the following sample script.
Sample script:
const res = Math.floor(Math.random() * parseInt("7FFFFFFF", 16)).toString(16).padStart(8, "0");
console.log(res);
In this sample script, the values of 00000000 to 7fffffff are randomly returned.
Missing documentation
I am afraid that since the Forms API is very new there is no documentation about the specific format the ID should have.
I have done a couple of tests with the API and the only thing I was able to figure out is that the ID needs an 8-character-long string to work, otherwise it would not work or would fill out the blank spaces with zeros.
When doing the testing I was also able to find out that sometimes the API would take a specific pattern of letters and numbers, but when changing the numbers and letters it stops working for no reason.
This seems like missing clarification from the documentation, and I would strongly recommend sending feedback about this problem on the API method page. You can do so by clicking the following option at the top right corner of the documentation:
Google tends to check that feedback a lot when talking about missing information. In addition to all that you can also fill out a report in Google's issue tracker so that they investigate the inconsistencies when using the batchUpdate method to update the ID.
References:
Forms Item
Method: forms.batchUpdate
Summary:
I am building my first large scale full stack application(MERN stack) that is trying to mimic a large scale clothing store. Each article of clothing has many 'tags' that represent its features, top/bottom/accessory/shoes/ect, and subcategories, for example on top there is shirt/outerwear/sweatshirt/etc, and sub-sub-categories within it, for example on shirt there is blouse/t-shirt/etc. Each article has tags for primary colors, hemline, pockets, technical features, the list goes on.
Main question:
how should I best organize the data in mongodb with mongoose schemas in order for it to be quickly searchable when I plan on having 50,000 or more articles? And genuinely curious, how do large clothing retailers typically design databases to be easily searchable by customers when items have so many identifying features?
Things I have tried or thought of:
On the mongoDB website there is a recommendation to use a tree structure with child references. here is the link: https://docs.mongodb.com/manual/tutorial/model-tree-structures-with-child-references/ I like this idea but I read here: https://developer.mongodb.com/article/mongodb-schema-design-best-practices/ that when storing over a few thousand pieces of data, using object ID references is no longer sufficient, and could create issues because of datalimits.
Further, each clothing item would fall into many different parts of the tree. For example it could be a blouse so it would be in the blouse "leaf" of the tree, and then if its blue, it would be in the blue "leaf" of the tree, and if it is sustainably sourced, it would fall into that "leaf" of the tree as well. Considering this, a tree like data structure seems not the right way to go. It would be storing the same ObjectID in many different leaves.
My other idea was to store the article information (description, price, and picture) seperate from the tagging/hierarchical information. Then each tagging object would have a ObjectID reference to the item. This way I could take advantage of the propogate method of mongoose if I wanted to collect that information.
I also created part of the large tree structure as a proof of concept for a design idea I had, and this is only for the front end right now, but this also creates bad searches cause they would look like taxonomy[0].options[0].options[0].options[0].title to get to 'blouse'. Which from my classes doesnt seem like a good way to make the code readable. This is only a snippet of a long long branching object. I was going to try to make this a mongoose schema. But its a lot of work and I wanna make sure that I do it well.
const taxonomy = [
{
title: 'Category',
selected: false,
options: [
{
title: 'top',
selected: false,
options: [
{
title: 'Shirt',
selected: false,
options: [
{
title: 'Blouse',
selected: false,
},
{
title: 'polo',
selected: false,
},
{
title: 'button down',
selected: false,
},
],
},
{
title: 'T-Shirt',
selected: false,
},
{
title: 'Sweater',
selected: false,
},
{
title: 'Sweatshirt and hoodie',
selected: false,
},
],
},
Moving forward:
I am not looking for a perfect answer, but I am sure that someone has tackled this issue before (all big businesses that sell lots of categorized products have) If someone could just point me in the right direction, for example, give me some terms to google, some articles to read, or some videos to watch, that would be great.
thank you for any direction you can provide.
MongoDB is a document based database. Each record in a collection is a document, and every document should be self-contained (it should contain all information that you need inside it).
The best practice would be to create one collection for each logical whole that you can think of. This is the best practice when you have documents with a lot of data, because it is scalable.
For example, you should create Collections for: Products, Subproducts, Categories, Items, Providers, Discounts...
Now, when you creating Schemas, instead of creating nested structure, you can just store a reference of one collection document as a property of another collection document.
NOTE: The maximum document size is 16 megabytes.
BAD PRACTICE
Let us first see what would be the bad practice. Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [{
"sub_product_name": "Subpoduct_name_1",
"sub_product_description": "Description",
"items": [{
"item_name": "item_name_1",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
}]
},
{
"item_name": "item_name_2",
"item_desciption": "Description",
"discounts": [{
"discount_name": "Discount_1",
"percentage": 25
},
{
"discount_name": "Discount_2",
"percentage": 50
}]
},
]
},
...
]
}
Here product document has sub_products property which is an array of sub_products. Each sub_product has items, and each item has discounts. As you can see, because of this nested structure, the maximum document size would be quickly exceeded.
GOOD PRACTICE
Consider this structure:
Product = {
"name": "Product_name",
"sub_products": [
'sub_product_1_id',
'sub_product_2_id',
'sub_product_3_id',
'sub_product_4_id',
'sub_product_5_id',
...
]
}
Subproduct = {
"id": "sub_product_1_id",
"sub_product_name": "Subroduct_name",
"sub_product_description": "Description",
"items": [
'item_1_id',
'item_2_id',
'item_3_id',
'item_4_id',
'item_5_id',
...
]
}
Item = {
"id": "item_1_id",
"item_name": "item_name_1",
"item_desciption": "Description",
"items": [
'discount_1_id',
'discount_2_id',
'discount_3_id',
'discount_4_id',
'discount_5_id',
...
]
}
Discount = {
"id": "discount_1_id",
"discount_name": "Discount_1",
"percentage": 25
}
Now, you have collection for each logical whole and you are just storing a reference of one collection document as a property of another collection document.
Now you can use one of the best features of the Mongoose that is called population. If you store a reference of one collection document as a property of another collection document, when performing querying of the database, Mongoose will replace references with the actual documents.
I have the following schema in mongoose:
const textSchema = new Schema({
title: { type:String, unique: true },
sentences: [SentenceSchema],
})
Let's suppose that a typical text has around 100 sentences. Now I've changed one sentence up a little bit in the front-end and need to send that change to my server/mongodb. How would I best do this with mongoose+rest api? Would I update the whole article because of a change in a single sentence? Or is there a way for me to target only that very sentence that I want to change?
PS: I am sorry, this question is probably quite basic, but it would help me out a lot! Thanks in advance for your answers!
No, you don't need to change whole article. What you can do is, you can find sentence that actually change by using sentence id and update particular sentence. You need to use findOneAndUpdate() method.
Please refer below query
Text.findOneAndUpdate(
{ "_id": textId, "sentences._id": updatedSentenceId },
{
"$set": {
"sentences.$": updatedSentence
}
},
function(err,doc) {
}
);
Currently I am working on a mobile app. Basically people can post their photos and the followers can like the photos like Instagram. I use mongodb as the database. Like instagram, there might be a lot of likes for a single photos. So using a document for a single "like" with index seems not reasonable because it will waste a lot of memory. However, I'd like a user add a like quickly. So my question is how to model the "like"? Basically the data model is much similar to instagram but using Mongodb.
No matter how you structure your overall document there are basically two things you need. That is basically a property for a "count" and a "list" of those who have already posted their "like" in order to ensure there are no duplicates submitted. Here's a basic structure:
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3")
"photo": "imagename.png",
"likeCount": 0
"likes": []
}
Whatever the case, there is a unique "_id" for your "photo post" and whatever information you want, but then the other fields as mentioned. The "likes" property here is an array, and that is going to hold the unique "_id" values from the "user" objects in your system. So every "user" has their own unique identifier somewhere, either in local storage or OpenId or something, but a unique identifier. I'll stick with ObjectId for the example.
When someone submits a "like" to a post, you want to issue the following update statement:
db.photos.update(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
"likes": { "$ne": ObjectId("54bb2244a3a0f26f885be2a4") }
},
{
"$inc": { "likeCount": 1 },
"$push": { "likes": ObjectId("54bb2244a3a0f26f885be2a4") }
}
)
Now the $inc operation there will increase the value of "likeCount" by the number specified, so increase by 1. The $push operation adds the unique identifier for the user to the array in the document for future reference.
The main important thing here is to keep a record of those users who voted and what is happening in the "query" part of the statement. Apart from selecting the document to update by it's own unique "_id", the other important thing is to check that "likes" array to make sure the current voting user is not in there already.
The same is true for the reverse case or "removing" the "like":
db.photos.update(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
"likes": ObjectId("54bb2244a3a0f26f885be2a4")
},
{
"$inc": { "likeCount": -1 },
"$pull": { "likes": ObjectId("54bb2244a3a0f26f885be2a4") }
}
)
The main important thing here is the query conditions being used to make sure that no document is touched if all conditions are not met. So the count does not increase if the user had already voted or decrease if their vote was not actually present anymore at the time of the update.
Of course it is not practical to read an array with a couple of hundred entries in a document back in any other part of your application. But MongoDB has a very standard way to handle that as well:
db.photos.find(
{
"_id": ObjectId("54bb201aa3a0f26f885be2a3"),
},
{
"photo": 1
"likeCount": 1,
"likes": {
"$elemMatch": { "$eq": ObjectId("54bb2244a3a0f26f885be2a4") }
}
}
)
This usage of $elemMatch in projection will only return the current user if they are present or just a blank array where they are not. This allows the rest of your application logic to be aware if the current user has already placed a vote or not.
That is the basic technique and may work for you as is, but you should be aware that embedded arrays should not be infinitely extended, and there is also a hard 16MB limit on BSON documents. So the concept is sound, but just cannot be used on it's own if you are expecting 1000's of "like votes" on your content. There is a concept known as "bucketing" which is discussed in some detail in this example for Hybrid Schema design that allows one solution to storing a high volume of "likes". You can look at that to use along with the basic concepts here as a way to do this at volume.
I hope I'm having a big brainfart moment. But here's my situation in a scraping szenario;
I'm wanting to be able to scrape over multiple machines and cores. Per site, I have different Front pages, I scrape (exmpl. for the site stackoverflow I'd have fronts stackoverflow.com/questions/tagged/javascript and stackoverflow.com/questions/tagged/nodejs).
An article could be on every Front and when I discover an article I want to create an Article if the url is unknown, if its known I want to make an Front entry in article.discover if Front is unknown and otherwise insert my FrontDiscovery to the apropriate Front.
Here are my Schemas;
FrontDiscovery = new Schema({
_id :{ type:ObjectId, auto:true },
date :{ type: Date, default:Date.now},
dims :{ type: Object, default:null},
pos :{ type: Object, default:null}
});
Front = new Schema({
_id :{ type:ObjectId, auto:true },
url :{type:String}, //front
found :[ FrontDiscovery ]
});
Article = new Schema({
_id :{ type:ObjectId, auto:true },
url :{ type: String , index: { unique: true } },
site :{ type: String },
discover:[ Front]
});
The Problem I am thinking I will eventually be running into is a race condition. When two job-runners (in parallel) find the same (before unknown) article and create a new one. Yes, I have a unique index on it and could handle it that way - quite inelegantly imho.
But lets go further; When - for what ever reason - my 2 job-runners are scraping the same front at the same time and both notice that for Front there is no entry yet and create a new one adding the FrontDiscovery, I'd end with two entry's for the same Front.
What are your strategies to circumvent such a situation? findByIdAndUpdate with the upsert:true for each document seperately? If so, how can I only push something to the embedded document collection and not overwrite everything else at the same time but still create the defaults if it hasnt been created?
Thank you for any help in directing me in the right direction! I really hope I'm having a massive brainfart..
Update with upsert=true can be used to perform an atomic "insert or update" (http://docs.mongodb.org/manual/core/update/#update-operations-with-the-upsert-flag).
For instance if we want to make sure a document in Front collection with specific url is inserted exactly once, we could run something like:
db.Front.update(
{url: 'http://example.com'},
{$set: {
url: 'http://example.com'},
found: true
}
)
Operations on a single document in MongoDB are always atomic. If you make updates that span over multiple documents, then no atomicity is guaranteed. In such cases, you can ask yourself: do I really need the operations to be atomic? If the answer is no, then you probably will find your way around working with potentially unconsistent data. If the answer is yes and you want to stick with MongoDB, check out the design pattern on Two Phase Commits.