I know there's a lot of threads about this subject - but i still didn't find anything that matches my situation.
There is a big database with companies and products schema's.
Each company uploads their products, (product has "company" field with the mongo id of the company)
My operation goes like this -
Company uploads array of products.
Each product has his own productId.
I need to insert all the products in to the DB but -
If i find product with the same productId - update it - if not - insert a new product.
Change some fields (change product "status" to "deleted") to all of the products that belongs to that company in the DB, that didn't match any of the product id's (from the array of products the user uploaded).
I'm searching for a better solution than the one i'm using, my process goes like this -
fetching all the company's products.
looping through all of the products - and updating a field (status = "deleted") building a mapping object for each product, so i can identify it and check if it exists in the uploaded array.
looping through the company's array (with async library), checking if product exists (in the mapping array), if exists - updating it, if not - create a new product.
It feels very sloppy and i'm sure there is a better way to do it.
the thing i tried to use is -
await Product.updateMany({ productStatus: { $ne: 'manual'} , company: {$eq: company._id}}, { $set: {status: 'deleted' }});
and then
Product.bulkWrite(
productsArray.map((row) =>
({
updateOne: {
filter: { productId: row.productId, company: row.company },
update: row,
upsert: true
}
})
)
)
It was super slow (my method was faster even thought it's more complicated)
If i find product with the same productId - update it - if not - insert a new product.
You can use upserts, which is a find-and-modify operation with upsert: true set. Pass the whole product document.
Change some fields (change product "status" to "deleted") to all of the products that belongs to that company in the DB, that didn't match any of the product id's (from the array of products the user uploaded).
You can use $nin to update the status on all products whose ids are not in the list of currently available products.
Related
I'm new to mongodb and to overall databases side of development.
I'm trying to make a product listing site where all the categories would be displayed with the number of products within that particular category and when clicked on a particular category, it would get me all the products in that category.
Some things to note are:
every product will have only one category
each category will have multiple products
I don't know how to go about this problem and tried searching it online but couldn't exactly find what I was looking for. I've also tried making the schema for this but I do not know if it's the right approach or not and this is how it looks:
const productsSchema = {
category: String,
name: String,
price: String,
description: String,
thumbnail: String,
};
Side note: I'm using MERN stack.(if its of any help)
If I've understand well your question, you can use something like this:
db.collection.aggregate([
{
"$match": {
"category": "category1"
}
},
{
"$count": "total"
}
])
With this query you will get the total $count for each category.
Example here
In your frontend you will need a call for every category.
Maybe if your DB has a lot of different categories this is not a good approach, but if the number is not large enough you can call this query a couple times and you will get the result you want.
MongoDB Documentation reference here
I would say you should have a product schema and a product category schema, where the product category schema has an array of product ids that belong to that category.
In the product schema, you could also have a pointer to the category object that a product is linked to (as opposed to just the name of the category as a string).
Maybe take a look at mongoose populate https://mongoosejs.com/docs/populate.html
I am new to MongoDB and mongoose. I am trying to create a Node & MongoDB auction app. So since it is actually an online auction, users should be able to bid for items. I successfully completed the user registration, sign in page and authentication process, however, I am a bit stuck in the bidding page.
I created a Schema using mongoose and each item for auction is saved in the database. I want to add name and price of each user who bid for the item in the same object in MongoDB, like this:
{
name: "valuable vase from 1700s",
owner: "John Doe",
itemId: 100029,
bids: {
100032: 30000,
100084: 34000
}
}
So each user will have ids like 100032: 30000, and when they bid, their "account id: price" will be added under bids in the database object of the item.
I made some research and found some ways to solve the problem but I want to know if what I want to do is possible and if it is the right solution to do.
Thanks for giving me your time!
There are indeed couple of ways to achieve what you want.
In my opinion, a collection called ItemBids, where each document includes this data structure, will benefit you the most.
{
itemId: ObjectId # reference to the item document
accountId: ObjectId # reference to the account
bid: Number # the bid value
}
This pattern is suitable for your case because you can easily query the bids by whatever you want -
You can get all the account bids, you can get all the item bids, and you can sort them with native Mongo by the bid price.
Every time there's a bid, you insert a new document to this collection.
Another option is embedding an array of Bids objects in the item Object.
Each Bid object should include:
bids: [{
account: ObjectId("") # This is the account
price: Number
}]
The cons here are that querying it and accessing it will require more complex queries.
You can read more about the considerations
here:
https://docs.mongodb.com/manual/core/data-model-design
https://coderwall.com/p/px3c7g/mongodb-schema-design-embedded-vs-references
The way you decided to implement your functionality is a little bit complicated.
It is not impossible to do that but, the better way is to use array of objects instead of a single object like this:
{
name: '',
..
..
bids: [{
user: 100032,
price: 30000
}, {
user: 100084,
price: 34000
}]
}
I am designing an item inventory system for a website that I am building.
The user's inventory is loaded from a Web API. This information is then processed so that it is more suited to my web app. I am trying to combine all the item records into one MongoDB collection - so other user inventories will be cached in the same place. What I have to deal with is deleting old item records if they are missing from the user's inventory (i.e. they sold it to someone) and also upserting the new items. Please note I have looked through several Stack Overflow questions about bulk upserts but I was unable to find anything about conditional updates.
Each item has two unique identifiers (classId and instanceId) that allow me to look them up (I have to use both IDs to match it) which remain constant. Some information about the item, such as its name, can change and therefore I want to be able to update those records when I fetch new inventory information. I also want new items that my site hasn't seen before to be added to my database.
Once the data returned from the Web API has been processed, it is left in a large array of objects. This means I am able to use bulk writing, however, I am unaware of how to upsert with conditions with multiple records.
Here is part of my item schema:
const ItemSchema = new mongoose.Schema({
ownerId: {
type: String,
required: true
},
classId: {
type: String,
required: true
},
instanceId: {
type: String,
required: true
},
name: {
type: String,
required: true
}
// rest of item attributes...
});
User inventories typically contain 600 or more items, with a max count of 2500.
What is the most efficient way of upserting this much data? Thank you
Update:
I have had trouble implementing the solution to the bulk insert problem. I made a few assumptions and I don't know if they were right. I interpreted _ as lodash, response.body as the JSON returned by the API and myListOfItems also as that same array of items.
import Item from "../models/item.model";
import _ from 'lodash';
async function storeInventory(items) {
let bulkUpdate = Item.collection.initializeUnorderedBulkOp();
_.forEach(items, (data) => {
if (data !== null) {
let newItem = new Item(data);
bulkUpdate.find({
classId: newItem.classId,
instanceId: newItem.instanceId
}).upsert().updateOne(newItem);
items.push(newItem);
}
});
await bulkUpdate.execute();
}
Whenever I run this code, it throws an error that complains about an _id field being changed, when the schema objects I created don't specify anything to do with schemas, and the few nested schema objects don't make a difference to the outcome when I change them to just objects.
I understand that if no _id is sent to MongoDB it auto generates one, but if it is updating a record it wouldn't do that anyway. I also tried setting _id to null on each item but to no avail.
Have I misunderstood anything about the accepted answer? Or is my problem elsewhere in my code?
This is how I do it :
let bulkUpdate = MyModel.collection.initializeUnorderedBulkOp();
//myItems is your array of items
_.forEach(myItems, (item) => {
if (item !== null) {
let newItem = new MyModel(item);
bulkUpdate.find({ yyy: newItem.yyy }).upsert().updateOne(newItem);
}
});
await bulkUpdate.execute();
I think the code is pretty readable and understandable. You can adjust it to make it work with your case :)
I am new to CouchDB and the NoSQL scene and coming from a SQL background. I have some questions on referential integrity, for example I have a product document as below
{
type: "product"
name: "Sweet Necklace"
category: "necklace"
}
And each category have their own document
{
type: "category",
name: "necklace",
custom_attr: ".."
}
Just for the sake of the argument, what happens when the stakeholder chose to rename the category from "necklace" to "accessories", what should happen on the products that have the category field set as "necklace"? Do I do a bulk update on all products with category equal to necklace? (I don't think CouchDB allows us to perform a "UPDATE ALL WHERE" kinda statement)
What is the best practice on handling such situation?
P/S: I chose to save the category name in the product document instead of a category ID since NoSQL encourages denormalization anyway.
If you're maintaining a separate document for the category then you've not denormalized your data at all. In fact, by doing what you're doing, you're getting the worst of both worlds - no normalization and no denormalization.
Try something like this:
Product document:
{
_id:"product:first_product",
name:"First Product"
category:"category:category_1"
}
Category document:
{
_id:"category:category_1",
name:"Category 1",
custom_attr: {}
}
This way, when you change the name of the category, you're still referring to the correct document from all the product documents that have this category.
Note: you can still have a type field and let the _id remain as it is currently.
Edit:
To get the product/category info, you can define a map function like so:
function(doc){
if(doc.id.indexOf('product:') === 0){
// or if(doc.type === 'product') if you use the type field
emit(doc, {'_id': doc.category});
}
}
Now whenever you use this view and you set include_docs to true, the category information will be included in your results.
I have a collection, say, "Things":
{ id: 1
creator: 1
created: Today }
{ id: 2
creator: 2
created: Today }
{ id: 3
creator: 2
created: Yesterday }
I'd like to create a query that'll return each Thing created by a set of users, but only their most recently created thing.
What would this look like? I can get search my collection with an array of creators and it works just fine - how can I also only get the most recently created object per user?
Thing.find({ _creator : { "$in" : creatorArray })...
You cannot find, sort and pick the most recent in just a single find() query. But you can do it using aggregation:
Match all the records where the creator is amongst the one who we are looking
for.
Sort the records in descending order based on the created field.
Group the documents based on the creator.
Pick each creator's first document from the group, which will also be
his latest.
Project the required fields.
snippet:
Thing.aggregate([
{$match:{"creator":{$in:[1,2]}}},
{$sort:{"created":-1}},
{$group:{"_id":"$creator","record":{$first:"$$ROOT"}}},
{$project:{"_id":0,
"id":"$record.id",
"creator":"$record.creator",
"created":"$record.created"}}
], function(err,data){
})