MongoDb: Insert or update multiple documents with a unique index - node.js

I have a MongoDB collection with a unique index.
I am trying to insert or update an array of documents into that collection.
If there is NO existing document in the collection matching the unique index of the document, that new document should be inserted into the collection.
However, if there IS already a document in the collection with that unique index, it should be updated with the fields of the new document. Any fields that are NOT present in the new document should be left untouched.
This is what I have currently which is working for inserting (but NOT for updating).
const mongojs = require('mongojs');
const db = mongojs('mongodb://username:password#address.mlab.com:37230/database');
// items is an array of documents
db.items.insert(items, (err, task) => {
if (err) {
console.log(err);
}
})
I understand this is wrong and it currently gives this error:
E11000 duplicate key error index: database.items.$upc_1 dup key:
What is the proper query for this?

You can try using mongodb bulkWrite api:
var ops = []
items.forEach(item => {
ops.push(
{
updateOne: {
filter: { _id: unique_id },
update: {
$set: { fields_to_set_if_exists },
$setOnInsert: { fileds_to_insert_if_does_not_exist }
},
upsert: true
}
}
)
})
db.collections('collection_name').bulkWrite(ops, { ordered: false });

I don't believe that you can update an entire array of documents at the same time. Therefore, you would have to update each item in the array individually.
items.forEach((item) => {
db.items.update({_id: item._id}, item, {upsert: true}, (err, task) => {
if (err) {
console.log(err);
}
});
}
The {upsert: true} option will update if a record exists and insert if not.

What are you looking for is an upsert, not an insert. It can be done by the following code:
db.collection.update(
<query>,
<updates>,
{
upsert: <boolean>,
multi: <boolean>,
writeConcern: <document>,
collation: <document>
}
)
Query will search for a document using the parameters of the query, if it finds, it will update the fields mentioned in . If it doesn't find, it will insert a new document with the fields and values of .
The last object (with multiple fields), contains a field to say if an upsert is desired and one called "multi" to say if an update on multiple documents is desired.
For example:
db.items.update({name:"item1"},{$set:{price:20}},{upsert:true})
This will search for a document with the name "item1" and set its price to 20. If it doesn't find, it will create a new one with price 20.
One thing to be noticed though is:
If you don't use the tag $set on the fields, it will substitute the whole document.
Supposing you have a document like this:
{_id:"1234",name:"item1",price:10}
If you run the following two queries:
db.items.update({name:"item1"},{$set:{price:20}},...)
and
db.items.update({name:"item1"},{price:20},...)
it will yeld different results:
First one:
{_id:"1234",name:"item1",price:20}
Second one:
{_id:"1234",price:20}
If you don't call $set, it will change the whole document.
More information on the manual:
https://docs.mongodb.com/manual/reference/method/db.collection.update/
Hope my answer was helpful

Related

MongoDB update query in subarray

An update in the array of objects inside another array of objects.
mongodb field that I'm working on:
otherFields: values,
tasks: [
{
_id: mongodb.objectID(),
title: string,
items:[{
_id: mongodb.objectID(),
title: string,
completed: boolean //field need to be update.
}]
},
{}...
],
otherFields: value
sample mongodb document
I need to find the document using the task_id and the item_id and update a completed field in item of a task. Using the mongoose findOneAndUpdate method
const path = "tasks.$.items." + item_id + "completed";
collectionName.findOneAndUpdate(
{ _id: req.user._id, "tasks._id": taskID },
{ $set: { [path]: true }});
The above query doesn't work!!!
There is no need to use multiple query conditions, since you'd like to update a specific item that has an unique ID. Therefore you could use something along the lines:
collectionName.findOneAndUpdate(
{ 'tasks.items._id': itemID },
...
);
Keep in mind this structure is far away from optimized as it would basically look through the entire database...
Also now that I think of it, you'd also have issue with the update, as there are two nested arrays within the document. Read more here: How to Update Multiple Array Elements in mongodb

How to improve the performance of query in mongodb?

I have a collection in MongoDB with more than 5 million documents. Whenever I create a document inside the same collection I have to check if there exists any document with same title and if it exists then I don't have to add this to the database.
Example: here is my MongoDB document:
{
"_id":ObjectId("3a434sa3242424sdsdw"),
"title":"Lost in space",
"desc":"this is description"
}
Now whenever a new document is being created in the collection, I want to check if the same title already exists in any of the documents and if it does not exists, then only I want to add it to the database.
Currently, I am using findOne query and checking for the title, if it not available only then it is added to the database. I am facing the performance issue in this. It is taking too much time to do this process. Please suggest a better approach.
async function addToDB(data){
let result= await db.collection('testCol').findOne({title:data.title});
if(result==null){
await db.collection('testCol').insertOne(data);
}else{
console.log("already exists in db");
}
}
You can reduce the network round trip time which is currently 2X. Because you execute two queries. One for find then one for update. You can combine them into one query as below.
db.collection.update(
<query>,
{ $setOnInsert: { <field1>: <value1>, ... } },
{ upsert: true }
)
It will not update if already exists.
db.test.update(
{"key1":"1"},
{ $setOnInsert: { "key":"2"} },
{ upsert: true }
)
It looks for document with key1 is 1. If it finds, it skips. If not, it inserts using the data provided in the object of setOnInsert.

Mongodb check if value is in a nested array

I have a collection in my database that contains a field which is composed of 3 arrays, like this :
use_homepage: {
home: [Array],
hidden: [Array],
archive: [Array]
}
This field represents the homepage of a user.
Each array contains an ObjectID that identifies projects shown on the user homepage.
I would like to check if my project id is in use_homepage.home or use_homepage.hidden, and if it is, remove the id from the array that match.
Can I do this with 1 (or 2 max) requests or do I have to make a request each time I have to check in another array ?
In case you expect to update one document at most, you can try this:
db.entities.findAndModify({
query: { $or : [
{ home: ObjectId('<HERE YOUR ID TO BE FOUND>') },
{ hidden: ObjectId('<HERE YOUR ID TO BE FOUND>') }
]},
update: { $pull: {
home: ObjectId('<HERE YOUR ID TO BE DELETED>'),
hidden: ObjectId('<HERE YOUR ID TO BE DELETED>')
}
}
});
As you can see, in general, you can search for some value and delete some other value.
The statement returns the original matching document (i.e. before the deletion is performed). If you want the modified document you can add the following attribute:
new: true
In case you search for many documents to update, this solution does not work, since findAndModify() works just on the first document matching the query condition.
Finaly, i used to make 2 requests to do the job :
db.User.find({"use_homepage.home": id}, {_id: 1}).toArray(function(err, result) {
// If some users have the id in the array home
db.User.updateMany({_id: {$in: users_match_ids}}, {
$pull: {"use_homepage.home": id}
}
});
// Do the same with 'hidden' array
If anyone see this post and have a better solution, I take it :)

Mongoose: best way to upsert data defined in conditions array

I trying to find best way for insert/update mongodb documents within one query, without calling save foreach element in result. Is it possible?
This code create document only with setData object.
conditions.idUser = {$in: uids};
db.Data.update(conditions, {$set: setData}, { multi: true, upsert: true }, function(err) {
But I need to save multiple documents, like {idUser : uid1, ..}, {idUser: uid2, ...}

mongoose distinct and populate with documents

I have the following model:
var followerSchema = new Schema({
id_follower: {type: Schema.Types.ObjectId, ref: 'Users'},
id_post: {type: Schema.Types.ObjectId, ref: 'Posts'}
});
I want to be able to find all posts for a list of followers. When I use find, it returns me of course multiple times the same post as multiple users can follow the same post.
So I tried to use distinct, but I have the feeling the "populate" does not work afterwards.
Here is my code:
followerModel
.distinct('id_post',{id_follower:{$in:followerIds}})
.populate('id_post')
.sort({'id_post.creationDate':1})
.exec(function (err, postFollowers) {
console.log(postFollowers);
})
It only returns me the array of the posts, and it is not populated.
I am new to mongoDB, but according to the documentation of mongoose, the "distinct" method should return a query, just as the "find" method. And on a query you can execute the "populate" method, so I don't see what I am doing wrong.
I also tried to use the .distinct() method of the query, so then my code was like this:
followerModel
.find({id_follower:{$in:followerIds}})
.populate('id_post')
.distinct('id_post')
.sort({'id_post.creationDate':1})
.exec(function (err, postFollowers) {
console.log(postFollowers);
})
In that case it works, but as in the documentation of mongoose you need to provide a callback function when you use the distinct method on a query, and so in my logs I get errors all over. A workaround would be to have a dummy callback function, but I want to avoid that...
Does anybody has an idea why the first attempt is not working? And if the second approach is acceptable by providing a dummy callback?
Would this be the right way considering the current lack of support in mongoose?
followerModel
.find({id_follower:{$in:followerIds}})
.distinct('id_post',function(error,ids) {
Posts.find({'_id':{$in : ids}},function(err,result) {
console.log(result);
});
});
You can simply use aggregate to group and populate the collection.
now we have the desired result
db.<your collection name>.aggregate([
{
$match: {<match your fields here>}
},
{
$group: {_id: <your field to group the collection>}
},
{
$lookup: {
from: "<your collection of the poupulated field or referenced field>",
localField: "<give the id of the field which yout want to populate from the collection you matched above cases>",
foreignField: "_id", //this references the id of the document to match the localField id in the from collection
as: 'arrayName', //<some name to the returned document, this is a single document array>
}
},
{
$project: {
//you really don't want the whole populated fields, you can select the fields you want
<field name>:
<1 or 0>, // 1 to select and 0 to not select
//you can add multiple fields here
//to select the fields that just returned from the last stage we can use
"arrayName._id": <1 or 0>,
}
}
])
//at last you can return the data
.then((data) =>{
console.log(data);
});
we have distinct() by $group and
populate() by $lookup
and we also select() by $project

Resources