I have an array of objects and I want to store them in a collection using only one I/O operation if it's possible. If any document already exists in the collection I want to replace it, or insert it otherwise.
These are the solutions that I found, but doesn't work exactly as I want:
insertMany(): this doesn't replace the document that already exists, but throws exception instead (This is what I found in the Mongodb documentation, but I don't know if it's the same as mongoose).
update() or updateMany() with upsert = true: this doesn't help me as well, because here I have to do the same updates to all the to stored documents.
There is no replaceMany() in mongodb or mongoose.
Is there anyone how knows any optimal way to do replaceMany using mongoose and node.js
There is bulkWrite (https://docs.mongodb.com/manual/reference/method/db.collection.bulkWrite/), which makes it possible to execute multiple operations at once. In your case, you can use it to perform multiple replaceOne operations with upsert. The code below shows how you can do it with Mongoose:
// Assuming *data* is an array of documents that you want to insert (or replace)
const bulkData = data.map(item => (
{
replaceOne: {
upsert: true,
filter: {
// Filter specification. You must provide a field that
// identifies *item*
},
replacement: item
}
}
));
db.bulkWrite(bulkData);
You need to query like this:
db.getCollection('hotspot').update({
/Your Condition/
}, {
$set: {
"New Key": "Value"
}
}, {
multi: true,
upsert: true
});
It fulfils your requirements..!!!
Related
I have a collection in MongoDB with more than 5 million documents. Whenever I create a document inside the same collection I have to check if there exists any document with same title and if it exists then I don't have to add this to the database.
Example: here is my MongoDB document:
{
"_id":ObjectId("3a434sa3242424sdsdw"),
"title":"Lost in space",
"desc":"this is description"
}
Now whenever a new document is being created in the collection, I want to check if the same title already exists in any of the documents and if it does not exists, then only I want to add it to the database.
Currently, I am using findOne query and checking for the title, if it not available only then it is added to the database. I am facing the performance issue in this. It is taking too much time to do this process. Please suggest a better approach.
async function addToDB(data){
let result= await db.collection('testCol').findOne({title:data.title});
if(result==null){
await db.collection('testCol').insertOne(data);
}else{
console.log("already exists in db");
}
}
You can reduce the network round trip time which is currently 2X. Because you execute two queries. One for find then one for update. You can combine them into one query as below.
db.collection.update(
<query>,
{ $setOnInsert: { <field1>: <value1>, ... } },
{ upsert: true }
)
It will not update if already exists.
db.test.update(
{"key1":"1"},
{ $setOnInsert: { "key":"2"} },
{ upsert: true }
)
It looks for document with key1 is 1. If it finds, it skips. If not, it inserts using the data provided in the object of setOnInsert.
as the title says, I want to perform a find (one) for a document, by _id, and if doesn't exist, have it created, then whether it was found or was created, have it returned in the callback.
I don't want to update it if it exists, as I've read findAndModify does. I have seen many other questions on Stackoverflow regarding this but again, don't wish to update anything.
I am unsure if by creating (of not existing), THAT is actually the update everyone is talking about, it's all so confuzzling :(
Beginning with MongoDB 2.4, it's no longer necessary to rely on a unique index (or any other workaround) for atomic findOrCreate like operations.
This is thanks to the $setOnInsert operator new to 2.4, which allows you to specify updates which should only happen when inserting documents.
This, combined with the upsert option, means you can use findAndModify to achieve an atomic findOrCreate-like operation.
db.collection.findAndModify({
query: { _id: "some potentially existing id" },
update: {
$setOnInsert: { foo: "bar" }
},
new: true, // return new doc if one is upserted
upsert: true // insert the document if it does not exist
})
As $setOnInsert only affects documents being inserted, if an existing document is found, no modification will occur. If no document exists, it will upsert one with the specified _id, then perform the insert only set. In both cases, the document is returned.
Driver Versions > 2
Using the latest driver (> version 2), you'll use findOneAndUpdate as findAndModify was deprecated. The new method takes 3 arguments, the filter, the update object (which contains your default properties, that should be inserted for a new object), and options where you have to specify the upsert operation.
Using the promise syntax, it looks like this:
const result = await collection.findOneAndUpdate(
{ _id: new ObjectId(id) },
{
$setOnInsert: { foo: "bar" },
},
{
returnOriginal: false,
upsert: true,
}
);
const newOrUpdatedDocument = result.value;
Its a bit dirty, but you can just insert it.
Be sure that the key has a unique index on it (if you use the _id it's ok, it's already unique).
In this way if the element is already present it will return an exception that you can catch.
If it isn't present, the new document will be inserted.
Updated: a detailed explanation of this technique on the MongoDB Documentation
Here's what I did (Ruby MongoDB driver):
$db[:tags].update_one({:tag => 'flat'}, {'$set' => {:tag => 'earth' }}, { :upsert => true })}
It will update it if it exists, and insert it if it doesn't.
I'm trying to limit the amount of elements a user can add to an array field on one of my schemas. I'm currently adding the elements to the array using Schema.findOneAndUpdate(); with the $push operator.
The first thing I tried was the solution given by another answer here on StackOverflow, namely: https://stackoverflow.com/a/29418656/6502807
This solution adds a validate function to the fields in the schema definition. By setting runValidators to true, I did get the function to run with Schema.findOneAndUpdate(). It was at that moment, however, that I stumbled upon the next problem. At the end of the Validation chapter in the Mongoose docs it says:
Also, $push, $addToSet, $pull, and $pullAll validation does not run any validation on the array itself, only individual elements of the array.
So attempting to check for array length did not work when using $pull. It simply supplied the validation function with an empty array every time, regardless of its actual contents in the database.
Next thing I tried was to use a pre hook. This was without any success as well. For some reason it did not execute the hook, even with runValidators set to true. This is how I defined said hook:
Settings.pre('update', async function (next) {
if (this.messages.length > MAX_MESSAGES) {
throw new Error('Too many messages');
} else {
next();
}
});
EDIT: The reason the function did not fire was because I was using findOneAndUpdate instead of update this is fixed and the function now runs. The solution code above, however, does not work.
The schema with the array looks like this:
const Settings = new mongoose.Schema({
// A lot more fields not relevant to this question
messages: {
type: [{
type: String
}]
}
});
Another thing worth mentioning is that these update statements are used in conjunction with other options. I need the update statement to behave like an update or insert so my complete set of options looks like this:
{
runValidators: true,
setDefaultsOnInsert: true,
upsert: true,
new: true
}
When executing queries with the pre hook set like this, the array limit can be exceeded without any validation error being thrown.
At this point I'm wondering if there is any sensible way to do a max length check like this without having to do it myself outside of mongoose's abstraction layer.
I am using Mongoose 5.2.6 running on node v9.11.1 with MongoDB 4.0.0.
Any help is much appreciated!
Well if you are using latest version from mongodb and mongoose then you can use $expr operator
const udpate = await db.collection.update(
{ $expr: { $gt: [{"$size": "$messages" }, MAX_MESSAGES] }},
{ update }
)
You should be able to do that with the pre update hook. The thing is that that hook would not by default give you the update being mage so you can verify etc. You have to take it via this.getUpdate():
Settings.pre('update', async function (next) {
var preUpdate = this.getUpdate()
// now inside of the preUpdate you would have your update being made and should have the array in there on which you can check the length
});
To give you an idea in my test schema I had to do something like this on an update with a $set:
this.getUpdate().$set.books.length // gave me 2 which was correct etc
I also had no issues running and hitting the update hook at all. It looks super simple out of the mongoose docs:
AuthorSchema.pre('update', function(next) {
console.log('UPDATE hook fired!')
console.log(this.getUpdate())
next();
});
I have to remove an item from an array of subschemas in a document.
SubSchema = new mongoose.Schema({...})
MySchema = new mongoose.Schema({someArray: [SubSchema]})
(...)
mydoc.somearray.pull(req.body.submodel_id);
However, I need the index of the element that has been removed to notify all connected clients.
Is there an elegant solution to this, or do I have to use _.findIndex or something like that? (I imagine that to have worse performance since it unnecessarily iterates the array twice)
Not sure if an elegant solution exists for this as MongoDB has no way of returning the index of the array element
being pulled within an update operation. One approach (though I would consider it a dirty hack) would be to get the original
array after the update operation and get the removed element index using Array.indexOf() within the update callback.
Consider the following update operation using findOneAndUpdate() to get the update document:
var submodel_id = req.body.submodel_id,
query = { "someArray": submodel_id };
Model.findOneAndUpdate(
query,
{ "$pull": { "someArray": submodel_id } },
{ "new": false },
function(err, doc) {
var removedIndex = doc.someArray.indexOf(submodel_id);
console.log(removedIndex);
}
);
as the title says, I want to perform a find (one) for a document, by _id, and if doesn't exist, have it created, then whether it was found or was created, have it returned in the callback.
I don't want to update it if it exists, as I've read findAndModify does. I have seen many other questions on Stackoverflow regarding this but again, don't wish to update anything.
I am unsure if by creating (of not existing), THAT is actually the update everyone is talking about, it's all so confuzzling :(
Beginning with MongoDB 2.4, it's no longer necessary to rely on a unique index (or any other workaround) for atomic findOrCreate like operations.
This is thanks to the $setOnInsert operator new to 2.4, which allows you to specify updates which should only happen when inserting documents.
This, combined with the upsert option, means you can use findAndModify to achieve an atomic findOrCreate-like operation.
db.collection.findAndModify({
query: { _id: "some potentially existing id" },
update: {
$setOnInsert: { foo: "bar" }
},
new: true, // return new doc if one is upserted
upsert: true // insert the document if it does not exist
})
As $setOnInsert only affects documents being inserted, if an existing document is found, no modification will occur. If no document exists, it will upsert one with the specified _id, then perform the insert only set. In both cases, the document is returned.
Driver Versions > 2
Using the latest driver (> version 2), you'll use findOneAndUpdate as findAndModify was deprecated. The new method takes 3 arguments, the filter, the update object (which contains your default properties, that should be inserted for a new object), and options where you have to specify the upsert operation.
Using the promise syntax, it looks like this:
const result = await collection.findOneAndUpdate(
{ _id: new ObjectId(id) },
{
$setOnInsert: { foo: "bar" },
},
{
returnOriginal: false,
upsert: true,
}
);
const newOrUpdatedDocument = result.value;
Its a bit dirty, but you can just insert it.
Be sure that the key has a unique index on it (if you use the _id it's ok, it's already unique).
In this way if the element is already present it will return an exception that you can catch.
If it isn't present, the new document will be inserted.
Updated: a detailed explanation of this technique on the MongoDB Documentation
Here's what I did (Ruby MongoDB driver):
$db[:tags].update_one({:tag => 'flat'}, {'$set' => {:tag => 'earth' }}, { :upsert => true })}
It will update it if it exists, and insert it if it doesn't.