This question already has answers here:
How to update if exists otherwise insert new document?
(4 answers)
Closed 3 years ago.
I'm getting usersfrom this API.
https://redmine-mock-api.herokuapp.com/api/v1/users
There are 100 in total (100 different ids). I added them to my mongodb database using mongoose but the problem is that it's always adding (i already have 5000 users (50 times repeated 100 users)
I want to add if the id does not exist or update if it exists.
What am I doing wrong? users is the array of users from the API
db.collection("users").insertMany(users, function (error, response) {
if (error) throw error;
console.log("Number of users inserted: " + response.insertedCount);
db.close();
});
try following format
db.collection.update({'_id': Id}, //Find with the unique identifier
{//put whatever you want to insert},
{upsert: true}
)
Looking at the mongo db docs it looks like insertMany is not what you want to use here. You probably want to use updateMany as it supports upserts where insertMany doesn't.
You should be able to use something like this:
db.collection("users").updateMany(
{},
users,
{
upsert: true
}
)
This is using an empty filter so will insert if the item doesn't exist or update if it does.
db.collection.updateMany(
<filter>,
<update>,
{
upsert: <boolean>
}
)
The updateMany() method takes the following parameters:
Filter
The selection criteria for the update. The same query selectors as in the find() method are available.
Specify an empty document { } to update all documents in the collection.
Update
The modifications to apply.
Use Update Operators such as $set, $unset, or $rename.
Upsert
Optional. When true, updateMany() either:
Creates a new document if no documents match the filter. For more details see upsert behavior.
Updates documents that match the filter.
To avoid multiple upserts, ensure that the filter fields are uniquely indexed.
Defaults to false.
db.users.updateMany(
{},
{$set: users},
{
upsert: true
}
)
Related
I have a collection in MongoDB with more than 5 million documents. Whenever I create a document inside the same collection I have to check if there exists any document with same title and if it exists then I don't have to add this to the database.
Example: here is my MongoDB document:
{
"_id":ObjectId("3a434sa3242424sdsdw"),
"title":"Lost in space",
"desc":"this is description"
}
Now whenever a new document is being created in the collection, I want to check if the same title already exists in any of the documents and if it does not exists, then only I want to add it to the database.
Currently, I am using findOne query and checking for the title, if it not available only then it is added to the database. I am facing the performance issue in this. It is taking too much time to do this process. Please suggest a better approach.
async function addToDB(data){
let result= await db.collection('testCol').findOne({title:data.title});
if(result==null){
await db.collection('testCol').insertOne(data);
}else{
console.log("already exists in db");
}
}
You can reduce the network round trip time which is currently 2X. Because you execute two queries. One for find then one for update. You can combine them into one query as below.
db.collection.update(
<query>,
{ $setOnInsert: { <field1>: <value1>, ... } },
{ upsert: true }
)
It will not update if already exists.
db.test.update(
{"key1":"1"},
{ $setOnInsert: { "key":"2"} },
{ upsert: true }
)
It looks for document with key1 is 1. If it finds, it skips. If not, it inserts using the data provided in the object of setOnInsert.
as the title says, I want to perform a find (one) for a document, by _id, and if doesn't exist, have it created, then whether it was found or was created, have it returned in the callback.
I don't want to update it if it exists, as I've read findAndModify does. I have seen many other questions on Stackoverflow regarding this but again, don't wish to update anything.
I am unsure if by creating (of not existing), THAT is actually the update everyone is talking about, it's all so confuzzling :(
Beginning with MongoDB 2.4, it's no longer necessary to rely on a unique index (or any other workaround) for atomic findOrCreate like operations.
This is thanks to the $setOnInsert operator new to 2.4, which allows you to specify updates which should only happen when inserting documents.
This, combined with the upsert option, means you can use findAndModify to achieve an atomic findOrCreate-like operation.
db.collection.findAndModify({
query: { _id: "some potentially existing id" },
update: {
$setOnInsert: { foo: "bar" }
},
new: true, // return new doc if one is upserted
upsert: true // insert the document if it does not exist
})
As $setOnInsert only affects documents being inserted, if an existing document is found, no modification will occur. If no document exists, it will upsert one with the specified _id, then perform the insert only set. In both cases, the document is returned.
Driver Versions > 2
Using the latest driver (> version 2), you'll use findOneAndUpdate as findAndModify was deprecated. The new method takes 3 arguments, the filter, the update object (which contains your default properties, that should be inserted for a new object), and options where you have to specify the upsert operation.
Using the promise syntax, it looks like this:
const result = await collection.findOneAndUpdate(
{ _id: new ObjectId(id) },
{
$setOnInsert: { foo: "bar" },
},
{
returnOriginal: false,
upsert: true,
}
);
const newOrUpdatedDocument = result.value;
Its a bit dirty, but you can just insert it.
Be sure that the key has a unique index on it (if you use the _id it's ok, it's already unique).
In this way if the element is already present it will return an exception that you can catch.
If it isn't present, the new document will be inserted.
Updated: a detailed explanation of this technique on the MongoDB Documentation
Here's what I did (Ruby MongoDB driver):
$db[:tags].update_one({:tag => 'flat'}, {'$set' => {:tag => 'earth' }}, { :upsert => true })}
It will update it if it exists, and insert it if it doesn't.
I have an array of objects and I want to store them in a collection using only one I/O operation if it's possible. If any document already exists in the collection I want to replace it, or insert it otherwise.
These are the solutions that I found, but doesn't work exactly as I want:
insertMany(): this doesn't replace the document that already exists, but throws exception instead (This is what I found in the Mongodb documentation, but I don't know if it's the same as mongoose).
update() or updateMany() with upsert = true: this doesn't help me as well, because here I have to do the same updates to all the to stored documents.
There is no replaceMany() in mongodb or mongoose.
Is there anyone how knows any optimal way to do replaceMany using mongoose and node.js
There is bulkWrite (https://docs.mongodb.com/manual/reference/method/db.collection.bulkWrite/), which makes it possible to execute multiple operations at once. In your case, you can use it to perform multiple replaceOne operations with upsert. The code below shows how you can do it with Mongoose:
// Assuming *data* is an array of documents that you want to insert (or replace)
const bulkData = data.map(item => (
{
replaceOne: {
upsert: true,
filter: {
// Filter specification. You must provide a field that
// identifies *item*
},
replacement: item
}
}
));
db.bulkWrite(bulkData);
You need to query like this:
db.getCollection('hotspot').update({
/Your Condition/
}, {
$set: {
"New Key": "Value"
}
}, {
multi: true,
upsert: true
});
It fulfils your requirements..!!!
I'm trying to update every document in an expanding Mongo database.
My plan is to start with the youngest, most recently created document and work back from there, one-by-one querying the next oldest document.
The problem is that my Mongoose query is skipping documents that were created in the same second. I thought greater than/less than operators would work on _ids generated in the same second. But though there are 150 documents in the database right now, this function gets from the youngest to the oldest document in only 8 loops.
Here's my Mongoose query within the recursive node loop:
function loopThroughDatabase(i, doc, sizeOfDatabase){
if (i < sizeOfDatabase) {
(function(){
myMongooseCollection.model(false)
.find()
.where("_id")
.lt(doc._id)
.sort("id")
.limit(1)
.exec(function(err, docs) {
if (err) {
console.log(err);
}
else {
updateDocAndSaveToDatabase(docs[0]);
loopThroughDatabase(i + 1, docs[0], sizeOfDatabase); //recursion here
}
});
})();
}
}
loopThroughDatabase(1, youngestDoc, sizeOfDatabase);
Error found.
In the Mongoose query, I was sorting by "id" rather than "_id"
If you read the MongoDB documentation, you will see that it depends on the process in which the item was created http://docs.mongodb.org/manual/reference/glossary/#term-objectid, therefore, to guarantee what you need, you need to add a Date stamp to the records and use that instead of the _id
as the title says, I want to perform a find (one) for a document, by _id, and if doesn't exist, have it created, then whether it was found or was created, have it returned in the callback.
I don't want to update it if it exists, as I've read findAndModify does. I have seen many other questions on Stackoverflow regarding this but again, don't wish to update anything.
I am unsure if by creating (of not existing), THAT is actually the update everyone is talking about, it's all so confuzzling :(
Beginning with MongoDB 2.4, it's no longer necessary to rely on a unique index (or any other workaround) for atomic findOrCreate like operations.
This is thanks to the $setOnInsert operator new to 2.4, which allows you to specify updates which should only happen when inserting documents.
This, combined with the upsert option, means you can use findAndModify to achieve an atomic findOrCreate-like operation.
db.collection.findAndModify({
query: { _id: "some potentially existing id" },
update: {
$setOnInsert: { foo: "bar" }
},
new: true, // return new doc if one is upserted
upsert: true // insert the document if it does not exist
})
As $setOnInsert only affects documents being inserted, if an existing document is found, no modification will occur. If no document exists, it will upsert one with the specified _id, then perform the insert only set. In both cases, the document is returned.
Driver Versions > 2
Using the latest driver (> version 2), you'll use findOneAndUpdate as findAndModify was deprecated. The new method takes 3 arguments, the filter, the update object (which contains your default properties, that should be inserted for a new object), and options where you have to specify the upsert operation.
Using the promise syntax, it looks like this:
const result = await collection.findOneAndUpdate(
{ _id: new ObjectId(id) },
{
$setOnInsert: { foo: "bar" },
},
{
returnOriginal: false,
upsert: true,
}
);
const newOrUpdatedDocument = result.value;
Its a bit dirty, but you can just insert it.
Be sure that the key has a unique index on it (if you use the _id it's ok, it's already unique).
In this way if the element is already present it will return an exception that you can catch.
If it isn't present, the new document will be inserted.
Updated: a detailed explanation of this technique on the MongoDB Documentation
Here's what I did (Ruby MongoDB driver):
$db[:tags].update_one({:tag => 'flat'}, {'$set' => {:tag => 'earth' }}, { :upsert => true })}
It will update it if it exists, and insert it if it doesn't.