I scouted around to find the right solution for inserting a large amount of documents to MongoDB using Mongoose.
My current solution looks like this:
MongoClient.saveData = function(RecordModel, data, priority, SCHID, callback){
var dataParsed = parseDataToFitSchema(data, priority, SCHID);
console.log("Model created. Inserting in batches.");
RecordModel.insertMany(dataParsed)
.then(function(mongooseDocuments) {
console.log("Insertion was successful.");
})
.catch(function(err) {
callback("Error while inserting data to DB: "+err);
return;
})
.done(function() {
callback(null);
return;
});
}
But it appears to me there are other offered solutions out there. Like this one:
http://www.unknownerror.org/opensource/Automattic/mongoose/q/stackoverflow/16726330/mongoose-mongodb-batch-insert
Using collection.insert. How is that different to the Model.insertMany?
Same goes for update, my previous question: What is the right approach to update many records in MongoDB using Mongoose
Asks how do I update big chunk of data with Mongoose, defined by _id. The answer suggests to use collection.bulkWrite while I am under impression Model.insertMany can do it too.
Related
I am trying out loopback-connector-couch database operator for deleting multiple records from CouchDB in one shot.
I have records _id in an array
Following the API spec for Loopback model as provided in the below api doc
https://apidocs.strongloop.com/loopback/#persistedmodel-destroyall
persist_model.destroyAll({id:{or:["49c199312b7dce75d69124f9e377a682","49c199312b7dce75d69124f9e377a682"]}},function(err,res){
console.log('records deleted success:'+res);
});
When executing the above code, it does not throw any error but its not deleting any documents
Would appreciate any help on this :)
The correct way is to use inq operator if you want to compare the value with multiple value stored in an array as follows:
persist_model.destroyAll({ id: { inq: [ "49c199312b7dce75d69124f9e377a682", "49c199312b7dce75d69124f9e377a682" ]}},
function(err, res) {
if (err) {
throw err;
}
console.log('records deleted success:' + res);
}
);
You can read more about inq operator here.
I'm working in the MEAN stack, and I'm stuck on a Mongoose issue. Updates are working fine for individual documents(single key:value), but when I attempt to update a nested object with a new object, it's just deleting the original object and not inserting the new one.
Using an identical query directly in mongo, within the terminal, works perfectly. But from my Mongoose model, I get the above behavior.
My final desired query is more complicated and uses many variables, so I've simplified the code to the following to highlight the issue:
Setup.update({name: "main"}, {$set: {"schedule.sunday.eleven_pm": { associates: 111, supervisors: 111}}}, function(err){
if(err){
console.log(err);
}
else{
console.log('successfully updated main schedule setup')
Setup.find({}, function(err, setup){
if(err){
console.log(err);
}
else{
res.json(setup);
}
})
}
});
Gives me this in my db:
"eleven_pm" : {
}
But from terminal, the same query (cut and pasted from my mongoose query, just added 'db.setups' to beginning:
db.setups.update({name: "main"}, {$set: {"schedule.sunday.eleven_pm": { associates: 111, supervisors: 111}}})
Gives me this, the desired result:
"eleven_pm" : {
"associates" : 111,
"supervisors" : 111
}
I've tried writing this as a findOneAndUpdate(), but encountered the same behavior. Am I doing something unorthodox here?
Any help greatly appreciated.
A small change in my database structure was not amended in my Mongoose "Schema", and the difference between the two led to the unusual query behavior explained above.
I'm trying to update every document in an expanding Mongo database.
My plan is to start with the youngest, most recently created document and work back from there, one-by-one querying the next oldest document.
The problem is that my Mongoose query is skipping documents that were created in the same second. I thought greater than/less than operators would work on _ids generated in the same second. But though there are 150 documents in the database right now, this function gets from the youngest to the oldest document in only 8 loops.
Here's my Mongoose query within the recursive node loop:
function loopThroughDatabase(i, doc, sizeOfDatabase){
if (i < sizeOfDatabase) {
(function(){
myMongooseCollection.model(false)
.find()
.where("_id")
.lt(doc._id)
.sort("id")
.limit(1)
.exec(function(err, docs) {
if (err) {
console.log(err);
}
else {
updateDocAndSaveToDatabase(docs[0]);
loopThroughDatabase(i + 1, docs[0], sizeOfDatabase); //recursion here
}
});
})();
}
}
loopThroughDatabase(1, youngestDoc, sizeOfDatabase);
Error found.
In the Mongoose query, I was sorting by "id" rather than "_id"
If you read the MongoDB documentation, you will see that it depends on the process in which the item was created http://docs.mongodb.org/manual/reference/glossary/#term-objectid, therefore, to guarantee what you need, you need to add a Date stamp to the records and use that instead of the _id
Save() giving me error like "Object has no method 'save'"
Country.update({id:req.param('country_id')},model).exec(function(err,cntry){
if(err) return res.json(err);
if(!cntry.image){
cntry.image = 'images/countries/'+filename;
cntry.save(function(err){ console.log(err)});
}
})
Any Idea about how to save model within update query . ??
Assuming you're using Waterline and sails-mongo, the issue here is that update returns an array (because you can update multiple records at once), and you're treating it like a single record. Try:
Country.update({id:req.param('country_id')},model).exec(function(err,cntry){
if(err) return res.json(err);
if(cntry.length === 0) {return res.notFound();}
if(!cntry[0].image){
cntry[0].image = 'images/countries/'+filename;
cntry[0].save(function(err){ console.log(err)});
}
});
This seems to me an odd bit of code, though; why not just check for the presence of image in model before doing Country.update and alter model (or a copy thereof) accordingly? That would save you an extra database call.
When using mongoose (3.8) to update the database directly the callback function receives 3 parameters, none of then is a mongoose object of the defined model. The parameters are:
err is the error if any occurred
numberAffected is the count of updated documents Mongo reported
rawResponse is the full response from Mongo
The right way is, first you fetch and then change the data:
Country.findOne({id: req.param('country_id')}, function (err, country) {
// do changes
})
Or using the update method, the way you intended:
Country.update({id: req.param('country_id'), image: {$exists: false}}, {image: newValue}, callback)
I have this:
exports.deleteSlide = function(data,callback){
customers.findOne(data.query,{'files.$':1},function(err,data2){
if(data2){
console.log(data2.files[0]);
data2.files[0].slides.splice((data.slide-1),1);
data2.files[0].markModified('slides');
data2.save(function(err,product,numberAffected){
if(numberAffected==1){
console.log("manifest saved");
var back={success:true};
console.log(product.files[0]);
callback(back);
return;
}
});
}
});
}
I get the "manifest saved" message and a callback with success being true.
When I do the console.log when I first find the data, and compare it with the console.log after I save the data, it looks like what I expect. I don't get any errors.
However, when I look at the database after running this code, it looks like nothing was ever changed. The element that I should have deleted, still appears?
What's wrong here?
EDIT:
For my query, I do {'name':'some string','files.name':'some string'}, and if the object is found, I get an array of files with one object in it.
I guess this is a subdoc.
I've looked around and it says the rules for saving subdocs are different than saving the entire collection, or rather, the subdocs are only applied when the root object is saved.
I've been going around this by grabbing the entire root object, then I do loops to find the actual subdoc I that I want, and after I manipulate that, I save the whole object.
Can I avoid doing this?
I'd probably just switch to using native drivers for this query as it is much simpler. (For that matter, I recently dropped mongoose on my primary project and am happy with the speed improvements.)
You can find documentation on getting access to the native collection elsewhere.
Following advice here:
https://stackoverflow.com/a/4588909/68567
customersNative.update(data.query, {$unset : {"slides.1" : 1 }}, function(err){
if(err) { return callback(err); }
customersNative.findAndModify(data.query, [],
{$pull: {'slides' : null } }, {safe: true, 'new' : true}, function(err, updated) {
//'updated' has new object
} );
});