i am having an issue with this piece of code.
(Using nodes, express, mongoose to save documents to mongodb)
// We have an array that contains objects (365) of type 'Day' to be stored in mongodb.
// In my code this array contains data :-)
let arrDays = []
// Empty array to hold reference to the _id of the created days
let arrDaysID = [];
// Store all days in the 'arrDays' array to mongodb using mongoose
Day.create(arrDays, function(err, days){
if(err){
console.log("Something went wrong " + err)
} else {
// Iterate over every 'day' object in mongodb and add a reference to
// the _id of the 'day' object inside the 'year' object
days.forEach(function(day){
year.days.push(day._id);
year.save();
});
}
});
The problem is that instead of adding every _id reference in the 'year' object once, the _id reference is added multiple times. When the code finished running, about 140.000 references are inside the year.days array while there are only 365 days defined...
Any hints or tips are welcome.
Dave
As per the documentation of model.create API, it is a Shortcut for saving one or more documents to the database. MyModel.create(docs) does new MyModel(doc).save() for every doc in docs.
This means, you are executing the forEach loop for every day added in create method making it an n*n complex function.
365 * 365 = 133225 which is approximately equal to 140000 records.
I think this explains the problem.
EDIT - Better Alternative
As per mongoose insertMany API, Model.insertMany() is faster than Model.create because it only sends one operation to the server, rather than one for each document
Related
I am using graphql to get some data from mongodb database. So, I was making an api which on running saves data in main collection but also saves data in some other collection with a couple of more data. I was trying to delete _id from the data that I get on saving on main collection but it's not working and I can't figure out why.
Here's what's happening
const data = await User.getResolver('updateById').resolve({ //default resolver from graphql-compose library to update a record. Returns the whole document in record //
args: {...rp.args}
})
const version = '4'
const NewObject = _.cloneDeep(data.record);
NewObject.version = version;
NewObject.oldId = data._id;
_.unset(NewObject, '_id'); //this doesn't work neither does delete NewObject._id//
console.log('new obj is', NewObject, version, NewObject._id, _.unset(NewObject, '_id')); //prints _id and surprisingly the unset in console.log is returning true that means it is successfully deleting the property//
I am very confused to what I am doing wrong.
Edit: Sorry should have mentioned, but according to lodash docs _.unset returns true if it successfully deletes a property.
Turn's out mongoDb makes _id as a Non configurable property. So, to do this I had to make use of toObject() method to make an editable copy of my object.
const NewObject = _.cloneDeep(data.record).toObject();
With this I was able to delete _id.
Alternatively _.omit of lodash also works.
I have an array like this on under user collection
How can i remove a single value from this array by using sails.js code
also i need to remove all post id from every users document if a post is deleted, is there any simplest and fastest way to remove from all documents too??
There are a couple of easy ways to do it if you're using the built-in ORM "Waterline", and they are basically equivalent.
The first is to run Model.update() and overwrite the array with the new value.
Model.update(id, { likes : [/* new array value */] }).then(...)
The other is to 1) find the object 2) pull the value off the array 3) save:
Model.findOne(id, function(err, document){
if(err) // handle err case
else {
document.likes = document.likes.filter(value => value !== 'stringToRemove')
document.save(function(err, saved){
... // do more stuff
})
}
})
Finally, unless you're using it for a very simple app, I cannot recommend using Waterline ORM with MongoDB. Save yourself a heap of headache and use Mongoose.
I got a generic GET function that worps for my entire app, considering I am using just absolute documents.
Now I get to a point that I need some properties of some of my documents reference others, and when executed, the GET function populate them (obviously). For that, I need to require the referenced schema, and populate with referenced model.
The point is: I want to my GET function stay generic, so I don't want to reference any of my schemas, unless it is needed. The same goes for the .populate() method.
To achieve that, I am iterating through each key of the resulting object of the .findOne() method, and trying to check if each specific key, is or is not a ObjectId/reference or not. something like this:
require('../schemas/mySchema').findOne({'slug': req.params.slug}, function(err, doc){
console.log(mongoose.Types.ObjectId.isValid(doc[key]));
});
But the only true value it returns is for the "id"and "__v" properties (no idea where these came from... I did not set them. _id is also false), all the rest comes as false (including a given property that IS a reference, tested and working)
Is there any way to do that?
Thanks in advance
I believe mongoose returns references with the objectId nested - in the same structure as a populated object but having only the _id key. Try this:
var item = doc[key];
if (typeof item === 'object') {
console.log(mongoose.Types.ObjectId.isValid(item._id));
}
I'm trying to update every document in an expanding Mongo database.
My plan is to start with the youngest, most recently created document and work back from there, one-by-one querying the next oldest document.
The problem is that my Mongoose query is skipping documents that were created in the same second. I thought greater than/less than operators would work on _ids generated in the same second. But though there are 150 documents in the database right now, this function gets from the youngest to the oldest document in only 8 loops.
Here's my Mongoose query within the recursive node loop:
function loopThroughDatabase(i, doc, sizeOfDatabase){
if (i < sizeOfDatabase) {
(function(){
myMongooseCollection.model(false)
.find()
.where("_id")
.lt(doc._id)
.sort("id")
.limit(1)
.exec(function(err, docs) {
if (err) {
console.log(err);
}
else {
updateDocAndSaveToDatabase(docs[0]);
loopThroughDatabase(i + 1, docs[0], sizeOfDatabase); //recursion here
}
});
})();
}
}
loopThroughDatabase(1, youngestDoc, sizeOfDatabase);
Error found.
In the Mongoose query, I was sorting by "id" rather than "_id"
If you read the MongoDB documentation, you will see that it depends on the process in which the item was created http://docs.mongodb.org/manual/reference/glossary/#term-objectid, therefore, to guarantee what you need, you need to add a Date stamp to the records and use that instead of the _id
NodeJS + Express, MongoDB + Mongoose
I have a JSON feed where each record has a set of "venue" attributes (things like "venue name" "venue location" "venue phone" etc). I want to create a collection of all venues in the feed -- one instance of each venue, no dupes.
I loop through the JSON and test whether the venue exists in my venue collection. If it doesn't, save it.
jsonObj.events.forEach(function(element, index, array){
Venue.findOne({'name': element.vname}, function(err,doc){
if(doc == null){
var instance = new Venue();
instance.name = element.vname;
instance.location = element.location;
instance.phone = element.vphone;
instance.save();
}
}
}
Desired: A list of all venues (no dupes).
Result: Plenty of dupes in the venue collection.
Basically, the loop created a new Venue record for every record in the JSON feed.
I'm learning Node and its async qualities, so I believe the for loop finishes before even the first save() function finishes -- so the if statement is always checking against an empty collection. Console.logging backs this claim up.
I'm not sure how to rework this so that it performs the desired task. I've tried caolan's async module but I can't get it to help. There's a good chance I'm using incorrectly.
Thanks so much for pointing me in the right direction -- I've searched to no avail. If the async module is the right answer, I'd love your help with how to implement it in this specific case.
Thanks again!
Why not go the other way with it? You didn't say what your persistence layer is, but it looks like mongoose or possibly FastLegS. In either case, you can create a Unique Index on your Name field. Then, you can just try to save anything, and handle the error if it's a unique index violation.
Whatever you do, you must do as #Paul suggests and make a unique index in the database. That's the only way to ensure uniqueness.
But the main problem with your code is that in the instance.save() call, you need a callback that triggers the next iteration, otherwise the database will not have had time to save the new record. It's a race condition. You can solve that problem with caolan's forEachSeries function.
Alternatively, you could get an array of records already in the Venue collection that match an item in your JSON object, then filter the matches out of the object, then iteratively add each item left in the filtered JSON object. This will minimize the number of database operations by not trying to create duplicates in the first place.
Venue.find({'name': { $in: jsonObj.events.map(function(event){ return event.vname; }) }}, function (err, docs){
var existingVnames = docs.map(function(doc){ return doc.name; });
var filteredEvents = jsonObj.events.filter(function(event){
return existingVnames.indexOf(event.vname) === -1;
});
filteredEvents.forEach(function(event){
var venue = new Venue();
venue.name = event.vname;
venue.location = event.location;
venue.phone = event.vphone;
venue.save(function (err){
// Optionally, do some logging here, perhaps.
if (err) return console.error('Something went wrong!');
else return console.log('Successfully created new venue %s', venue.name);
});
});
});