I have millions of record in source collection, I am trying to perform operation bulkWirte with $setOnInsert, like if incoming record already exists in destination collection then just skip it.
But somehow this operation is not working, I don't know if there is change in behavior of $setOnInsert or it doesn't work in bulk update case.
If I perform same operation without bulkWrite then its working as expected.
This is code part of code where I tried with batch update
bulkArr.push({
updateOne: {
filter: {
date: { $gte: new Date(from).getTime(), $lte: new Date(to).getTime() }, "uid": mergedObj.uid, "baseId": mergedObj.baseId,
"lessonId": mergedObj.lessonId, "pageId": mergedObj.pageId, "site": mergedObj.site
},
update: {
$setOnInsert: mergedObj
},
upsert: true
}
});
if (bulkArr.length % batchSize === 0) {
batchCounter = batchCounter + 1;
stream.pause();
db.collection('saq_temp_jan').bulkWrite(bulkArr, (err, result) => {
if (err) {
console.log(err);
} else {
bulkArr = [];
console.log('Upserted count>>>', result.nUpserted);
console.log('Matched count>>>', result.nMatched);
console.log('Batch counter>>>', batchCounter);
stream.resume();
}
});
Any suggestion would be most welcome.
Related
In my code, I have to push some value in the reported array of collections of activity doc. so i use this code
return new Promise(async (resolve, reject) => {
const logData = {
'reportedBy': body.reportedBy,
'reportedAt': new Date().toISOString(),
'reportReason': body.reportReason
}
await model.findOneAndUpdate({ _id: body.id },
{
$addToSet: { 'reported.log': [logData] },
$inc: {
'reported.count': 1
}
},
{ new: true }, (err, activity) => {
createError(err, reject);
resolve(activity);
});
});
}
After successfully submit it show me count increase by 1 but in MongoDB, it shows a count 2 can any please guide me about what I miss in the above code.
You are using async & callback styles which makes the operation execute twice. Pick one style.
See also UpdateMany in MongoDB running twice with $inc
I have an application developed in NodeJS, which works as a REST API and consumes data from MongoDB
In MongoDB I have a collection called 'ftp' with more than 10 million documents with the following structure
{
"_id" : ObjectId("59e7c66911506bd1725cf145"),
"ip" : "72.32.177.76",
"timestamp" : "2017-10-16T02:30:26-04:00",
"data" : {
"banner" : "220-FileZilla Server version 0.9.41 beta\r\n"
}
}
The "data.banner" field is a hased index
From NoodeJs I make an aggregate query that filters a string of text using a regular expression, groups and counts the results.
function getData(timeInit, table, textSearch, res) {
MongoClient.connect(url, function (err, db) {
if (err) throw err;
db.collection(table).aggregate([
{
$match: { 'data.banner': $regex:textSearch}
}, {
$group: {
_id: '$data.banner',
num: { $sum: 1 },
}
},
{
$sort: {
num: -1
}
},{
$limit:5
}
], {
allowDiskUse: true
}
).toArray(function (err, result) {
if (err) throw err;
var timeFinal = new Date();
var data = {
result: result,
timeLapse: (timeFinal - timeInit) / 1000,
numResult: result.length
};
res.send(data);
db.close();
});
});
};
The query with regular expression takes about 8 seconds to return results, an excessive time in my opinion, since the regular expressions are not optimal.
My question is how should I make the filter to search for documents that contain text in an optimal way reducing the response time.
If someone knows how to optimize this type of query I would appreciate it a lot.
How do I make a query in mongoose to find if a user has 50 documents then remove the oldest one and add in the new one, if not just add in the new one?
This is my attempt:
Notifications.findOne({_id: userId}, function(err, results) {
if(err) throw err;
if(results.length < 50) {
saveNotification();
} else {
Notifications.findByIdAndUpdate(userId, {pull: //WHAT GOES HERE),
function(err, newNotify) {
if(error) throw error;
saveNotification();
});
}
});
function saveNotification() {
var new_notification = new Notification ({
notifyToUserId: creatorId,
notifyFromUserId: null,
notifyMsg: newmsg,
dateNotified: dateLastPosted
});
new_notification.save(function(err, results){
if(!err) {
console.log('notification has been added to the db');
cb(null, resultObject);
} else {
console.log("Error creating notification " + err);
}
});
}
As #Pio mentioned I don't think you can do it in one query with your current schema. But if you have chance to change the schema, you can use fixed size array pattern that is described in the following article Limit Number of Elements in an Array after an Update
Basically you can keep the notifications of users in one document. Key of the document will be userId, and notifications will be stored in an array. Then the following query would achieve your goal.
Notifications.update(
{ _id: userId },
{
$push: {
notifications: {
$each: [ notificationObject ], // insert your new notification
$sort: { dateNotified: 1 }, // sort by insertion date
$slice: -50 // retrieve the last 50 notifications.
}
}
}
)
I am not sure you can do it in one query, but you can
.count({user: yourUser'}) then depending on the count .insert(newDocument) or update the oldest one so you won't remove + insert.
Capped collections do what you want by nature. If you define a capped collection with size 50 it will only keep 50 documents and will overwrite old data when you insert more.
check
http://mongoosejs.com/docs/guide.html#capped
http://docs.mongodb.org/manual/core/capped-collections/
new Schema({..}, { capped: { size: 50, max: 50, autoIndexId: true } });
Remember that when working with capped collection you can only make inplace updates. Updating whole document may change the size of collection that will remove other documents.
I ended up using cubbuk's answer and expanding it to add a notification if there is no array to start with along with upsert...
Notification.findOneAndUpdate({notifyToUserId: to}, {
$push: {
notifyArray: {$each: [newNotificationObject], // insert your new notification
$sort: { dateNotified: 1 }, // sort by insertion date
$slice: -50 // retrieve the last 50 notifications.
}
}
}, {upsert: true}, function(err, resultOfFound) {
if(err) throw err;
if(resultOfFound == null) {
var new_notification = new Notification ({
notifyToUserId: to,
notifyArray: [newNotificationObject]
});
new_notification.save(function(err, results){
if(!err) {
console.log('notification has been added to the db');
cb(null, resultObject);
} else {
console.log("Error creating notification " + err);
}
});
} else {
cb(null, resultObject);
}
});
I am trying to get by code the next sequence number but it always says "undefined".
I did this in my mongoDB before:
db.PresentationCollection.insert(
{
_id: "editorID",
seq: 0
}
)
my code (name is editorID):
function getNextSequence(name, db) {
var collection = db.get('PresentationCollection');
var ret = collection.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
You're missing the callback. Callback-based asynchronous functions generally do not return anything meaningful. See the documentation for findAndModify in the node binding's readme.
I had the same problem from following this link and it is indeed the callback not being specified and your code not waiting for the returned result - mongo db documents create auto increment
Here is what I did to solve it. Keep in mind I am using Q for promise helping but you could use straight up javascript promises.
function _getNextSequence(name) {
var deferred = Q.defer();
db.counters.findAndModify(
{ _id: name }, //query
[], //sort
{ $inc: { seq: 1 } }, //update
{ new:true }, //options
function(err, doc) { //callback
if (err) deferred.reject(err.name + ': ' + err.message);
if (doc){
deferred.resolve(doc.value.seq);
}
});
return deferred.promise;
}
I am struggling with very weird problem in MongoDB, what I want is that the query updates document if the condition matches and if not creates new (upsert).
The problem:
I get the correct results from the callback, as this returns the newly inserted document. But what appears the query gets never inserted in database.
What I think causes this, is the $set part, is looking for array but instead this must be object:
boards [
// Looks for element 48 and inserts there
]
{ '$set':
{ 'boards.48.acl': 'ReadWrite',
'boards.48.status': 'goal' } }
Here is the query:
// the 11 is example input
var scoreKey = util.format('scores.' + 11),
aclKey = scoreKey + '.acl',
statusKey = scoreKey + '.status',
setQuery = {
$set: { }
};
setQuery['$set'][aclKey] = acl;
setQuery['$set'][statusKey] = status;
db.sessions.findAndModify({
query: { '_id': sid },
update: setQuery,
upsert: true,
new: true
}, function (err, res) {
if (err) return cb(err);
else {
console.log(res);
return cb(null, res);
}
});
// Update
Well it seems all kind of the changes to database are silently failing, so the query seems to be correct.