I am in situation where I have to update either two documents or none of them, how is it possible to implement such behavior with mongo?
// nodejs mongodb driver
Bus.update({
"_id": { $in: [ObjectId("abc"), ObjectId("def")] },
"seats": { $gt: 0 }
}, {
$inc: { "seats": -1 }
}, { multi: true }, function(error, update) {
assert(update.result.nModified === 2)
})
The problem with code above it will update even if only one bus matched. In my case I try to book ticket for bus in both directions and should fail if at least one of them already fully booked.
Thank you
Related
I am encountering a delay of 5 to 10 seconds from when the operation happens in MongoDB until I capture it in a Change Stream in NodeJS.
Are these times normal, what parameters could I check to see if any are impacting this?
Here are a couple of examples and some suspicions (to be tested).
Here we try to catch changes only in the fields of the Users collection that interest us, I do not know if doing this to avoid unwanted events may be causing delay in the reception of the ChangeStream and it would be convenient to receive more events and filter in code the updated fields.
I do not know, also if the "and" of the type of operation would have to be put before or it is irrelevant.
userChangeStreamQuery: [{
$match: {
$and: [
{$or:[
{ "updateDescription.updatedFields.name": { $exists: true } },
{ "updateDescription.updatedFields.email": { $exists: true } },
{ "updateDescription.updatedFields.organization": { $exists: true } },
{ "updateDescription.updatedFields.displayName": { $exists: true } },
{ "updateDescription.updatedFields.image": { $exists: true } },
{ "updateDescription.updatedFields.organizationName": { $exists: true } },
{ "updateDescription.updatedFields.locationName": { $exists: true } }
]},
{ operationType: "update" }]
}
}],
Of this other one, that waits for events on the Plans collection, I worry that it does not have aggregate defined and it is when receiving the event where it is filtered if the operation arrives type 'insert', 'update', 'delete'. This one is giving us a delay of 7~10 seconds.
startChangeStream({
streamId: 'plans',
collection: 'plans',
query: '',
resumeTokens
});
...
const startChangeStream = ({ streamId, collection, query, resumeTokens }) => {
const resumeToken = resumeTokens ? resumeTokens[streamId] || undefined : undefined;
nativeMongoDbFactory.setChangeStream({
streamId,
collection,
query,
resumeToken
});
}
In no case are massive operations, normally they are operations performed by the user through web forms.
when the collection is sharding, using change streams the mongos server need to wait until all shards have data to return, if some shards no data to write, the idle primary mongod writes a no-op to the oplog every 10 (idlewriteperiodms) seconds. that is why you delay is 7~10 seconds.
So I'm currently trying to perform this operation
return this.model.findByIdAndUpdate(id, { $push: { certifiedBy: certifier } }, { $inc: {score: 1}}, { new: true })
The issue here is that score will grow without limit, I would like to prevent this and make it so when this is happening it cannot increment if score <= 5 but still add certifier into my certifiedBy array.
Can it be done with mongoose directly or do I have to get the object first check if it over 5 and call a different query in that case ?
Thanks
You can't change the $inc behaviour but you can do a checkpoint to stop it before 5
return this.model.findOneAndUpdate({
_id: id,
score: {
$lte: 5
}
}, {
$push: { certifiedBy: certifier },
$inc: { score: 1 }
},
{
new: true
})
I am looking to update X documents all at once. The short is I basically need to randomly select N documents and then update them as "selected". I'm trying to design an API that needs to randomly distribute questions. I can not find a way to do this in mongoose I have tried:
update ends up selecting everything
Question
.update({}, {
$inc: {
answerCount: 1,
lockedCount: 1
},
$push:{
devices: deviceID
}
}, {multi:true})
.limit(4)
--- I also tried
Question
.find()
.sort({
answerCount: 1,
lockedCount: 1
})
.limit(req.query.limit || 4)
.update({}, {
$inc: {
answerCount: 1,
lockedCount: 1
},
$push:{
devices: deviceID
}
}, { multi: true }, callback);
Both resulted in updating all docs. Is there a way to push this down to mongoose without having to use map ? The other thing I did not mention is .update() without multi resulted in 1 document being updated.
You could also pull an array of _ids that you'd like to update then run the update query using $in. This will require two calls to the mongo however the updates will still be atomic:
Question.find().select("_id").limit(4).exec(function(err, questions) {
var q = Question.update({_id: {$in: questions}}, {
$inc: {answerCount: 1, lockedCount:1},
$push: {devices: deviceid}
}, {multi:true});
q.exec(function(err) {
console.log("Done");
});
});
So I did an simple map implementation and will use it unless someone can find a more efficient way to do it.
Question
.find({
devices: { $ne: deviceID}
},
{ name: true, _id: true})
.sort({
answerCount: 1,
lockedCount: 1
})
.limit(req.query.limit || 4)
.exec(updateAllFound );
function updateAllFound(err, questions) {
if (err) {
return handleError(res, err);
}
var ids = questions.map(function(item){
return item._id;
});
return Question.update({ _id: { $in: ids} } ,
{
$inc: {
answerCount: 1,
lockedCount: 1
},
$push:{
devices: deviceID
}
}, { multi: true }, getByDeviceID);
function getByDeviceID(){
return res.json(200, questions);
}
}
I am trying to aggregate some records in a mongo database using the node driver. I am first matching to org, fed, and sl fields (these are indexed). If I only include a few companies in the array that I am matching the org field to, the query runs fine and works as expected. However, when including all of the clients in the array, I always get:
MongoError: getMore: cursor didn't exist on server, possible restart or timeout?
I have tried playing with the allowDiskUse, and the batchSize settings, but nothing seems to work. With all the client strings in the array, the aggregation runs for ~5hours before throwing the cursor error. Any ideas? Below is the pipeline along with the actual aggregate command.
setting up the aggregation pipeline:
var aggQuery = [
{
$match: { //all clients, from last three days, and scored
org:
{ $in : array } //this is the array I am talking about
,
frd: {
$gte: _.last(util.lastXDates(3))
},
sl : true
}
}
, {
$group: { //group by isp and make fields for calculation
_id: "$gog",
count: {
$sum: 1
},
countRisky: {
$sum: {
$cond: {
if :{
$gte: ["$scr", 65]
},
then: 1,
else :0
}
}
},
countTimeZoneRisky: {
$sum: {
$cond: {
if :{
$eq: ["$gmt", "$gtz"]
},
then: 0,
else :1
}
}
}
}
}
, {
$match: { //show records with count >= 500
count: {
$gte: 500
}
}
}
, {
$project: { //rename _id to isp, only show relevent fields
_id: 0,
ISP: "$_id",
percentRisky: {
$multiply: [{
$divide: ["$countRisky", "$count"]
},
100
]
},
percentTimeZoneDiscrancy: {
$multiply: [{
$divide: ["$countTimeZoneRisky", "$count"]
},
100
]
},
count: 1
}
}
, {
$sort: { //sort by percent risky and then by count
percentRisky: 1,
count: 1
}
}
];
Running the aggregation:
var cursor = reportingCollections.hitColl.aggregate(aggQuery, {
allowDiskUse: true,
cursor: {
batchSize: 40000
}
});
console.log('Writing data to csv ' + currentFileNamePrefix + '!');
//iterate through cursor and write documents to CSV
cursor.each(function (err, document) {
//write each document to csv file
//maybe start a nuclear war
});
You're calling the aggregate method which doesn't return the cursor by default (like e.g. find()). To return query as a cursor, you must add the cursor option in the options. But, the timeout setting for the aggregation cursor is (currently) not supported. The native node.js driver only supports the batchSize setting.
You would set the batchOption like this:
var cursor = coll.aggregate(query, {cursor: {batchSize:100}}, writeResultsToCsv);
To circumvent such problems, I'd recommend aggregation or map-reduce directly through mongo client. There you can add the notimeout option.
The default timeout is 10 minutes (obviously useless for long time-consuming queries) and there's no way currently to set a different one as far as I know, only infinite by aforementioned option. The timeout hits you especially for high batch sizes, because it will take more than 10 mins to process the incoming docs and before you ask mongo server for more, the cursor has been deleted.
IDK your use case, but if it's a web view, there should be only fast queries/aggregations.
BTW I think this didn't change with 3.0.*
I'm trying to pop and retrieve an element out of an array stored in a document. I can't use $pop since it doesn't return the POPed element. I'm trying to use findAndModify() instead. It works in the shell console, but I'm having troubles getting it to work using the mongodb node.js driver (https://www.npmjs.org/package/mongodb).
my document structure looks like so:
{ _id: '1', queue: [1,2,3]}
In mongo shell, I do:
> db.collection.findAndModify({ query: { _id: 1 },
update: { $pop: { queue: -1 } },
fields: { queue: { $slice: 1 } }, new: false })
$slice ensures that the returning document shows the element that has just been poped. To clarify, I'm not interested in what is in the queue, I'm only interested in what I have just popped out of the queue.
returns:
< {_id: 1, "queue": [1]} // yes, it works!
Using the mongodb library, I don't know how to specify the $slice: 1, it doesn't seem to be supported in the options(?):
> db.collection('collection').findAndModify(
{ _id: 1 },
[],
{ $pop: { queue: -1 }, queue: { $slice: 1 } },
{ new: false },
function(error, results) {
if(error) {
console.log(error);
}
console.log(results);
}
);
returns:
< MongoError: exception: Field name duplication not allowed with modifiers
Basically - where should I put the "queue: {$slice: 1}" part in the nodejs query to make this work? Is it even supported in the node.js driver implementation?
Also, it doesn't seem like findAndModify() is meant to be used this way. If $pop was returning the POPed value, it would be ideal. Any suggestions on how to do this?
Thanks,
- Samir
It seems like that the node.js implementation does not support the 'fields' operand at all.
We've figured out this work around:
1) We store each element in it's own document, instead of an array within the same document.
2) Now findAndModify works like so:
db.collection('collection').findAndModify(
{}, // findAndModify will match it to the first document, if multiple docs are found
[],
{},
{remove: true, new: false }, // returns & removes document from collection
function(error, results) {
if(error) {
console.log(error);
}
console.log(results);
}
);
Some good links that helped us and might help you if you have a similar issue:
https://blog.serverdensity.com/queueing-mongodb-using-mongodb/
http://www.slideshare.net/mongodb/mongodb-as-message-queue