Does CouchDB rebuild an index when a design document has been 'updated' without any change to its map or reduce functions?
EG, will the following code cause 5 re-indexes of the database,
const db = new pouch('some-database-url');
const ddoc = {
_id: '_design/findStuff',
views: {
'findStuff': {
map: `function(doc) {
if (doc._id.indexOf('stuff') === 0) {
emit(doc._id, doc);
}
}`
}
}
}
for (let i=0; i<5; i++) {
db.get(ddoc._id).then( d => {
db.put( {
...ddoc,
_rev: d._rev
})
})
}
or does CouchDB make a comparison with the previous versions and decide not to bother?
It's not immediately obvious to me how to test this because the re-indexing is a background process whose status I don't know how to check, and because it results in the same built index.
Yes... and not exactly.
Changing the rev on a ddoc triggers an index rebuild, but that rebuild happens on read, not on update.
That is to say, index updates happen the first time the index is consulted, after a change (whether that change be brought on by a ddoc update, or by changing the documents in the database).
Related
I have a forEach loop where I am querying a document, doing some simple math calculations and then updating a document in the collection and move on to the next iteration.
The problem is, alot of times randomly some of the UPDATE operations will not update the document. I don't know why is it happening. Is it because of the lock?
I have tried logging things just before the update operation. The data is all correct but when it comes to update, it will randomly not update at all. Out of 10 iterations, lets say 8 will correctly work
const name = "foo_game";
players.forEach(({ id, team, username }) => {
let updatedStats = {};
Users.findOne({ id }).then(existingPlayer => {
if (!existingPlayer) return;
const { stats } = existingPlayer;
const existingStats = stats[pug.name];
if (!existingStats) return;
const presentWins = existingStats.won || 0;
const presentLosses = existingStats.lost || 0;
updatedStats = {
...existingStats,
won:
team === winningTeam
? presentWins + 1
: changeWinner
? presentWins - 1
: presentWins,
lost:
team !== winningTeam
? presentLosses + 1
: changeWinner
? presentLosses - 1
: presentLosses,
};
// THE CALCULATIONS ARE ALL CORRECT TILL THIS POINT
// THE UPDATE WIILL RANDOMLY NOT WORK
Users.findOneAndUpdate(
{ id, server_id: serverId },
{
$set: {
username,
stats: { ...stats, [name]: updatedStats },
},
},
{
upsert: true,
}
).exec();
});
});
Basically what you are missing here is the asynchronous operations of both the findOne() and the findOneAndUpdate() are not guaranteed to complete before your foreach() is completed. Using forEach() is not a great choice for a loop with async operations in it, but the other main point here is that it's completely unnecessary since MongoDB has a much better way of doing this and in one request to the server.
In short, instead of "looping" you actually want to provide an array of instructions to bulkWrite():
let server_id = serverId; // Alias one of your variables or just change it's name
Users.bulkWrite(
players.map(({ id, team, username }) =>
({
"updateOne": {
"filter": { _id, server_id },
"update": {
"$set": { username },
"$inc": {
[`stats.${name}.won`]:
team === winningTeam ? 1 : changeWinner ? - 1 : 0,
[`stats.${name}.lost`]:
team !== winningTeam ? 1 : changeWinner ? - 1 : 0
}
},
"upsert": true
}
})
)
)
.then(() => /* whatever continuation here */ )
.catch(e => console.error(e))
So rather than looping, that Array.map() produces one "updateOne" statement within the bulk operation for each array member and sends it to the server. The other change of course is you simply do not need the findOne() in order to read existing values. All you really need is to use the $inc operator in order to either increase or decrease the current value. Noting here that if nothing is currently recorded at the specified path, then it would create those with whatever value of 1/-1/0 was determined by the logic and handed to $inc.
Note this is how you actually should be doing things in general, as aside from avoiding uneccesary loops of async calls the main thing here is to actually use the atomic operators like $inc that MongoDB has. Reading data state from the database in order to make changes is an anti-pattern and best avoided.
I'm developing a small NodeJS web app using Mongoose to access my MongoDB database. A simplified schema of my collection is given below:
var MySchema = mongoose.Schema({
content: { type: String },
location: {
lat: { type: Number },
lng: { type: Number },
},
modifierValue: { type: Number }
});
Unfortunately, I'm not able to sort the retrieved data from the server the way it is more convenient for me. I wish to sort my results according to their distance from a given position (location) but taking into account a modifier function with a modifierValue that is also considered as an input.
What I intend to do is written below. However, this sort of sort functionality seems to not exist.
MySchema.find({})
.sort( modifierFunction(location,this.location,this.modifierValue) )
.limit(20) // I only want the 20 "closest" documents
.exec(callback)
The mondifierFunction returns a Double.
So far, I've studied the possibility of using mongoose's $near function, but this doesn't seem to sort, not allow for a modifier function.
Since I'm fairly new to node.js and mongoose, I may be taking a completely wrong approach to my problem, so I'm open to complete redesigns of my programming logic.
Thank you in advance,
You might have found an answer to this already given the question date, but I'll answer anyway.
For more advanced sorting algorithms you can do the sorting in the exec callback. For example
MySchema.find({})
.limit(20)
.exec(function(err, instances) {
let sorted = mySort(instances); // Sorting here
// Boilerplate output that has nothing to do with the sorting.
let response = { };
if (err) {
response = handleError(err);
} else {
response.status = HttpStatus.OK;
response.message = sorted;
}
res.status(response.status).json(response.message);
})
mySort() has the found array from the query execution as input and the sorted array as output. It could for instance be something like this
function mySort (array) {
array.sort(function (a, b) {
let distanceA = Math.sqrt(a.location.lat**2 + a.location.lng**2);
let distanceB = Math.sqrt(b.location.lat**2 + b.location.lng**2);
if (distanceA < distanceB) {
return -1;
} else if (distanceA > distanceB) {
return 1;
} else {
return 0;
}
})
return array;
}
This sorting algorithm is just an illustration of how sorting could be done. You would of course have to write the proper algorithm yourself. Remember that the result of the query is an array that you can manipulate as you want. array.sort() is your friend. You can information about it here.
I am building an array with data and want to push that array to sub-document.
var pubArray = [];
var count = 5
for (i = 0; i < count; i++) {
pubArray.push({publicationName: req.body.publicationName[i], dateSent:req.body.dateSent[i]});
};
Students.findOne({studentNumber: filter}, function (err, student) {
student.publications.push({pubArray});
student.save();
});
If I use the {publicationName: req.body.publicationName[i], dateSent:req.body.dateSent[i]} inside the student.publications.push it works fine. If I try to push the array, nothing happens.
Note that the .push() method in mogoose works just like it's JavaScript equivalent in that it is "pushing" a single element onto the array, rather than a whole array. So you can either assign the whole array or just construct in the loop:
student.publications = pubArray;
or:
// Construct with .push in loop:
Students.findOne({ "studentNumber": filter },function(err,student) {
for ( var i = 0; i < count: i++ ) {
student.publications.push({
"publicationName": req.body.publicationName[i],
"dateSent": req.body.dateSent[i]
});
}
student.save(function(err) {
// Complete
});
});
But really you would be better off using an "atomic" operator of $push with $each in a direct update. This is then just one trip to the server, rather than two:
Students.update(
{ "studentNumber": filter },
{ "$push": { "publications": { "$each": pubArray } } },
function(err,numAffected) {
}
);
That is generally worlds better than the "find/modify/save" pattern, and not only in being more efficient, but it also avoids potential conflicts or overwriting data since the object and array is modified "in-place" in the database, with the state current to the time of modification.
Atomic operators should always be favoured for the performance benefits as well as lack of conflicts in modification.
The publications property of the student object is an Array. You can simply assign this property to the pubArray created earlier:
Students.findOne({studentNumber: filter}, function (err, student) {
student.publications = pubArray;
student.save();
});
I have a mongodb model called User, which has a mixed schema type variable called "inventory" (contains all the items a user contains). I would like to loop through all the users, and change the name of each item in their inventory. In particular, I would like to convert strings in the format of "10_alex_magician" or "3_maia_princess" to "alex_magician" and "maia_princess" respectively. The string conversion is relatively straightforward, and I'm using x.split('').slice(1).join('') to accomplish the conversion.
Where I'm having trouble is even when console.log shows that the conversion has been applied, it doesn't seem to be updating correctly to mongodb, yet no error message is being thrown. Does anyone know how to solve this?
Node.js function
//function to change old naming of items "10_alex_magician" to "alex_magician"
function modifyUser() {
User.find({}, function(err, results) {
_.map(results, function(result) {
var regex = /^\d+_[A-Za-z]+_[A-Za-z]+$/
for (var i = 0, len = result.inventory.length; i < len; i++) {
if(regex.test(result.inventory[i].itemName)) {
result.inventory[i].itemName = result.inventory[i].itemName.split('_').slice(1).join('_');
result.save(function(err, r) {
if(err) console.log(err);
//logging r shows that the text has been correctly updated
console.log(r)
});
}
}
})
})
}
Format of inventory variable
"inventory": [
{
"type": "sticker",
"numberOwned": 2,
"itemName": "1_alex_magician"
},
{
"type": "sticker",
"numberOwned": 1,
"itemName": "10_alex_scuba"
}
],
Mongoose only has automatic change detection for top-level properties and you are modifying a nested property, so mongoose doesn't know anything changed. Use markModified to tell mongoose you are mucking with inventory.
result.inventory[i].itemName = result.inventory[i].itemName.split('_').slice(1).join('_');
result.markModified('inventory');
result.save(....)
For efficiency, you may want to consider both .lean() and .stream() for this type of query and just do your updates with findByIdAndUpdate, passing just the updated inventory property.
We have an order system where every order has an id. For accounting purposes we need a way to generate invoices with incremening numbers. What is the best way to do this without using an sql database?
We are using node to implement the application.
http://www.mongodb.org/display/DOCS/How+to+Make+an+Auto+Incrementing+Field
The first approach is keeping counters in a side document:
One can keep a counter of the current _id in a side document, in a
collection dedicated to counters. Then use FindAndModify to atomically
obtain an id and increment the counter.
The other approach is to loop optimistically and handle dup key error code of 11000 by continuing and incrementing the id for the edge case of collisions. That works well unless there's high concurrency writes to a specific collection.
One can do it with an optimistic concurrency "insert if not present"
loop.
But be aware of the warning on that page:
Generally in MongoDB, one does not use an auto-increment pattern for
_id's (or other fields), as this does not scale up well on large database clusters. Instead one typically uses Object IDs.
Other things to consider:
Timestamp - unique long but not incrementing (base on epoch)
Hybrid Approach - apps don't necessarily have to pick one storage option.
Come up with your id mechanism based on things like customer, date/time parts etc... that you generate and handle collisions for. Depending on the scheme, collisions can be much less likely. Not necessarily incrementing but is unique and has a well defined readable pattern.
I did not find any working solution, so I implemented the "optimistic loop" in node.js to get Auto-Incrementing Interger ID fields. Uses the async module to realize the while loop.
// Insert the document to the targetCollection. Use auto-incremented integer IDs instead of UIDs.
function insertDocument(targetCollection, document, callback) {
var keepRunning = true;
var seq = 1;
// $type 16/18: Integer Values
var isNumericQuery = {$or : [{"_id" : { $type : 16 }}, {"_id" : { $type : 18 }}]};
async.whilst(testFunction, mainFunction, afterFinishFunction);
// Called before each execution of mainFunction(). Works like the stop criteria of a while function.
function testFunction() {
return keepRunning;
}
// Called each time the testFunction() passes. It is passed a function (next) which must be called after it has completed.
function mainFunction(next) {
findCursor(targetCollection, findCursorCallback, isNumericQuery, { _id: 1 });
function findCursorCallback(cursor) {
cursor.sort( { _id: -1 } ).limit(1);
cursor.each(cursorEachCallback);
}
function cursorEachCallback(err, doc) {
if (err) console.error("ERROR: " + err);
if (doc != null) {
seq = doc._id + 1;
document._id = seq;
targetCollection.insert(document, insertCallback);
}
if (seq === 1) {
document._id = 1;
targetCollection.insert(document, insertCallback);
}
}
function insertCallback(err, result) {
if (err) {
console.dir(err);
}
else {
keepRunning = false;
}
next();
}
}
// Called once after the testFunction() fails and the loop has ended.
function afterFinishFunction(err) {
callback(err, null);
}
}
// Call find() with optional query and projection criteria and return the cursor object.
function findCursor(collection, callback, optQueryObject, optProjectionObject) {
if (optProjectionObject === undefined) {
optProjectionObject = {};
}
var cursor = collection.find(optQueryObject, optProjectionObject);
callback(cursor);
}
Call with
insertDocument(db.collection(collectionName), documentToSave, function() {if(err) console.error(err);});