I have read the other problems/answers people are having with the version key but for some reason I do not understand why this specifically happens in my case.
So I have the following mocha test:
it('should be able to save one job', function (done) {
Promise.join(user.saveAsync(), company.saveAsync(),
function (savedUser, savedCompany) {
user = savedUser[0];
user.jobs.push(job);
user.saveAsync()
.spread(function (savedUserWithJob) {
user = savedUserWithJob;
user.jobs.should.have.length(1);
done();
})
.catch(function (err) {
done(err);
});
});
});
which is all fine and passes. I did not have any other problems even at runtime.
Now when I try to run the same test again right after the first one i.e.:
it('should be able to save one job', function (done) {
.....
});
it('should be able to save one job again', function (done) {
.....
});
The second one fails with the error:
VersionError: No matching document found.
I really do not understand why that happens as I am already pushing to the array the first time with no versioning problems. Why is it failing the second consecutive time?
Mongoose versionKey saves an version number on document, usually named __v.
This value is atomically incremented whenever a modification to an array potentially changes any array’s elements position. This value is also sent along in the where clause for any updates that require the use of positional notation. If our where clause still matches the document, it ensures that no other operations have changed our array elements position and it is ok to use use positional syntax. (read more here)
In your tests, after first save(), the doc has an incremented __v, that should be updated on your doc to be used by mongoose as part of the where clause on second save().
Related
Solved: I can only guess that delirium got me.
sorry for not figuring it out before posting, but here's the solution for the next persons sake. The try/catch is not necessary, but it did help me get out some errors that were not being printed to console. If there is a better way to document solving my own problem, I'd appreciate the feedback.
Old Question
I have docs stored in my mongo db that I want to modify before being returned to a find() or findOne(). I want to do this without changing the values and a single hook for all find variants. Call it a masking. I'm actually normalizing a numeric value to a scale of 0 to 1. I do not want to modify the value in the doc, just change the values when the doc is returned via find, findOne. E.G. "Change Model values after load in Mongoose"
Solution
schema.post('init', function(doc){
try{
console.log(doc); // prints the doc to console, complete
doc.property = "new property value"; // changes the document value
console.log(doc); // returns the modified document
return doc;
}
catch(error){
console.log(error);
}
}
Old Notes
Other threads note usage of the post init hook. I have tried the methods in the following related topics with no success.
Mongoose - how to tap schema middleware into the 'init' event?
Change Model values after load in Mongoose (this method has changed with 5.x)
schema.post('init', function(doc){
console.log('Tell me i fired'); // works, prints to console
console.log(doc); // works, prints full model to console
doc.valueField = doc.valueField2+doc.valueField3; // fails, no doc returned, throws no errors
});
schema.post('init', function(doc, next){
console.log('Tell me i fired'); // works, prints to console
console.log(doc); // works, prints full model to console
doc.valueField1 = doc.valueField2+doc.valueField3; // fails, no doc returned, throws no errors
next();
});
I've tried adding return(doc);, doc.save() and just about every mutation of the above code I can compile in my shrinking mental capacity. I keep throwing crap at the wall and nothing is sticking. The documentation over at mongoose is poor and the behavior changed with version 5.x with the removal of Asynch call backs (dropped the next() ).
This is a bug submission on mongoose where the dropping of next() is noted.
https://github.com/Automattic/mongoose/issues/6037
release notes from the 5.x release
https://github.com/Automattic/mongoose/blob/master/migrating_to_5.md#init-hook-signatures
~ Edited, because i solved my own problem ~
After a lot of googling I have not been able to confirm the correct approach to this problem. The following code runs as expected but I have a grave feeling that I am not approaching this in the correct way, and I am setting myself up for problems.
The following code is initiated by the main app.js file and is passed a location to start loading XML files from and processing into a mongoDB
exports.processProfiles = function(path) {
var deferrer = q.defer();
q(dataService.deleteProfiles()) // simple mongodb call to empty the Profiles collection
.then(function(deleteResult) {
return loadFilenames(path); // method to load all filenames in the given path using fs
})
.then(function(filenames) {
// now we have all the file names lets load and save
filenames.forEach(function(filename) {
// Here is where i think the problem is!
// kick off another promise chain for the dynamically sized array of files to process
q(loadFileContent(path, filename)) // first we load the data in the file
.then(function(inboundFile) {
// then parse XML structure to my new shiny JSON structure
// and ask Mongo to store it for me
return dataService.createProfile(processProfileXML(filename, inboundFile));
})
.done(function(result) {
console.log(result);
})
});
})
.catch(function(err) {
deferrer.reject('Unable to Process Profile records : ' + err);
})
.done(function() {
deferrer.resolve('Profile Processing Completed');
});
return deferrer.promise;
}
Whilst this code works these are my main concerns but cannot solve them on my own after a few hours of Google and reading.
1) Is this blocking? The read out to the console is difficult to understand if this is running asynchronously as i want it to - i think it is but advice on if I am doing something fundamentally wrong would be great
2) Is having a nested promise a bad idea, should I be linking it to the outter promise - I have tried but could not get anything to compile or run.
I haven't used Q in a really long time, but I think that you'd need to do is let it know you're about to hand back an array of promises that need to all be satisfied before moving on.
Additionally as you're waiting for multiple promises on one section of code, rather than nesting further, throw the 'set' of promises back up once they're all satisfied.
q(dataService.deleteProfiles()) // simple mongodb call to empty the Profiles collection
.then(function (deleteResult) {
return loadFilenames(path); // method to load all filenames in the given path using fs
})
.then(function (filenames) {
return q.all(
filenames.map(function (filename) {
return q(loadFileContent(path, filename)) { /* Do stuff with your filenames */ });
})
);
.then(function (resultsOfLoadFileContentsPromises) {
console.log('I did stuff with all the things');
)
.catch(function(err) {});
What you have is not 'blocking'. But really what you're doing with promises is moving things into a new 'block'ing section. The more blocks you have, the more async-ish your code will appear. If nothing else is running apart from this promise, it will still appear procedural.
But inner promises must still resolve before the parent promises resolve thereafter.
Inner promises like what you have aren't an inherently bad, personally I will break them out into seperate files to makes easier to reason about, but I wouldn't define that as 'bad' unless there's no need for that inner promise to exist, however where possible (and in your example here) I've adjusted so I throw back up the next set of promises for a new section to deal with the data after it's gotten it.
(I'm not great with Q though, this code will probably require a little further tweaking).
I have to implement a program in node.js which looks like the following code snippet. It has an array though which I have to traverse and match the values with database table entries. I need to wait till the loop ends and send the result back to the calling function:
var arr=[];
arr=[one,two,three,four,five];
for(int j=0;j<arr.length;j++) {
var str="/^"+arr[j]+"/";
// consider collection to be a variable to point to a database table
collection.find({value:str}).toArray(function getResult(err, result) {
//do something incase a mathc is found in the database...
});
}
However, as the str="/^"+arr[j]+"/"; (which is actually a regex to be passed to find function of MongoDB in order to find partial match) executes asynchronously before the find function, I am unable to traverse through the array and get required output.
Also, I am having hard time traversing through array and send the result back to calling function as I do not have any idea when will the loop finish executing.
Try using async each. This will let you iterate over an array and execute asynchronous functions. Async is a great library that has solutions and helpers for many common asynchronous patterns and problems.
https://github.com/caolan/async#each
Something like this:
var arr=[];
arr=[one,two,three,four,five];
asych.each(arr, function (item, callback) {
var str="/^"+item+"/";
// consider collection to be a variable to point to a database table
collection.find({value:str}).toArray(function getResult(err, result) {
if (err) { return callback(err); }
// do something incase a mathc is found in the database...
// whatever logic you want to do on result should go here, then execute callback
// to indicate that this iteration is complete
callback(null);
});
} function (error) {
// At this point, the each loop is done and you can continue processing here
// Be sure to check for errors!
})
I am using node-sqlite3, but I am sure this problem appears in another database libraries too. I have discovered a bug in my code with mixing transactions and async code.
function insertData(arrayWithData, callback) {
// start a transaction
db.run("BEGIN", function() {
// do multiple inserts
slide.asyncMap(
arrayWithData,
function(cb) {
db.run("INSERT ...", cb);
},
function() {
// all done
db.run("COMMIT");
}
);
});
}
// some other insert
setInterval(
function() { db.run("INSERT ...", cb); },
100
);
You can also run the full example.
The problem is that some other code with insert or update query can be launched during the async pause after begin or insert. Then this extra query is run in the transaction. This is not a problem when the transaction is committed. But if the transaction is rolled back the change made by this extra query is also rolled back. Hoops we've just unpredictably lost data without any error message.
I thought about this issue and I think that one solution is to create a wrapper class that will make sure that:
Only one transaction is running at the same time.
When transaction is running only queries which belong to the transaction are executed.
All the extra queries are queued and executed after the current transaction is finished.
All attempts to start a transaction when one is already running will also get queued.
But it sounds like too complicated solution. Is there a better approach? How do you deal with this problem?
At first, I would like to state that I have no experience with SQLite. My answer is based on quick study of node-sqlite3.
The biggest problem with your code IMHO is that you try to write to DB from different locations. As I understand SQLite, you have no control of different parallel "connections" as you have in PostgreSQL, so you probably need to wrap all your communication with DB. I modified your example to use always insertData wrapper. Here is the modified function:
function insertData(callback, cmds) {
// start a transaction
db.serialize(function() {
db.run("BEGIN;");
//console.log('insertData -> begin');
// do multiple inserts
cmds.forEach(function(item) {
db.run("INSERT INTO data (t) VALUES (?)", item, function(e) {
if (e) {
console.log('error');
// rollback here
} else {
//console.log(item);
}
});
});
// all done
//here should be commit
//console.log('insertData -> commit');
db.run("ROLLBACK;", function(e) {
return callback();
});
});
}
Function is called with this code:
init(function() {
// insert with transaction
function doTransactionInsert(e) {
if (e) return console.log(e);
setTimeout(insertData, 10, doTransactionInsert, ['all', 'your', 'base', 'are', 'belong', 'to', 'us']);
}
doTransactionInsert();
// Insert increasing integers 0, 1, 2, ...
var i=0;
function doIntegerInsert() {
//console.log('integer insert');
insertData(function(e) {
if (e) return console.log(e);
setTimeout(doIntegerInsert, 9);
}, [i++]);
}
...
I made following changes:
added cmds parameter, for simplicity I added it as last parameter but callback should be last (cmds is an array of inserted values, in final implementation it should be an array of SQL commands)
changed db.exec to db.run (should be quicker)
added db.serialize to serialize requests inside transaction
ommited callback for BEGIN command
leave out slide and some underscore
Your test implementation now works fine for me.
I have end up doing full wrapper around sqlite3 to implement locking the database in a transaction. When DB is locked all queries are queued and executed after the current transaction is over.
https://github.com/Strix-CZ/sqlite3-transactions
IMHO there are some problems with the ivoszz's answer:
Since all db.run are async you cannot check the result of the whole transaction and if one run has error result you should rollback all commands. For do this you should call db.run("ROLLBACK") in the callback in the forEach loop. The db.serialize function will not serialize async run and so a "cannot start transaction within transaction occurs".
The "COMMIT/ROLLBACK" after the forEach loop has to check the result of all statements and you cannot run it before all the previous run finished.
IMHO there are only one way to make a safe-thread (obv referred to the background thread pool) transaction management: create a wrapper function and use the async library in order to serialize manually all statements. In this way you can avoid db.serialize function and (more important) you can check all single db.run result in order to rollback the whole transaction (and return the promise if needed).
The main problem of the node-sqlite3 library related to transaction is that there aren't a callback in the serialize function in order to check if one error occurs
Whenever I define a Firebase transaction in NodeJS I notice it always runs three times - the first two times with null data, then finally a third time with actually data. Is this normal/intended?
For example this code:
firebaseOOO.child('ref').transaction(function(data) {
console.log(data);
return data;
});
outputs the following:
null
null
i1: { a1: true }
I would have expected that it only print the last item.
To answer a question in the comments, here is the same with a callback:
firebaseOOO.child('ref').transaction(function(data) {
console.log(data);
return data;
}, function(error, committed, snapshot) {
if (error)
console.log('failed');
else if (!committed)
console.log('aborted');
else
console.log('committed');
console.log('fin');
});
Which yields the following output:
null
null
i1: { a1: true }
committed
fin
I had read the details of how transactions work before posting the question, so I had tried setting applyLocally to false like this:
firebaseOOO.child('ref').transaction(function(data) {
console.log('hit');
return data;
}, function(){}, false);
But it still hits 3 times (just double-checked) so I thought it was something different. Getting the 'value' before transacting does "work" as expected, in that it only hits once, and that's regardless of what applyLocally is set to, so I'm not sure what applyLocally does? This is what I mean by getting the value before transacting:
firebaseOOO.child('ref').once('value', function(data) {
console.log('1');
firebaseOOO.child('ref').transaction(function(data) {
console.log('2');
return data;
});
});
Outputs:
1
2
#Michael: How can one make use of this behavior? Transactions are primarily for having data use itself to modify itself - the prototypical increment++ scenario. So if I need to add 1 to the existing value of 10, and continue working with the result of 11, the first two times the function hits I will have an erroneous result of 1 that I need to handle, and finally the correct result of 11 on the third hit. How can I make use of those two initial 1's? Another scenario (and maybe I shouldn't be using transactions for this, but if it worked like I expected it makes for cleaner code) is to insert a value if it does not yet exist. If transactions only hit once, a null value would mean the value does not exist, and so you could, for example, init the counter to 1 in that case, otherwise add 1 to whatever the value is. With the noisy nulls, this is not possible.
It seems the takeaway from all this is to simply use the 'once' pattern more often than not?
ONCE TRANSACTION PATTERN:
firebaseOOO.child('ref').once('value', function(data) {
console.log('1');
firebaseOOO.child('ref').transaction(function(data) {
console.log('2');
return data;
});
});
The behavior you're seeing here is related to how Firebase fires local events and then eventually synchronizes with the Firebase servers. In this specific example, the "running three times" will only happen the very first time you run the code—after that, the state has been completely synchronized and it'll just trigger once from then on out. This behavior is detailed here: https://www.firebase.com/docs/transactions.html (See the "When a Transaction is run, the following occurs" section.)
If, for example, you have an outstanding on() at the same location and then, at some later time, run this same transaction code, you'll see that it'll just run once. This is because everything is in sync prior to the transaction running (in the ideal case; barring any normal conflicts, etc).
transaction() will be called multiple times and must be able to handle null data. Even if there is existing data in your database it may not be locally cached when the transaction function is run.
firebaseOOO.child('ref').transaction(function(data) {
if(data!=null){
console.log(data);
return data;
}
else {
return data;
}
}, function(error, committed, snapshot) {
if (error)
console.log('failed');
else if (!committed)
console.log('aborted');
else
console.log('committed');
console.log('fin');
});