I'm using Couchbase NoSQL DB but I guess this can happen with any NoSQL DB. Here's what happens:
I'm checking if a specific key exists and I'm catching the keyNotFound error to now ADD this key into the database. See the code:
// retrieve the document for this connection_id
db.get(connection_id, function(err, result) {
if (err && err.code === 13) {
// Catched a keyNotFound -> define a new document for the voice connection
var voice_c = {
voice_count: '1',
voice_duration: call_duration,
last_contact: call_start
};
// Add this new Voice_c document for this connection_id to DB
db.add(connection_id, voice_c, function(err, result) {
if (err)
throw err; // whilst adding a new voice connection
});
When I get to the db.add step I get an error "Key exists (with a different CAS value)" even though I just checked fragments of a millisecond before if the same key exist (and it didn't exist).
I couldn't replicate the error at the same place in my data feed but the second time it happened even earlier, indicating it's a random event. I'm puzzled as to how this can happen unless there's a big bug in my code which I just don't see.
There is no other code running and altering the documents in the NoSQL DB, it all runs locally on my MBP. The DB was flushed and was empty before I started to run my script.
I've checked manually in the data feed and when it happened the first time there was indeed the same connection_id about 50 records earlier. But in the second instance the error was thrown when the connection_id was showing up the first time (though it comes up some 19000 records later). Very strange, hope someone can help me how I can avoid getting this error.
have you tried using the "stale:false" option of couchbase for renewing the index? i also would recommend using async.waterfall for both statements, it increases readability and decreases callback hell.
Related
Let me be real simple. I am running node.js server. When I receive data from patch request, first I need to check in database. If the row exists I will update it, otherwise I will create that record. Here is my code. This is what I am calling in the request handler callback.
let dbCollection = db$master.collection('users');
createUser(req.body).then(user => {
dbCollection.updateMany(
{ rmisId: req.body.rmisId },
{ $set: { ...user } },
{ upsert: true }
).then((result) => {
log.debug("RESULTS", user);
return result;
})
.catch((err => {
log.debug(err);
return err;
}));
})
This is working fine in sequential requests. But its creating duplicate record when I receive 10 concurrent request. I am running on my local machine and replicating concurrent request using Apache JMeter. Please help me if you have experienced this kind of problem.
Thank you !
UPDATE
I have tested another approach that reads the database like dbCollection.find({rmisId: req.body.rmisId}) the database for determine its existing or no. But it has no difference at all.
You cannot check-and-update. Mongodb operations are atomic at the document level. After you check and see that the record does not exist, another request may create the document you just checked, and after that you can recreate the same record if you don't have unique indexes or if you're generating the IDs.
Instead, you can use upsert, as you're already doing, but without the create. It looks like you're getting the ID from the request, so simply search using that ID, and upsert the user record. That way if some other thread inserts it before you do, you'll update what the previous thread inserted. If this is not something you prefer, add a unique index for that user ID field.
Same code works fine when letting couch auto generate UUID's. I am starting off with a new completely empty database yet I keep getting this
error: conflict
reason: Document update conflict
To reiterate I am posting new documents to an empty database so not sure how I can get update conflicts when nothing is being updated. Even stranger the conflicting documents still show up in the DB with only a single revision, but overall there are missing records.
I am trying to insert about 38,000 records with _bulk_docs in batches of 100. I am getting these records (100 at a time) from a RETS server, each record already has a unique ID that I want to use for the couchDB _id instead of their UUID's. I am using a promised based library to get the records and axios to insert them into couch. After getting the first batch of 100 I then run this code to add an _id to each of the 100 records before inserting
let batch = [];
batch = records.results.map((listing) => {
let temp = listing;
temp._id = listing.ListingKey;
return temp;
});
Then insert:
axios.post('http://127.0.0.1:5984/rets_store/_bulk_docs', { docs: batch })
This is all inside of a function that I call recursively.
I know this probably wont be enough to see the issue but thought Id start here. I know for sure it has something to do with my map() and adding the _id = ListingKey
Thanks!
I am using the Mongoose driver with NodeJS. I have quite a simple update call whose purpose is to sync an external source of meetings to my database:
collection.update({ meeting_id: doc.meeting_id}, newDoc, {upsert:true})
The object returned determines whether or not an update or an insert occurred. This works perfectly. My issue is that I must determine if an actual change occurred. When you update a document with itself, MongoDB treats this in exactly the same way as if all fields were changed.
So my question is: Is there any good way to tell if anything actually changed? I could search for each document then compare each field manually, but this seems like a poor (and slow) solution.
you can use findAndModify which will return updated results as compared to update which will return no of updated records.
collection.findAndModify(
{ meeting_id: doc.meeting_id},
newDoc,
{ new: true },
function (err, documents) {
res.send({ error: err, affected: documents });
}
);
My server application (using node.js, mongodb, mongoose) has a collection of documents for which it is important that two client applications cannot modify them at the same time without seeing each other's modification.
To prevent this I added a simple document versioning system: a pre-hook on the schema which checks if the version of the document is valid (i.e., not higher than the one the client last read). At first sight it works fine:
// Validate version number
UserSchema.pre("save", function(next) {
var user = this
user.constructor.findById(user._id, function(err, userCurrent) { // userCurrent is the user that is currently in the db
if (err) return next(err)
if (userCurrent == null) return next()
if(userCurrent.docVersion > user.docVersion) {
return next(new Error("document was modified by someone else"))
} else {
user.docVersion = user.docVersion + 1
return next()
}
})
})
The problem is the following:
When one User document is saved at the same time by two client applications, is it possible that these interleave between the pre-hook and the actual save operations? What I mean is the following, imagine time going from left to right and v being the version number (which is persisted by save):
App1: findById(pre)[v:1] save[v->2]
App2: findById(pre)[v:1] save[v->2]
Resulting in App1 saving something that has been modified meanwhile (by App2), and it has no way to notice that it was modified. App2's update is completely lost.
My question might boil down to: Do the Mongoose pre-hook and the save method happen in one atomic step?
If not, could you give me a suggestion on how to fix this problem so that no update ever gets lost?
Thank you!
MongoDB has findAndModify which, for a single matching document, is an atomic operation.
Mongoose has various methods that use this method, and I think that they will suit your use case:
Model.findOneAndUpdate()
Model.findByIdAndUpdate()
Model.findOneAndRemove()
Model.findByIdAndRemove()
Another solution (one that Mongoose itself uses as well for its own document versioning) is to use the Update Document if Current pattern.
I want to update an existing document in couchdb. I have an image and i want to add it to an existing document in the db without lose the previus fields.
I'm using nodejs with nano.
tanks, this put me in the right orientation. At the end i do it in this way:
db.get(id,{ revs_info: true }, function (error, objeto) {
if(error)
console.log('wrong id');
fs.readFile('image.jpg', function(err, data)
{
if (!err)
{
db.attachment.insert(id, 'imagen.jpg', data, 'image/jpg',{ rev: objeto._rev}, function(err, body) {
if (!err)
console.log(body);
});
}
});
});
Your question is not really clear about the specific problem. So here just some general guidance on updating documents.
When designing the database make sure you set the ID rather than allowing couchdb to edit it. This way you can access the document directly when updating it.
When updating, you are required to prove that you are updating the most recent version of the document. I usually retrieve the document first and make sure you have the most recent '_rev' in the document you'll insert.
finally the update may fail if a different process has edited the document in the time between retrieving and updating it. So you should capture a failure in the insert and repeat the process until you succeed.
That being said, there are two ways you can store an image:
As an attachment: I believe nano support the attachment.insert() and attachment.get() functions to do so.
As a reference: I would usually rather store the images elsewhere and just store the url or filepath to access them. I've not used nano much but believe you can do this by doing the below.
doc = db.get(docname); // get the document with all existing elements
doc['mynewimage'] = myimageurl; // update the document with the new image
// this assumes it's a dictionary
db.insert(doc); // inserts the document with the correct _id (= docname)
// and _rev