I just need a simple mutex, stored in MongoDB. I want a lock given a unique id. There seem to be many popular solutions with Redis, but in this case, since we are already using MongoDB, I am looking for some sort of library that I can use for locking with MongoDB, but I can't find any good packages. Is there a way to do a simple lock with Mongoose or the official MongoDB node.js driver?
I am especially looking for some mutex in MongoDB that has a built-in TTL (time to live). With Redis, you can give a key a TTL and it will remove itself after a period of time, that's an essential feature.
When I google "mongodb + ttl" this is what I see:
https://docs.mongodb.com/manual/core/index-ttl/
To recap our discussion in the comments...
DBMS Transaction Locking
If you're asking about locking at the DBMS transaction level, I think you will find that most DBMS (SQL or NoSQL) handle transactions / locking on their own (i.e. a read operation on a record will wait until a write operation is finished). In MongoDB, since each operation is a single transaction, they've provided a specifically helpful atomic operation called "findAndUpdate".
Domain Specific Locking
Nothing is stopping you from creating some sort of "locks" collection which must be checked before certain operations are made. You will definitely need to consider and take note of the "edge" cases that could result in illegal state or data inconsistency. This is a good time to also reevaluate your architecture (hint: microservices).
TTL
Mongo supports specifying a TTL index on any date field. So, in your case you could consider adding an index like so: db.my_locks.createIndex( { "deleteAt": 1 }, { expireAfterSeconds: 1 } ) and specifying "deleteAt" on insert.
Related
I am currently working on an application using Neo4j as an embedded database.
And I wondering how it would be possible to make sure that separate threads use separate transactions. Normally, I would assign database operations to a transaction, but the code examples I found, don't allow for making sure that write operations use separate transactions:
try (Transaction tx = graphDb.beginTx()) {
Node node = graphDb.createNode();
tx.success();
}
As graphDB shall be used as a thread-safe singleton, I really don't see, how that shall work... (E.g. for several users creating a shopping list in separate transactions.)
I would be grateful for pointing out where I misunderstand the concept of transactions in Neo4j.
Best regards and many thanks in advance,
Oliver
The code you posted will run in separate transactions if executed by multiple threads, one transaction per thread.
The way this is achieved (and it's quite a common pattern) is storing transaction state against ThreadLocal (read the Javadoc and things will become clear).
Neo4j Transaction Management
In order to fully maintain data integrity and ensure good transactional behavior, Neo4j supports the ACID properties:
atomicity: If any part of a transaction fails, the database state is left unchanged.
consistency: Any transaction will leave the database in a consistent state.
isolation: During a transaction, modified data cannot be accessed by other operations.
durability: The DBMS can always recover the results of a committed transaction.
Specifically:
-All database operations that access the graph, indexes, or the schema must be performed in a transaction.
Here are the some useful links to understand Neo4j transactions
http://neo4j.com/docs/stable/rest-api-transactional.html
http://neo4j.com/docs/stable/query-transactions.html
http://comments.gmane.org/gmane.comp.db.neo4j.user/20442
I would like to update a document that involves reading other collection and complex modifications, so the update operators in findAndModify() cannot serve my purpose.
Here's what I have:
Collection.findById(id, function (err, doc) {
// read from other collection, validation
// modify fields in doc according to user input
// (with decent amount of logic)
doc.save(function (err, doc) {
if (err) {
return res.json(500, { message: err });
}
return res.json(200, doc);
});
}
My worry is that this flow might cause conflict if multiple clients happens to modify the same document.
It is said here that:
Operations on a single document are always atomic with MongoDB databases
I'm a bit confused about what Operations mean.
Does this means that the findById() will acquire the lock until doc is out of scope (after the response is sent), so there wouldn't be conflicts? (I don't think so)
If not, how to modify my code to support multiple clients knowing that they will modify Collection?
Will Mongoose report conflict if it occurs?
How to handle the possible conflict? Is it possible to manually lock the Collection?
I see suggestion to use Mongoose's versionKey (or timestamp) and retry for stale document
Don't use MongoDB altogether...
Thanks.
EDIT
Thanks #jibsales for the pointer, I now use Mongoose's versionKey (timestamp will also work) to avoid committing conflicts.
aaronheckmann — Mongoose v3 part 1 :: Versioning
See this sample code:
https://gist.github.com/anonymous/9dc837b1ef2831c97fe8
Operations refers to reads/writes. Bare in mind that MongoDB is not an ACID compliant data layer and if you need true ACID compliance, you're better off picking another tech. That said, you can achieve atomicity and isolation via the Two Phase Commit technique outlined in this article in the MongoDB docs. This is no small undertaking, so be prepared for some heavy lifting as you'll need to work with the native driver instead of Mongoose. Again, my ultimate suggestion is to not drink the NoSQL koolaid if you need transaction support which it sounds like you do.
When MongoDB receives a request to update a document, it will lock the database until it has completed the operation. Any other requests that MongoDB receives will wait until the locking operation has completed and the database is unlocked. This lock/wait behavior is automatic, so there aren't any conflicts to handle. You can find a lot more information about this behavior in the Concurrency section of the FAQ.
See jibsales answer for links to MongoDB's recommended technique for doing multi-document transactions.
There are a couple of NoSQL databases that do full ACID transactions, which would make your life a lot easier. FoundationDB is one such database. Data is stored as Key-Value but it supports multiple data models through layers.
Full disclosure: I'm an engineer at FoundationDB.
In my case I was wrong when "try to query the dynamic field with the upsert option". This guide helped me: How to solve error E11000 duplicate
In above guide, you're probably making one of two mistakes:
Upsert a document when findOneAndupdate() but the query finds a non-unique field.
Use insert many new documents in one go but don't use "ordered = false"
I am writing a web application where more users can perform simultaneous operation in the same document in mongodb.
I use the mean.io stack, but I am quite new to it.
I was wondering how does mongoose manage concurrency. Every "user click" operation performs first a read to get the document, and a save after some calculations. Of course the sequence read-calculate-save is not atomic.
Does mongoose work with 'last change wins' policy, or does it throw a versioning error?
Does it make sense in this case to use a queue?
Thanks, best regards.
Yes the last change will win.
A queue could be a good option to solve the problem but I'll suggest 2 other ways:
You could use more advanced mongodb commands, such as $inc (http://docs.mongodb.org/manual/reference/operator/update/inc/) to compute attomically (if your computation are too complicated maybe it is not possible)
If you don't necessarily need to have the correct count available at any time, you could use a 'big data' approach and just store the raw clicks information.
Whenever you need the data (or say every hour or day), you could then use the mongodb aggregate framework, or their mapreduce feature, to compute the correct count.
So I've been trying to wrap my head around this one for weeks, but I just can't seem to figure it out. So MongoDB isn't equipped to deal with rollbacks as we typically understand them (i.e. when a client adds information to the database, like a username for example, but quits in the middle of the registration process. Now the DB is left with some "hanging" information that isn't assocaited with anything. How can MongoDb handle that? Or if no one can answer that question, maybe they can point me to a source/example that can? Thanks.
MongoDB does not support transactions, you can't perform atomic multistatement transactions to ensure consistency. You can only perform an atomic operation on a single collection at a time. When dealing with NoSQL databases you need to validate your data as much as you can, they seldom complain about something. There are some workarounds or patterns to achieve SQL like transactions. For example, in your case, you can store user's information in a temporary collection, check data validity, and store it to user's collection afterwards.
This should be straight forwards, but things get more complicated when we deal with multiple documents. In this case, you need create a designated collection for transactions. For instance,
transaction collection
{
id: ..,
state : "new_transaction",
value1 : values From document_1 before updating document_1,
value2 : values From document_2 before updating document_2
}
// update document 1
// update document 2
Ooohh!! something went wrong while updating document 1 or 2? No worries, we can still restore the old values from the transaction collection.
This pattern is known as compensation to mimic the transactional behavior of SQL.
Is it possible to cache, say, mongoose document obejcts in Redis,
perhaps for implementing a write-back or write-through cache with timeout-based cache flush mechanisms?
P.S.:
I am familiar with mongoose-redis-cache, but I guess it supports only lean queries, which do not quite serve the purpose here. (But I may be wrong).
As Mongoose objects wrap a MongoDB document, there'd be no reason that you couldn't call
JSON.stringify(mongooseObject.toJSON())
which would return a string representing the MongoDB document. (toJSON) You could then store that result in a key/value in redis.
Where it starts to get more complex is that you'd need to first override the normal save and update functionality to save any modifications to your redis store rather than to the database. While doable, Mongoose wasn't designed for that and you'd be probably more successful to just use the native mongodb drivers and manage general document interactions that way. There are a number of extremely handy operators that you'd need to independently handle (like $push for example, which would add a single value to an array).
The real issue though is that you loose nearly all of the power of MongoDB by not being able to use the query engine or aggregation framework if all of the data isn't already stored in MongoDB (and even if it is, you're still bypassing your caching layer). And, if you're not using any of that functionality, then MongoDB may not be the best match for your needs (and you might instead consider something like CouchDB).
While I can see the potential value of using a caching layer for a high performance MongoDB system, the complexity of a write-back style cache may be more complex than it's worth (and not necessarily safe).
Of course, a write-through cache would be simpler (although you have the complexity of two data-stores and making sure writes are committed consistently if you're going to trust the cache and DB).
(As an aside, I'm not sure how you'd actually manage timeouts, as I thought redis deleted the values associated with keys if they were assigned a lifetime/timeout? I wouldn't want to loose data to the redis cache if you were doing write-back).
In Redis you can only cache raw JSONs, but to cache the whole Mongoose.Documents objects, You can use my library, which handles both - caching results in Redis and mongoose documents in memory. It also has event-based logic to clear both caches, if some related changes appear.
https://www.npmjs.com/package/speedgoose