So I've been trying to move data from one database to another. I've already move them but I need to clear the documents which I've already moved from the old database. I've been using ektorp's execute bulk to perform bulk operations. But for some reason I keep getting document update conflict when I try to delete bulk by inserting _deleted.
I might be doing it wrong, here is what I did.
Fetch by bulk with include docs. (For some reason, this doesn't work with just id and rev.)
Then include the _deleted field to each document.
Post using executebulk.
It works for some documents but keeps getting document update conflict for some documents.
Any solution/suggestions please..
This is the preferred way of deleting docs in bulk:
List<Object> bulkDocs = ...
MyClass toBeDeleted = ...
bulkDocs.add(BulkDeleteDocument.of(toBeDeleted));
db.executeBulk(bulkDocs);
If you only need a way to delete/update docs in bulk and you don't need to necessarily implement it in your own software, you can use the great couchapp at:
https://github.com/harthur/costco
You need to upload it to your own server with a couchapp deployment tool, and use a function like
function(doc) {
if(doc.istodelete) // replace this or remove to delete all docs
return null;
}
Read instructions and examples
Related
I have a cloud function which listens for document updates within a collection. If a special field is updated, I want to move this document to another collection.
Can I simply update the location of the document?
Or do I have to write a transaction which contains a get(), write() and delete() of the document, or is there a better solution?
That seems to be a common use case but I cant find any documentation.
Firestore currently doesn't offer a "move" operation. You'll have to do what you proposed in your question: copy the document yourself in a transaction.
Firebase Firestore do not support moving data; Its better to copy to new and delete existing.
If your data is kind of transaction data;
make sure execute operations in transaction block (delete only when
copy is success)
Also you can ensure using [ .validate() ] delete only if its exist in copied node
I am working on a node.js app, and I've been searching for a way around using the Model.save() function because I will want to save many documents at the same time, so it would be a waste of network and processing doing it one by one.
I found a way to bulk insert. However, my model has two properties that makes them unique, an ID and a HASH (I am getting this info from an API, so I believe I need these two informations to make a document unique), so, I wanted that if I get an already existing object it would be updated instead of inserted into the schema.
Is there any way to do that? I was reading something about making concurrent calls to save the objects, using Q, however I still think this would generate an unwanted load on the Mongo server, wouldn't it? Does Mongo or Mongoose have a method to bulk insert or update like it does with insert?
Thanks in advance
I think you are looking for the Bulk.find(<query>).upsert().update(<update>) function.
You can use it this way:
bulk = db.yourCollection.initializeUnorderedBulkOp();
for (<your for statement>) {
bulk.find({ID: <your id>, HASH: <your hash>}).upsert().update({<your update fields>});
}
bulk.execute(<your callback>)
For each document, it will look for a document matching the {ID: <your id>, HASH: {your hash}} criteria. Then:
If it finds one, it will update that document using {<your update fields>}
Otherwise, it will create a new document
As you need, it will not make a connection to the mongo server on each iteration of the for loop. Instead a single call will be made on the bulk.execute() line.
I want to delete all documents where foo equals x. Seems like a pretty basic operation, but I just can't figure it out.
I know how to delete an individual document, but that's not good enough - I may have to delete a few thousand at a time.
How do I bulk delete documents in CouchDB?
I don't know if it's the right way but make a view that exposes the foo field, query the view for the doc._ids of all your documents that you want to delete, and make a bulk update against all your documents. So two (ideally) calls to couch.
http://comments.gmane.org/gmane.comp.db.couchdb.user/11222
Has a similar way to go about it.
Do a bulk update on all the documents you want to delete and update doc._deleted=true following the example in Bulk deletion of documents
It's quite easy with bulk delete: https://wiki.apache.org/couchdb/HTTP_Bulk_Document_API
Just POST to _all_docs a list of JSONs that look like:
{"_id": "0", "_rev": "1-62657917", "_deleted": true}
I also needed something to handle that and, since there was nothing at the time, I decided to make my own implementation.
You can find it here.
Update
Since it was very helpful to me and in order to protect myself from mistakes, I added a backup/restore feature to this tool that can now be found on version 0.2
I tried a somewhat long method to delete documents. I first created a view called map_fun that called the documents i wanted to get deleted. I then iterated through the view and stored the keys of allt he documents and used del db['_id'] to delete them
map_fun = function(doc){
if (doc.doc_type == 'classic'){
emit(doc._id, doc)
}}
deldoclist = []
for row in db.query(map_fun):
deldoclist.append(row.key)
for item in deldoclist:
del db[item]
How to get Post with Comments Count in single query with CouchDB?
I can use map-reduce to build standalone view [{key: post_id, value: comments_count}] but then I had to hit DB twice - one query to get the post, another to get comments_count.
There's also another way (Rails does this) - count comments manually, on the application server and save it in comment_count attribute of the post. But then we need to update the whole post document every time a new comment added or deleted.
It seems to me that CouchDB is not tuned for such a way, unlike RDBMS when we can update only the comment_count attribute in CouchDB we are forced to update the whole post document.
Maybe there's another way to do it?
Thanks.
The view's return json includes the document count as 'total_rows', so you don't need to compute anything yourself, just emit all the documents you want counted.
{"total_rows":3,"offset":0,"rows":[
{"id":...,"key":...,value:doc1},
{"id":...,"key":...,value:doc2},
{"id":...,"key":...,value:doc3}]
}
Is it possible to have couch update or change fields on the fly when you create/update a doc? For example in the design view.... validate_doc_update:
function(newDoc, oldDoc, userCtx) {
}
Within that function I can throw errors like:
if(!newDoc.user_email && !newDoc.user_name && !newDoc.user_password){
throw({forbidden : 'all fields required'});
}
My Question is how would I reassign a field? I tried this:
newDoc.user_password ="changed";
with changed being some new value or hashed value. My overall goal is to build a user registration/login system with node and couchdb and have not found very good examples.
The validate_doc_update function cannot have any side effects and cannot change the document before storage. It only has the power to block an update or to let it through. This is important, because the function is not only called when a user requests an update, but also when changes are replicated from one CouchDB instance to another. So the function can be called multiple times for one document.
However, CouchDB now supports Document Update Handlers that can modify a document or even build it from scratch. These can be used to convert non-JSON input data into usable documents. You can find some documentation in the CouchDB Wiki.
Before you build your own user registration/login system, I'd suggest you look into the built-in CouchDB security features (if you haven't - some information here). They might not be enough for you (e.g. if you need email validation or something similar), but maybe you can build on them.