I am evaluating couchdb for a persistent cart functionality. If I create one docuemnt per user and have each cart item as a field, how many items can I store? In current scenario I can have upto 500 items in a cart.
doc-per-cart or a doc-per-item are both fine choices, neither document sounds like it would get very large (JSON encoding/decoding is slower for very large documents and they must be held entirely in memory). On balance, I'd prefer doc-per-item. Of course, you will need to create a (simple) view to display the cart if you went with doc-per-item.
One good reason to prefer doc-per-item is CouchDB's MVCC. Adding an item to a cart will always create a new document, so you will not need to know the current _rev of the item. When a user wants to delete an item, you will have the _id and the _rev and can easily delete it. If you went with doc-per-cart then you will be constantly updating a document, which requires you to have the current _rev all the time.
Note that doc-per-item will allow duplicates in your cart (the user hits Reload and makes two additions instead of one) but as long as the display of the cart shows this, and the final checkout page does too, then I think it's a reasonable failure mode.
A quick review of the CouchDB overview should make it clear that there is no inherent limit on the number of fields in a CouchDB document, and therefore no limit (aside from available memory) to the number of items you can store in your cart.
Related
I'm working on a project that will have a large number (thousands, possibly millions) of documents on a firebase collection, I need to access the average of the value by day of documents that are the same type, each one of them has a field "registered_value", "date" and a "code" to identify its value and type and registered date.
I need to show users the average value by day of the documents that have the same code.
Users can add new documents, edit existing ones or delete the ones created by them
Since I need to get this data frequently it will be too expensive to always read the entire collection every time a user loads the pages that display this info is there a better way store or get the avarege?
I'm working with ReactJS and Node.js
There's nothing built into Firestore to get aggregated values like that. The common approach is to store the aggregated value in a separate document somewhere, and update that document upon every relevant write operation. You can do this either from client-side code, or from Cloud Functions.
For more on this, see the Firebase documentation on aggregation queries and on distributed counters.
I need to add a field to each document of a very large collection in Firestore via the admin sdk for node.js. Some of the documents already have the field so I need to check for its value before setting/updating it. The collection is around 150k documents. Even trying to get the documents with the code below times out.
const documents = await db.collection('collectionName').get()
Is there a special way to handle very large collections?
You should use pagination in order to avoid loading all the documents into memory with a single query. This will let you process documents in batches by specifying the last document snapshot you saw in the prior query in order to make a new query to get the next page of data.
You will want to place a reasonable limit on each query to make sure you're getting a reasonable number of documents in each page.
I am building a custom control to do a lookup and provide a summary of the status of several items in database. There are 20 different statuses, and in order to determine the number for each status, I am doing a NotesDatabase.search to count each status.
This was fine when there were only 2 statuses to check, however the business now want all of them displayed. :)
I'm concerned about the time it will take to do the search, and want to do this in the most efficient manner possible.
Things I have taken into account:
The documents are updated regularly, so I can't really have an agent doing the calcs and the custom control run a lookup for those static values. This would mean data is old.
The results are dependant on the user logged in, doing counts based on their login ID so I can't really have seperate views per person.
Does anyone have a clean suggested solution?
I am about to start testing the 20 searches and will update this with those results, but am expecting it to be very slow.
A
the other option: instead of #DBLookup you go into the view and just run through it end to end using a navigator. That's pretty fast and should be faster than 20x search.
Of course you could update tallies in the QuerySave event and write it into a user specific in memory profile.
So in your QuerySave you would see what Users are loaded in a ApplicationBean and update those. If a user logs in newly then a search in the database is done into the application bean. When a session expires (Session listener) the entry in the ApplicationBean is cleared out.
Instead of 20 searches you actually might be better off with ONE Ajax call. Create a view that is categorized by your status and is collapsed. Then make an Ajax call ...statusview?ReadViewEntries&Outputformat=JSON&count=100. This will give you the 100 status summary entries with a childcount property.
Would that work for you?
Would it be possible to add 20 Status documents, and to update one or more of these documents whenever some condition is met? Each time a document is updated, an agent runs to match with those conditions, in order to update the status.
If there are many updates per day, it's not really efficient.
Is it possible to delete all documents in a couchdb database, except design documents, without creating a specific view for that?
My first approach has been to access the _all_docs standard view, and discard those documents starting with _design. This works but, for large databases, is too slow, since the documents need to be requested from the database (in order to get the document revision) one at a time.
If this is the only valid approach, I think it is much more practical to delete the complete database, and create it from scratch inserting the design documents again.
I can think of a couple of ideas.
Use _all_docs
You do not need to fetch all the documents, only the ID and revisions. By default, that is all that _all_docs returns. You can make a pretty big request in a batch (10k or 100k docs at a time should be fine).
Replicate then delete
You could use an _all_docs query to get the IDs of all design documents.
GET /db/_all_docs?startkey="_design/"&endkey="_design0"
Then replicate them somewhere temporary.
POST /_replicator
{ "source":"db", "target":"db_ddocs", "create_target":true
, "user_ctx": {"roles":["_admin"]}
, "doc_ids": ["_design/ddoc_1", "_design/ddoc_2", "etc..."]
}
Now you can just delete the original database and replicate the temporary one back by swapping the "source" and "target" values.
Deleting vs "deleting"
Note, these are really apples vs. oranges techniques. By deleting a database, you are wiping out the edit history of all its documents. In other words, you cannot replicate those deletion events to any other database. When you "delete" a document in CouchDB, it stores a record of that deletion. If you replicate that database, those deletions will be reflected in the target. (CouchDB stores "tombstones" indicating the document ID, its revision history, and its deleted state.)
That may or may not be important to you. The first idea is probably considered more "correct" however I can see the value of the second. You can visualize the entire program to accomplish this in your head. It's only a few queries and you're done. No looping through _all_docs batches, no headache. Your specific situation will probably make it obvious which is better.
Install couchapp, pull down the design doc to your hard disk, delete the db in futon, push the design doc back up to your recreated database. =)
You could write a shell script that goes through the list of all documents and deletes them all one by one except design docs. Apparently couch-batch can do that. Note that you don't need to fetch the whole docs to do that, just the id and revision.
Other than that, I think filtered replication (or the replication proposed by JasonSmith) is your best bet.
Whenever a document is accessed, I would like to add last access time into the document.
How do I update a document in a view whenever there is GET request?
You can't. A GET (when used correctly) does not modify data; CouchDB uses GET correctly.
If you really want to record an access time like this you'll need to update the document with the new timestamp and PUT the document back to CouchDB. However, if more than a few people are accessing a document you're quite likely to get contention over it and get conflict errors from CouchDB.
One option is to create a new "document accessed" document in CouchDB on each access but that would rapidly increase the size of the database. You'd actually have a history of access times if that's useful?
Personally, I would look at simply logging document access to a file or queue and process the file/queue in the background. You could have one "document accessed" document per real document as there's little or no chance of contention and a failed update probably wouldn't really matter (you could always try again anyway).