I have a view which looks like this:
function(doc) {
if (doc.type === 'article' && (Date.parse(doc.published) < (Date.now() - 30 * 60 * 1000))) {
emit(doc._id, doc._rev);
}
}
The view basically emits articles that are stale (i.e. {{published date}} < {{present - 30 minutes}}.
Now, the issue is as follows: The view does not update itself after the first read. The first access builds the view on all documents as expected. But thereafter it seems like it only updates itself on change (delete, create or update of new documents).
This is however an issue and not what I desire. I have other articles that are getting stale as time progresses therefore I would like couch to return these articles too but since they are not changed they don't come up in the view.
This, what I just described, seems to be expected couchdb behavior (?). But, is there a way to show aging artciles too ?
PS: An easy way to test this is to insert a document with published=Date.now() and type="article" and run this view. After 30 minutes you will see the document is actually stale as per the view definition but it will not show up in the view.
30 minutes is just a number. You can reduce it to a smaller time frame if you want. Thanks in advance for your help !
Yes, this is by design. The view index is only updated on the Create Update and Delete parts of CRUD operations.
Filtering by something dynamic during a Read is done with a key.
In your case you would probably want to emit as follows:
emit(doc.published, doc);
Then in your call to CouchDB you would add parameters that would further filter on the published date.
Related
I have a filter function filtering based on document property, e.g. "version: A" and it works fine, until there a document update at some point in time when this property "version: A" removed (or updated to "version: B").
At this point i would like to be notified that the document been updated, similar to one when the document get deleted, but couldn't find an effective way (without listening and processing all documents changes).
Hope i'm just missing something and it's not a design limitation.
While my other answer is a valid approach, I had this same situation yesterday and decided to look at making this work using Mango selectors. I did the following:
Establish a changes feed filtered by the query selector (see the "_selector" filter for /db/_changes)
Perform the query (db/_find) and record the results
Establish a second changes feed that filters for just in the documents returned in (2) (see the "_doc_ids" filter for /db/_changes)
The feed at (1) lets you know when new documents match your query along with edits to documents that matched your query both before and after the change.
The feed at (2) lets you know when a change is made to a document that previously matched your query, irrespective of if it matches your query after the change has been made.
The combination of these feeds covers all cases, though with some false positives. On a change in either feed, tear down the changes feed at (3) and redo steps (2) and (3).
Now, some notes on this approach:
This is really only suitable in cases where the number of documents returned by the query is small because if the filtering by _id in the second feed.
Care must be taken to ensure that the second feed is established correctly if there are lots of changes coming in from the first changes feed.
There are cases where a change will appear in both feeds. It would be good to avoid reacting twice.
If changes are expected to happen frequently, then employ debouncing or rate limiting if your client does not need to process each and every change notification.
This approach worked well for me and the cases I had to deal with.
References:
http://docs.couchdb.org/en/stable/api/database/find.html
http://docs.couchdb.org/en/stable/api/database/changes.html
The behaviour that you described is correct.
CouchDB will populate the changes feed with the docs that accomplish with the filter function. If you remove/modify the information that is used by the filter function the filtered changes feed will ignore those updates.
The closest you will come to this is to use a view and filter the changes feed based on that view - see [1] for details.
You can create a simple view that includes the "version" as part of the key using a map function such as:
function (doc) {
emit(doc.version, 1);
}
A changes feed filtered by this view will notify you of the insert or deletion of documents that have a "version" field as well as changes to the "version" field of existing documents. You can not, however, determine the previous value of the "version" field from the changes feed.
Depending on your requirements, you can make the view more targeted. For example, if you only cared about transition form "A" to "B" then you could include only documents that have "A" or "B" as their "Version":
function (doc) {
if( doc.version === "A" || doc.version === "B") {
emit(doc.version, 1);
}
}
But be aware that this will not trigger a change notification on transition from, say, "A" to "C" (or any other value for "version", including when the document is deleted) because change notifications are only send when the map function emit()'s at least one value for a document. It doesn't not notify you when the map function used to emit at least one value for a give document, but no longer does!
You can also filter the changes feed using Mango selectors, so if Mango queries work for you then perhaps this is simpler than using a view, but I'm not sure that you can be notified of deletions via Mango selectors...
EDIT:
May claim about the simple map function above is not quite right as it will notify you of all document insertions and deletions, not just ones with a "version" field. You can do this to avoid some of those false positive notifications:
function (doc) {
if ( doc.hasOwnProperty( 'version' ) || doc.hasOwnProperty( '_deleted' ) ) {
emit(doc.version, 1);
}
}
That will give notifications for new documents with a "version" field, or an update that adds a "version" field to an existing document, but it will still notify of all deletions.
[1] http://docs.couchdb.org/en/stable/api/database/changes.html#changes-filter-view
I am trying to access a field in custom request handler. I am accessing it like this for each document:
Document doc;
doc = reader.document(id);
DocFields = doc.getValues("state");
There are around 600,000 documents in the solr. For a query running on all the docs, it is taking more than 65 seconds.
I have also tried SolrIndexSearcher.doc method, but it is also taking around 60 seconds.
Removing the above lines of code bring down the qtime to milliseconds. But, I need to access that field for my algo.
Is there a more optimised way to do this?
It seems that you querying one document at a time which is slow.
If you need to query all documents try to query *:*(instead of asking for a specific id) and then iterate over the results.
Trying to update an existing CouchDB map function so that it only returns docs created in the past 24 hours.
The current map is very simple
function(doc) {
if(doc.email && doc.type == 'user')
emit(doc.email, doc);
}
I'd like to get the current linux timestamp value and compare that to the creationTime.unix value stored in the doc.
Is that possible?
N.B I'm building the view in futon
I do not know if you can do that, but it if you can that would be very bad for CouchDB database sanity.
Map functions for same document should always emit same values, each time you invoke it (provided that document has not changed in the mean time). This is important since CouchDB stores this emited data in the index, and does not recalculate it again until it is necessary. If map functions could emit different values for the same doc, that would render index unusable.
So, no, do not try that.
Good news is that you can easily achieve what you need without that. If you emit creation time, than you can query your view just for docs with creation time in certain interval like in:
/blog/_design/docs/_view/by_date?startkey="2010/01/01 00:00:00"&endkey="2010/02/00 00:00:00"
Read more how you can query your views in CouchDB The Definitive Guide
At the moment I'm pointless how do achieve pagination with ember-data. I found out, that i can return 'meta' property in response and ember-data does not throw an error. But I just don't know how to access it, nor what is intended purpose of this property.
The few examples on internet assume that i have whole collection already loaded to ember, or they do little trick and do infinite scroll, which does'nt require information about page count.
I think that loading all records it's ok if I would have < 1k of them, but sometimes I'll be dealing with massive amounts of data (let's say apache logs). What then?
So basically I'm at the point in which I would like to use ember and ember-data to build my first real-life application, but I just think that it is not a good idea.
Ok, so anybody has any idea how to solve this basic, yet complicated, problem? :)
Ok, so here are some ideas to get you started.
First, you have to start with a route and take an page number as a dynamic parameter.
this.resource('posts', { path: '/posts/:page' };
Then as I have no experience with Silex, you need to support some kind of server side parameters that could be used for pagination. For example offset and limit where first means how many records you want to skip and second how many record you want in select from there. Ideally you should implement them as query parameters like ?offset=0&limit=10.
Then you just implement your table route as follows:
App.TableRoute = Ember.Route.extend({
model: function (params) {
return App.Post.find({ offset: (params.page - 1) * 10, limit: 10 });
}
});
You can then start doing some more magic and create your items per page parameter or validate the page number by fetching number of all records in advance.
There's a bunch of questions on here related to pagination using CouchDB, but none that quite fit what I'm wondering about.
Basically, I have a result set ranked by number of votes, and I want to page through the set in descending order.
Here's the map for reference.
function(doc) {
emit(doc.votes);
}
Now, the problem. I found out that startkey_docid doesn't work on it's own. You have to use it in combination with startkey. The thing is, for the query, I don't use a startkey parameter (I'm not looking to restrict the results, just get the most->least). I was thinking I could just use startkey={{doc.votes}}&startkey_docid={{doc._id}} instead, but the number of votes for a document could have changed by the time someone clicks the "Next Page" link.
The way to solve this seemed obvious: just set startkey=99999999 so that it will return all documents in the database and I can just use startkey_docid to start at the one where we left off last time. Oddly, when I do that, the startkey_docid stopped working and just allowed all results to be returned again. Apparently startkey needs to exactly equal the key on the document whose _id is used in startkey_docid.
What I'm asking is whether anyone knows a workaround for using startkey_docid to page when the actual startkey could have changed by the time you want to use it? Should my application just lookup the document by _id and immediately use the doc.votes value hoping it hasn't changed in the few milliseconds between requests? Even that doesn't seem very reliable.
EDIT: Ended up switching to Mongo for the speed, so this question turned out to be kinda moot.
I have never done something like this but I think I have some idea how to do it. What you can do is to take a snapshot of the ratings and refer to it in every page. You probably want your view not to consume to much space, so you should not map separate copies of the documents with votes not changed after taking the snapshot. So, you can do the following:
Add some history of ratings with timestamp to your document.
Map the ratings AND history like this.
In your app get the current time: start_time = Date.now() and query all pages.
Cleanup the history older then the oldest active sessions.
The problem is that if you emit [votes, date] and try to paginate you will never know how many document you have to fetch to get desired number per page. There can always be some older version which you will have to skip, and you will have make next get from DB. Thats why you can consider emitting: [date, votes], read the view always twice -- for start_time and current time, and merge and sort the result (like in merge-sort).
Ad.1:
{ ...,
votes: 12,
history: [
{date: 1357390271342, votes: 10},
{date: 1357390294682, votes: 11}
]
}
Ad.2:
function (doc) {
emit([{}, doc.votes], null);
doc.history && doc.history.forEach(function(h) {
emit([h.date, h.votes], null);
});
}
Ad.3:
?startkey=[start_time, votes]&limit=items_per_page_plus1
?startkey=[{}, votes]&limit=items_per_page_plus1
Merge lists, sort by votes in your app (on in a list function).
If you will have problems with using start_docid then you can emit [date, votes, id] and query with the ID explicitly. Even when this particular doc changes its votes it will still be available in the history.
Ad.4:
If you emit [date, votes] then you can just get outdated history width: ?startkey=[0]&endkey=[oldest_active_session_time]&inclusive_end=false and update them with update handler:
function(doc, req) {
if (!doc || !doc.history) return [null, 'Error'];
var history = new Array();
var oldest = +(req.query.date);
doc.history.forEach(function(h) {
if (h.date >= oldest)
history.push(h);
});
doc.history = history;
return [doc, 'OK'];
}
Note: I have not tested it, so it is expected not to run without modifications :)
As far as I know CouchDB uses b-tree shadowing to make updates and in principle is should be possible to access older revisions of the view. I am not into the CouchDB design, so it is just a guess and there seems not to be any (documented) API for this.
I can't figure out any simple solution by now, but there are options:
Replicate not-so-often your sorting list to small dedicated db so it will be much more stale than stale=ok
Modify your schema in a way that you'll be able to sort by some more stable data. Look at the banking/ledger example in CouchDb guide: http://guide.couchdb.org/draft/recipes.html#banking. Try to log every vote and reduce them hourly for example. As a bonus you'll get a history/trends :)
I'm kind of surprised this question has been left unanswered because the functionality of CouchDB Futon basically does this when you are paginating through the results of a map function. I opened up firebug to see what was happening in the javascript console as I paginated and saw that for every set of paginated results it is passing the startkey along with startkey_docid. So although the question is how do I paginate without including startkey, CouchDB specifies that the startkey is required and demonstrates how it can work. The endkey is not specified, so if there is only one result for the specified startkey, the next set of paginated results will also contain the next key of the sorted results that do not match the startkey.
So to clarify a bit, the answer to this problem is that as you are paginating and keeping track of the startkey_docid, you also need to capture the startkey of the same document that will be the start of the next set of results. When you are calling the paginated results use both the captured startkey and startkey_docid as couchdb requires. Leave endkey off so that the results will continue on to the next key of the sorted results.
The usecase scenario for wanting to be able to paginate without specifying a key is kind of odd. So let's say that the start docid of the next paginated result did change it's key value drastically from a 9 to a 3. And we are also assuming that there is only one instance of the docid existing in the map results, even though it could potentially appear multiple times (which I believe is why the startkey needs to be specified). As the user is clicking the next button, the user's paginated results will have now moved from looking at rank 9 to rank 3. But if you are including the startkey in addition to the startkey_docid, the paginated results would just start all over at the beginning of the rank 9 results which is a more logical progression than potentially jumping over a large set of results.