I'm creating a page system using CouchDB, showing:
10 items per page
a link to the previous page (if any)
a link to the next page (if any)
From this article on the topic, I understand that using skip is suboptimal, and that I should instead use the startkey property to specify the first document, read 11 documents from there, display the first 10 and use the key of the 11th to display the link to the next page. What troubles me is the link to the previous page. The article says:
Populating the link to the previous page is as simple as carrying the current startkey over to the next page. If there’s no previous startkey, we are on the first page.
This works when going to the next page: when I move from page 4 to page 5 I can remember that the previous page was 4. But when I move back from page 5 to page 4, I have no way of carrying over the startkey of page 3. How can this work?
Is it possible (and recommended) to use endkey along with skip=10 and limit=1 to find the first element on the previous page, so that I may create a link back to it?
You can in fact only ask for 11 documents with no skip, and that is what Futon does (look at CouchDB logs).
The trick
Both the next and previous page link will be similar: startkey is the first or last element, with a skip=1 to avoid overlapping. You then have to correctly use the descending parameter to get previous documents or next documents.
The execution
Whenever you're asking for a page, CouchDB answers with eleven documents. Let's say the key of the first one is first and the key of the last one is last. The pagination links will look like:
"next": /db/_view/myview?descending=true&limit=11&startkey=last&skip=1
"back": /db/_view/myview?descending=false&limit=11&startkey=first&skip=1
Et voilà! You just have to reverse the documents before displaying them when descending is false. ("Finding your data with views" from the CouchDB guide explains nicely the relation between those parameters and B-Trees.)
Bonus
You can easily get the docid of the first or last page (limit=1 and descending true or false), and get a pagination system that looks a lot like something you would have with a classical database (first, last, previous, next).
Read 21 documents instead of 11 - One extra going forward, and ten going backwards. The very first one holds the key to the previous page.
Related
Numerous readings indicate that skip should REALLY be avoided when doing pagination. As in the link most cite that using startKey and limit is the way to go. After I get the first page I know the startKey of that page, the lastKey of that page and the total entries. If I have a pagination control with page numbers as buttons and the user selects page 3, how do I get there? I have no idea what the startKey of page3. Perhaps, I do a simple view up front to go get the start keys for the each page.
This page nicely describes pagination:
http://docs.couchdb.org/en/1.6.1/couchapp/views/pagination.html
So, you can't really have a "Go to page 298", but to have links to the previous and next 5 pages, you can look up a larger number of preceeding and following documents and generate links accordingly. For example, if you have 10 posts per page, look up 50 following keys and take every 10th one.
As for making a "Go to page X", perhaps a background script that generates somesort of cache?
I have around 53,00,000 documents in MarkLogic server and I am building a simple search application. User enters a search term and MarkLogic server searches that term in all the nodes in all the documents and returns the matching documents as the result. I have implemented a custom paging to show results per page. I am showing 10 results per page.
I am using search api for this as:-
import module namespace search="http://marklogic.com/appservices/search" at "/Marklogic/appservices/search/search.xqy";
declare variable $options:=
<options xmlns="http://marklogic.com/appservices/search">
<transform-results apply="raw"/>
</options>;
search:search($p, $options, $noRecFrom, 10)/search:result
where $p is the input from the user $noRecFrom is the number which indicates from where we have to show records. For example for page 1 $noRecFrom will be 1, for page 2 $noRecFrom will be 11, for page 3 $noRecFrom will be 21 and so on. For paging there are hyperlinks to go to First, Next, Prev and Last pages.
To calculate the total number of records returned I am using:-
for $x in search:search($p, $options)
return $x//#total;
While First, Next and Prev hyperlink works perfectly but if someone clicks Last the application stops responding and the query does not show any output. Is it due to the large number of documents in the database or I am implementing it wrongly.
Is there any efficient way for pagination in MarkLogic (for search:search) so that the user can go the Last page without delay in query result for such a large database ?
The way you've implemented it, you're running the search repeatedly in your for loop. And that would indeed be slow.
Instead, you should be calculating a $start parameter based on the #total and number of documents per page, and passing that in as an argument (I think it's the third one) to search:search.
I would also recommend making sure you can run in unfiltered mode. There is good information about optimizing for fast pagination (indexes, etc) on the developer site; the idea is to resolve queries out of indexes to give very good, accurate unfiltered performance.
If you do it that
There is a tutorial on paginated search at http://developer.marklogic.com/learn/2006-09-paginated-search
Once you have resolved the issues mentioned by cwhit above, if you still want to get to the last page of data in a faster manner, you could make your code smart enough to reverse the sort order and pull the correct offset of records.
Here's another tip:
To get better insight to what MarkLogic is doing with search:search, call
search:get-default-options()
to see the starting point for common search applications.
In my sample project, I set the entity page size to 20. Then I have an entity set with result count which is divisible to the page size. For example, the Categories set which has 100 items. When I go to:
http://localhost/Sample.svc/Categories?$skiptoken=80
I got 81st to 100th categories and the page has the "next" link
http://localhost/Sample.svc/Categories?$skiptoken=100
I tried to go to that page and it returns nothing.
What's the explanation for that?
The paging simply takes the next PageSize items. If it finds less than that, then it's clear there are no more items to return so you don't get the next link. If the query returns the requested number of items, the runtime doesn't try to figure out if this is the last page or not, it simply returns a next link. It might happen that such a link will return no results.
In fact the next link is not bound to return any results, but as long as the response constains another next link, there are potentially more results. The standard built in paging will return pages of the predefined size (except for the last one), but services are free to use any other kind of paging which might return different sizes for each page (including empty pages).
To directly answer your question "Why is the last page empty?":
The runtime doesn't "look ahead" so it can't tell if a given page is the last one except for when it gets less than the expected number of results. Looking ahead would be both costly (asking for more than necessary) and potentially wrong (what if the extra result causes an error...).
In my web application I got a search results list (SR). The search is heavily parametrized. Each element on the list can be clicked and then the element's own page (EP) is displayed.
Now, the customer wants to have the ability to go to previous and next element from the search list that was used to enter the element page.
How would you implement this? I can either pass the search conditions to the EP and the element's index on the list, then prev/next would just mean to rerun the search query, get previous / next index and display it (still passing the conditions and new index).
Or is there a better approach?
How intensive is your search process? It sounds like something you don't want to execute anymore than necessary. What if you when you render the search results you also store in a list the unqiue EP IDs on the server. You can then navigate through that using indexes for prev/next and the unique ID of the EP element to load details? You can then also store the query term and repopulate the search results with a 'Back to Search' link?
What's a good way to paginate a list that is constantly growing?
For example, a user may open page 2 of some listing, keep looking at it for a while and then open page 3. Meanwhile, new records have been added to the beginning of the list (page 1) and caused page 3 to contain items that could have been on page 2 previously.
Should such a list be paginated from the other end? Then how to deal with the front page and the fact that it would, without special attention, contain TOTAL NUMBER OF ITEMS % PAGE SIZE items (ie. not constant)?
Assuming you have a fixed page size, some specified ordering, and the user is specifying the page they wish to view, I would simply fetch the data accordingly. Trying to do anything more complicated than that will just end up causing you unnecessary headache and confuse the user.
I think the element you're missing here is the ordering. If you specify an ordering then the user will intuitively understand. Also, that's how the majority of pagination is done, so you're not deviating from what you're user really expects.
I'd add a warning to advise the user that new items have come in and allow them to refresh the list. If they choose not to then maintain list in the state it was in when they clicked on the "Page 2" button. You can use a timestamp on the URL to determine which version of the list to serve to which user.
I do not think it's a good idea to page through a growing list. You'd better recalculate items that should be displayed on a page everytime user performs an action.