https://api.github.com/search/issues?q=stress+test+label:bug+language:python+state:closed
the above query is suppose to return 76 results, and when I try to run it, it only returns 30. I guess GitHub return results in portions when it is over 30. Any idea how I can get the rest of the results?
You need to use page parameter, e.g. for next 30 page = 2
https://api.github.com/search/issues?q=stress+test+label:bug+language:python+state:closed&page=2
You can also use per_page parameter to change the default size of 30. It supports max size of 100. Like this:
https://api.github.com/search/issues?q=stress+test+label:bug+language:python+state:closed&per_page=100
More detail can be found here
The Problem: Github api response doesn't contain all the relevant data.
Solution: The api from server is limiting the amount of items the user gets and splitting it into pages (pagination). You should explicitly Specify in your request how many items you'd like to receive from server pagination engine ,using formula for Github pagination api
?page=1&per_page=<numberOfItemsYouSpecify>"
For example: I'd like to get all my collaborators info in my private repo. I'm performing curl request to Github contains: username, authentication token , Organization and repository name and api call with pagination magic.
curl -u johnDoe:abc123$%^ https://api.github.com/repos/MyOrganizationName/MyAwesomeRepo/collaborators?page=1&per_page=1000"
Explanation:
What is Pagination: Pagination is the process of splitting the contents or a section of a website into discrete pages. Users tend to get lost when there's bunch of data and with pagination splitting they can concentrate on a particular amount of content. Hierarchy and paginated structure improve the readability score of the content. Loading pages is due to the less content on each item and each page has a separate URL which is easy to refer.
In this use case Github api splits the result into 30 items per resonse, depends on the request
Github reference:
Different API calls respond with different defaults. For example, a
call to List public repositories provides paginated items in sets of
30, whereas a call to the GitHub Search API provides items in sets of
100
Related
I am using the marketing API
I have an issue with the pagination, it seems like it ignores the start and count.
I am using the start and count query parameters and no matter what number I put in the count, I get a response with all the results
I followed this document for pagination:
https://learn.microsoft.com/en-us/linkedin/shared/api-guide/concepts/pagination
I use this endpoint:
https://api.linkedin.com/v2/adAnalyticsV2
my parameters include:
start=0&count=10&q=statistics&timeGranularity=MONTHLY ....
in the response, I received 821 elements without any pagination. instead of 10 per page.
if I use the logic in the docs the values for start and count will not affect the results or the query
what am I doing wrong?
I don't want to use it without pagination and find out later that I missed records.
Thanks,
Roiy
I have used pagination to retrieve the posts of the organization, and its working for me
my api call is
api_url = "https://api.linkedin.com/v2/shares?q=owners&owners=urn%3Ali%3Aorganization%3A123456&sharesPerOwner=1000&count=100&start=0"
one thing importantly i noticed is this does not requires restli2.0 version(dont pass this in header)
I have more than 500 records in my blockchain network and i want to write a chaincode for the pagination on frontend. I have used getQueryResultWithPagination and getStateByRangeWithPagination but my concern is that i want to fetch records starting from 90th to 100th means 10 records but not starting from beginning neither i want to fetch the whole record. My chaincode is written in nodejs. I'm stuck on this and would really appreciate the community to give me their valuable suggestions.
Are you looking for directly jumping to a page if so, then its not possible. One drawback of the linked list style pagination is that you can’t pre-compute the rows for a particular page from the page number and the rows per page. Jumping to a specific page doesn’t really work.
Just see here for the reference: https://docs.couchdb.org/en/stable/ddocs/views/pagination.html
getQueryResultWithPagination of course only works with CouchDB ... so when using the CouchDB query language you should be able to specify the skip parameter in your actual query. Note that per the documentation, the limit parameter is not honored as the page size parameter is used:
If a pageSize is specified using the paginated query APIs (GetStateByRangeWithPagination(), GetStateByPartialCompositeKeyWithPagination(), and GetQueryResultWithPagination()), a set of results (bound by the pageSize) will be returned to the chaincode along with a bookmark. The bookmark can be returned from chaincode to invoking clients, which can use the bookmark in a follow on query to receive the next “page” of results.
Of course to build a front-end client which pages through results, you need to pass both the pageSize and bookmark parameters to getQueryResultWithPagination and your chain code function will need to return the bookmark to the caller so that it can be passed in to fetch the next page of results.
I am building a typical pagination that allows the user to click on a particular page number and view the results (similar to the google search result view). I am using the cloudant search API for this. The cloudant search API provides the limit option but no skip option. How can I skip n number of results if the user is on page 1 and clicks on page 4 ?
I can see that the pagination is implemented using bookmarks. Does it mean that I need to first get the bookmark for page 4 by sending 3 additional requests one after another to the search api ?
There are a couple of different ways of handling this - one is the one you already suggested, which is just to fetch the pages as needed to get the bookmarks. I'm not sure there are many alternatives for search results where we can't pre-calculate the results.
Another alternative, and this depends a bit on the details of what you are trying to do, is to create a view containing the data and use the keys to narrow down the view to the results you need. View outputs support use of limit and skip which would enable you to implement pagination.
There's also a good example of pagination in the docs: http://docs.couchdb.org/en/2.1.0/ddocs/views/pagination.html
I'm working on an instagram scraper for something and I'm trying to figure out if it's possible to get all photos for a tag that have an id or timestamp later than the last one I have.
The instagram API docs are useless in that they don't have any real info on pagination (which I presume I'll have to abuse).
Does anyone have any ideas?
I've been slogging through the Instagram API for the last couple of days so here's my 2 cents worth:
As far as I can see it if you call the api with /tags/tag-name/media/recent it only return a list if items. If the amount exceeds about 25 you have to make another request with the pagination value returned in the previous request.
In order to gain some control I am initially iterating through all images and storing the results (just the URL not the actual image) to a database. Now I can manipulate however I want. When I feel like updating (I'm doing it manually now but could be a cron job or use the real-time api) I re-read all the images, compare to what I have in my DB and add possible new images. My app then reads out the url and info from my DB (which btw is a heck of a lot faster than going through the instagram api, which will only return about 25 images per request - regardless of any 'count' parameter value you put in the request url) and displays it.
I am developing this for a client who is afraid of people posting nsfw or whatever pics using their dedicated hashtag (for a contest) - with the above set up I can offer them an interface where they can check and mark images that are then displayed in the app.
One thing to watch out for is when a user deletes his picture; you will have to find a way to check for this. Currently (since I'm lazy) I load all images and use jquery to check for an error loading the image. If there is one I delete the image from the DB (via ajax).
I'm not sure the pagination is going to help you: as far as I can see the pagination response has no relation to the id's of the actual image objects on each page - so theoretically a pagination id that jumps to a certain page (i.e. date) might not work tomorrow if enough images have been deleted in the mean time.
to get all images instead of latest 20, just append &count=-1 to your api call - it's that simple.
In either case, there is a timestamp on each json object - or if you prefer, you can use max_tag_id
check out my post here: there any way to show more than 20 photos of the instagram API?
* Update April 2014: count=-1 is no longer available.
I have around 53,00,000 documents in MarkLogic server and I am building a simple search application. User enters a search term and MarkLogic server searches that term in all the nodes in all the documents and returns the matching documents as the result. I have implemented a custom paging to show results per page. I am showing 10 results per page.
I am using search api for this as:-
import module namespace search="http://marklogic.com/appservices/search" at "/Marklogic/appservices/search/search.xqy";
declare variable $options:=
<options xmlns="http://marklogic.com/appservices/search">
<transform-results apply="raw"/>
</options>;
search:search($p, $options, $noRecFrom, 10)/search:result
where $p is the input from the user $noRecFrom is the number which indicates from where we have to show records. For example for page 1 $noRecFrom will be 1, for page 2 $noRecFrom will be 11, for page 3 $noRecFrom will be 21 and so on. For paging there are hyperlinks to go to First, Next, Prev and Last pages.
To calculate the total number of records returned I am using:-
for $x in search:search($p, $options)
return $x//#total;
While First, Next and Prev hyperlink works perfectly but if someone clicks Last the application stops responding and the query does not show any output. Is it due to the large number of documents in the database or I am implementing it wrongly.
Is there any efficient way for pagination in MarkLogic (for search:search) so that the user can go the Last page without delay in query result for such a large database ?
The way you've implemented it, you're running the search repeatedly in your for loop. And that would indeed be slow.
Instead, you should be calculating a $start parameter based on the #total and number of documents per page, and passing that in as an argument (I think it's the third one) to search:search.
I would also recommend making sure you can run in unfiltered mode. There is good information about optimizing for fast pagination (indexes, etc) on the developer site; the idea is to resolve queries out of indexes to give very good, accurate unfiltered performance.
If you do it that
There is a tutorial on paginated search at http://developer.marklogic.com/learn/2006-09-paginated-search
Once you have resolved the issues mentioned by cwhit above, if you still want to get to the last page of data in a faster manner, you could make your code smart enough to reverse the sort order and pull the correct offset of records.
Here's another tip:
To get better insight to what MarkLogic is doing with search:search, call
search:get-default-options()
to see the starting point for common search applications.