When I give the odata service url,https://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/People in browser,I only get 8 records.Why do I get only 8 records when there are total of 20 records in People entity?Is PageSize set in Trippinservice?Can anyone help me to understand this?
Yes, this service implements server-side paging.
Firstly we identify that server-side pagination is in effect from the presense of the #odata.nextLink property in the response, this is in the root of the response:
{
"#odata.context": "http://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/$metadata#People",
"#odata.nextLink": "https://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/People?%24skiptoken=8",
"value": [
... 8 records ...
]
}
We can only assUme that the page size is 8 by counting the number of records in the response or by consulting the documentation for the given service. In this case there is a $skiptoken query parameter inside the next link, in this case it happens to have a value of 8 and this corresponds to the number of records, but only by coincidence.
NOTE: %24 is a dollar sign $ that has been url encoded
~/TripPinServiceRW/People?%24skiptoken=8
In the case of the TripPin service documented in the OData Basic Tutorial, the $skiptoken value represents the number of records to skip and is a common implementation, but it is not a standard.
There is no specific mention of the server page size for the People feed in the documentation, nor is there a standard way to document $skiptoken logic at all in the $metatdata. Have a read on how and why ASP.NET WebAPI implements skiptoken: Use $skiptoken for server-driven paging
We can demonstrate this by navigating the next nextLink or altering the $orderby:
GET: ~/TripPinServiceRW/People?$skiptoken=8
{
"#odata.context": "http://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/$metadata#People",
"#odata.nextLink": "https://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/People?%24skiptoken=16",
"value": [
... 8 more records ...
]
}
The $skiptoken is now 16 in the new nextLink: ~/TripPinServiceRW/People?%24skiptoken=16. In many implementations the $skiptoken will represent the key value of the last record that was sent, but TripPin does not use this convention, we can verify that by changing the order:
Both of these responses will have $skiptoken=8 in the nextLink
GET: ~/TripPinServiceRW/People?$orderby=UserName
{
"#odata.context": "http://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/$metadata#People",
"#odata.nextLink": "https://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/People?%24orderby=UserName&%24skiptoken=8",
"value": [
... 8 records ...
]
}
GET: ~/TripPinServiceRW/People?$orderby=UserName desc
{
"#odata.context": "http://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/$metadata#People",
"#odata.nextLink": "https://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/People?%24orderby=UserName+desc&%24skiptoken=8",
"value": [
... 8 records ...
]
}
According to the specification, $skiptoken is an arbitrary token that services can use to retrieve the next page of data from a previously prepared set. The value of the $skiptoken will have special significance to the server itself and it may be an arbitrary token or reference pointing to a page in a cached resultset.
11.2.5.7 Server-Driven Paging
Responses that include only a partial set of the items identified by the request URL MUST contain a link that allows retrieving the next partial set of items. This link is called a next link; its representation is format-specific. The final partial set of items MUST NOT contain a next link.
OData clients MUST treat the URL of the next link as opaque, and MUST NOT append system query options to the URL of a next link. Services may not allow a change of format on requests for subsequent pages using the next link. Clients therefore SHOULD request the same format on subsequent page requests using a compatible Accept header. OData services may use the reserved system query option $skiptoken when building next links. Its content is opaque, service-specific, and must only follow the rules for URL query parts.
It is worth highlighting this very specific note in the specification:
OData clients MUST NOT use the system query option $skiptoken when constructing requests.
The $skiptoken is a server-side implementation and in many cases you couldn't even guess what a correct value might be. The TripPin service is a very simple demonstration API, it uses a page size of 8 to illustrate the behaviour of server-side paging, given the small size of the overall dataset (20) this is a nice arbitrary number that will result in multiple pages with the last page only partially full. That's enough to test basic compliance of server-side supporting data interfaces.
Server-side paging is designed to encourage search driven interfaces where users formulate more precise search criteria in preference to browsing through the pages 1 by 1. Virtual Scrolling is a common user interface paradigm that exploits server-side paging. Different languages and runtimes have different implementations but basically the user can scroll through a list or grid of data and when they get to the bottom there might be a link to "load more" records (behind the scenes, this will load the response from the nextLink). Sometimes this link is not displayed and the data is automatically loaded as the user approaches or reaches the end of the list.
You can still use traditional client-side paging using the $top and $skip query parameters, however some service implementations of server-side paging will still constrain the results to the fixed number of rows as defined by that servers internal logic. If you are implementing client-side paging then you may still need to use the nextLink to retrieve all the results for each client-side page of results.
Lets compare by first getting page 2, of a client-side page size of 5:
GET: ~/TripPinServiceRW/People?$skip=5&$top=5
{
"#odata.context": "http://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/$metadata#People",
"value": [
... 5 records ...
]
}
Notice that there is no #odata.nextLink property in the response, that is because the requested number of items does not exceed the server page size logic. So lets try a page size of 9. This time, to retrieve all the records for the page, we will need to make multiple queries.
The general guidance here is to recursively call the service for each of the nextLink urls in the responses, if they contain a nextLink
GET: ~/TripPinServiceRW/People?$skip=9&$top=9
{
"#odata.context": "http://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/$metadata#People",
"#odata.nextLink": "https://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/People?%24skip=9&%24top=9&%24skiptoken=8",
"value": [
... 8 records ...
]
}
GET: ~/TripPinServiceRW/People?%24skip=9&%24top=9&%24skiptoken=8
{
"#odata.context": "http://services.odata.org/V4/(S(ysqt4lcalbsipb1qkoc04ryb))/TripPinServiceRW/$metadata#People",
"value": [
... 1 record ...
]
}
When you are implementing your own OData-v4 conformant API it is important to understand this quirk and to document specifically in your API documentation what your policy or convention is with regard to server-side paging and which collections it is enabled on.
In my own implementations, I will often disable server-side paging if the request contains the client-paging token $top (and the value for $top is <= 200) but
From a client-side implementation if you see a nextLink property in the response and you didn't recieve the expected number of records, then you should query for the subsequent server-pages to fulfil your request if you are not able to implement a virtual Scroll enabled user experience.
NOTE: Normally when we discuss paging in OData v4 services the examples would include use of the $count query option.
[11.2.9 Requesting the Number of Items in a Collection]: On success, the response body MUST contain the exact count of items matching the request after applying any $filter or $search system query options...
The returned count MUST NOT be affected by $top, $skip, $orderby, or $expand.
The TripPin service DOES NOT CONFORM to this particular (and many other) clause in the specification, so I have not used that query option in this explanation.
Related
While using security graph api in azure, when i reach the last set of data in the payload #data.nextlink is missing,so that i'm unable to fetch the next set of data?
I read the Graph api and Security Graph api carefully, and I found that when the current page data is fully loaded, there should be no #odata.nextlink parameter.
When you see my test results, I think this api design may need to be improved, when $top is equal to odata count.
Because all the data has been loaded, there is no next page of data you mentioned.
Sample
Query this api
I know that this api will return 63 data after access.
https://graph.microsoft.com/beta/security/secureScores?$top=63&$count=true
Result
Query #odata.nextLink.
When $top=63, although there is no data on the next page, there will still be the #odata.nextLink parameter.
When $top=64, we will see that the #odata.nextLink parameter is gone.
I'm building an application which pulls down incident listings for my org via Pagerduty's REST API.
The GET /incidents endpoint does respond with more, offset, and other keys that are indicative of pagination being supported, and it does make intuitive sense on this endpoint, but I haven't been able to actually paginate these results:
Passing offset or limit as a query param returns a 403
Passing these in various forms in request headers just gets ignored entirely
Is there a way to paginate these results at all?
it might help to include the code you're using to make the request, or a curl request from the command line. Including pagination parameters shouldn't lead to a 403, so I'm thinking something else might be missing.
You should be able to paginate the lists using GET parameters, e.g
https://api.pagerduty.com/incidents?limit=20&offset=100
limit has a maximum value of 100, and limit + offset together must be less than 10,000. That might be why you were getting an error?
See here for additional details on the pagination parameters
Yes, it's possible to paginate the results.
After invoking the API method for the first time, you need to check the more response field value. If true, then you can call the API method again with an updated offset.
offset is related to the total results, and not the total pages.
The 403 error code response you're getting is most likely related to the user permissions and not with paginating results.
I'm reading tutorials on ASP .NET Core and Razor Pages.
One of them, when dealing with the BindProperty attribute, has remarks I find hardly understandable:
Razor Pages, by default, bind properties only with non-GET verbs.
Binding to properties can reduce the amount of code you have to write.
Binding reduces code by using the same property to render form fields
(<input asp-for="Customer.Name" />) and accept the input.
For security reasons, you must opt in to binding GET request data to
page model properties. Verify user input before mapping it to
properties. Opting in to this behavior is useful when addressing
scenarios which rely on query string or route values.
To bind a property on GET requests, set the [BindProperty] attribute's
SupportsGet property to true: [BindProperty(SupportsGet = true)]
(emphasis mine) Source: Introduction to Razor Pages in ASP.NET Core ยง Writing a basic form
I do not understand. Why do extra security measures need to be taken when dealing specifically with GET requests?
As far as I can tell, GET requests are supposed to be safer, not less safe than POST requests, because GET only retrieves data, while POST sends data to the server. So, more often than not, POST requests need extra security measures GET reqs don't need.
And yet now I'm reading that it's fine to do X with POST but careful, don't do this with GET recklessly, you must opt-in and be warned, take precautions!
Why are these warnings necessary? What kind of security vulnerabilities can be introduced by binding GET request data to page model properties? And why are these vulnerabilites not applicable to POST requests?
Binding is two-way model, meaning it binds a given element for both rendering ("show this value here") and submitting back ("save this value there"). For example, you'd use binding to populate an input field with a value that a user could then update and POST back. Binding handles dealing with the value automatically.
If you simply want to display a value, you don't need to use binding at all. Simply make it a public property of your model and reference it directly ({Model.PropertyName}).
There are a bunch of reasons for avoiding [BindProperty(SupportsGet = true)] but I think HTTP's RFC 7231, Section 9.4 covers it well:
URIs are intended to be shared, not secured, even when they identify secure resources. URIs are often shown on displays, added to templates when a page is printed, and stored in a variety of unprotected bookmark lists. It is therefore unwise to include information within a URI that is sensitive, personally identifiable, or a risk to disclose.
Authors of services ought to avoid GET-based forms for the submission of sensitive data because that data will be placed in the request-target. Many existing servers, proxies, and user agents log or display the request-target in places where it might be visible to third parties. Such services ought to use POST-based form submission instead.
Since the Referer header field tells a target site about the context that resulted in a request, it has the potential to reveal information about the user's immediate browsing history and any personal information that might be found in the referring resource's URI.
https://api.github.com/search/issues?q=stress+test+label:bug+language:python+state:closed
the above query is suppose to return 76 results, and when I try to run it, it only returns 30. I guess GitHub return results in portions when it is over 30. Any idea how I can get the rest of the results?
You need to use page parameter, e.g. for next 30 page = 2
https://api.github.com/search/issues?q=stress+test+label:bug+language:python+state:closed&page=2
You can also use per_page parameter to change the default size of 30. It supports max size of 100. Like this:
https://api.github.com/search/issues?q=stress+test+label:bug+language:python+state:closed&per_page=100
More detail can be found here
The Problem: Github api response doesn't contain all the relevant data.
Solution: The api from server is limiting the amount of items the user gets and splitting it into pages (pagination). You should explicitly Specify in your request how many items you'd like to receive from server pagination engine ,using formula for Github pagination api
?page=1&per_page=<numberOfItemsYouSpecify>"
For example: I'd like to get all my collaborators info in my private repo. I'm performing curl request to Github contains: username, authentication token , Organization and repository name and api call with pagination magic.
curl -u johnDoe:abc123$%^ https://api.github.com/repos/MyOrganizationName/MyAwesomeRepo/collaborators?page=1&per_page=1000"
Explanation:
What is Pagination: Pagination is the process of splitting the contents or a section of a website into discrete pages. Users tend to get lost when there's bunch of data and with pagination splitting they can concentrate on a particular amount of content. Hierarchy and paginated structure improve the readability score of the content. Loading pages is due to the less content on each item and each page has a separate URL which is easy to refer.
In this use case Github api splits the result into 30 items per resonse, depends on the request
Github reference:
Different API calls respond with different defaults. For example, a
call to List public repositories provides paginated items in sets of
30, whereas a call to the GitHub Search API provides items in sets of
100
I have developed a Web Interface for a db. The db and Web Interface are for my own use in my hobby running on my private intranet. Currently the db has 1800+ records which is going to increase with usage. Ver 1 of the Web Interface listed all records (~2.5KB) on a single page requiring a ton of scrolling. Vers 2 of the Web interface introduced pagination where records are grouped into a non-fixed size of roughly 100 records. On page load all 1800+ records are still transferred to the client but only the first page is "visible", the other 17 are hidden. I use a series of "non-submit" buttons with JS on-click functionality to hid the current page and make the selected page visible. Better in that scrolling is limited to ~100 records. Vers 3 only transfers the first page and the paging buttons on page load. Now, the on-click function using fetch() API sends a GET Request with parameters to fetch the desired page then swaps it into the DOM. The parameters specify a starting and ending points for the page. These values come from the paging buttons supplied by the server on page load. Works well with significantly reduced data transfer size. In Vers 4 I am generalize the fetch() API GET Request parameters to send user specified parameters to allow the user to choose any page starting and ending point. (Note: The user can not specify a page size directly.) So if the user selects a start and end point that no records fall into my plan is to use HTTP Response Code 204 "No Content" to tell the JS code that there are no matching records and nothing to swap. Is this the appropriate Response Code? Should I be including any other Header information in the Response with the 204 code?
Take a look at what the RFC says about 204:
https://www.rfc-editor.org/rfc/rfc7231#section-6.3.5
It's really intended for PUT requests. I think for what you are doing, it's fine to return 200 and no body, with a Content-Length header of 0.