I'm trying to figure out how to work with a REST api call that is paginated. The JSON response from each page of pagination has a flag "lastPage": True or "lastPage": False to specify if you've reached the last page.
In the Data Factory REST Connector article, the pagination section mentions several supported pagination schemes:
Absolute or relative path of next page in response body or header
Query parameter for the next page in response body or header
Header value for next page in response body or header
None of these three approaches seem to describe the type of response I'm dealing with. What solution would work in this case?
Here's the documentation for the API I'm working with.
About your request, it's not supported for now.
You could post the feedback here: https://feedback.azure.com/forums/270578-data-factory
We can help you vote it up to make product team know.
Related
I have been trying to get the "preview_image_url" from a users timeline of tweets. I have been using V2 and paginator to get all the tweets as API have limit of 100 tweets per request. I am not able to access the media_fields as it's not present or pops up using dir method as well in python. This is the code I am using right now. the code only provides the media key for a particular tweet.
client = tweepy.Client(bearer_token=my_keys.BEARER_TOKEN)
username_list = ["coinfessions"]
user_id = client.get_user(username="coinfessions").data.id
print(user_id)
for tweet in tweepy.Paginator(client.get_users_tweets, id= str(user_id), exclude=['retweets', 'replies'], expansions = "attachments.media_keys", media_fields = ["url","preview_image_url"],max_results=5 ).flatten(10):
if tweet.attachments is not None:
print(tweet.attachments)
In one of the question posted on stack overflow. Someone suggested using the response as the return type. This method includes a "include" key where you can find the type, media_key and preview_image_url. I tried this method and it worked fine with normal response. However, it's not working well with the paginator. I think, I am doing something wrong here.
Anothor response on similar question points to this official FAQ page on tweepy documentation. If anyone can help me in understanding, how to access Models in V2 response.
You can't access the includes if you are using the flatten methods.
The only way to access them is to iterates through the responses themselves.
paginator = tweepy.Paginator(client.get_users_tweets, [...]) # Without the flatten
for page in paginator: # Each page is a response from the API
page.data # All the tweets in each page
page.includes # All the includes in each page
On a side note, I don't understand why you are using max_results=5. Because of that, you retrieve only 5 results per request and you cancel the advantages of the paginator.
While using security graph api in azure, when i reach the last set of data in the payload #data.nextlink is missing,so that i'm unable to fetch the next set of data?
I read the Graph api and Security Graph api carefully, and I found that when the current page data is fully loaded, there should be no #odata.nextlink parameter.
When you see my test results, I think this api design may need to be improved, when $top is equal to odata count.
Because all the data has been loaded, there is no next page of data you mentioned.
Sample
Query this api
I know that this api will return 63 data after access.
https://graph.microsoft.com/beta/security/secureScores?$top=63&$count=true
Result
Query #odata.nextLink.
When $top=63, although there is no data on the next page, there will still be the #odata.nextLink parameter.
When $top=64, we will see that the #odata.nextLink parameter is gone.
I'm building an application which pulls down incident listings for my org via Pagerduty's REST API.
The GET /incidents endpoint does respond with more, offset, and other keys that are indicative of pagination being supported, and it does make intuitive sense on this endpoint, but I haven't been able to actually paginate these results:
Passing offset or limit as a query param returns a 403
Passing these in various forms in request headers just gets ignored entirely
Is there a way to paginate these results at all?
it might help to include the code you're using to make the request, or a curl request from the command line. Including pagination parameters shouldn't lead to a 403, so I'm thinking something else might be missing.
You should be able to paginate the lists using GET parameters, e.g
https://api.pagerduty.com/incidents?limit=20&offset=100
limit has a maximum value of 100, and limit + offset together must be less than 10,000. That might be why you were getting an error?
See here for additional details on the pagination parameters
Yes, it's possible to paginate the results.
After invoking the API method for the first time, you need to check the more response field value. If true, then you can call the API method again with an updated offset.
offset is related to the total results, and not the total pages.
The 403 error code response you're getting is most likely related to the user permissions and not with paginating results.
I have troubles paging through message search results with the rest API.
I have a request looking like this:
outlook.office.com/api/v2.0/me/messages/?$search="deni"
the request returns a proper result and also includes a 'next page' looking like this:
"#odata.nextLink": "https://outlook.office.com/api/v2.0/me/messages/?%24search=%22deni%22&%24top=10&%24skiptoken=aT01NjMzYWQ3OS02MmJjLTQ5ZDEtODg4ZC0zYTgwNDlhOTY3Nzkmcz0xMA%3d%3d"
I guess this link is URL encoded so I URL decode it to get this:
outlook.office.com/api/v2.0/me/messages/?$search="deni"&$top=10&$skiptoken=aT01NjMzYWQ3OS02MmJjLTQ5ZDEtODg4ZC0zYTgwNDlhOTY3Nzkmcz0xMA==
However, when i try to make request with the next link I'm getting 405 Method Not Allowed with the following error:
"The OData request is not supported."
I've tried it in the sandbox as well (oauthplay.azurewebsites.net) - same result. What could it be that I am doing wrong. What is the right way to page through the search results?
I know that there is a limit of 250 messages that could be searched, but this is not the case here. I have 10 and I am trying to read the next 10.
Of course I have tried paging with the $skip and $top parameters, but $kip is not supported together with $search.
I can't seem to find a definitive answer in the documentation on how to page through search results and is it possible at all.
Thanks to anyone willing to help.
I'm trying to get paginated results from Youtube Data API v3,
https://www.googleapis.com/youtube/v3/playlistItems?part=snippet&playlistId=UUCj956IF62FbT7Gouszaj9w&key={YOUR_API_KEY}
now, in the response there's the handy parameter nextPageToken that links to the next page of the results set
"nextPageToken": "CAUQAA"
Is there any way to jump to a specific page of the results, say the 6th?
Apologies, but no, there is not. This is intentional. You can only move forward one page of results at a time.