Related
I am trying to handle an API response with pagination. The first page provides a pagination token to reach the next one, but when I try to feed this back into the params parameter of the requests.get method it seems to slightly encode the token in the wrong way.
My attempt to retrieve the next page (using the response output of the first requests.get method):
# Initial request
response = requests.get(url=url, headers=headers, params=params)
params.update({"paginationToken": response.json()["paginationToken"]})
# Next page
response = requests.get(url=url, headers=headers, params=params)
This fails with status 500: Internal Server Error and message Padding is invalid and cannot be removed.
An example pagination token:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
The url attribute of response seems to show a slightly different token if you look carefully, especially around the '%' signs:
https://www.wikiart.org/en/Api/2/DictionariesByGroup?group=1&paginationToken=gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d
For example, the pagination token and url end differently: 226M%3d and 226M%253d. When I manually copy the first part of the url and add in the correct pagination token it does retrieve the information in a browser.
Am I missing some kind of encoding I should apply to the request.get parameters before feeding them back into a new request?
You are right it is some form of encoding, percentage encoding to be precise. It is frequently used to encode URLs. It is easy to decode:
from urllib.parse import unquote
pagination_token="gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d"
pagination_token = unquote(pagination_token)
print(pagination_token)
Outputs:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
But I expect that is half your problem, use a requests session object https://requests.readthedocs.io/en/master/user/advanced/#session-objects to make the requests as there is most likely a cookie which will be sent with the request to be used in conjunction with the pagination token. I can not tell for sure as the website is currently down.
I do not understand why resp.json() needs to be awaited. From my understanding async/await is useful when dealing with I/O. But when I call resp.json() in the example below, has the web request not already been processed with session.get() in the line above?
async with session.get('https://api.github.com/events') as resp:
print(await resp.json())
But when I call resp.json() in the example below, has the web request not already been processed with session.get() in the line above?
No, it reads only HTTP headers, to get response body you need to read the rest of the response.
It's pretty useful since you can check HTTP headers and avoid reading the rest of the response if, let's say, server returned wrong HTTP code.
Another example: if you expect response body to be big, you can read it by chunks to avoid RAM overusage (check note here).
I'm using nock to mock http responses in my integration tests.
I want to return different responses for sequential requests.
For example, for the first 10 requests to /api/movies I want to return 200,
then I want to return 404, and then 200 again.
Something like this:
nock('http://app.com')
.get('/api/movies')
.times(10)
.reply(200)
.get('/api/movies')
.times(1)
.reply(404)
.get('/api/movies')
.times(20)
.reply(200)
But it seems it just execute the first definition.
How can I do it?
I am using node js as reverse proxy mostly using http and http-proxy module. While sending the request to to nodejs to redirect to one of my application, i have to pass request headers which will all be in upper case. However, nodejs or rather http is converting all upper case to lower case, because of which one of the validation of my application is failing.
My code snippet is:
http.createServer(function (request, response) {
var redirection = 'http://localhost:8000';
var path = url.parse(request.url).path;
switch (path) {
case '/health':
proxy.web(request, response, { target: redirection });
break;
}).listen(8080);
Request headers passed are:
curl -H "X-AUTH: PBxqcEm5sU743Cpk" -X GET http://localhost:8080/health
Now what is happening is, header "X-AUTH" is getting transformed into "x-auth" and my application is not able to validate it. In my application the header matching is case sensitive.
The request headers printed from node js request object are:
{ host: 'localhost:8080',
'user-agent': 'curl/7.47.1',
accept: '*/*',
'x-auth': 'PBxqcEm5sU743Cpk' }
My requirement is to retain the upper case of the header passed in request so that my application can validate and authorize it.
Please let me know if there is any way to achieve this
Thanks a lot
FWIW HTTP header field names are case-insensitive so the case really should not matter.
However, node does provide access to the raw headers (including duplicates) via req.rawHeaders. Since req.rawHeaders is an array (format is [name1, value1, name2, value2, ...]), you will need to iterate over it to find the header(s) you are looking for.
Is it possible to change the url of a HTTP-request without redirection?
For example instead of:
request 1
GET /user/abc123/ HTTP/1.1
HTTP/1.1 301 Moved Permanently
Location: /files/abc123
request 2
GET /files/abc123 HTTP/1.1
HTTP/1.1 200 OK
.
.
[filecontent]
I could could respond the file directly, but letting the client know that he got redirected:
single request
GET /user/abc123/ HTTP/1.1
HTTP/1.1 200 OK
Location: /files/abc123
.
.
[filecontent]
As far as I know, it's not possible to do this with HTTP. Redirection in HTTP specifically means the the client is supposed to send a second a request.
I think what you want is more akin to specifying a "canonical url" for some resources, and then having this canonical url displayed in the browsers location bar.
RFC 6596 specifies a way to specify canonical urls with <link rel="canonical">. However, it does not specify what a browser should do with it, if anything. Google uses it to make better choices about which urls to index.
Other than using <link> tags, it's also possible to specify relationships between resources via the HTTP Link header, i.e. Link: </better-url>; rel=canonical. See http://www.w3.org/wiki/LinkHeader . I'm not sure if this would be picked up by Google though. The page at http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394 doesn't mention Google supports it. Browsers surely will disregard it, as they do with practically any link tag, stylesheets being the notable exception.
If the content in question is a HTML document, you could use the HTML5 history API for this. Specifically, use the history.replaceState method. I don't think achieving something similar is possible with other types of content.
Edit
Content-Location header may actually fit what you want quite well.
From section 14.14 of HTTP 1.1 RFC:
The Content-Location entity-header field MAY be used to supply the resource location for the entity enclosed in the message when that entity is accessible from a location separate from the requested resource's URI. A server SHOULD provide a Content-Location for the variant corresponding to the response entity; especially in the case where a resource has multiple entities associated with it, and those entities actually have separate locations by which they might be individually accessed, the server SHOULD provide a Content-Location for the particular variant which is returned.
Content-Location = "Content-Location" ":"
( absoluteURI | relativeURI )
The value of Content-Location also defines the base URI for the entity.
The Content-Location value is not a replacement for the original requested URI; it is only a statement of the location of the resource corresponding to this particular entity at the time of the request. Future requests MAY specify the Content-Location URI as the request- URI if the desire is to identify the source of that particular entity.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
See also What is the purpose of the HTTP header field “Content-Location”?
Well, it is possible, but it feels a bit dirty though.
A quick demo:
var express = require('express');
var app = express();
app.get('/user/abc123', function(req, res, next) {
req.path = req.url = '/files/abc123';
next();
});
app.get('/files/abc123', function(req, res) {
res.set('Location', req.url);
res.send('files!');
});
app.listen(3012);
This is my simple approach, not just change the originalUrl but the path too. My method suggestions:
app.use(function(req, res, next) {
console.log("request", req.originalUrl);
const removeOnRoutes = '/not-wanted-route-part';
req.originalUrl = req.originalUrl.replace(removeOnRoutes,'');
req.path = req.path.replace(removeOnRoutes,'');
return next();
});
By this way /not-wanted-route-part/users will became /users