Client
Python Version - 3.9,
Python Requests module version - 2.25
Server
Java 13,
Tomcat 9.
I have a Tomcat+Java based server exposing REST APIs. I am writing a client in python to consume those APIs. Everything is fine until I send empty body in POST request. It is a valid use case for us. If I send empty body I get 400 bad request error - Invalid character found in method name [{}POST]. HTTP method names must be tokens. If I send empty request from POSTMAN or Java or CURL it works fine, problem is only when I used python as a client.
Following is python snippet -
json_object={}
header = {'alias': 'A', 'Content-Type' : 'application/json', 'Content-Length' : '0'}
resp = requests.post(url, auth=(username, password), headers=header, json=json_object)
I tried using data as well instead of json param to send payload with not much of success.
I captured the wireshark dumps to undertand it further and found that, the request tomcat received is not as per RFC2616 (https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html). Especially the part -
Request-Line = Method SP Request-URI SP HTTP-Version CRLF
Because I could see in from wireshark dumps it looked like - {}POST MY-APP-URI HTTP/1.1
As we can see the empty body is getting prefixed with http-method, hence tomcat reports that as an error.
I then looked at python http library code -client.py. Following are relevant details -
File - client.py
Method - _send_output (starting at line # 1001) - It first sends the header at line #1010 and then the body somewhere down in the code. I thought(I could be wrong here) perhaps in this case header is way longer 310 bytes than body 2 bytes, so by the time complete header is sent on wire body is pushed and hence TCP frames are order in such a way that body appears first. To corroborate this I added a delay of 1 second just after sending header line#1011 and bingo, the error disappeared and it started working fine. Not sure if this is completely correct analysis, but can someone in the know can confirm or let me know how to fix this.
Related
I am trying to handle an API response with pagination. The first page provides a pagination token to reach the next one, but when I try to feed this back into the params parameter of the requests.get method it seems to slightly encode the token in the wrong way.
My attempt to retrieve the next page (using the response output of the first requests.get method):
# Initial request
response = requests.get(url=url, headers=headers, params=params)
params.update({"paginationToken": response.json()["paginationToken"]})
# Next page
response = requests.get(url=url, headers=headers, params=params)
This fails with status 500: Internal Server Error and message Padding is invalid and cannot be removed.
An example pagination token:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
The url attribute of response seems to show a slightly different token if you look carefully, especially around the '%' signs:
https://www.wikiart.org/en/Api/2/DictionariesByGroup?group=1&paginationToken=gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d
For example, the pagination token and url end differently: 226M%3d and 226M%253d. When I manually copy the first part of the url and add in the correct pagination token it does retrieve the information in a browser.
Am I missing some kind of encoding I should apply to the request.get parameters before feeding them back into a new request?
You are right it is some form of encoding, percentage encoding to be precise. It is frequently used to encode URLs. It is easy to decode:
from urllib.parse import unquote
pagination_token="gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d"
pagination_token = unquote(pagination_token)
print(pagination_token)
Outputs:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
But I expect that is half your problem, use a requests session object https://requests.readthedocs.io/en/master/user/advanced/#session-objects to make the requests as there is most likely a cookie which will be sent with the request to be used in conjunction with the pagination token. I can not tell for sure as the website is currently down.
I am new to Robot Framework. I was trying to access an URL and generate a token.
I am getting the below error.
Test 1 | FAIL |
405 != 200
upon checking the log
00:52:49.347 INFO POST Response :
url=XXX
status=405, reason=Method Not Allowed
body=None
URL is getting passed correctly and the same is working fine in Postman.
Create Session myssion ${auth_url}
${response}= Post Request myssion /oauth2/token auth=${auth} headers=${headers}
grant_type=${grant_type}
Should Be Equal As Strings ${response.status_code} 200
${token}= evaluate $resp.json().get("access_token")
Please help.
It appears that the problem is a missing space. You only have one space between auth=${auth} and headers=${headers}. You need two or more spaces:
${response}= Post Request myssion /oauth2/token auth=${auth} headers=${headers}
^^
I am trying to send a post request through the request module with headers["Transfer-encoding"] = "chunked", but I am getting back:
<BODY><h2>Bad Request - Invalid Content Length</h2><hr><p>HTTP Error 400. There is an invalid content length or chunk length in the request.</p>
I am sending a json string. headers["Content-Type"] = "application/json" is also given.
Does anybody know if I am missing some setting? Maybe I should set the chunk-size somewhere?
Analysing the headers of the request attached to the response I actually get a content-length header different from zero.
I also tried to create a custom generator from the json string, and pass it to the post method as data=, but it it seems to simply hang there (also above the given timeout=).
Your error says you didn't create the request properly (it's 4xx error, not 5xx which would indicate server issue).
Transfer-Encoding: chunked serves for sending data in chunks. When the body of your message consists of unspecified number of chunks and you send them in lets say - stream. I would suggest reading this.
Each chunk should have it's size in front of the data. For instance:
HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked
9\r\n
Some data\r\n
6\r\n
Python\r\n
If you want to send chunked requests with python requests module. You probably need a generator method for that. Please see this. With such few information I can't help you more.
We have azure function which is running with azure v1 (.NET Framework 4.7). In http response, we are trying to set the content length but it is not flowing to client. It sets the Transfer encoding header with value as chunked.
To give glimpse of how we are setting it.
var body = <some byte array data>
var response = new HttpResponseMessage();
response.Content = new ByteArrayContent(body);
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
response.Content.Headers.ContentLength = body.Length;
I went through some online discussion and Azure/azure-functions-host#3765
There they talked about setting up,
response.Headers.TransferEncoding.Add(TransferCodingHeaderValue.Parse("identity"))
But it seems to be of no use.
Would anyone faced such kind of issue before?
BTW, when I ran the same code on azure function v2 (.NET Core 2.2) it works.
Update: Transfer-Encoding header we could see through packet capture using wireshark.
Wireshark screenshot depicting Transfer-Encoding header even if content-length is set through code
Update-2 : This is happening for function running locally. When we publish the function on azure it does provide the content length header. Locally, I guess it (v1) uses nodejs simulator which seems to have some issue. Anyway, will update here once we find out any workaround.
Content-length can't be set if the Transfer-Encoding is set to be chunked. Transfer-Encoding: chunked isn't needed for progressive rendering. However, it is needed when the total content length is unknown before the first bytes are sent.
Basically, the decimal value of the content-length in OCTETs represents both the entity-length and the transfer-length. If there is some transfer-encoding, then these two lengths will be different and consequently the content-length value should be ignored.
I test with your code in function v1 without Transfer-Encoding it work well too.
So if you want to set specific content length, you could remove Transfer-Encoding.
I'm writing a Python 3 (3.5) script that will act as a simple command line interface to a user's stikked install. The API is pretty straight forward, and its documentation is available.
My post function is:
def submit_paste(paste):
global settings
data = {
'title': settings['title'],
'name': settings['author'],
'text': paste,
'lang': settings['language'],
'private': settings['private']
}
data = bytes(urllib.parse.urlencode(data).encode())
print(data)
handler = urllib.request.urlopen(settings['url'], data)
print(handler.read().decode('utf-8'))
When I run the script, I get the printed output of data, and the message returned from the API. The data encoding looks correct to me, and outputs:
b'private=0&text=hello%2C+world%0A&lang=text&title=Untitled&name=jacob'
As you can see, that contains the text= attribute, which is the only one actually required for the API call to successfully work. I've been able to successfully post to the API using curl as shown in that link.
The actual error produced by the API is:
Error: Missing paste text
Is the text attribute somehow being encoded incorrectly?
Turns out the problem wasn't with the post function, but with the URL. My virtual host automatically forwards http traffic to https. Apparently, Apache drops the post variables when it forwards.