It used to work like this:
import requests
u = "https://uza3fcl1odf.typeform.com/app/form/result/token/tCsWUtMB/default"
j = requests.post(u, headers={"Accept": "application/json"}).json()
# {'token': '20903331626f396431746f397777693832713331626f39643136747572367978706434313339363936363463366336363533373434333536343134353337366237613339343233303664373236613737353935613431363436393336343833313335333833383332333733393336333333373633653233363637346132653432343938383963326537663165326535633262373837326336613437343539666638653932613131393266386236393635626131353838323739363337', 'landed_at': '1588279637'}
Also here are the topics about this:
Post request to Typeform failing due to Invalid payload
Python post request, problem with posting
error with signature token when filling a typeform
How to get JSON data from a Python POST request
Here is my test form: https://uza3fcl1odf.typeform.com/to/tCsWUtMB
But now this solution does not work, I do not know why. What kind of POST request should I make to get "signature" and" landed_at " data in response?
It is desirable that your example can be easily checked in Postman.
Related
I am using a GET request to fetch what I expect to be an XML document from an endpoint. The response has the following structure:
' <itunes:explicit>clean</itunes:explicit>\n' +
' <itunes:episode>11</itunes:episode>\n' +
' <itunes:episodeType>full</itunes:episodeType>\n' +
(This is from a console log in a Node.js function).
I haven't encountered a response like this before and am having trouble doing anything useful with it. I've tried:
Changing the response type and encoding of my GET function
Parsing the response with an XML parser - this throws an error
Removing the newline and + characters manually with regex (I'd like to avoid this, but it doesn't seem to work anyway)
It's worth saying that the response looks as you'd expect in a browser window:
Am I missing something fundamental about how this data is encoded / structured and what is the best way to turn it into something I can work with?
Rookie error. In case anyone else stumbles across this; I expected the response from my Axios GET request to be the xml. The response was actually in a data property in the response:
const response = await axios.get(url);
const myXML = response.data;
I am trying to handle an API response with pagination. The first page provides a pagination token to reach the next one, but when I try to feed this back into the params parameter of the requests.get method it seems to slightly encode the token in the wrong way.
My attempt to retrieve the next page (using the response output of the first requests.get method):
# Initial request
response = requests.get(url=url, headers=headers, params=params)
params.update({"paginationToken": response.json()["paginationToken"]})
# Next page
response = requests.get(url=url, headers=headers, params=params)
This fails with status 500: Internal Server Error and message Padding is invalid and cannot be removed.
An example pagination token:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
The url attribute of response seems to show a slightly different token if you look carefully, especially around the '%' signs:
https://www.wikiart.org/en/Api/2/DictionariesByGroup?group=1&paginationToken=gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d
For example, the pagination token and url end differently: 226M%3d and 226M%253d. When I manually copy the first part of the url and add in the correct pagination token it does retrieve the information in a browser.
Am I missing some kind of encoding I should apply to the request.get parameters before feeding them back into a new request?
You are right it is some form of encoding, percentage encoding to be precise. It is frequently used to encode URLs. It is easy to decode:
from urllib.parse import unquote
pagination_token="gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d"
pagination_token = unquote(pagination_token)
print(pagination_token)
Outputs:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
But I expect that is half your problem, use a requests session object https://requests.readthedocs.io/en/master/user/advanced/#session-objects to make the requests as there is most likely a cookie which will be sent with the request to be used in conjunction with the pagination token. I can not tell for sure as the website is currently down.
Client
Python Version - 3.9,
Python Requests module version - 2.25
Server
Java 13,
Tomcat 9.
I have a Tomcat+Java based server exposing REST APIs. I am writing a client in python to consume those APIs. Everything is fine until I send empty body in POST request. It is a valid use case for us. If I send empty body I get 400 bad request error - Invalid character found in method name [{}POST]. HTTP method names must be tokens. If I send empty request from POSTMAN or Java or CURL it works fine, problem is only when I used python as a client.
Following is python snippet -
json_object={}
header = {'alias': 'A', 'Content-Type' : 'application/json', 'Content-Length' : '0'}
resp = requests.post(url, auth=(username, password), headers=header, json=json_object)
I tried using data as well instead of json param to send payload with not much of success.
I captured the wireshark dumps to undertand it further and found that, the request tomcat received is not as per RFC2616 (https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html). Especially the part -
Request-Line = Method SP Request-URI SP HTTP-Version CRLF
Because I could see in from wireshark dumps it looked like - {}POST MY-APP-URI HTTP/1.1
As we can see the empty body is getting prefixed with http-method, hence tomcat reports that as an error.
I then looked at python http library code -client.py. Following are relevant details -
File - client.py
Method - _send_output (starting at line # 1001) - It first sends the header at line #1010 and then the body somewhere down in the code. I thought(I could be wrong here) perhaps in this case header is way longer 310 bytes than body 2 bytes, so by the time complete header is sent on wire body is pushed and hence TCP frames are order in such a way that body appears first. To corroborate this I added a delay of 1 second just after sending header line#1011 and bingo, the error disappeared and it started working fine. Not sure if this is completely correct analysis, but can someone in the know can confirm or let me know how to fix this.
I know that it is not an advisable solution to use GET however I am not in control of how this server works and have very little experience with requests.
I'm looking to add a dictionary via a GET request and was told that the server had been set up to accept this but I'm not sure how that works. I have tried using
import requests
r = request.get('www.url.com', data = 'foo:bar')
but this leaves the webpage unaltered, any ideas?
To use request-body with a get request, you must override the post method. e.g.
request_header={
'X-HTTP-Method-Override': 'GET'
}
response = requests.post(request_uri, request_body, headers=request_header)
Use requests like this pass the the data in the data field of the requests
requests.get(url, headers=head, data=json.dumps({"user_id": 436186}))
It seems that you are using the wrong parameters for the get request. The doc for requests.get() is here.
You should use params instead of data as the parameter.
You are missing the http in the url.
The following should work:
import requests
r = request.get('http://www.url.com', params = {'foo': 'bar'})
print(r.content)
The actual request can be inspected via r.request.url, it should be like this:
http://www.url.com?foo=bar
If you're not sure about how the server works, you should send a POST request, like so:
import requests
data = {'name', 'value'}
requests.post('http://www.example.com', data=data)
If you absolutely need to send data with a GET request, make sure that data is in a dictionary and instead pass information with params keyword.
You may find helpful the requests documentation
In Power Query, I can download data from Web using the Web.Contents function, but there's an api that required the request to contains multipart/form data in the following format
"__rdxml"=<*Some data*>
So how do you do this using the Web.Contents function?
I tried, doing
...
PostContent = "__rdxml=<*Some data*>",
Source Web.Contents(url,Content=Text.ToBinary(PostContent))
...
But server response with 400 Bad Request.
I checked the raw request with Fiddler, it seem like the request is not sending with content-type=multipart/form-data header.
I tried manually adding the content-type header with content-type=multipart/form-data, but that doesn't work either. Same 400 Bad Request in the response.
Any idea?
multipart/form-data is a fairly complicated encoding, requiring a bunch of MIME-specific headers. I would first try to see if you can use application/x-www-form-urlencoded instead:
let
actualUrl = "http://some.url",
record = [__rdxml="some data"],
body = Text.ToBinary(Uri.BuildQueryString(record)),
options = [Headers =[#"Content-type"="application/x-www-form-urlencoded"], Content=body],
result = Web.Contents(actualUrl, options)
in
result
EDIT: I've come up with an example of using multipart/form-data with Power Query. It's at https://gist.github.com/CurtHagenlocher/b21ce9cddf54e3807317