I know that it is not an advisable solution to use GET however I am not in control of how this server works and have very little experience with requests.
I'm looking to add a dictionary via a GET request and was told that the server had been set up to accept this but I'm not sure how that works. I have tried using
import requests
r = request.get('www.url.com', data = 'foo:bar')
but this leaves the webpage unaltered, any ideas?
To use request-body with a get request, you must override the post method. e.g.
request_header={
'X-HTTP-Method-Override': 'GET'
}
response = requests.post(request_uri, request_body, headers=request_header)
Use requests like this pass the the data in the data field of the requests
requests.get(url, headers=head, data=json.dumps({"user_id": 436186}))
It seems that you are using the wrong parameters for the get request. The doc for requests.get() is here.
You should use params instead of data as the parameter.
You are missing the http in the url.
The following should work:
import requests
r = request.get('http://www.url.com', params = {'foo': 'bar'})
print(r.content)
The actual request can be inspected via r.request.url, it should be like this:
http://www.url.com?foo=bar
If you're not sure about how the server works, you should send a POST request, like so:
import requests
data = {'name', 'value'}
requests.post('http://www.example.com', data=data)
If you absolutely need to send data with a GET request, make sure that data is in a dictionary and instead pass information with params keyword.
You may find helpful the requests documentation
Related
I am new to the whole backend stuff I understood that both bodyparser and express.json() will parse the incoming request(body from the client) into the request object.
But what happens if I do not parse the incoming request from the client ?
without middleware parsing your requests, your req.body will not be populated. You will then need to manually go research on the req variable and find out how to get the values you want.
Your bodyParser acts as an interpreter, transforming http request, in to an easily accessible format base on your needs.
You may read more on HTTP request here ( You can even write your own http server )
https://nodejs.org/api/http.html#http_class_http_incomingmessage
You will just lose the data, and request.body field will be empty.
Though the data is still sent to you, so it is transferred to the server, but you have not processed it so you won't have access to the data.
You can parse it yourself, by the way. The request is a Readable stream, so you can listen data and end events to collect and then parse the data.
You shall receive what you asked for in scenarios where you do not convert the data you get the raw data that looks somewhat like this username=scott&password=secret&website=stackabuse.com, Now this ain't that bad but you will manually have to filter out which is params, what is a query and inside of those 2 where is the data..
unless it is a project requirement all that heavy lifting is taken care of by express and you get a nicely formatted object looking like this
{
username: 'scott',
password: 'secret',
website: 'stackabuse.com'
}
For Situation where you DO need to use the raw data express gives you a convenient way of accessing that as well all you need to do is use this line of code
express.raw( [options] ) along with express.json( [options] )
I am trying to handle an API response with pagination. The first page provides a pagination token to reach the next one, but when I try to feed this back into the params parameter of the requests.get method it seems to slightly encode the token in the wrong way.
My attempt to retrieve the next page (using the response output of the first requests.get method):
# Initial request
response = requests.get(url=url, headers=headers, params=params)
params.update({"paginationToken": response.json()["paginationToken"]})
# Next page
response = requests.get(url=url, headers=headers, params=params)
This fails with status 500: Internal Server Error and message Padding is invalid and cannot be removed.
An example pagination token:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
The url attribute of response seems to show a slightly different token if you look carefully, especially around the '%' signs:
https://www.wikiart.org/en/Api/2/DictionariesByGroup?group=1&paginationToken=gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d
For example, the pagination token and url end differently: 226M%3d and 226M%253d. When I manually copy the first part of the url and add in the correct pagination token it does retrieve the information in a browser.
Am I missing some kind of encoding I should apply to the request.get parameters before feeding them back into a new request?
You are right it is some form of encoding, percentage encoding to be precise. It is frequently used to encode URLs. It is easy to decode:
from urllib.parse import unquote
pagination_token="gyuqfh%252bqyNrV9SI1%252bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%253d"
pagination_token = unquote(pagination_token)
print(pagination_token)
Outputs:
gyuqfh%2bqyNrV9SI1%2bXulE6MXxJgb1VmOu68eH4YZ6dWUgRItb7yJPnO9bcEXdwg6gnYStBuiFhuMxILSB2gpZCLb2UjRE0pp9RkDdIP226M%3d
But I expect that is half your problem, use a requests session object https://requests.readthedocs.io/en/master/user/advanced/#session-objects to make the requests as there is most likely a cookie which will be sent with the request to be used in conjunction with the pagination token. I can not tell for sure as the website is currently down.
I would like to send a POST request using the requests library in Python 3. The only difficulty I am having is in how to embed the urlencoded form part. The raw form looks like:
URLEncoded form:
data[type]: portion_send
data[attributes][style]: first_style
data[nodes][front][data][id]:1111
data[nodes][back][data][id]:1115
The best I have been able to do is to have:
data = {"data":{"attributes":{"style":"first_style"},"nodes":{"front":{"data":{"id":"1111"}},"back":{"data":{"id":"1115"}}},"type":"portion_send"}}
and to embed this inside a requests.post function with params = data, params = json.dumps(data), and data = json.dumps(data), each to no avail.
Does anyone have any ideas on how I can put this into a post request? The strange part is that the nesting jumps deeply and it is throwing me off. Thanks!
Python Requests Module
I need to know what the error message was to help you with the python requests module.
Using Linux or MacOS Curl`
If the above fails, you could try this:
os.execv('curl -d data=value http://YourWebsite.com')
It executes a separate process on your PC. If you run Windows, you could install curl from here: curl.haxx.se.
Right, hello, so I'm trying to implement opticard (loyality card services) with my webapp using trio and asks (https://asks.readthedocs.io/).
So I'd like to send a request to their inquiry api:
Here goes using requests:
import requests
r = requests.post("https://merchant.opticard.com/public/do_inquiry.asp", data={'company_id':'Dt', 'FirstCardNum1':'foo', 'g-recaptcha-response':'foo','submit1':'Submit'})
This will return "Invalid ReCaptcha" and this is normal, and what I want
Same thing using aiohttp:
import asyncio
import aiohttp
async def fetch(session, url):
async with session.post(url, data={'company_id':'Dt', 'FirstCardNum1':'foo', 'g-recaptcha-response':'foo','submit1':'Submit'} ) as response:
return await response.text()
async def main():
async with aiohttp.ClientSession() as session:
html = await fetch(session, 'https://merchant.opticard.com/public/do_inquiry.asp')
print(html)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Now this also returns "Invalid ReCaptcha", so that's all nice and good.
However now, using trio/asks:
import asks
import trio
async def example():
r = await asks.post('https://merchant.opticard.com/public/do_inquiry.asp', data={'company_id':'Dt', 'FirstCardNum1':'foo', 'g-recaptcha-response':'foo','submit1':'Submit'})
print(r.text)
trio.run(example)
This returns a completely different response with 'Your session has expired to protect your account. Please login again.', this error/message can be accessed normally when inputting an invalid url such as 'https://merchant.opticard.com/do_inquiry.asp' instead of 'https://merchant.opticard.com/public/do_inquiry.asp'.
I have no idea where this error is coming from, I tried setting headers, cookies, encoding, nothing seems to make it work. I tried replicating the issue, but the only way I managed to replicate the result with aiohttp and requests is by setting an incorrect url like 'https://merchant.opticard.com/do_inquiry.asp' instead of 'https://merchant.opticard.com/public/do_inquiry.asp'.
This must be an issue from asks, maybe due to encoding or formatting, but I've been using asks for over a year and never had an issue where a simple post request with data would return differently on asks compared to everywhere else. And I'm baffled as I can't understand why this is happening, it couldn't possibly be a formatting error on asks' part because if so how come this is the first time something like this has ever happened after using it for over a year?
This is a bug how asks handles redirection when a non-standard location is received.
The server returns a 302 redirection with Location: inquiry.asp?... while asks expects it to be a full URL. You may want to file a bug report to asks.
How did I find this? A good way to go is to use a proxy (e.g. mitmproxy) to inspect the traffic. However asks doesn't support proxies. So I turned to wireshark instead and use a program to extract TLS keys so wireshark can decrypt the traffic.
I'm trying to use python's requests library to log in to a website. It's a pretty simple code, and you can really get the gist of requests just by going on its website. I, however, want to check if I'm successfully logged in via the url. The problem I've encountered is when I initiate the post requests and give it (the variable p) a url, whether the html has changed or not I'm still passed the same url when I type print(p.url). Is there any way for me to refresh the browser or update the url to whatever it's currently set at?
(I can add a line for checking the url against itself later, but for now I just want to get the correct url)
#!usr/bin/env python3
import requests
payload = {'login': 'USERNAME,
'password': 'PASSWORD'}
with requests.Session() as s:
p = s.post('WEBSITE', data=payload)
#print p.text
print(p.url)
The usuage of python-requests may not as complex as you think. It will automatically handle the redirect of your post ( or session.get()).
Here, session.post() method return a response object:
r = s.post('website', data=payload)
which means r.url is current url you are looking for.
If you still want to refresh current page, just use:
s.get(r.url)
To verify whether you has login successfully, one solution is to do the login in your browser.
Based on the title or content of the webpage returned (i.e, use the content in r.text), you can judge whether you have made it.
BTW, python-requests is a great library, enjoy it.