How do we use POST method in Python using urllib.request? - python-3.x

I have to make use of POST method using urllib.request in Python and have written the following code for POST method.
values = {"abcd":"efgh"}
headers = {"Content-Type": "application/json", "Authorization": "Basic"+str(authKey)}
req = urllib.request.Request(url,values,headers=headers,method='POST')
response = urllib.request.urlopen(req)
print(response.read())
I am able to make use of 'GET' and 'DELETE' but not 'POST'.Could anyone help me out in solving this?
Thanks

If you really have to use urllib.request in POST, you have to:
Encode your data using urllib.parse.urlencode()(if sending a form)
Convert encoded data to bytes
Specify Content-Type header (application/octet-stream for raw binary data, application/x-www-form-urlencoded for forms , multipart/form-data for forms containing files and application/json for JSON)
If you do all of this, your code should be like:
req=urllib.request.Request(url,
urllib.parse.urlencode(data).encode(),
headers={"Content-Type":"application/x-www-form-urlencoded"}
)
urlopen=urllib.request.urlopen(req)
response=urlopen.read()
(for forms)
or
req=urllib.request.Request(url,
json.dumps(data).encode(),
headers={"Content-Type":"application/json"}
)
urlopen=urllib.request.urlopen(req)
response=urlopen.read()
(for JSON).
Sending files is a bit more complicated.
From urllib.request's official documentation:
For an HTTP POST request method, data should be a buffer in the
standard application/x-www-form-urlencoded format. The
urllib.parse.urlencode() function takes a mapping or sequence of
2-tuples and returns an ASCII string in this format. It should be
encoded to bytes before being used as the data parameter.
Read more:
Python - make a POST request using Python 3 urllib
RFC 7578 - Returning Values from Forms: multipart/form-data

You can use the requests module for this.
import requests
...
url="https://example.com/"
print url
data = {'id':"1", 'value': 1}
r = requests.post(url, data=data)
print(r.text)
print(r.status_code, r.reason)

You can send calls without installing any additional packages.
Call this function with your input data and url. function will return the response.
from urllib import request
import json
def make_request(input_data, url):
# dict to Json, then convert to string and then to bytes
input_data = str(json.dumps(input_data)).encode('utf-8')
# Post Method is invoked if data != None
req = request.Request(url, data=input_data)
return request.urlopen(req).read().decode('utf-8')

Related

How to mock an API that returns a file

I have a function (api_upload) that accepts a file and HTTP headers, it then sends the file along with the HTTP headers to an API service (example.com/upload), which processes the file, and returns a URL of the processed file.
I am trying to write a unit test for this function (api_upload), and I am feeling a bit lost since this is my first time playing with unit-tests. I have installed pip install requests-mock and also pip install pytest-mock. I guess my question is how do I mock the example.com/upload, because I don't need to actually call example.com/upload in my testing. But I need to test that the function api_upload is "working" as expected.
def api_upload(input_file, headers):
# covert the input_file into a dictionary, since requests library wants a dictionary
input_file = {'file': input_file}
url = 'https://example.com/upload'
files = {'file': open(input_file['file'], 'rb')}
response = requests.post(url, headers=headers, files=files)
saved_file_url = response.json()['url']
return saved_file_url
What I tried is:
def test_api_upload(requests_mock):
requests_mock.get("https://example.com/upload", text='What do I put here?')
#I am not sure what else to write here?
The first thing you need to alter is the (HTTP) method you call. api_upload() does a POST request so that needs to be matched by requests_mock call.
The second thing is that you need to mock the response that the mock would return for your requests.post() call. api_upload() apparently expects JSON that consists of an object with url as one of the attributes. That's what you need to return then.
As to what you should do inside your test function, I'd say:
Mock the POST call and the expected response.
Call api_upload() and give it a mock file (see tmp_path that returns an instance of pathlib.Path).
Assert that the returned URL is what you mocked.
An untested example:
def test_api_upload(requests_mock, tmp_path):
file = tmp_path / "a.txt"
file.write_text("aaa")
mocked_response = {"url": "https://example.com/file/123456789"}
requests_mock.post("https://example.com/upload", json=mocked_response)
assert api_upload(input_file=str(file), headers={}) == mocked_response["url"]

Using request_mock to dynamically set response based on request

I am trying to mock a simple POST request that creates a resource from the request body, and returns the resource that was created. For simplicity, let's assume the created resource is exactly as passed in, but given an ID when created. Here is my code:
def test_create_resource(requests_mock):
# Helper function to generate dynamic response
def get_response(request, context):
context.status_code = 201
# I assumed this would contain the request body
response = request.json()
response['id'] = 100
return response
# Mock the response
requests_mock.post('test-url/resource', json=get_response)
resource = function_that_creates_resource()
assert resource['id'] == 100
I end up with runtime error JSONDecodeError('Expecting value: line 1 column 1 (char 0)'). I assume this is because request.json() does not contain what I am looking for. How can I access the request body?
I had to hack up your example a little bit as there is some information missing - but the basic idea works fine for me. I think as mentioned something is wrong with the way you're creating the post request.
import requests
import requests_mock
with requests_mock.mock() as mock:
# Helper function to generate dynamic response
def get_response(request, context):
context.status_code = 201
# I assumed this would contain the request body
response = request.json()
response['id'] = 100
return response
# Mock the response
mock.post('http://example.com/test-url/resource', json=get_response)
# resource = function_that_creates_resource()
resp = requests.post('http://example.com/test-url/resource', json={'a': 1})
assert resp.json()['id'] == 100
This example is not complete and so we cannot truly see what is happening.
In particular, it would be useful to see a sample function_that_creates_resource.
That said, I think your get_response code is valid.
I believe that you are not sending valid JSON data in your post request in function_that_creates_resource.

Formatting Urllib Requests to Get Data From a Server

I am trying to use python to access pull and plot data from a server (ERRDAP if you're interested).
Here is the code that I am trying to use:
url = 'https://scoos.org/erddap/tabledap/autoss.csv?query'
values = {'time>=' : '2016-07-10T00:00:00Z',
'time<' : '2017-02-10T00:00:00Z',
'temperature' : 'temperature',
'time' :'time',
'temperature_flagPrimary'}
data = urllib.parse.urlencode(values)
data.encode('utf-8')
resp = urllib.request.urlopen(req)
The error message that I get is: "POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str." I think this either has something to do with me trying to request a csv file or improperly using urllib. Any help would be greatly appreciated!

Google Translate API : Multiple input texts - Python

I am struggling to find a way to input multiple texts in Google Translate API.
My setup includes the following things.
Using urllib.request.build_opener (Python3)
Google Translate API https://translation.googleapis.com/language/translate/v2
I know that we can pass multiple parameters (Multiple "q"), but I don't know how to do it with Python.
I referred Google Translate documents. I found this.
My Question :
How to add multiple texts to the input. ? Because the following code is not making any sense to me.
data = {'q':'cat', 'q':'dog','source':source,'target':target,'format':'html'}
This is my code.
data = {'q':'This is Text1', 'q':'This is Text2', 'q':'This is Text3', source':source,'target':target,'format':'html'}
_req = urllib.request.Request("https://translation.googleapis.com/language/translate/v2?key="+API_KEY)
_req.add_header('Content-length', len(data))
_req.data = urllib.parse.urlencode(data).encode("utf-8")
response = Connector._get(_req,_session)
Connector._get() is in some other file and it internally calls urllib.request.build_opener with data.
Thanks!
To post multiple parameters (with the same name) in Python for an HTTP request, you can use a list for the values. They'll be added to the URL like q=dog&q=cat.
Example:
headers = { 'content-type': 'application/json; charset=utf-8' }
params = {'q': ['cat', 'dog'],'source':source,'target':target,'format':'html'}
response = requests.post(
"https://translation.googleapis.com/language/translate/v2?key=",
headers=headers,
params=params
)
Specifically, params = {'q': ['cat', 'dog']} is relevant to your question.
I do not have tested by myself, but it seems that you should build the data string to give as data argument to your python urllib.request method. So something like data = "{\n \'q\':{}\n \'q\':{} {} etc.".format(qstr,qstr, etc...)
After that you could want to make it more painfull to have several qs.
You could make a loop and building your string with += operations.

TypeError('not a valid non-string sequence or mapping object',)

I am using aiohttp get request to download some content from another web api
but i am receiving:
exception = TypeError('not a valid non-string sequence or mapping object',)
Following is the data which i am trying to sent.
data = "symbols=LGND-US&exprs=CS_EVENT_TYPE_CD_R(%27%27,%27now%27,%271D%27)"
How to resolve it?
I tried it in 2 ways:
r = yield from aiohttp.get(url, params=data) # and
r = yield from aiohttp.post(url, data=data)
At the same time i am able to fetch data using:
r = requests.get(url, params=data) # and
r = requests.post(url, data=data)
But i need async implementation.
And also suggest me some way if i can use import requests library instead of import aiohttp to make async http request, because in many cases aiohttp post & get request are not working but the same are working for requests.get & post requests.
The docs use bytes (i.e. the 'b' prefix) for the data argument.
r = await aiohttp.post('http://httpbin.org/post', data=b'data')
Also, the params argument should be a dict or a list of tuples.

Resources