I have a python server setup with aiohttp that is accepting files POST'd to a specific endpoint. I only want to accept a json body, or gzip'd json files. My code is as follows:
class Uploader(View):
async def post(self):
if not self.request.can_read_body:
return json_response({'message': 'Cannot read body'}, status=400)
elif self.request.content_type != 'application/json' and self.request.content_type != 'multipart/form-data':
return json_response({'message': 'Incorrect data type sent to the server'}, status=400)
try:
json_body = await self.request.json()
# Other bits of code using the json body
except RequestPayloadError as e:
# Internal logging here
return json_response({'message': 'Unable to read payload'}, status=400)
# Other code for handling ValidationError, JSONDecodeError, Exception
return json_response({'message': 'File successfully uploaded'}, status=201)
When I test this by uploading something that isn't json or gzip'd json, the RequestPayloadError exception is correctly being hit, the internal logging is being done as expected, and the client is being returned the expected response. However, I'm also seeing the following unhandled exception:
Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 428, in start
await payload.readany()
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 325, in readany
raise self._exception
File "/web_api/app/views/resources/Uploader.py", line 49, in post
json_body = await self.request.json()
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_request.py", line 512, in json
body = await self.text()
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_request.py", line 506, in text
bytes_body = await self.read()
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_request.py", line 494, in read
chunk = await self._payload.readany()
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 325, in readany
raise self._exception
aiohttp.web_protocol.RequestPayloadError: 400, message='Can not decode content-encoding: gzip'
How am I supposed to handle this currently unhandled exception given that it doesn't seem to be originating in my code, and I'm already handling the one that I'm expecting? Can I suppress aiohttp exceptions somehow?
EDIT: I'm using version 3.1.1 of aiohttp
Can not decode content-encoding: gzip points on the problem source.
Your peer sends data with Content-Encoding: gzip HTTP header but actually the data is not gzip compressed (other compressor is used or no compressor at all).
As result aiohttp fails on decompressing such data with RequestPayloadError exception.
Related
I have a azure python function. I am trying to post data to an API endpoint through my function. Below is the code I have for the same,
import logging
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
try:
req_body = req.get_json()
url = "https://example.com/msgs"
payload= req.get_body()
headers = {
'Authorization': 'mytoken ',
}
response = func.HttpRequest(method="POST", url=url, headers=headers, body=payload,params=None,route_params=None)
return func.HttpResponse("", response)
except :
func.HttpResponse(
"This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
status_code=200
)
Every time I invoke my function, I get below error,
Executed 'Functions.HttpExample' (Failed, Id=bc8a184c-7b20-4946-a92c-ed2afad66e56, Duration=17ms)
[2021-10-15T20:14:11.494Z] System.Private.CoreLib: Exception while executing function: Functions.HttpExample. System.Private.CoreLib: Result: Failure
Exception: TypeError: unable to encode outgoing TypedData: unsupported type "<class 'azure.functions.http.HttpResponseConverter'>" for Python type "NoneType"
Stack: File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.9\WINDOWS\X64\azure_functions_worker\dispatcher.py", line 427, in _handle__invocation_request
return_value = bindings.to_outgoing_proto(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.9\WINDOWS\X64\azure_functions_worker\bindings\meta.py", line 116, in to_outgoing_proto
datum = get_datum(binding, obj, pytype)
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.9\WINDOWS\X64\azure_functions_worker\bindings\meta.py", line 107, in get_datum
raise TypeError(
Can anyone help me to fix this issue? or tell me a way to make a POST call from azure python function?
Thanks,
Tintu
I had the same error, but my problem was different. I had commented out a block of code, which included the following line:
return func.HttpResponse(“Request processed successfully.”, status_code=200)
Commenting out this line meant that the function would end without an HTTP response, therefore returning None. That is where the “None” type in my error came from. I hope this helps somebody else!
I had the same problem!
I get this error:
Exception while executing function: Functions.DouMonitorHttpStart <--- Result: Failure Exception: TypeError: unable to encode outgoing TypedData: unsupported type "<class 'azure.functions.http.HttpResponseConverter'>" for Python type "dict" Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 427, in _handle__invocation_request return_value = bindings.to_outgoing_proto( File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/bindings/meta.py", line 116, in to_outgoing_proto datum = get_datum(binding, obj, pytype) File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/bindings/meta.py", line 107, in get_datum raise TypeError(
But after a re-run, it eventually succeeds.
This happened with me with I had a try block and a finally block, without a except or a else block. I added an except and a else block and it worked fine.
I am trying to handle the specific exception type for a post request but during the execution the URL is throwing the error
Connection refused
When the control reached to the code
except requests.exceptions.ConnectionError as connErr:
this itself throws an exception.
"During handling of the above exception, another exception occurred:"
Where as I know print statement will not write the error to the server logs so I have to use the logger to publish the logs to the server.
So my questions are:
how do define exception handling in a correct way in Python.
Connection refused- what should I conclude by this error. Is it requires some credentials for the post request?? or that IP itself is not working??
Presently I am using Azure insight to trace the errors in the published log.
import requests
import logging
data={'number': 12524, 'type': 'issue', 'action': 'show'}
def GetPost(data):
logging.info('-----------GetPost Function Starts Here-----------')
try:
headers = {
'user-agent': 'customize header string',
'Content-Type': 'application/json; charset=utf-8'
}
response = requests.post('http://dummyurl.org', data= data, headers=headers, timeout=3)
logging.info('Http received response code: ', response.status_code)
response.raise_for_status()
except requests.exceptions.HTTPError as httpErr:
logging.error("Http Error: ", exc_info=httpErr)
except requests.exceptions.ConnectionError as connErr:
logging.error("Error Connecting: ", exc_info=connErr)
except requests.exceptions.Timeout as timeOutErr:
logging.error("Timeout Error: ", exc_info=timeOutErr)
except requests.exceptions.RequestException as reqErr:
logging.error("Something Else: ", exc_info=reqErr)
except Exception as err:
logging.error("Other error occurred: ", exc_info=err)
logging.info('-----------GetPost Function Ends Here-----------')
Following is my error log taken from the Azure Insight:
Error Connecting:
Traceback (most recent call last):
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/util/connection.py", line 96, in create_connection
raise err
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/connection.py", line 353, in connect
conn = self._new_conn()
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f53b8c1a040>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/site/wwwroot/.python_packages/lib/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/home/site/wwwroot/.python_packages/lib/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dummyip', port=443): Max retries exceeded with url: /test.gateway/rest/RestEndpoint (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f53b8c1a040>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/site/wwwroot/PWO_EventHubTrigger/postCall.py", line 13, in GetPost
response = requests.post('https://dummyip:443/test.gateway/rest/RestEndpoint', data= data, headers=headers, timeout=3)
File "/home/site/wwwroot/.python_packages/lib/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/home/site/wwwroot/.python_packages/lib/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/site/wwwroot/.python_packages/lib/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/site/wwwroot/.python_packages/lib/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/site/wwwroot/.python_packages/lib/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='dummyip', port=443): Max retries exceeded with url: /test.gateway/rest/RestEndpoint (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f53b8c1a040>: Failed to establish a new connection: [Errno 111] Connection refused'))
I am calling GetPost function from the main function.
I am new to python development and trying to write a Azure functions. There are multiple similar posts are available but unable to identify the correct answer :(
TL;DR
If you want to avoid During handling of the above exception, another exception occurred catch the first exception from your trace.
Your exception handling is correct it's just that you aren't catching all of them :)
In general it is okay to do that, catch exceptions which you want to handle differently otherwise just raise/handle it commonly.
Your Code:
I am getting a socket error which then raises a ConnectionError, in order to fix that add the socket error as the first exception you expect:
def GetPost(data):
logging.info('-----------GetPost Function Starts Here-----------')
try:
headers = {
'user-agent': 'customize header string',
'Content-Type': 'application/json; charset=utf-8'
}
response = requests.post('http://dummyurl.org', data= data, headers=headers, timeout=3)
logging.info('Http received response code: ', response.status_code)
response.raise_for_status()
except socket.error as exc:
logging.error(f"Caught exception socket.error : {exc}")
except requests.exceptions.HTTPError as httpErr:
logging.error("Http Error: ", exc_info=httpErr)
except requests.exceptions.ConnectionError as connErr:
logging.error("Error Connecting: ", exc_info=connErr)
except requests.exceptions.Timeout as timeOutErr:
logging.error("Timeout Error: ", exc_info=timeOutErr)
except requests.exceptions.RequestException as reqErr:
logging.error("Something Else: ", exc_info=reqErr)
except Exception as err:
raise RuntimeError(f"Something bad happened {err}") from None
logging.info('-----------GetPost Function Ends Here-----------')
GetPost(data)
Refer: Handling Exceptions
how do define exception handling in a correct way in Python.
From your description it seems your exception handling is right.
During handling of the above exception, another exception occurred:
This problem always occur when you raise an exception or the except part has some problem in the except part.
Have you defined the 'httpErr'?
Below code seems on problem on my side:
try:
x = 2
y = 0
result = x / y
except ZeroDivisionError as e:
logging.error("Http Error: ", exc_info="<some info>")
Connection refused- what should I conclude by this error. Is it
requires some credentials for the post request?? or that IP itself is
not working??
There are many possible causes of this problem. Ports, firewalls, virtual networks, ip, etc. In fact, this should be considered a completely different problem, and you need to analyze it according to the specific circumstances of the things you use.
I am trying to create a unit test where I need to upload a CSV file. Here is a snippet I am trying to do,
from tornado.testing import AsyncHTTPTestCase
import json
class TestCSV(AsyncHTTPTestCase):
def test_post_with_duplicates_csv_returns_400(self, *args, **kwargs):
dup_file = open("test.csv", 'r')
body = {'upload': dup_file.read()}
request_config = {
'method': 'POST',
'headers': {
'Content-Type': 'application/json',
'Origin': 'localhost'
},
'body': json.dumps(payload)
}
response = self.fetch('http://localhost/file_upload', **request_config)
self.assertEqual(response.code, 400)
and the actual code looks for the uploaded file like this,
...
file = self.request.files['upload'][0]
...
This returns 500 status code with the following message,
HTTPServerRequest(protocol='http', host='127.0.0.1:46243', method='POST', uri='/v2/files/merchants/MWBVGS/product_stock_behaviors', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 191, in wrapper
result = func(*args, **kwargs)
File "/usr/app/src/handlers/merchants.py", line 463, in post
file = self.request.files['upload'][0]
KeyError: 'upload'
Can some one help me on why the file is not getting detected?
Env: Python 3.6, tornado
You're encoding the file as JSON, but the request.files fields are used for HTML multipart uploads. You need to decide which format you want to use (in addition to those formats, you can often just upload the file as the HTTP PUT body directly) and use the same format in the code and the test.
Tornado doesn't currently provide any tools for producing multipart uploads, but the python standard library's email.mime package does.
Im trying to accelerate multiple get requests to a web service using asyncio and aiohttp.
For that im fetching my data from a postgresql database using psycopg2 module .fetchmany() inside a function and constructing a dictionary of 100 records to send as lists of dictionary urls to an async function named batch() . batch by batch process.
The problem im facing in batch() function is that some requests are logging the message below although the script continues and dont fail but im not able to catch and log this exceptions to later reprocess them.
Task exception was never retrieved
future: <Task finished coro=<batch.<locals>.fetch() done, defined at C:/PythonProjects/bindings/batch_fetch.py:34> exception=ClientOSError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)>
Traceback (most recent call last):
File "C:/PythonProjects/bindings/batch_fetch.py", line 36, in fetch
async with session.get(url) as resp:
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
self._resp = await self._coro
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 497, in _request
await resp.start(conn)
File "C:\Miniconda3\lib\site-packages\aiohttp\client_reqrep.py", line 844, in start
message, payload = await self._protocol.read() # type: ignore # noqa
File "C:\Miniconda3\lib\site-packages\aiohttp\streams.py", line 588, in read
await self._waiter
aiohttp.client_exceptions.ClientOSError: [WinError 10054] An existing connection was forcibly closed by the remote host
Task exception was never retrieved
future: <Task finished coro=<batch.<locals>.fetch() done, defined at C:/PythonProjects/bindings/batch_fetch.py:34> exception=ClientConnectorError(10060, "Connect call failed ('xx.xxx.xx.xxx', 80)")>
Traceback (most recent call last):
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 924, in _wrap_create_connection
await self._loop.create_connection(*args, **kwargs))
File "C:\Miniconda3\lib\asyncio\base_events.py", line 778, in create_connection
raise exceptions[0]
File "C:\Miniconda3\lib\asyncio\base_events.py", line 765, in create_connection
yield from self.sock_connect(sock, address)
File "C:\Miniconda3\lib\asyncio\selector_events.py", line 450, in sock_connect
return (yield from fut)
File "C:\Miniconda3\lib\asyncio\selector_events.py", line 480, in _sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
TimeoutError: [Errno 10060] Connect call failed ('xx.xxx.xx.xxx', 80)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/PythonProjects/bindings/batch_fetch.py", line 36, in fetch
async with session.get(url) as resp:
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
self._resp = await self._coro
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 476, in _request
timeout=real_timeout
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 522, in connect
proto = await self._create_connection(req, traces, timeout)
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 854, in _create_connection
req, traces, timeout)
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 992, in _create_direct_connection
raise last_exc
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 974, in _create_direct_connection
req=req, client_error=client_error)
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 931, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cms-uat.cme.in.here.com:80 ssl:None [Connect call failed ('xx.xxx.xx.xxx', 80)]
Im just entering into asyncio world as you can depict from my code, so all the advises on the full code approach for this scenario are very welcomme.
Thank you
full code below.
import psycopg2.extras
import asyncio
import json
from aiohttp import ClientSession
from aiohttp import TCPConnector
base_url = 'http://url-example/{}'
def query_db():
urls = []
# connection to postgres table , fetch data.
conn = psycopg2.connect("dbname='pac' user='user' host='db'")
cursor = conn.cursor('psycopg2 request', cursor_factory=psycopg2.extras.NamedTupleCursor)
sql = "select gid, paid from table"
cursor.execute(sql)
while True:
rec = cursor.fetchmany(100)
for item in rec:
record = {"gid": item.gid, "url": base_url.format(item.paid)}
urls.append(record.get('url'))
if not rec:
break
# send batch for async batch request
batch(urls)
# empty list of urls for new async batch request
urls = []
def batch(urls):
async def fetch(url):
async with ClientSession() as session:
async with session.get(url) as resp:
if resp.status == 200:
response = await resp.json()
# parse the url to fetch the point address id.
paid = str(resp.request_info.url).split('/')[4].split('?')[0]
# build the dictionary with pa id and full response.
resp_dict = {'paid': paid, 'response': response}
with open('sucessful.json', 'a') as json_file:
json.dump(resp_dict, json_file)
json_file.write("\n")
elif resp.status is None:
print(resp.status)
elif resp.status != 200:
print(resp.status)
response = await resp.json()
# parse the url to fetch the paid.
paid = str(resp.request_info.url).split('/')[4].split('?')[0]
# build the dictionary with paid and full response.
resp_dict = {'paid': paid, 'response': response}
with open('failed.json', 'a') as json_file:
json.dump(resp_dict, json_file)
json_file.write("\n")
loop = asyncio.get_event_loop()
tasks = []
for url in urls:
task = asyncio.ensure_future(fetch(url))
tasks.append(task)
try:
loop.run_until_complete(asyncio.wait(tasks))
except Exception:
print("exception consumed")
if __name__ == "__main__":
query_db()
Task exception was never retrieved
You see this warning when you've created some task, it finished with exception, but you never explicitly retrieved (awaited) for its result. Here's related doc section.
I bet in your case problem is with the line
loop.run_until_complete(asyncio.wait(tasks))
asyncio.wait() by default just waits when all tasks are done. It doesn't distinguish tasks finished normally or with exception, it just blocks until everything finished. In this case it's you job to retrieve exceptions from finished tasks and following part won't help you with this since asyncio.wait() will never raise an error:
try:
loop.run_until_complete(asyncio.wait(tasks))
except Exception:
print('...') # You will probably NEVER see this message
If you want to catch error as soon as it happened in one of tasks I can advice you to use asyncio.gather(). By default it will raise first happened exception. Note however that it is you job to cancel pending tasks if you want their graceful shutdown.
I have an http server that uses an handle with similar logic to this:
MyHTTPHandler(http.server.BaseHTTPRequestHandler):
def do_PUT(self):
content_length = int(self.headers.get('Content-Length', 0))
if content_length:
body = self.rfile.read(content_length)
if not body:
# socket is closed
return
# save body
but when the server is running it always crashes on:
File "/usr/local/lib/python3.5/http/server.py", line 524, in end_headers
self.flush_headers()
File "/usr/local/lib/python3.5/http/server.py", line 528, in flush_headers
self.wfile.write(b"".join(self._headers_buffer))
File "/usr/local/lib/python3.5/socket.py", line 593, in write
return self._sock.send(b)
BrokenPipeError: [Errno 32] Broken pipe
when I return after receiving no body.
I understand that the error is probably because the handler tries to write a response to the socket that was closed but I can't put my finger on where I can catch this in order to prevent the server crash and just close the connection on the server side
Well the stack trace points to the end_headers() method, not your own code. So what you can try is something like this:
def end_headers(self):
try:
super().end_headers()
except BrokenPipeError as e:
pass # insert cleanup here