I have batched and executed my request using the following code:
batch = BatchHttpRequest()
for msg_id in message_ids:
batch.add(service.users().messages().get(userId = 'me', id = msg_id['id']), callback = mycallbackfunc)
batch.execute()
How can I access the responses for each of the requests? I have looked through the documentation but there is no public method to get the responses.
You access the response of each request in the callback function, which is called mycallbackfunc in your example:
def process_message(request_id, response, exception):
if exception is not None:
# Do something with the exception
pass
else:
# Do something with the response
pass
batch = BatchHttpRequest()
for msg_id in message_ids:
batch.add(service.users().messages().get(userId = 'me', id = msg_id['id']), callback=process_message)
batch.execute()
Related
I have created a route with type Json in odoo 14.
#http.route('/test', auth='public', methods=['POST'], type="json", csrf=False)
def recieve_data(self, **kw):
headers = request.httprequest.headers
args = request.httprequest.args
data = request.jsonrequest
So when I receive request using this route everything is fine, but suppose I want to return a different status code like 401, I could not do that with the route which type is json.
I have also tried the bellow method but the problem with this method is that it stops all odoo future requests.
from odoo.http import request, Response
#http.route('/test', auth='public', methods=['POST'], type="json", csrf=False)
def recieve_data(self, **kw):
headers = request.httprequest.headers
args = request.httprequest.args
data = request.jsonrequest
Response.status = "401 unauthorized"
return {'error' : 'You are not allowed to access the resource'}
So in the above example If I set the response status code to 401 all other requests even if they are to different routes will be stopped and its status code changes 401 .
I have checked this problem in both odoo14 and odoo13.
You cannot specify the code of the response, as you can see the status code is hardcoded in http.py:
def _json_response(self, result=None, error=None):
# ......
# ......
return Response(
body, status=error and error.pop('http_status', 200) or 200,
headers=[('Content-Type', mime), ('Content-Length', len(body))]
)
If the result is not an error the status code is always 200. but you can change the code of the method directly or use a monkey patch witch if it's not really important to specify the code of status in the result don't do ti ^^.
A simple monkey patch put this in __init__.py of the module that you want this behavior in it.
from odoo.http import JsonRequest
class JsonRequestPatch(JsonRequest):
def _json_response(self, result=None, error=None):
response = {
'jsonrpc': '2.0',
'id': self.jsonrequest.get('id')
}
default_code = 200
if error is not None:
response['error'] = error
if result is not None:
response['result'] = result
# you don't want to remove some key of another result by mistake
if isinstance(response, dict)
defautl_code = response.pop('very_very_rare_key_here', default_code)
mime = 'application/json'
body = json.dumps(response, default=date_utils.json_default)
return Response(
body, status=error and error.pop('http_status', defautl_code ) or defautl_code,
headers=[('Content-Type', mime), ('Content-Length', len(body))]
)
JsonRequest._json_response = JsonRequestPatch._json_response
and in the result of JSON api you can change the code of status like this
def some_controler(self, **kwargs):
result = ......
result['very_very_rare_key_here'] = 401
return result
The risk in monkey patching is you override the method compeletly and if some change is done in Odoo in new version you have to do you it in your method too.
The above code is from odoo 13.0 I did a lot of this when I build a restfull module to work with a website uses axios witch don't allow sending body in GET request. and odoo expect the argument to be inside the body in json request.
I want to terminate rest request coming to server without further processing if input params are missing.
Currently this is the implementation, which I think is not very good for verify_required_params().
I want to terminate this request without returning any value from verify_required_params() in case of missing params. else flow should continue.
Running this on flask server and open to include any new package for best/ optimized approach.
Can please someone suggest an optimize way for this?
#app.route('/is_registered', methods=['POST'])
def is_registered():
_json = request.get_json()
keys = _json.keys()
customer = Customer()
if verify_required_params(['mobile'], keys) is True:
_mobile = _json['mobile']
validated = validate_mobile(_mobile)
registered = customer.find(_mobile)
if not validated:
response = get_response('MOBILE_NUMBER_NOT_VALID')
return jsonify(response)
if not registered:
response = get_response('MOBILE_NUMBER_NOT_REGISTERED')
return jsonify(response)
response = get_response('MOBILE_NUMBER_REGISTERED')
return jsonify(response)
else:
return verify_required_params(['mobile'], keys)
def verify_required_params(required, received):
required = set(required)
received = set(received)
missing = list(sorted(required - received))
data = {"missing_key(s)": missing}
# response = app.response_class(
# response=json.dumps(data),
# status=200,
# mimetype='application/json'
# )
if missing:
return jsonify(data)
return True
🎶 You say it works in a RESTful way, then your errors come back as 200 OK 🎶
In REST, your URL should encode all the information about your entity. In your case, you are identifying a client by their phone number, and you are getting rather than updating information about them, so your endpoint should look like GET /client/<phonenumber>/registered. That way, a request can't not provide this information without going to a different endpoint.
In short, your code will be replaced with:
#app.route('/client/<mobile>/registered', methods=['GET'])
def is_registered(mobile):
if not mobile.is_decimal():
return jsonify({'error': 'mobile is not number'}), 400 # Bad Request
customer = Customer()
registered = bool(customer.find(mobile))
# does it make sense to have a customer who is not registered yet?
# if not, use:
if not registered:
return jsonify({'error': 'client not found'}), 404 # Not Found
validated = validate_mobile(mobile)
return jsonify( {'validated': validated, 'registered': registered} )
In addition, it's better to have the validation function be a decorator. That way it gets called before the actual business logic of the function. For your example of checking whether request.get_json() contains the proper fields, this is how it would look like:
import functools
def requires_fields(fields):
required_fields = set(fields)
def wrapper(func):
#functools.wraps(decorated)
def decorated(*args, **kwargs):
current_fields = set(request.get_json().keys())
missing_fields = required_fields
if missing_fields:
return jsonify({'error': 'missing fields', 'fields': list(missing_fields)}), 400 # Bad Request
resp = func(*args, **kwargs)
return resp
return wrapper
# usage:
#app.route('/comment', methods=['POST'])
#requires_fields(['author', 'post_id', 'body'])
def create_comment():
data = request.get_json()
id = FoobarDB.insert('comment', author=data['author'], post_id=data['post_id'], body=data['body'])
return jsonify({'new_id': id}), 201 # Created
If you must leave it the way it is now, then in order to not return data from the validation function, you must raise an HTTPException. The default function to do it is flask.abort(code).
I am trying to read request body using django but, it throws an error:
You cannot access body after reading from request's data stream
Here is my code:
#csrf_exempt
def update_profile(request):
"""
"""
if request.method == 'POST':
try:
# Validate
payload = json.loads(request.body)
# get files
profile_pic = request.FILES.get('profile_pic')
user_data = util.update_profile(obj_common, user_id, payload,profile_pic)
return user_data
I have seen many answer on the stackoverflow, They advice me to replace request.body with request.data.
but when it tried i got another error
{AttributeError}'WSGIRequest' object has no attribute 'data'
Looks like request.POST is being processed somewhere before calling this function. Try searching for it
I would simply like to associate responses from aiohttp asynchronous HTTP requests with an identifier. I am using the following code to hit the API and extract contactproperty object which requires an external field (contacid) in order to call its API:
def get_contact_properties(self, office_name, api_key, ids, chunk_size=100, **params):
properties_pages = []
batch = 0
while True:
chunk_ids = [ids[i] for i in range(batch * chunk_size + 1, chunk_size * (1 + batch) + 1)]
urls = ["{}/{}".format(self.__get_base_url(), "contacts/{}/properties?api_key={}".format(contactid, api_key))
for contactid in chunk_ids]
responses_raw = self.get_responses(urls, self.get_office_token(office_name), chunk_size)
try:
responses_json = [json.loads(response_raw) for response_raw in responses_raw]
except Exception as e:
print(e)
valid_responses = self.__get_valid_contact_properties_responses(responses_json)
properties_pages.append(valid_responses)
if len(valid_responses) < chunk_size: # this is how we know there are no more pages with data
break
else:
batch = batch + 1
ids is a list of ids. The problem is that I do not know which response corresponds to which id so that later I can link it to contact entity using contacid. This is my fetch() function so I was wondering how to edit this function to return the contactid along with output.
async def __fetch(self, url, params, session):
async with session.get(url, params=params) as response:
output = await response.read()
return (output)
async def __bound_fetch(self, sem, url, params, session):
# Getter function with semaphore.
async with sem:
output = await self.__fetch(url, params, session)
return output
You can return the url (or whatever key identifies your request) together with the output.
Regarding using the data, I think you should read the response directly as JSON, especially since aiohttp can do this for you automatically.
async def __fetch(self, url, params, session):
async with session.get(url, params=params) as response:
try:
data = await response.json()
except ValueError as exc:
print(exc)
return None
return data
async def __bound_fetch(self, sem, url, params, session):
# Getter function with semaphore.
async with sem:
output = await self.__fetch(url, params, session)
return {"url": url, "data": data}
You did not post the get_responses function but I'm guessing something like this should work:
responses = self.get_responses(urls, self.get_office_token(office_name), chunk_size)
Responses will be a list of {"url": url, data: "data"} (data can be None for invalid responses); however with the code above one invalid request will not affect the others.
I have a streaming application that almost continuously takes the data given as input and sends an HTTP request using that value and does something with the returned value.
Obviously to speed things up I've used asyncio and aiohttp libraries in Python 3.7 to get the best performance, but it becomes hard to debug given how fast the data moves.
This is what my code looks like
'''
Gets the final requests
'''
async def apiRequest(info, url, session, reqType, post_data=''):
if reqType:
async with session.post(url, data = post_data) as response:
info['response'] = await response.text()
else:
async with session.get(url+post_data) as response:
info['response'] = await response.text()
logger.debug(info)
return info
'''
Loops through the batches and sends it for request
'''
async def main(data, listOfData):
tasks = []
async with ClientSession() as session:
for reqData in listOfData:
try:
task = asyncio.ensure_future(apiRequest(**reqData))
tasks.append(task)
except Exception as e:
print(e)
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
responses = await asyncio.gather(*tasks)
return responses #list of APIResponses
'''
Streams data in and prepares batches to send for requests
'''
async def Kconsumer(data, loop, batchsize=100):
consumer = AIOKafkaConsumer(**KafkaConfigs)
await consumer.start()
dataPoints = []
async for msg in consumer:
try:
sys.stdout.flush()
consumedMsg = loads(msg.value.decode('utf-8'))
if consumedMsg['tid']:
dataPoints.append(loads(msg.value.decode('utf-8')))
if len(dataPoints)==batchsize or time.time() - startTime>5:
'''
#1: The task below goes and sends HTTP GET requests in bulk using aiohttp
'''
task = asyncio.ensure_future(getRequests(data, dataPoints))
res = await asyncio.gather(*[task])
if task.done():
outputs = []
'''
#2: Does some ETL on the returned values
'''
ids = await asyncio.gather(*[doSomething(**{'tid':x['tid'],
'cid':x['cid'], 'tn':x['tn'],
'id':x['id'], 'ix':x['ix'],
'ac':x['ac'], 'output':to_dict(xmltodict.parse(x['response'],encoding='utf-8')),
'loop':loop, 'option':1}) for x in res[0]])
simplySaveDataIntoDataBase(id) # This is where I see some missing data in the database
dataPoints = []
except Exception as e:
logger.error(e)
logger.error(traceback.format_exc())
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
logger.error(str(exc_type) +' '+ str(fname) +' '+ str(exc_tb.tb_lineno))
if __name__ == '__main__':
loop = asyncio.get_event_loop()
asyncio.ensure_future(Kconsumer(data, loop, batchsize=100))
loop.run_forever()
Does the ensure_future need to be awaited ?
How does aiohttp handle requests that come a little later than the others? Shouldn't it hold the whole batch back instead of forgetting about it altoghter?
Does the ensure_future need to be awaited ?
Yes, and your code is doing that already. await asyncio.gather(*tasks) awaits the provided tasks and returns their results in the same order.
Note that await asyncio.gather(*[task]) doesn't make sense, because it is equivalent to await asyncio.gather(task), which is again equivalent to await task. In other words, when you need the result of getRequests(data, dataPoints), you can write res = await getRequests(data, dataPoints) without the ceremony of first calling ensure_future() and then calling gather().
In fact, you almost never need to call ensure_future yourself:
if you need to await multiple tasks, you can pass coroutine objects directly to gather, e.g. gather(coroutine1(), coroutine2()).
if you need to spawn a background task, you can call asyncio.create_task(coroutine(...))
How does aiohttp handle requests that come a little later than the others? Shouldn't it hold the whole batch back instead of forgetting about it altoghter?
If you use gather, all requests must finish before any of them return. (That is not aiohttp policy, it's how gather works.) If you need to implement a timeout, you can use asyncio.wait_for or similar.