aiohttp response.json raises getaddrinfo failed error if after response.status but works ok if before - python-3.x

---UPDATED 10/20/2022---
After some further digging, it turns out the problem described below has nothing to do with aiohttp. The issue is that the response (which is a FHIR resource bundle) contains paginated data. And the URL that the server was returning for the pagination links was invalid (it has the hshweb domain referenced in my question below; my guess is it referred to some internal address that's not exposed outside their firewall). My sample code below shows fetching the initial response, what's not shown is that there is additional processing to get subsequent pages. And because the pagination links were invalid, it was those subsequent page fetches that were failing.
---ORIGINAL POST---
Background
I've been using aiohttp to fetch JSON data from APIs of different companies; my code has been running fine for over a year now.
In the code snippet below, I query the endpoint, and if the status is OK I then get the JSON content using response.json
async with await session.get(url, headers=self.headers) as response:
logger.info(f'Received response to {url}',
extra={**self.log_dict, 'httpstatus': response.status}
)
if response.status in [200, 206]:
try:
response_json = await response.json(content_type=None)
break
except json.decoder.JSONDecodeError:
response_json = None
logger.warning((f'Invalid JSON querying {url}; '
f'status code {response.status}. Skipping.'),
extra=self.log_dict
)
break
This has been working as expected so far.
Problem
With the latest company whose API I'm attempting to integrate with, I'm hitting API https://api.mvphealthcare.com.... and am getting a 200 response status, however when attempting to then get the JSON payload with response.json() I get the following error:
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host hshweb.mvphealthcare.com:443 ssl:default [getaddrinfo failed]
The identical GET request from Postman works exactly as expected.
Note that the URL in this error message (hshweb.mvphealthcare.com) is different from the original that I queried (api.mvphealthcare.com).
Attempting to ping hshweb.mvphealthcare.com from the command line results in Ping request could not find host hshweb.mvphealthcare.com. Please check the name and try again. so the getaddrinfo failed error itself sort of makes sense. What's unclear is why aiohttp is attempting to hit that other address.
While not exactly the same issue, this SO question suggested that there are instances where using response.json() after response.ok could be problematic. So I temporarily modified my code to call await response.json() before the call to response.status and was able to get the response! As the SO answer suggested the issue is fixed in aiohttp 3.8, I upgraded my library (was previously using 3.7.4). However, that did not resolve the problem if calling response.json() was after response.status.
Questions
Why does calling reponse.json() work if it is before response.status, but not after?
What's going on behind the scenes that's causing the URL to change (and if hshweb.mvphealthcare.com is not resolvable anyway, why does it work if response.json() is called before response.status)
What do I need to do to make this work?

Related

(Asana) XML is invalid: Headers are not supported on this surface

I am pretty sure I am using exactly the same code in the following two cases, as I tested it on Postman first.
The first time I tried to create a task from Postman, it has no issue.
The next time I tried to do the same from my NodeJS server, and it shows this error. I tried the client SDK package and manually calling through Axios, both are not working and throwing the same error.
Is the Asana API not callable from NodeJS server?
It turns out the "Headers" does not indicate the Headers sent with the request, but the "Headers" HTML Element inside the rich text field. Ensure there's no h1-h6 element in your html_notes can solve this issue.

When I use pycurl to execute a curl command, I get error 3 "illegal characters found in URL" but when pasting said URL in Chome, it can be resolved

Good day. I'm writing a python program that requests some posts from my Facebook page. To do so, Facebook offers a tool that they call "Graph API Explorer". Using something similar to a GET request, I can get anything that I want (granted that I have access and a valid token). I've come up with my own solution for the Graph API Explorer and that is generating my URLs. After generating a URL, I use pycurl to get a JSON object from Facebook that contains all of my data.
When I use pycurl, I get the following error:
pycurl.error: (3, 'Illegal characters found in URL')
but when printing said URL and pasting it to a browser, I got a valid response.
URL: https://graph.facebook.com/v7.0/me?fields=posts%7Bmessage%2Cfrom%7D&access_token=<and my access token which is valid>
my code looks like this:
def get_posts_curl(nodes=['posts'], fields=[['message', 'from']], token_file='Facebook/token.txt'):
curl = pycurl.Curl()
response = BytesIO()
token = get_token_from_file(token_file)
# constructing request.
url = parse_facebook_url_request(nodes, fields, token)
url = convert_to_curl(url)
print("---URL---: " + url)
# curl session and settings.
curl.setopt(curl.CAINFO, certifi.where())
curl.setopt(curl.URL, url)
curl.setopt(curl.WRITEDATA, response)
curl.perform()
curl.close()
return response.getvalue().decode('utf-8')
The error pops up at curl.perform()
Some info that might be relevant:
All was working great a while ago. After transferring my program from my workstation (that is running Windows 10) to my server (Ubuntu 18.04 Server) still, all was working fine and I placed that project to the side. Only now that error pops up and I haven't touched the project in a while.
It seems that the token is causing the issue. I've tried about 100 tokens and some cause the problem and some don't. Also, a fix that solved it all was using urllib3.unquote
from urllib.parse import unquote
...
url = unquote(url)

Python requests.put() is giving error: Invalid API call, Wrong method or URL

I have been trying to do a put request via BambooHR api to add a time off request. But it gives 404 and the header gives "Invalid API call, Wrong method or URL"
I couldn't understand what's wrong with my code. Add a time off request from Bamboo HR documentation is /api/gateway.php/{company}/v1/employees/{employee id}/time_off/request/ Sample: PUT /api/gateway.php/test/v1/employees/1/time_off/request/
import requests
url = 'https://api.bamboohr.com/api/gateway.php/johnsnowlabs/v1/employees/96/time_off/requests/?status=requested'
response1= requests.put(url, auth=('68d4165c9262fcf2302745a6d791b23dsfsd4107','John11')
print(response1.status_code)
it gives 404 and when response1.headers then the error message says Invalid API call, Wrong method or URL
I tried different combination of inputs in url but none seems to be working. However, all the GET requests are working, it's only the PUT one is not working.

Unicode-objects must be encoded before hashing when requesting data using Flask-OAuth

I'm integrating Google's login with a Flask site using Flask-OAuth.
Everything is working fine. I can authorise the login and get a token back etc without any difficulties. But when I use Flask-OAuth's get method to request the logged in user's email address I get an error saying:
TypeError: Unicode-objects must be encoded before hashing
I'm using Python3 and this has the smell of a Python version issue but I can't figure out what I'd need to change.
The code I'm using is this:
def get_additional_data(self):
access_token = session.get('oauth_token')
headers = {'Authorization': 'OAuth ' + access_token[0]}
return self.service.get(
'https://www.googleapis.com/oauth2/v1/userinfo', None,
headers=headers)
I'm not sure what I can encode in that request. Even if I don't pass the headers I get the same error (rather than an invalid request or something like that).
I've run 2to3 on oauth2/__init__.py and the tweaks is suggests are very minor and shouldn't prevent the code from running in Python 3. Also, everything else OAuth2 related is working.
The bad news is that the solution to this problem is switching to Flask-OAuthlib.
The good news is it required very few changes from Flask-OAuth to get it working.

How to send POST variables with Nipple on NodeJS

I am trying to use nipple to post to an url within my nodejs application, which itself is running on hapi.js
The documentation essentially doesn't seem to spell it out.
(https://www.npmjs.com/package/nipple)
I tried passing it as payload inside options but that, while not returning an error, returns a 400. Can someone provide a correct example doing a post using nipple?
Essentially, I have two variables that I need to send - let's call the var1 and var2.
Thanks!
That link says that the project has been renamed to wreck. On wreck's github, several of the tests are for a post requests, including this one:
https://github.com/hapijs/wreck/blob/master/test/index.js#L68
If you are still scratching your head, you could also try using curl or postman to sanity check your URL, regardless of any nipple/wreck errors. If that also gives you a 400, nipple/wreck may not be the culprit.

Resources