Python requests too slow compared to Postman or cURL - python-3.x

I am trying to make a single API call using different approaches in Python3 but they are all incredibly slow if I compare with the same request in Postman and/or cURL.
This is what I am trying:
headers = {
"Authorization": "Bearer " + self.access_token,
"Content-Type": "application/json",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Accept": "*/*",
"User-Agent": "PostmanRuntime/7.28.2",
"Cache-Control": "no-cache"
}
session = requests.Session()
api_res = session.get(self.api_endpoint, headers=headers, timeout=self.req_timeout,)
When running this call, it gets stuck for a few minutes until I receive a response. If I use Postman for example, I get the result in 1 second or less.
I also tried using http.client, urllib3 but I still see a huge delay in the call.
I also tried debugging the call:
import http.client
http.client.HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
Debug response:
[DEBUG] Starting new HTTPS connection (1): ...:443
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 61
header: Connection: keep-alive
header: Date: ...
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Expires: -1
header: WWW-Authenticate: Bearer
header: Access-Control-Allow-Origin: *
header: P3P: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
header: Access-Control-Allow-Methods: GET, POST, OPTIONS, HEAD
header: X-XSS-Protection: 1; mode=block
header: Access-Control-Allow-Headers: accept, authorization, content-type, Cache-Control, P3P, GE-IGNORE-CACHE, Signature, fromMobile, ssoToken, fromAdmin, fromGameapp, fromGtv, fromGameapp, ManagerOrgUnitUserName, ManagerOrgUnitId, g-s-x, g-s-x-t, g-s-i-i, User-Agent, Referer, Origin, Access-Control-Allow-Headers
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Referrer-Policy: same-origin
header: Strict-Transport-Security: max-age=31536000; includeSubDomains
header: X-Cache: Error from cloudfront
header: Via: 1.1 ....cloudfront.net (CloudFront)
header: X-Amz-Cf-Pop: ORD52-C1
header: X-Amz-Cf-Id: ...
Any ideas about why is it so slow? What it doesn't happen when I replicate to Postman for example?
Even from Google is taking a lot of time:
requests.get("https://www.google.com")
I also realized that it's working with ipv4 and NOT with ipv6.
Thanks!

I found out that IPv6 was not working but IPv4 was. I had to force calls to IPv4 like this:
import socket
import requests.packages.urllib3.util.connection as urllib3_cn
urllib3_cn.allowed_gai_family = lambda: socket.AF_INET

Related

CouchDB, EventSource and OPTIONS method (405 Method Not Allowed)

I would like to implement Event Source feed of _changes for some documents in publicly readable database in CouchDB. Unfortunately, I get 405 error and:
Access to fetch at 'https://COUCHDOMAIN:6984/DBNAME/_changes?feed=eventsource&filter=_doc_ids&doc_ids=[%22DOCID%22]&since=now&include_docs=true&heartbeat=44000' from origin 'https://CLIENTDOMAIN' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.
I do not understand why I get Allow: GET,POST,HEAD in response while in CORS config, there is also OPTIONS... EventSource constructed with withCredentials: true/false, no difference...
I will appreciate any suggestion.
My CORS configuration:
[cors]
origins = https://CLIENTDOMAIN
headers = accept, authorization, content-type, origin, referer
methods = GET, PUT, POST, HEAD, DELETE, OPTIONS
credentials = true
Request Headers:
OPTIONS /DBNAME/_changes?feed=eventsource&filter=_doc_ids&doc_ids=[%22DOCID%22]&since=now&include_docs=true&heartbeat=44000 HTTP/1.1
Host: COUCHDOMAIN:6984
Connection: keep-alive
Accept: */*
Access-Control-Request-Method: GET
Access-Control-Request-Headers: cache-control
Origin: https://CLIENTDOMAIN
User-Agent: Mozilla/5.0 (Linux; Android 10; LM-V405) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Mobile Safari/537.36
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
Sec-Fetch-Dest: empty
Referer: https://CLIENTDOMAIN/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,pl-PL;q=0.8,pl;q=0.7,en-US;q=0.6
Response Headers:
HTTP/1.1 405 Method Not Allowed
Access-Control-Allow-Origin: https://CLIENTDOMAIN
Access-Control-Expose-Headers: content-type, cache-control, accept-ranges, etag, server, x-couch-request-id, x-couch-update-newrev, x-couchdb-body-time
Allow: GET,POST,HEAD
Cache-Control: must-revalidate
Content-Length: 69
Content-Type: application/json
Date: Fri, 11 Feb 2022 20:04:06 GMT
Server: CouchDB/3.2.1 (Erlang OTP/20)
X-Couch-Request-ID: ab68310597
X-CouchDB-Body-Time: 0
General (from chrome dev tools):
Request URL: https://COUCHDOMAIN:6984/DBNAME/_changes?feed=eventsource&filter=_doc_ids&doc_ids=[%22DOCID%22]&since=now&include_docs=true&heartbeat=44000
Request Method: OPTIONS
Status Code: 405 Method Not Allowed
Remote Address: COUCHIP:6984
Referrer Policy: strict-origin-when-cross-origin
Solution (inspired by: https://stackoverflow.com/a/50914749/2626200):
[cors]
origins = https://CLIENTDOMAIN
headers = accept, authorization, content-type, origin, referer, cache-control
...

Python POST request to retrieve base64 encode File

Im trying to POST request using Python to retreive a specific File. Since the URL is behind a server with authorized access theres no use posting it here
However the form data contains a field called base64 and lengthy which I cant figure out if its a form data value or base64 encoding of post request
Here are browser parameters
General:
Request URL: http://exampleapi.com/api/Document/Export
Request Method: POST
Status Code: 200 OK
Remote Address: XX.XXX.XXX.XX:XX
Referrer Policy: no-referrer-when-downgrade
Response Headers:
Access-Control-Allow-Origin: http://example.com
Cache-Control: no-cache
Content-Disposition: attachment; filename=location-downloads.xlsx
Content-Length: 7148
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Date: Tue, 23 Jul 2019 21:00:18 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Request Headers :
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 10162
Content-Type: application/x-www-form-urlencoded
Cookie: abcConnection=!UA7tkC3iZCmVNGRUyRpDWARVBWk/lY6SZvgxLlaygsQKk+vuwA1NxvhwE9ph4i+3NZlKeepIfuHhUvyQjl68fhhrT9ueqMx/3mBKUDcT
DNT: 1
Host: exampleapi.com
Origin: http://example.com
Referer: http://example.com/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
Form Data:
fileName: location-downloads.xlsx
contentType: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
base64: UEsDBAoAAAAAAAh4904AAAAAAAAAAAAAAAAJAAAAZG9jUHJvcHMvUEsDBAoAAAAIAAh490(shortened for simplicity)
Here is what I tried
url='http://example.com'
urllib3.disable_warnings()
headers = {
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0",
}
with requests.session() as s:
r=s.get(url,headers={"User-Agent":"Mozilla/5.0"},verify=False)
data=r.content
soup=BeautifulSoup(data,'html.parser')
form_data = {
"fileName":"location-downloads.xlsx",
"contentType":"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
}
r2=s.post('http://exampleapi.com/api/Document/Export',data=json.dumps(form_data,ensure_ascii=True).encode('utf-8'),headers=headers,verify=False)
print(r2.status_code)
Any idea how i should proceed. My status code also shows 500 for the post here

After I getting a set-cookie in response, is not saved and transmitted in requests

Basically after an auth, I setting a cookie, but apparently after page refresh on the cookie that was set by cloudflare is saved
And the cookie that I transmitted with set-cookie is not used in after set-cookie requests
# Response headers
HTTP/2.0 200 OK
date: Thu, 18 Jul 2019 10:03:25 GMT
content-type: application/json; charset=utf-8
content-length: 29
set-cookie: __cfduid=d578c7a5e4378dc1b1946964a08ebc4ec1563444205; expires=Fri, 17-Jul-20 10:03:25 GMT; path=/; domain=.doc.io; HttpOnly; Secure
set-cookie: __doc=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlIjoiNzQ2NTczNzQ0MDY2NjE3Mzc0NmQ2MTY5NmMyZTZkNzgiLCJwIjoiNTUzMjQ2NzM2NDQ3NTY2YjU4MzEzODMzNTU0NjUwNTU2ZjRiMzkzMTY3NDUzNDY5NDc3MzM3MzgzOTU5MzczMDUxNjk0ZjQxNjQ0OTM5Nzg0YjZiNzU1Njc3Nzk0NDc0NjE3NDMxNTE0NzcwMzE0YjQxNmY1MjU5MzM3YTZhNDU2NDJiNmU0ZTc0NGE3NTMyNTQ1ODc2NjI1YTczNDc1MTQ1Njc0MjVhNGQ0MTNkM2QiLCJkIjoiMzEzNTM2MzMzNDM0MzQzMjMwMzUzNTM2MzMiLCJpYXQiOjE1NjM0NDQyMDV9.go1jDpc2rBe5FjK2sKX4ybW4PhCPFq1xT1WIX-mSI84; Domain=.doc.io; Path=/; Expires=Thu, 18 Jul 2019 16:03:25 GMT; HttpOnly; Secure
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 4f83a02a3dc36455-FRA
X-Firefox-Spdy: h2
reply
.code(200)
.header('Access-Control-Allow-Origin', '*')
.header('Content-Type', 'application/json; charset=utf-8')
.setCookie('__doc', token, {
domain: '.doc.io',
path: '/',
secure: true,
httpOnly: true,
expires: new Date(new Date().setHours(new Date().getHours() + 6))})
.send({ 'success': 'Sign In success' })
All my websites are https
First I do POST request for an auth on /auth, and you could see response in response headers above and after I do GET on (trying to load page) from /page and get cookies, but with reply.log.info(request.cookies) I see only cookies from cloudflare. Surely I tried to refresh and go to address in different table, there just no any cookies, but from cloudflare.
# Request headers
Host: test.doc.io
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Cookie: __cfduid=d578c7a5e4378dc1b1946964a08ebc4ec1563444205
Upgrade-Insecure-Requests: 1
TE: Trailers
After I tested it with an actual page with js code for a XHR request it works fine. However cookie was only displayed, but not actually saved in the storage when I was sending request directly in Firefox's inspector. Lost a day to figure out that created in inspector cookies seems preventing from installing

Has anyone an idea about IBM OpenWhisk and Cors settings?

i have the following problem. When run my openWhisk function through the API gateway all is fine. Also calling it as a cross domain request works well.
But i need to call it with credentials and it seems there is no option to do this.
What i tried:
just enabling the API CORS setup without any headers in the function itself
enabling cors in the API und having CORS headers in the function as well ->the relevant headers (access-origin) in the function get overwritten!
disabling CORS in the API and having CORS headers in the function -> the relevant headers (access-origin) of the function get deleted
This is the code that normally should work:
return {
headers: {
'Access-Control-Allow-Headers': '*',
'Access-Control-Allow-Origin': domain,
'Access-Control-Allow-Credentials': 'true',
'Content-Type': 'text/xml'
},
body: xml
}
Would be great if anyone has an idea because the support is not answering my ticket for many days now.
Thanks and best, André
Using your sample:
> cat t.js
function main() {
return {
headers: {
'Access-Control-Allow-Headers': '*',
'Access-Control-Allow-Origin': 'domain',
'Access-Control-Allow-Credentials': 'true',
'Content-Type': 'text/xml'
},
body: "<hi></hi>"
}
}
Against Bluemix:
> wsk action update t t.js -a web-custom-options true --web true
ok: updated action t
And curling the web action
> curl -v https://openwhisk.ng.bluemix.net/api/v1/web/myspace/default/t.http
< HTTP/1.1 200 OK
< X-Backside-Transport: OK OK
< Connection: Keep-Alive
< Transfer-Encoding: chunked
< Server: nginx/1.11.13
< Date: Tue, 18 Jul 2017 14:00:40 GMT
< Content-Type: text/xml
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Origin: domain
< Access-Control-Allow-Credentials: true
<hi></hi>
Without the annotation:
> wsk action update t t.js -a web-custom-options false --web true
Repeating the curl will show these headers:
< Access-Control-Allow-Origin: *,domain
< Access-Control-Allow-Methods: OPTIONS, GET, DELETE, POST, PUT, HEAD, PATCH
< Access-Control-Allow-Headers: Authorization, Content-Type,*
< Access-Control-Allow-Credentials: true

github api v3 update reference returns a 422 "Object does not exist"

For the context I'm trying to update a file through the GitHub API.
Everything was fine until I tried to update the reference.
According to the doc, below are the requests I forged and their returns.
If anyone has an idea, I did find nothing to make it work.
$ curl -i -XPATCH -d '{"sha": "69d0a253406585d8faf616ce3ae0ff2453b346d7"}' -H "Authorization: token AUTH-TOKEN" https://api.github.com/repos/Trax-air/TraxIT/git/refs/heads/ci-migrate-quay
HTTP/1.1 422 Unprocessable Entity
Server: GitHub.com
Date: Wed, 18 Nov 2015 14:08:49 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 128
Status: 422 Unprocessable Entity
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4948
X-RateLimit-Reset: 1447856141
X-OAuth-Scopes: gist, read:repo_hook, repo, user
X-Accepted-OAuth-Scopes:
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-GitHub-Request-Id: 4EC2914C:94AC:15486DB6:564C8671
{
"message": "Object does not exist",
"documentation_url": "https://developer.github.com/v3/git/refs/#update-a-reference"
}
I tried to update the reference by itself, it worked:
$ curl -i -XPATCH -d '{"sha": "694973310d80edfe9ca08bd2fd5a06a6407b08ad"}' -H "Authorization: token AUTH-TOKEN" https://api.github.com/repos/Trax-air/TraxIT/git/refs/heads/ci-migrate-quay
HTTP/1.1 200 OK
Server: GitHub.com
Date: Wed, 18 Nov 2015 14:10:20 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 337
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4947
X-RateLimit-Reset: 1447856141
Cache-Control: private, max-age=60, s-maxage=60
ETag: "25641a46e3d517196995aec80669dcd2"
X-OAuth-Scopes: gist, read:repo_hook, repo, user
X-Accepted-OAuth-Scopes:
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: c6c65e5196703428e7641f7d1e9bc353
X-GitHub-Request-Id: 4EC2914C:94AB:F33F280:564C86CC
{
"ref": "refs/heads/ci-migrate-quay",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/refs/heads/ci-migrate-quay",
"object": {
"sha": "694973310d80edfe9ca08bd2fd5a06a6407b08ad",
"type": "commit",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/commits/694973310d80edfe9ca08bd2fd5a06a6407b08ad"
}
}
I then tried to confirm my commit exist:
$curl -i -XGET -H "Authorization: token AUTH-TOKEN" https://api.github.com/repos/Trax-air/TraxIT/git/commits/69d0a253406585d8faf616ce3ae0ff2453b346d7
HTTP/1.1 200 OK
Server: GitHub.com
Date: Wed, 18 Nov 2015 14:03:29 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1028
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4950
X-RateLimit-Reset: 1447856141
Cache-Control: private, max-age=60, s-maxage=60
Last-Modified: Wed, 18 Nov 2015 11:58:58 GMT
ETag: "4823502d472e3b3fe873841fcd60d3c6"
X-OAuth-Scopes: gist, read:repo_hook, repo, user
X-Accepted-OAuth-Scopes:
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: 8a5c38021a5cd7cef7b8f49a296fee40
X-GitHub-Request-Id: 4EC2914C:94AA:AE467E1:564C8530
{
"sha": "69d0a253406585d8faf616ce3ae0ff2453b346d7",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/commits/69d0a253406585d8faf616ce3ae0ff2453b346d7",
"html_url": "https://github.com/Trax-air/TraxIT/commit/69d0a253406585d8faf616ce3ae0ff2453b346d7",
"author": {
"name": "traxbot",
"email": "traxbot#trax-air.com",
"date": "2015-11-18T11:58:58Z"
},
"committer": {
"name": "traxbot",
"email": "traxbot#trax-air.com",
"date": "2015-11-18T11:58:58Z"
},
"tree": {
"sha": "ca47cb13f520913e643b15e6d0776f38ba577091",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/trees/ca47cb13f520913e643b15e6d0776f38ba577091"
},
"message": "Updated api_gateway to 0.15",
"parents": [
{
"sha": "694973310d80edfe9ca08bd2fd5a06a6407b08ad",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/commits/694973310d80edfe9ca08bd2fd5a06a6407b08ad",
"html_url": "https://github.com/Trax-air/TraxIT/commit/694973310d80edfe9ca08bd2fd5a06a6407b08ad"
}
]
}
This may be due to caching.
I asked to Github support and here is their answer:
Thanks for reaching out. The commit in question
(69d0a253406585d8faf616ce3ae0ff2453b346d7) doesn't exist in that repository,
so you're not allowed to update the branch to point to it.
As far as I can tell, it did exist in the repository at some point, but was pruned
because it was no longer reachable. I think the API was telling you that it still exists
in the repository due to caching.
I just cleared our caches and I think you should see that it's no longer available
if you try to fetch that commit. I'm sorry for the confusion about that --
I'll ask the team to investigate why this caching problem happened.
This solved it for me:
'{"sha": "new_sha", "force": true }'

Resources