I wonder when i request a url for example :
and inspect with dev tool i can see a custom header response value set by the application ,
x-powered-by: streamA
cache-control: public, max-age=1800
content-encoding: gzip
content-type: text/html; charset=utf-8
date: Thu, 28 Jul 2022 17:43:47 GMT
expect-ct: enforce; max-age=86400; report-uri=https://3697d3b054cce6a97c9c759c82571b41.report-uri.com/r/d/ct/enforce
feature-policy: autoplay 'none'; encrypted-media 'none'; microphone 'none'; midi 'none'; payment 'none'; vr 'none'; ambient-light-sensor 'none'; magnetometer 'none'; picture-in-picture 'none'; sync-xhr 'none'; usb 'none';
referrer-policy: no-referrer-when-downgrade
server: istio-envoy
vary: Accept-Encoding
via: 1.1 df153902fc47e450893ee30df220e710.cloudfront.net (CloudFront)
x-amz-cf-id: BMXC92lZlNdehRGC3hc_b9_ANQc8-M8aKFaaz8pwLx3HGpc2Ls5tHw==
x-amz-cf-pop: DUB56-P1
x-cache: RefreshHit from cloudfront
x-content-type-options: nosniff
x-envoy-upstream-service-time: 337
x-pf-trace-id: 8589755742930307890
**x-powered-by: StreamA**
x-xss-protection: 1; mode=block
when i curl the same url in the output i dont see the customer header response value ,
i used curl --head http://urlhere
what did i missed here in curl to miss the custom header
Related
I am trying to make a single API call using different approaches in Python3 but they are all incredibly slow if I compare with the same request in Postman and/or cURL.
This is what I am trying:
headers = {
"Authorization": "Bearer " + self.access_token,
"Content-Type": "application/json",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Accept": "*/*",
"User-Agent": "PostmanRuntime/7.28.2",
"Cache-Control": "no-cache"
}
session = requests.Session()
api_res = session.get(self.api_endpoint, headers=headers, timeout=self.req_timeout,)
When running this call, it gets stuck for a few minutes until I receive a response. If I use Postman for example, I get the result in 1 second or less.
I also tried using http.client, urllib3 but I still see a huge delay in the call.
I also tried debugging the call:
import http.client
http.client.HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
Debug response:
[DEBUG] Starting new HTTPS connection (1): ...:443
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 61
header: Connection: keep-alive
header: Date: ...
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Expires: -1
header: WWW-Authenticate: Bearer
header: Access-Control-Allow-Origin: *
header: P3P: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
header: Access-Control-Allow-Methods: GET, POST, OPTIONS, HEAD
header: X-XSS-Protection: 1; mode=block
header: Access-Control-Allow-Headers: accept, authorization, content-type, Cache-Control, P3P, GE-IGNORE-CACHE, Signature, fromMobile, ssoToken, fromAdmin, fromGameapp, fromGtv, fromGameapp, ManagerOrgUnitUserName, ManagerOrgUnitId, g-s-x, g-s-x-t, g-s-i-i, User-Agent, Referer, Origin, Access-Control-Allow-Headers
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Referrer-Policy: same-origin
header: Strict-Transport-Security: max-age=31536000; includeSubDomains
header: X-Cache: Error from cloudfront
header: Via: 1.1 ....cloudfront.net (CloudFront)
header: X-Amz-Cf-Pop: ORD52-C1
header: X-Amz-Cf-Id: ...
Any ideas about why is it so slow? What it doesn't happen when I replicate to Postman for example?
Even from Google is taking a lot of time:
requests.get("https://www.google.com")
I also realized that it's working with ipv4 and NOT with ipv6.
Thanks!
I found out that IPv6 was not working but IPv4 was. I had to force calls to IPv4 like this:
import socket
import requests.packages.urllib3.util.connection as urllib3_cn
urllib3_cn.allowed_gai_family = lambda: socket.AF_INET
I am trying to use Electron Updater with a GitLab Private Repository.
Main Electron File (partial):
autoUpdater.requestHeaders = { 'PRIVATE-TOKEN': process.env.VUE_APP_GITLABSECRET }
autoUpdater.autoDownload = true
autoUpdater.setFeedURL({
provider: 'generic',
url: 'https://gitlab.com/SmellydogCoding/mchd-electronic-field-guide/-/jobs/artifacts/master/raw/dist_electron?job=build'
})
autoUpdater.on('checking-for-update', function () {
console.log('Checking for update...')
})
When I start the app I get this error message:
Error: Error: Cannot parse update info from latest.yml in the latest release artifacts (https://gitlab.com/SmellydogCoding/mchd-electronic-field-guide/-/jobs/artifacts/master/raw/dist_electron/latest.yml?job=build): YAMLException: end of the stream or a document separator is expected at line 3, column 17:
<head prefix="og: http://ogp.me/ns#">
What is happening is that the server is responding with a string of HTML, which is the Gitlab login page.
If I curl
--header 'PRIVATE-TOKEN': 'mygitlabprivatetoken' https://gitlab.com/SmellydogCoding/mchd-electronic-field-guide/-/jobs/artifacts/master/raw/dist_electron/latest.yml?job=build
The server returns:
Header
HTTP/1.1 302 Found
Server: nginx
Date: Tue, 19 Mar 2019 17:57:21 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 98
Cache-Control: no-cache
Location: https://gitlab.com/users/sign_in
Set-Cookie: _gitlab_session=da00cbc69f2d50ea4192f4e3002f84a9; path=/; secure; HttpOnly
X-Request-Id: dGkxtbboHy7
X-Runtime: 0.049129
Strict-Transport-Security: max-age=31536000
Content-Security-Policy: object-src 'none'; worker-src https://assets.gitlab-static.net https://gl-canary.freetls.fastly.net https://gitlab.com blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://assets.gitlab-static.net https://gl-canary.freetls.fastly.net https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://www.gstatic.com/recaptcha/ https://apis.google.com; style-src 'self' 'unsafe-inline' https://assets.gitlab-static.net https://gl-canary.freetls.fastly.net; img-src * data: blob:; frame-src 'self' https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://*.codesandbox.io; frame-ancestors 'self'; connect-src 'self' https://assets.gitlab-static.net https://gl-canary.freetls.fastly.net wss://gitlab.com https://sentry.gitlab.net https://customers.gitlab.com https://snowplow.trx.gitlab.net
Body
<html><body>You are being redirected.</body></html>
It seems like i'm not authenticating properly. I'm really not sure what i'm doing incorrectly.
I'm trying to provide the correlation id for this issue but I'm not sure how to obtain it. I assume it should be in response, is that correct? Is it the client-request-id maybe?
HTTP Response
HTTP/1.1 403 Forbidden
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.5
ocp-aad-diagnostics-server-name: CGa9FFPZASZM+usTGvTzaOZ4zSMxhpbWh2mO9+eJZWs=
request-id: 6ddd7a86-02c9-4dcf-9932-b17afc976107
client-request-id: 50478d75-542d-4e0e-96d9-de913d632471
x-ms-dirapi-data-contract-version: 1.6
ocp-aad-session-key: some-really-long-value
X-Content-Type-Options: nosniff
DataServiceVersion: 3.0;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Access-Control-Allow-Origin: *
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Duration: 782602
X-Powered-By: ASP.NET
Date: Wed, 09 Aug 2017 14:22:21 GMT
Content-Length: 139
{"odata.error":{"code":"Authorization_RequestDenied","message":{"lang":"en","value":"Insufficient privileges to complete the operation."}}}
Token Request Response
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.5
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
client-request-id: 8c500700-b75c-403a-a2c9-c6dfa27049d3
x-ms-request-id: 75627b63-fbd6-4fa8-95b5-b6a325d60900
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
Set-Cookie: esctx=some-value; domain=.login.microsoftonline.com; path=/; secure; HttpOnly
Set-Cookie: x-ms-gateway-slice=006; path=/; secure; HttpOnly
Set-Cookie: stsservicecookie=ests; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
Date: Wed, 09 Aug 2017 18:33:33 GMT
Content-Length: 1437
{"token_type":"Bearer","expires_in":"3599","ext_expires_in":"262800","expires_on":"1502307214","not_before":"1502303314","resource":"https://graph.windows.net/","access_token":"some-token-value"}
The correlation ID (also called client-request-id) can be found in interactions with the Azure AD service.
The request immediately before this when you were asking the Azure AD endpoints for tokens will include the ID in the response.
https://login.microsoftonline.com/...
https://login.windows.net/...
I'm trying to setup the caching of my ExpressApp and its rendered Jade files with Nginx.
When I have a look at the pm2 logs of my application while running the google Pagespeed Insight test, I see that the Images and HTML are still served by the app instead of the Nginx proxy.
Two Questions:
How do I have to change my configuration to have the desired effect?
How do I effectively validate that the Setup/Caching is working?
Thanks!
My current Nginx Configuration:
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=backcache:8m max_size=50m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
proxy_cache_valid 200 302 10m;
proxy_cache_valid 404 1m;
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
expires 1w;
proxy_cache backcache;
#proxy_cache_bypass $http_cache_control;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass http://localhost:3030;
}
}
Headers:
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: ###
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server certificate: AddTrust External CA Root
> GET / HTTP/1.1
> Host: ###
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.10.0 (Ubuntu)
< Date: Thu, 24 Nov 2016 13:15:37 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 66761
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
< ETag: W/"104c9-8+4lo531tUk9k3SJq4sPHg"
< Expires: Thu, 01 Dec 2016 13:15:37 GMT
< Cache-Control: max-age=604800
< X-Proxy-Cache: MISS
Headers of two consecutive Requests to an Image
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:02 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:02 GMT
X-Proxy-Cache: MISS
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:08 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:08 GMT
X-Proxy-Cache: MISS
The headers you show as the request headers are most probably not the original ones, because it says "Provisional headers shown".
The actual request headers probably contain the "cache-control" header and your nginx configuration uses that header to bypass the cache, as indicated by the 'X-Proxy-Cache:BYPASS' response header.
If there is no reason to keep it, you should remove the proxy_cache_bypass-directive from your config.
For the context I'm trying to update a file through the GitHub API.
Everything was fine until I tried to update the reference.
According to the doc, below are the requests I forged and their returns.
If anyone has an idea, I did find nothing to make it work.
$ curl -i -XPATCH -d '{"sha": "69d0a253406585d8faf616ce3ae0ff2453b346d7"}' -H "Authorization: token AUTH-TOKEN" https://api.github.com/repos/Trax-air/TraxIT/git/refs/heads/ci-migrate-quay
HTTP/1.1 422 Unprocessable Entity
Server: GitHub.com
Date: Wed, 18 Nov 2015 14:08:49 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 128
Status: 422 Unprocessable Entity
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4948
X-RateLimit-Reset: 1447856141
X-OAuth-Scopes: gist, read:repo_hook, repo, user
X-Accepted-OAuth-Scopes:
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-GitHub-Request-Id: 4EC2914C:94AC:15486DB6:564C8671
{
"message": "Object does not exist",
"documentation_url": "https://developer.github.com/v3/git/refs/#update-a-reference"
}
I tried to update the reference by itself, it worked:
$ curl -i -XPATCH -d '{"sha": "694973310d80edfe9ca08bd2fd5a06a6407b08ad"}' -H "Authorization: token AUTH-TOKEN" https://api.github.com/repos/Trax-air/TraxIT/git/refs/heads/ci-migrate-quay
HTTP/1.1 200 OK
Server: GitHub.com
Date: Wed, 18 Nov 2015 14:10:20 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 337
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4947
X-RateLimit-Reset: 1447856141
Cache-Control: private, max-age=60, s-maxage=60
ETag: "25641a46e3d517196995aec80669dcd2"
X-OAuth-Scopes: gist, read:repo_hook, repo, user
X-Accepted-OAuth-Scopes:
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: c6c65e5196703428e7641f7d1e9bc353
X-GitHub-Request-Id: 4EC2914C:94AB:F33F280:564C86CC
{
"ref": "refs/heads/ci-migrate-quay",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/refs/heads/ci-migrate-quay",
"object": {
"sha": "694973310d80edfe9ca08bd2fd5a06a6407b08ad",
"type": "commit",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/commits/694973310d80edfe9ca08bd2fd5a06a6407b08ad"
}
}
I then tried to confirm my commit exist:
$curl -i -XGET -H "Authorization: token AUTH-TOKEN" https://api.github.com/repos/Trax-air/TraxIT/git/commits/69d0a253406585d8faf616ce3ae0ff2453b346d7
HTTP/1.1 200 OK
Server: GitHub.com
Date: Wed, 18 Nov 2015 14:03:29 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1028
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4950
X-RateLimit-Reset: 1447856141
Cache-Control: private, max-age=60, s-maxage=60
Last-Modified: Wed, 18 Nov 2015 11:58:58 GMT
ETag: "4823502d472e3b3fe873841fcd60d3c6"
X-OAuth-Scopes: gist, read:repo_hook, repo, user
X-Accepted-OAuth-Scopes:
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
X-GitHub-Media-Type: github.v3
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: 8a5c38021a5cd7cef7b8f49a296fee40
X-GitHub-Request-Id: 4EC2914C:94AA:AE467E1:564C8530
{
"sha": "69d0a253406585d8faf616ce3ae0ff2453b346d7",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/commits/69d0a253406585d8faf616ce3ae0ff2453b346d7",
"html_url": "https://github.com/Trax-air/TraxIT/commit/69d0a253406585d8faf616ce3ae0ff2453b346d7",
"author": {
"name": "traxbot",
"email": "traxbot#trax-air.com",
"date": "2015-11-18T11:58:58Z"
},
"committer": {
"name": "traxbot",
"email": "traxbot#trax-air.com",
"date": "2015-11-18T11:58:58Z"
},
"tree": {
"sha": "ca47cb13f520913e643b15e6d0776f38ba577091",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/trees/ca47cb13f520913e643b15e6d0776f38ba577091"
},
"message": "Updated api_gateway to 0.15",
"parents": [
{
"sha": "694973310d80edfe9ca08bd2fd5a06a6407b08ad",
"url": "https://api.github.com/repos/Trax-air/TraxIT/git/commits/694973310d80edfe9ca08bd2fd5a06a6407b08ad",
"html_url": "https://github.com/Trax-air/TraxIT/commit/694973310d80edfe9ca08bd2fd5a06a6407b08ad"
}
]
}
This may be due to caching.
I asked to Github support and here is their answer:
Thanks for reaching out. The commit in question
(69d0a253406585d8faf616ce3ae0ff2453b346d7) doesn't exist in that repository,
so you're not allowed to update the branch to point to it.
As far as I can tell, it did exist in the repository at some point, but was pruned
because it was no longer reachable. I think the API was telling you that it still exists
in the repository due to caching.
I just cleared our caches and I think you should see that it's no longer available
if you try to fetch that commit. I'm sorry for the confusion about that --
I'll ask the team to investigate why this caching problem happened.
This solved it for me:
'{"sha": "new_sha", "force": true }'