When i use the curl command all the headers are correct but files are not caching as
CF-Cache-Status: MISS
is shown by curl command .
> < HTTP/1.1 200 OK < Date: Sat, 02 Jun 2018 05:57:28 GMT <
> Content-Type: image/png < Content-Length: 2293 < Connection:
> keep-alive < Set-Cookie:
> __cfduid=d3cf179afd212660124585350d19e88c81527919048; expires=Sun, 02-Jun-19 05:57:28 GMT; path=/; domain=.mayan.cf; HttpOnly <
> X-Powered-By: Express < Accept-Ranges: bytes < Cache-Control: public,
> max-age=86400 < Last-Modified: Fri, 01 Jun 2018 15:39:06 GMT < ETag:
> W/"8f5-163bc009a0c" < CF-Cache-Status: MISS < Expires: Sun, 03 Jun
> 2018 05:57:28 GMT < Server: cloudflare < CF-RAY: 4247aec4300930f6-SIN
> < { [1905 bytes data]
> * Connection #0 to host mayan.cf left intact
Cloudflare uses some heuristics to try to guess, based on the request alone, whether the content is static (cacheable) or dynamic. If it guesses wrong at request time, it won't cache the response. You can disable this heuristic by setting the Page Rule "Cache Level" to "Cache Everything" -- then Cloudflare will assume that the response will always be cacheable.
Try setting that page rule and see if it helps.
If the Cache-Control header is set to "private", "no-store",
"no-cache", or "max-age=0", or if there is a cookie in the response,
then Cloudflare will not cache the resource.
https://support.cloudflare.com/hc/en-us/articles/202775670-How-Do-I-Tell-Cloudflare-What-to-Cache-
Related
I am trying to download some playlists off soundcloud and found a site that does this for you. Of course if the playlist is long, then it's super tedious to click each link to download. So I saved the HTML of the page and have parsed out the links. The idea is to use urllib or requests to download the files.
Here's my code:
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
track_url = 'https://scdownloader.io/download?track=zandex-hazerback-erox-stroke-bth-release&token=be1bc7997695495f756312886f566110'
track_name = 'BANG_THE_HOUSE___zandex-hazerback-erox-stroke-bth-release.mp3'
output_file = '/Users/ms/Desktop/playlist/{}'.format(track_name)
urllib.request.urlretrieve(track_url, output_file)
When I run the above code, it does save the file, but it arrives as a 1 byte file only.
I've tried other permutations using requests but basically either it doesn't work, downloads and saves a zero byte file, or does work to download and save a 1 byte file... just can't get the whole thing!
Also note, I have to send headers b/c otherwise I get a 403 error.
Any help is greatly appreciated!
Thank you!
EDIT:
Per the comments below, here's what the urlretrieve http response is:
Date: Fri, 15 Mar 2019 23:52:44 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
Set-Cookie: __cfduid=dcc5f95391fac83973cc77648c0e8c0391552693964; expires=Sat, 14-Mar-20 23:52:44 GMT; path=/; domain=.scdownloader.io; HttpOnly; Secure
X-Powered-By: PHP/5.6.36
Set-Cookie: PHPSESSID=fsnrrrtpnrav3vq5u2t9vfvrp7; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding,User-Agent
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4b82671d38067790-LAX
I have two Redhat Linux servers, say 'original' and 'demo'. Both have apache running on them. On original server, I have a link 'http://original.com/abc' on a page which I want redirect it to http://demo.com.
What I have done is-
Open httpd.conf on original server and added following lines and restarted apache service:
ProxyPass /abc http://demo.com/
ProxyPassReverse /abc http://demo.com/
But when I am trying to access http://original.com/abc, it is redirecting me to the original server i.e. on http://original.com (not on http://demo.com)
I have tried to find the solutions on various sites but unable to find out what the problem is.
Edit:
Output of curl -v http://original.com/abc:
curl -v http://original.com/abc
* About to connect() to original.com port 80 (#0)
* Trying 10.100.100.100...
* Connected to original.com (10.100.100.100) port 80 (#0)
> GET /demo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: original.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Tue, 24 Apr 2018 06:30:23 GMT
< Server: Apache-Coyote/1.1
< X-Frame-Options: SAMEORIGIN
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Location: login.action
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Set-Cookie: JSESSIONID=0BAA246C7F2505D2F5A0335CB0542CAA; Path=/; HttpOnly
<
* Connection #0 to host original.com left intact
Output of curl -v http://demo.com/ :
#curl -v http://demo.com
* About to connect() to 10.100.100.101 port 80 (#0)
* Trying 10.100.100.101...
* Connected to 10.100.100.101 (10.100.100.101) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: demo.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: Apache-Coyote/1.1
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: JSESSIONID=FED19458459131F456D638EC57278C2A; Path=/; HttpOnly
< Location: login.action
< Content-Type: text/html
< Content-Length: 0
< Date: Tue, 24 Apr 2018 06:31:47 GMT
<
* Connection #0 to host demo.com left intact
The textual order of ProxyPass statements is relevant. The first matched entry stops further matching.
ProxyPass / http://some.tomcat:8080/
ProxyPassReverse / http://some.tomcat:8080/
...
ProxyPass /abc http://demo.com/
ProxyPassReverse /abc http://demo.com/
Here, demo.com would be fed nothing. The some.tomcat would be fed both / and /abc
I'm trying to provide the correlation id for this issue but I'm not sure how to obtain it. I assume it should be in response, is that correct? Is it the client-request-id maybe?
HTTP Response
HTTP/1.1 403 Forbidden
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.5
ocp-aad-diagnostics-server-name: CGa9FFPZASZM+usTGvTzaOZ4zSMxhpbWh2mO9+eJZWs=
request-id: 6ddd7a86-02c9-4dcf-9932-b17afc976107
client-request-id: 50478d75-542d-4e0e-96d9-de913d632471
x-ms-dirapi-data-contract-version: 1.6
ocp-aad-session-key: some-really-long-value
X-Content-Type-Options: nosniff
DataServiceVersion: 3.0;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Access-Control-Allow-Origin: *
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Duration: 782602
X-Powered-By: ASP.NET
Date: Wed, 09 Aug 2017 14:22:21 GMT
Content-Length: 139
{"odata.error":{"code":"Authorization_RequestDenied","message":{"lang":"en","value":"Insufficient privileges to complete the operation."}}}
Token Request Response
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.5
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
client-request-id: 8c500700-b75c-403a-a2c9-c6dfa27049d3
x-ms-request-id: 75627b63-fbd6-4fa8-95b5-b6a325d60900
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
Set-Cookie: esctx=some-value; domain=.login.microsoftonline.com; path=/; secure; HttpOnly
Set-Cookie: x-ms-gateway-slice=006; path=/; secure; HttpOnly
Set-Cookie: stsservicecookie=ests; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
Date: Wed, 09 Aug 2017 18:33:33 GMT
Content-Length: 1437
{"token_type":"Bearer","expires_in":"3599","ext_expires_in":"262800","expires_on":"1502307214","not_before":"1502303314","resource":"https://graph.windows.net/","access_token":"some-token-value"}
The correlation ID (also called client-request-id) can be found in interactions with the Azure AD service.
The request immediately before this when you were asking the Azure AD endpoints for tokens will include the ID in the response.
https://login.microsoftonline.com/...
https://login.windows.net/...
I'm trying to setup the caching of my ExpressApp and its rendered Jade files with Nginx.
When I have a look at the pm2 logs of my application while running the google Pagespeed Insight test, I see that the Images and HTML are still served by the app instead of the Nginx proxy.
Two Questions:
How do I have to change my configuration to have the desired effect?
How do I effectively validate that the Setup/Caching is working?
Thanks!
My current Nginx Configuration:
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=backcache:8m max_size=50m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
proxy_cache_valid 200 302 10m;
proxy_cache_valid 404 1m;
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
expires 1w;
proxy_cache backcache;
#proxy_cache_bypass $http_cache_control;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass http://localhost:3030;
}
}
Headers:
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: ###
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server certificate: AddTrust External CA Root
> GET / HTTP/1.1
> Host: ###
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.10.0 (Ubuntu)
< Date: Thu, 24 Nov 2016 13:15:37 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 66761
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
< ETag: W/"104c9-8+4lo531tUk9k3SJq4sPHg"
< Expires: Thu, 01 Dec 2016 13:15:37 GMT
< Cache-Control: max-age=604800
< X-Proxy-Cache: MISS
Headers of two consecutive Requests to an Image
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:02 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:02 GMT
X-Proxy-Cache: MISS
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:08 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:08 GMT
X-Proxy-Cache: MISS
The headers you show as the request headers are most probably not the original ones, because it says "Provisional headers shown".
The actual request headers probably contain the "cache-control" header and your nginx configuration uses that header to bypass the cache, as indicated by the 'X-Proxy-Cache:BYPASS' response header.
If there is no reason to keep it, you should remove the proxy_cache_bypass-directive from your config.
Varnish (running on fastly) is caching objects for longer than I (think I) have specified. To debug this issue, I'm adding headers everywhere in the following fashion:
sub vcl_miss {
set req.http.Debugmiss = "vcl_miss";
#FASTLY miss
return(fetch);
}
sub vcl_deliver {
#FASTLY deliver
set resp.http.Debugmiss = req.http.Debugmiss;
return(deliver);
}
I've done this for vcl_recv, vcl_hit, vcl_miss, vcl_pass, vcl_hash and vcl_fetch; however, the only functions that seem to be called according to the headers are vcl_recv (which does a lookup) and vcl_deliver. For example, according to the state diagram (https://www.varnish-software.com/book/3/_images/vcl.png) after vcl_recv returns the lookup code, the vcr_hash function should be called. If it is, it doesn't seem to set any headers.
Is there something I'm overlooking?
These are the response headers that curl returns:
< HTTP/1.1 200 OK
< Server: Cowboy
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Etag: W/"9fbfd39142780bd83fed663b051c83d9"
< X-Request-Id: a2a9b83a-4143-4b40-b788-94969dd5ce91
< X-Runtime: 0.020195
< X-Rack-Cache: miss
< Via: 1.1 vegur
< Content-Length: 434
< Accept-Ranges: bytes
< Date: Wed, 02 Sep 2015 13:41:41 GMT
< Via: 1.1 varnish
< Age: 4
< Connection: keep-alive
< X-Served-By: cache-lhr6332-LHR
< X-Cache: HIT
< X-Cache-Hits: 1
< X-Timer: S1441201301.048931,VS0,VE2
< Cache-Control: no-cache, no-store, private, must-revalidate, max-age=0, max-stale=0, post-check=0, pre-check=0
< Expires: 0
< Pragma: no-cache
< Debugrecv: vcl_recv
< Debugrecvreturn: lookup
< debugme: vcl_deliver; desperate8
The entire vcl can be found here: https://gist.github.com/mdemare/2e0fa52e62691806e0a0
I think the problem is in the vcl functions where you set headers for req which is the request object, not the response. So when you look at the response headers, the ones set to the req object won't show up.
You can try either looking at the request headers or you can use varnishlog (more info)