Varnish (running on fastly) is caching objects for longer than I (think I) have specified. To debug this issue, I'm adding headers everywhere in the following fashion:
sub vcl_miss {
set req.http.Debugmiss = "vcl_miss";
#FASTLY miss
return(fetch);
}
sub vcl_deliver {
#FASTLY deliver
set resp.http.Debugmiss = req.http.Debugmiss;
return(deliver);
}
I've done this for vcl_recv, vcl_hit, vcl_miss, vcl_pass, vcl_hash and vcl_fetch; however, the only functions that seem to be called according to the headers are vcl_recv (which does a lookup) and vcl_deliver. For example, according to the state diagram (https://www.varnish-software.com/book/3/_images/vcl.png) after vcl_recv returns the lookup code, the vcr_hash function should be called. If it is, it doesn't seem to set any headers.
Is there something I'm overlooking?
These are the response headers that curl returns:
< HTTP/1.1 200 OK
< Server: Cowboy
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Etag: W/"9fbfd39142780bd83fed663b051c83d9"
< X-Request-Id: a2a9b83a-4143-4b40-b788-94969dd5ce91
< X-Runtime: 0.020195
< X-Rack-Cache: miss
< Via: 1.1 vegur
< Content-Length: 434
< Accept-Ranges: bytes
< Date: Wed, 02 Sep 2015 13:41:41 GMT
< Via: 1.1 varnish
< Age: 4
< Connection: keep-alive
< X-Served-By: cache-lhr6332-LHR
< X-Cache: HIT
< X-Cache-Hits: 1
< X-Timer: S1441201301.048931,VS0,VE2
< Cache-Control: no-cache, no-store, private, must-revalidate, max-age=0, max-stale=0, post-check=0, pre-check=0
< Expires: 0
< Pragma: no-cache
< Debugrecv: vcl_recv
< Debugrecvreturn: lookup
< debugme: vcl_deliver; desperate8
The entire vcl can be found here: https://gist.github.com/mdemare/2e0fa52e62691806e0a0
I think the problem is in the vcl functions where you set headers for req which is the request object, not the response. So when you look at the response headers, the ones set to the req object won't show up.
You can try either looking at the request headers or you can use varnishlog (more info)
Related
Our React Development build runs flawless with Express Gateway setup on localhost. After build React for production and when we run serve -s build login page comes as it is the entry point of the app. It gets 200 ok response when we put sign-in credential. But when we looked into it we can see the request to server was not successful cause token it saves to browser application is undefined and we checked the response, It is "You need to enable javascript...". JS is enabled no doubt. I have checked By using
axios.post('http://localhost:8080/api/v1/auth/sign-in', userData)
It works fine but when setup proxy:
axios.post('/auth/sign-in', userData)
react doesn’t run
Here is the part of yml for express gateway setup:
http:
port: 8080
apiEndpoints:
auth-service:
host: "*"
paths: ["/api/v1/auth/*", "/api/v1/auth"]
mail-service:
host: "*"
paths: ["/api/v1/mail/*", "/api/v1/mail"]
serviceEndpoints:
auth-service-endpoint:
url: http://localhost:3003/
mail-service-endpoint:
url: http://localhost:3005/
policies:
- proxy
pipelines:
auth-service-pipeline:
apiEndpoints:
- auth-service
policies:
- proxy:
action:
serviceEndpoint: auth-service-endpoint
changeOrigin: true
stripPath: true
mail-service-pipeline:
apiEndpoints:
- mail-service
policies:
- proxy:
action:
serviceEndpoint: mail-service-endpoint
changeOrigin: true
stripPath: true
I put the setupProxy.js on src directory of React:
const { createProxyMiddleware } = require('http-proxy-middleware');
module.exports = function(app) {
app.use(createProxyMiddleware('/api/v1',
{ target: 'http://localhost:8080',
secure: false,
changeOrigin: true,
// pathRewrite: {
// "^/api": "/api/v1",
// }
}
));
}
Currently everything is on same machine. We are not using docker.
The application runs on Dev environment but shows 200 ok response in production build
Any help will be appreciated.
[Edit]
krypton:admin-dashboard-server hasan$ curl -v http://localhost:3001/find_all_services/1/10
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 3001 (#0)
> GET /find_all_services/1/10 HTTP/1.1
> Host: localhost:3001
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< X-DNS-Prefetch-Control: off
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15552000; includeSubDomains
< X-Download-Options: noopen
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Content-Type: application/json; charset=utf-8
< Content-Length: 1833
< ETag: W/"729-LM91B3vCUrbvesBrp32ykiXXkQo"
< Date: Tue, 12 Jan 2021 14:57:24 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
[{"id":1,"name":"Laser Hair Remove"},
{"id":2,"name":"Facial Treatments"}
]
krypton:admin-dashboard-server hasan$ curl -v
http://localhost:8080/api/v1/services/find_all_services/1/10
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /api/v1/services/find_all_services/1/10 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< access-control-allow-origin: *
< x-dns-prefetch-control: off
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=15552000; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< content-type: application/json; charset=utf-8
< content-length: 1833
< etag: W/"729-LM91B3vCUrbvesBrp32ykiXXkQo"
< date: Tue, 12 Jan 2021 15:03:45 GMT
< connection: keep-alive
<
* Connection #0 to host localhost left intact
[{"id":1,"name":"Laser Hair Remove"},
{"id":2,"name":"Facial Treatments"}
]
krypton:admin-dashboard-server hasan$ curl -v -H "Content-Type: application/json" -X POST -d
'{"email":"mh.mithun#gmail.com","password":"safe123"}'
http://localhost:8080/api/v1/auth/sign-in
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> POST /api/v1/auth/sign-in HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 52
>
* upload completely sent off: 52 out of 52 bytes
< HTTP/1.1 200 OK
< access-control-allow-origin: *
< x-dns-prefetch-control: off
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=15552000; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< content-type: application/json; charset=utf-8
< content-length: 270
< etag: W/"10e-S+kd8b4Yfl7un04FVGe3MFLFEaY"
< date: Tue, 12 Jan 2021 15:40:12 GMT
< connection: keep-alive
<
* Connection #0 to host localhost left intact
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhbGdvcml0aG0iOiJIUzI1N"
krypton:admin-dashboard-server hasan$ curl -v -H "Content-Type: application/json" -X POST -d '{"email":"mh.mithun#gmail.com","password":"safe123"}' http://localhost:3003/sign-in
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 3003 (#0)
> POST /sign-in HTTP/1.1
> Host: localhost:3003
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 52
>
* upload completely sent off: 52 out of 52 bytes
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< X-DNS-Prefetch-Control: off
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15552000; includeSubDomains
< X-Download-Options: noopen
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Content-Type: application/json; charset=utf-8
< Content-Length: 270
< ETag: W/"10e-LW/1l5fXf5BaiF3KJMvG60xRthE"
< Date: Tue, 12 Jan 2021 15:45:33 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhbGdvcml0aG0i"
When i use the curl command all the headers are correct but files are not caching as
CF-Cache-Status: MISS
is shown by curl command .
> < HTTP/1.1 200 OK < Date: Sat, 02 Jun 2018 05:57:28 GMT <
> Content-Type: image/png < Content-Length: 2293 < Connection:
> keep-alive < Set-Cookie:
> __cfduid=d3cf179afd212660124585350d19e88c81527919048; expires=Sun, 02-Jun-19 05:57:28 GMT; path=/; domain=.mayan.cf; HttpOnly <
> X-Powered-By: Express < Accept-Ranges: bytes < Cache-Control: public,
> max-age=86400 < Last-Modified: Fri, 01 Jun 2018 15:39:06 GMT < ETag:
> W/"8f5-163bc009a0c" < CF-Cache-Status: MISS < Expires: Sun, 03 Jun
> 2018 05:57:28 GMT < Server: cloudflare < CF-RAY: 4247aec4300930f6-SIN
> < { [1905 bytes data]
> * Connection #0 to host mayan.cf left intact
Cloudflare uses some heuristics to try to guess, based on the request alone, whether the content is static (cacheable) or dynamic. If it guesses wrong at request time, it won't cache the response. You can disable this heuristic by setting the Page Rule "Cache Level" to "Cache Everything" -- then Cloudflare will assume that the response will always be cacheable.
Try setting that page rule and see if it helps.
If the Cache-Control header is set to "private", "no-store",
"no-cache", or "max-age=0", or if there is a cookie in the response,
then Cloudflare will not cache the resource.
https://support.cloudflare.com/hc/en-us/articles/202775670-How-Do-I-Tell-Cloudflare-What-to-Cache-
I have two Redhat Linux servers, say 'original' and 'demo'. Both have apache running on them. On original server, I have a link 'http://original.com/abc' on a page which I want redirect it to http://demo.com.
What I have done is-
Open httpd.conf on original server and added following lines and restarted apache service:
ProxyPass /abc http://demo.com/
ProxyPassReverse /abc http://demo.com/
But when I am trying to access http://original.com/abc, it is redirecting me to the original server i.e. on http://original.com (not on http://demo.com)
I have tried to find the solutions on various sites but unable to find out what the problem is.
Edit:
Output of curl -v http://original.com/abc:
curl -v http://original.com/abc
* About to connect() to original.com port 80 (#0)
* Trying 10.100.100.100...
* Connected to original.com (10.100.100.100) port 80 (#0)
> GET /demo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: original.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Tue, 24 Apr 2018 06:30:23 GMT
< Server: Apache-Coyote/1.1
< X-Frame-Options: SAMEORIGIN
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Location: login.action
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Set-Cookie: JSESSIONID=0BAA246C7F2505D2F5A0335CB0542CAA; Path=/; HttpOnly
<
* Connection #0 to host original.com left intact
Output of curl -v http://demo.com/ :
#curl -v http://demo.com
* About to connect() to 10.100.100.101 port 80 (#0)
* Trying 10.100.100.101...
* Connected to 10.100.100.101 (10.100.100.101) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: demo.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: Apache-Coyote/1.1
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: JSESSIONID=FED19458459131F456D638EC57278C2A; Path=/; HttpOnly
< Location: login.action
< Content-Type: text/html
< Content-Length: 0
< Date: Tue, 24 Apr 2018 06:31:47 GMT
<
* Connection #0 to host demo.com left intact
The textual order of ProxyPass statements is relevant. The first matched entry stops further matching.
ProxyPass / http://some.tomcat:8080/
ProxyPassReverse / http://some.tomcat:8080/
...
ProxyPass /abc http://demo.com/
ProxyPassReverse /abc http://demo.com/
Here, demo.com would be fed nothing. The some.tomcat would be fed both / and /abc
sendFile is for sending files and it also figures out some interesting headers from the file (like content length). For a HEAD request I would ideally want the exact same headers but just skip the body.
There doesn't seem to be an option for this in the API. Maybe I can override something in the response object to stop it from sending anything?
Here's what I got:
res.sendFile(file, { headers: hdrs, lastModified: false, etag: false })
Has anyone solved this?
As Robert Klep has already written, the sendFile already has the required behavior of sending the headers and not sending the body if the request method is HEAD.
In addition to that, Express already handles HEAD requests for routes that have GET handlers defined. So you don't even need to define any HEAD handler explicitly.
Example:
let app = require('express')();
let file = __filename;
let hdrs = {'X-Custom-Header': '123'};
app.get('/file', (req, res) => {
res.sendFile(file, { headers: hdrs, lastModified: false, etag: false });
});
app.listen(3322, () => console.log('Listening on 3322'));
This sends its own source code on GET /file as can be demonstrated with:
$ curl -v -X GET localhost:3322/file
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3322 (#0)
> GET /file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:3322
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< X-Custom-Header: 123
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Content-Type: application/javascript
< Content-Length: 267
< Date: Tue, 11 Apr 2017 10:45:36 GMT
< Connection: keep-alive
<
[...]
The [...] is the body that was not included here.
Without adding any new handler this will also work:
$ curl -v -X HEAD localhost:3322/file
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3322 (#0)
> HEAD /file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:3322
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< X-Custom-Header: 123
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Content-Type: application/javascript
< Content-Length: 267
< Date: Tue, 11 Apr 2017 10:46:29 GMT
< Connection: keep-alive
<
This is the same but with no body.
Express uses send to implement sendFile, which already does exactly what you want.
I'm trying to setup the caching of my ExpressApp and its rendered Jade files with Nginx.
When I have a look at the pm2 logs of my application while running the google Pagespeed Insight test, I see that the Images and HTML are still served by the app instead of the Nginx proxy.
Two Questions:
How do I have to change my configuration to have the desired effect?
How do I effectively validate that the Setup/Caching is working?
Thanks!
My current Nginx Configuration:
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=backcache:8m max_size=50m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
proxy_cache_valid 200 302 10m;
proxy_cache_valid 404 1m;
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
expires 1w;
proxy_cache backcache;
#proxy_cache_bypass $http_cache_control;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass http://localhost:3030;
}
}
Headers:
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: ###
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server certificate: AddTrust External CA Root
> GET / HTTP/1.1
> Host: ###
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.10.0 (Ubuntu)
< Date: Thu, 24 Nov 2016 13:15:37 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 66761
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
< ETag: W/"104c9-8+4lo531tUk9k3SJq4sPHg"
< Expires: Thu, 01 Dec 2016 13:15:37 GMT
< Cache-Control: max-age=604800
< X-Proxy-Cache: MISS
Headers of two consecutive Requests to an Image
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:02 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:02 GMT
X-Proxy-Cache: MISS
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:08 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:08 GMT
X-Proxy-Cache: MISS
The headers you show as the request headers are most probably not the original ones, because it says "Provisional headers shown".
The actual request headers probably contain the "cache-control" header and your nginx configuration uses that header to bypass the cache, as indicated by the 'X-Proxy-Cache:BYPASS' response header.
If there is no reason to keep it, you should remove the proxy_cache_bypass-directive from your config.