Issue with CDN strong caching without Cache-Control or Expires headers - azure

We're using Azure CDN to serve images, and I'm trying to understand why images are being strong-cached by web browsers even though there is no Cache-Control or Expires headers in the image response.
For example, the following response headers are returned for an image from Azure CDN:
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Content-MD5: KuQCJm6GQyEjejWwmRgRwQ==
Content-Type: image/jpeg
Date: Tue, 21 Nov 2017 00:15:57 GMT
Etag: 0x8D523228F0F4C42
Last-Modified: Sat, 04 Nov 2017 01:22:47 GMT
Server: ECAcc (meb/A744)
X-Cache: HIT
x-ms-blob-type: BlockBlob
x-ms-lease-status: unlocked
x-ms-request-id: 00822b7c-001e-0045-4194-61d246000000
x-ms-version: 2009-09-19
Content-Length: 5143
<<image data>>
As you can see there is an Etag header returned, but no Cache-Control or Expires headers.
When tracing the network traffic (using Fiddler) from the browser (Chrome), we are NOT seeing any subsequent requests for these images.
My understanding of Etags is that subsequent requests for this image should be sent back to the server (weak caching), and then the server can return a 304 not modified response if the file has not changed.
Can anyone shed any light on this?

I think you need the header cache-control: must-revalidate to get the browser to check the source and have 304 mot modified returned if there is no change.
This is not optimal in terms of caching though.
You are better to invalidate the js with QS changes ("v=??") or set a short expires / max-age header (60 / 120 seconds, or whatever you can handle in terms of deployment, 5 minutes???).
Having an expires header combined with etags should still mean the browser receives a 304 not modified from the server after expiration.

Related

juiceshop heroku forge unsigned token challenge not solved

I recently deployed Juiceshop on Heroku and am trying to solve the challenge of forging an unsigned JWT token using Burpsuite. I followed the hints provided in the ebook starting from using burpsuite to get a valid JWT token from the request header using the proxy tab on the /rest/user/whoami URL and sent it to the repeater. In the repeater tab, I changed the alg property in the token to "none" and the email to "jwtn3d#juice-sh.op". I then encoded them in base64 and put a period after the header and payload respectively in the token value. When I send the request, I get a HTTP 200 OK status and the jwtn3d#juice-sh.op email is reflected as well, but the juiceshop challenge doesn't seem to be solved. When I reload the website, I'm not logged in as jwtn3d#juice-sh.op either.
Server: Cowboy
Connection: close
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Feature-Policy: payment 'self'
Set-Cookie: token=ewogICJ0eXAiOiAiSldUIiwKICAiYWxnIjogIm5vbmUiCn0.ewogICJzdGF0dXMiOiAic3VjY2VzcyIsCiAgImRhdGEiOiB7CiAgICAiaWQiOiAxLAogICAgInVzZXJuYW1lIjogIiIsCiAgICAiZW1haWwiOiAiand0bjNkQGp1aWNlLXNoLm9wIiwKICAgICJwYXNzd29yZCI6ICIwMTkyMDIzYTdiYmQ3MzI1MDUxNmYwNjlkZjE4YjUwMCIsCiAgICAicm9sZSI6ICJhZG1pbiIsCiAgICAiZGVsdXhlVG9rZW4iOiAiIiwKICAgICJsYXN0TG9naW5JcCI6ICIwLjAuMC4wIiwKICAgICJwcm9maWxlSW1hZ2UiOiAiYXNzZXRzL3B1YmxpYy9pbWFnZXMvdXBsb2Fkcy9kZWZhdWx0QWRtaW4ucG5nIiwKICAgICJ0b3RwU2VjcmV0IjogIiIsCiAgICAiaXNBY3RpdmUiOiB0cnVlLAogICAgImNyZWF0ZWRBdCI6ICIyMDIyLTAyLTA2IDExOjUyOjAxLjQwOSArMDA6MDAiLAogICAgInVwZGF0ZWRBdCI6ICIyMDIyLTAyLTA2IDExOjUyOjAxLjQwOSArMDA6MDAiLAogICAgImRlbGV0ZWRBdCI6IG51bGwKICB9LAogICJpYXQiOiAxNjQ0MTQ4NDcxLAogICJleHAiOiAxNjQ0MTY2NDcxCn0.; Path=/
Content-Type: application/json; charset=utf-8
Content-Length: 133
Etag: W/"85-PWOL8M3mYJv9caLvST18mm7K+2g"
Vary: Accept-Encoding
Date: Sun, 06 Feb 2022 11:57:25 GMT
Via: 1.1 vegur
{"user":{"id":1,"email":"jwtn3d#juice-sh.op","lastLoginIp":"0.0.0.0","profileImage":"assets/public/images/uploads/defaultAdmin.png"}}
I tried opening DevTools on the website and manually changing the cookie value to the forged token, and when I reloaded the website it reflected the jwtn3d#juice-sh.op email on the profile. But it still didn't solve the challenge. I've looked for tutorials everywhere and the challenges all seem to have just been solved once they sent the forged token over using the repeater, I'm at a loss as to why it isn't working in my case.
I found an alternative to using repeater, I used proxy intercept instead and changed the token value to the forged token on the /rest/user/whoami requests. That seemed to do the trick, though the notif pop up for the solved challenge didn't appear but it did show as solved in the scoreboard.

How do I get set cookie value when there are multiple set cookie request in varnish?

I am trying to unset cookie if my cookie has certain value.
I tried getting the value using beresp.http.Set-Cookie but this seems to get first value set in Set-Cookie on response. Is there any way I can pull the second Set-Cookie's value ?
Edit
This is my Response header
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Fri, 19 Feb 2021 09:47:53 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12297
Connection: keep-alive
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Encoding: br
Set-Cookie: VarnishCustomerIsGuest=True; path=/
Set-Cookie: .Nop.Customer=62a224b9-3b5b-4e74-8200-cf341df111af; expires=Sat, 19 Feb 2022 09:47:53 GMT; path=/; httponly
Set-Cookie: .Nop.TempData=; expires=Thu, 01 Jan 1970 00:00:00 GMT; path=/; samesite=lax; httponly
Vary: Accept-Encoding
X-MiniProfiler-Ids: ["4f9a095d-bbd1-4ffe-81e1-31761363af25"]
X-TMP1000: VarnishCustomerIsGuest=True; path=/
X-Varnish: 163854
Age: 0
Via: 1.1 varnish (Varnish/5.2)
X-Cache: MISS
Accept-Ranges: bytes
What I am trying to do is get the customer cookie at vcl_backend_response. But
beresp.http.Set-Cookie
gives me the first set-cookie value of VarnishCustomerIsGuest and I am not sure if I can get the customer cookie .Nop.Customer.
My application backend always sets cookie for every request so what I am trying to do is unset the cookies depending on .Nop.Customer value so that for some particular customer the page is always served from the varnish cache.
Varnish's vmod_std has the std.collect() function that collapses headers into a single header.
In your case, the following VCL snippet could be used to collapse all occurrences of the Set-Cookie header into a single Set-Cookie:
vcl 4.0;
import std;
sub vcl_backend_response {
std.collect(beresp.http.Set-Cookie);
}
Of course your existing VCL logic should be part of the VCL file as well. This example just shows the need to import the VMOD and to call std.collect() within vcl_backend_response.
An important disclaimer that comes directly from the std.collect() documentation page:
Care should be taken when collapsing headers. In particular collapsing Set-Cookie will lead to unexpected results on the browser side.
Once the Set-Cookie header is collapsed, you can use regsub() logic to extract the necessary values.

Upgrading to Azure (Microsoft) websocket not acknowledged

Due to constraints of current project, I am having to write the WebSocket protocol by hand in C++. I am able to get the authorization key, but when I try to upgrade the next socket connection, the server stalls after getting a completed MIME header. Then when I send anything after it, I get a 400 error. I do not get an acknowledgement from the server that the connection has been upgraded to a WebSocket. Here is a dump:
=========================================================================
POST /sts/v1.0/issueToken HTTP/1.1
Accept: */*
Connection: Keep-Alive
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
Host: api.cognitive.microsoft.com
Ocp-Apim-Subscription-Key: 21cedc8aaab847369294240b2122b08d
Origin: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapi.cognitive.microsoft.com&data=04%7C01%7Cv-lufil%40microsoft.com%7C427ffe760b7a4f6ffe6a08d4d8fa0613%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636372015735060988%7CUnknown%7CVW5rbm93bnx7IlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiT3RoZXIifQ%3D%3D%7C-1&sdata=823HpmiJeZ54tzq6CpX86ZS8B0yUiOYSNMXvrmDSunA%3D&reserved=0
User-Agent: Gideon/0.0.1
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 495
Content-Type: application/jwt; charset=us-ascii
Expires: -1
Server: Microsoft-IIS/8.5 Microsoft-HTTPAPI/2.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
apim-request-id: 00fe24bc-ba53-4d91-9363-ea7fddfe2a5a
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Operation-Location
Date: Tue, 01 Aug 2017 15:21:40 GMT
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzY29wZSI6Imh0dHBzOi8vc3BlZWNoLnBsYXRmb3JtLmJpbmcuY29tIiwic3Vic2NyaXB0aW9uLWlkIjoiZmMwOGVlNGM5ZmNkNGI0MWFmNTZiNzJmZDliZTE4ZWEiLCJwcm9kdWN0LWlkIjoiQmluZy5TcGVlY2guUHJldmlldyIsImNvZ25pdGl2ZS1zZXJ2aWNlcy1lbmRwb2ludCI6Imh0dHBzOi8vYXBpLmNvZ25pdGl2ZS5taWNyb3NvZnQuY29tL2ludGVybmFsL3YxLjAvIiwiYXp1cmUtcmVzb3VyY2UtaWQiOiIiLCJpc3MiOiJ1cm46bXMuY29nbml0aXZlc2VydmljZXMiLCJhdWQiOiJ1cm46bXMuc3BlZWNoIiwiZXhwIjoxNTAxNjAxNDk5fQ.2RQhid_B45fN5M2BmUlodhIe4Xxx71Ws1b03JylERUw
=========================================================================
POST /speech/recognition/dictation/cognitiveservices/v1?language=en-US HTTP/1.1
Accept: */*
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzY29wZSI6Imh0dHBzOi8vc3BlZWNoLnBsYXRmb3JtLmJpbmcuY29tIiwic3Vic2NyaXB0aW9uLWlkIjoiZmMwOGVlNGM5ZmNkNGI0MWFmNTZiNzJmZDliZTE4ZWEiLCJwcm9kdWN0LWlkIjoiQmluZy5TcGVlY2guUHJldmlldyIsImNvZ25pdGl2ZS1zZXJ2aWNlcy1lbmRwb2ludCI6Imh0dHBzOi8vYXBpLmNvZ25pdGl2ZS5taWNyb3NvZnQuY29tL2ludGVybmFsL3YxLjAvIiwiYXp1cmUtcmVzb3VyY2UtaWQiOiIiLCJpc3MiOiJ1cm46bXMuY29nbml0aXZlc2VydmljZXMiLCJhdWQiOiJ1cm46bXMuc3BlZWNoIiwiZXhwIjoxNTAxNjAxNDk5fQ.2RQhid_B45fN5M2BmUlodhIe4Xxx71Ws1b03JylERUw
Connection: upgrade
Content-Length: 8002
Content-Type: audio/wav; codec=audio/pcm; samplerate=16000
Host: speech.platform.bing.com
Path: audio
Sec-WebSocket-Key: Z2lkZW9ucm9ja3MK
Transfer-Encoding: chunked
Upgrade: websocket
User-Agent: Gideon/0.0.1
X-RequestId: 21cedc8aaab847369294240b2122b08d
X-Timestamp: 2017-08-01T15:21:40
HTTP/1.1 400 Bad Request
Exception: 4xx Client failure
Note that the server does not reply despite getting two "\r\n" to indicate an end of MIME header. When I send anything afterwards I get a 400 error.
According to your dump logs, it seems that you want to use Microsoft's Speech Service to convert the speech to text.
By using Microsoft's Speech Service,the 400 error means you don't have applied all the required parameters and HTTP headers and that the values are correct.
I found your request missed the X-ConnectionId in your request.
According to this article:
The Microsoft Speech Service requires that all clients include a unique id to identify the connection. Clients must include the X-ConnectionId header when starting a web socket handshake. The X-ConnectionId header value must be a universally unique identifier. Web socket upgrade requests that do not include the X-ConnectionId, that do not specify a value for the X-ConnectionId header, or that do not include a valid universally unique identifier value will be rejected by the service with a 400 Bad Request response.
So I suggest you could add the identify id and test again.

Where can I see Server http header?

One of our applications is tested by Whitehat Sentinel and one of their findings was that in some cases our response header for Server is set to:
Microsoft-HTTPAPI/2.0
I have tried accessing the URL they identified with Postman and Fiddler but I do not see the Server header. I have also tried an online web sniffer http://web-sniffer.net/
Can someone advise how I can see this header?
In Chrome Network tab I see these headers
HTTP/1.1 404 Not Found
Content-Type: text/html
Strict-Transport-Security: max-age=300
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Date: Thu, 13 Jul 2017 13:59:15 GMT
Content-Length: 1245
No Server header.
The URL reported by Whitehat was not working for me, I changed the target URL to domain.com/%% and this caused the request to be handled by http.sys and it returned the Server attribute.
That is not the name of the header. That is the value found in the Server header when an application serves files over HTTP via http.sys, which is the kernel-mode HTTP server built into Windows.
For example, when serving a file via a C# HttpListener, I get this header:
Server: Microsoft-HTTPAPI/2.0
This header can be disabled by setting the following registry value:
Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters
Name: DisableServerHeader
Type: DWORD
Value: 1

Browser Cache Control, Dynamic Content

Problem: I can't seem to get FireFox to cache images sent from a dynamic server
Setup: Static Apache Server with reverse proxy to a dynamic server (mod_perl2) at backend.
Here is the request URL for the server. It is sent to the the dynamic server, where the cookie is used to validate access to the image:
Request Headers
Host: <OBSCURED>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102815 Ubuntu/9.04 (jaunty) Firefox/3.0.15
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: <OBSCURED>
Cookie: pz_cred=4KCNr0RM15%2FJCOt%2BEa6%2BL62z%2Fxvbp2xNQHY5pJw5d6Q
Pragma: no-cache
Cache-Control: no-cache
The dynamic server streams the image back to the server, and provides the following response:
Response Headers
Date: Tue, 24 Nov 2009 04:28:07 GMT
Server: Apache/2.2.11 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0
Cache-Control: public, max-age=31536000
Content-Length: 25496
Content-Type: image/jpeg
Via: 1.1 127.0.1.1:8081
Keep-Alive: timeout=15, max=75
Connection: Keep-Alive
So far, so good (me thinks). However, on reload of the page, the image does not appear cached, and a request is again sent:
Request Headers
Host: <OBSCURED>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102815 Ubuntu/9.04 (jaunty) Firefox/3.0.15
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: <OBSCURED>
Cookie: pz_cred=4KCNr0RM15%2FJCOt%2BEa6%2BL62z%2Fxvbp2xNQHY5pJw5d6Q
Cache-Control: max-age=0
It doesn't seem that request should happen as the browser should have cached the image. As it is, a 200 response is received, same as the first, and the image appears to be re-fetched (although the browser does appear to be using the cached images).
The problem appears to be hinted at by the Cache-Control: max-age=0 in the reload request header, above.
Does anyone know why this is happening? Perhaps it is the Via header in the response that is causing the problem?
The original request has
Cache-Control: no-cache
which tells all the intermediate HTTP caches (including Firefox's) that you don't want to use a cached response, you want to get the response from the origin web server itself.
The response says:
Cache-Control: public, max-age=31536000
which tells everyone that as far as the origin server is concerned, the response may be cached. The server seems to be configured to enable the PNG image to be cached: HTTP 1.1 (section 14.21) says:
Note: if a response includes a
Cache-Control field with the max-age
directive (see section 14.9.3), that
directive overrides the Expires field.
Your second request says:
Cache-Control: max-age=0
which tells all the intermediate HTTP caches that you won't take any cached response older than 0 seconds.
One thing to watch out for: if you hit the Reload button in Firefox, you are asking to reload from the origin web server. To test the caching of the image, navigate away from the page and back, or open it up in a new tab. Not sure why you saw no-cache the first time and max-age=0 the second though.
BTW, I like the FireBug plug-in for Firefox. You can take a look at the request and response headers with it and all sorts of other good stuff.
My previous answer was only partially correct.
The problem is the way FireFox 3 handles reload events. Apparently, it almost always requests content again from the origin server. Thus the Cache-Control: max-age=0 request header.
Firefox does use cached images to render a page on reload, but then it still makes all the requests to update them "in the background". It then replace them as they come in.
Therefore, the page renders fast, YSlow reports cached content. But the server is still getting nailed.
The resolution is to interrogate the incoming headers in the dynamic server script and determine if a 'If-Modified-Since' header is provided. If this is the case, and it is determined the content has not changed, an HTTP_NOT_MODIFIED (304) response is returned.
This is not optimal -- I'd rather Firefox not make the requests at all -- but it cuts the page load time in half, and greatly reduces bandwidth. Given the way Firefox works on reload, this appears the best solution.
Other Comments: Jim Ferran's point about navigating away from page and returning has merit -- the cache is always used, and no requests are outgoing (+1 to Jim). Also, content that is dynamically added (e.g. AJAX calls after the initial load) appear to use the cache as well.
Hope this helps someone besides me :)
Looks like solved it:
Removed the proxy via header
Added a Last-Modified header
Added a far-future expires date
Firebug still shows 200 responses from the origin server, however, YSlow recognizes the images as cached. According to YSlow, total image download size when fresh is greater than 500K; with the cache primed, it shows 0K download size.
Here is the response header from the Origin server which does the trick:
Date: Tue, 24 Nov 2009 08:54:24 GMT
Server: Apache/2.2.11 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0
Last-Modified: Sun, 22 Nov 2009 07:28:25 GMT
Expires: Tue, 30 Nov 2010 19:00:25 GMT
Content-Length: 10883
Content-Type: image/jpeg
Keep-Alive: timeout=15, max=89
Connection: Keep-Alive
Because of the way I'm requesting the images, it really should not matter if these dates are static; my app knows the last mod time before requesting the image and appends this to the request URL on the client side to create a unique URL for each image version, e.g. http://myserver.com/img/125.jpg?20091122 (the info comes from a AJAX JSON feed). I could, for example, make the last modified date 01 Jan 2000, and the Expires date sometime in the year 2050.
If YSlow is correct -- and performance testing implies it is -- then FireBug should really report these local cache hits instead of a 200 response.

Resources