NodeJS/express: Cache and 304 status code - node.js

When I reload a website made with express, I get a blank page with Safari (not with Chrome) because the NodeJS server sends me a 304 status code.
How to solve this?
Of course, this could also be just a problem of Safari, but actually it works on all other websites fine, so it has to be a problem on my NodeJS server, too.
To generate the pages, I'm using Jade with res.render.
Update: It seems like this problem occurs because Safari sends 'cache-control': 'max-age=0' on reload.
Update 2: I now have a workaround, but is there a better solution?
Workaround:
app.get('/:language(' + content.languageSelector + ')/:page', function (req, res)
{
// Disable caching for content files
res.header("Cache-Control", "no-cache, no-store, must-revalidate");
res.header("Pragma", "no-cache");
res.header("Expires", 0);
// rendering stuff hereā€¦
}
Update 3:
So the complete code part is currently:
app.get('/:language(' + content.languageSelector + ')/:page', pageHandle);
function pageHandle (req, res)
{
var language = req.params.language;
var thisPage = content.getPage(req.params.page, language);
if (thisPage)
{
// Disable caching for content files
res.header("Cache-Control", "no-cache, no-store, must-revalidate");
res.header("Pragma", "no-cache");
res.header("Expires", 0);
res.render(thisPage.file + '_' + language, {
thisPage : thisPage,
language: language,
languages: content.languages,
navigation: content.navigation,
footerNavigation: content.footerNavigation,
currentYear: new Date().getFullYear()
});
}
else
{
error404Handling(req, res);
}
}

Easiest solution:
app.disable('etag');
Alternate solution here if you want more control:
http://vlasenko.org/2011/10/12/expressconnect-static-set-last-modified-to-now-to-avoid-304-not-modified/

Try using private browsing in Safari or deleting your entire cache/cookies.
I've had some similar issues using chrome when the browser thought it had the website in its cache but actually had not.
The part of the http request that makes the server respond a 304 is the etag. Seems like Safari is sending the right etag without having the corresponding cache.

I had the same problem in Safari and Chrome (the only ones I've tested) but I just did something that seems to work, at least I haven't been able to reproduce the problem since I added the solution. What I did was add a metatag to the header with a generated timstamp. Doesn't seem right but it's simple :)
<meta name="304workaround" content="2013-10-24 21:17:23">
Update
P.S
As far as I can tell, the problem disappears when I remove my node proxy (by proxy i mean both express.vhost and http-proxy module), which is weird...

As you said, Safari sends Cache-Control: max-age=0 on reload. Express (or more specifically, Express's dependency, node-fresh) considers the cache stale when Cache-Control: no-cache headers are received, but it doesn't do the same for Cache-Control: max-age=0. From what I can tell, it probably should. But I'm not an expert on caching.
The fix is to change (what is currently) line 37 of node-fresh/index.js from
if (cc && cc.indexOf('no-cache') !== -1) return false;
to
if (cc && (cc.indexOf('no-cache') !== -1 ||
cc.indexOf('max-age=0') !== -1)) return false;
I forked node-fresh and express to include this fix in my project's package.json via npm, you could do the same. Here are my forks, for example:
https://github.com/stratusdata/node-fresh
https://github.com/stratusdata/express#safari-reload-fix
The safari-reload-fix branch is based on the 3.4.7 tag.

Old question, I know. Disabling the cache facility is not needed and not the best way to manage the problem. By disabling the cache facility the server needs to work harder and generates more traffic. Also the browser and device needs to work harder, especially on mobile devices this could be a problem.
The empty page can be easily solved by using Shift key+reload button at the browser.
The empty page can be a result of:
a bug in your code
while testing you served an empty page (you can't
remember) that is cached by the browser
a bug in Safari (if so,
please report it to Apple and don't try to fix it yourself)
Try first the Shift keyboard key + reload button and see if the problem still exists and review your code.

Operating system: Windows
Browser: Chrome
I used Ctrl + F5 keyboard combination. By doing so, instead of reading from cache, I wanted to get a new response. The solution is to do hard refresh the page.
On MDN Web Docs:
"The HTTP 304 Not Modified client redirection response code indicates
that there is no need to retransmit the requested resources. It is an
implicit redirection to a cached resource."

// just add * in URL
app.get('/api*', (req, res)=>{
// do something
});

Related

How to cache response from google drive API in NodeJS

I want to reproduce music files from google drive on a web page. I have the link for each file but the response cache headers for the calls are 'no-cache, no-store, max-age=0, must-revalidate" so it will never be saved on the browser cache. Is there any way to add cache to google drive files requests?
The problem:
When you use the drive link for a music file (mp3) https://drive.google.com/a/pucp.pe/uc?id=1kYYS9FZ9Vxif5WJM9ZQcY4SR35NMgoIE&export=download the GET API call receives a 302 code which generates a redirect to another URL, in this case, to 'https://doc-0o-28-docs.googleusercontent.com/docs/securesc/bgp95l3eabkkpccn0qi0qopvc4e7d4mq/us95e8ush1v4b7vvijq1vj1d7ru4rlpo/1556330400000/01732506421897009934/01732506421897009934/1kYYS9FZ9Vxif5WJM9ZQcY4SR35NMgoIE?h=14771753379018855219&e=download'. Each of these calls has no-cache in headers.
I tried using workbox (cache API) but I don't find a way to cache redirects, probably I need to cache both calls (the first GET and the redirect). However, if I use the redirected URL the caching works, but I don't have access to that URL until the first call is made.
I tried to use a proxy server from a NodeJS server
app.get("/test", (req, res) => {
try {
https.get(
URL,
function(response) {
res.writeHead(response.statusCode, {...response.headers,
"Cache-Control": "public, max-age=120",
"Expires": new Date(Date.now() + 120000).toUTCString() })
response.pipe(res);
}
);
} catch (e) {
console.log(e);
}
});
I tried using the first URL with no luck.
I tried using the redirect URL but I get a "Status Code: 302 Found"
One solution could be to download the file and serve it directly from my server but I will be missing the point of using the drive storage. I really want to use google drive storage and not duplicate all files on my server.
Is there a recommended way to do the caching for this case? maybe there is some google drive configuration that I'm missing. Or do you know another approach I could take in this case?
You should be able to cache redirected responses using Workbox (and the Cache Storage API in general). HTTP 30x redirects are followed automatically by default, and you should only have to route the original, non-redirected URL.
Here's a live example of a Glitch project that uses Workbox to cache that MP3 file: https://glitch.com/edit/#!/horn-sausage?path=sw.js:9:0
The relevant snippet of code, that also accounts for the fact that there is no CORS used when serving the files (so you'll get back an opaque response with a status of 0) is:
workbox.routing.registerRoute(
new RegExp('https://drive.google.com/'),
new workbox.strategies.CacheFirst({
plugins: [
new workbox.cacheableResponse.Plugin({statuses: [0, 200]})
],
})
);

Express JS - Want to cache static resources but NOT rendered HTML

I'm working on a dynamic application where we don't want to cache HTML (i.e. cart contents can change from one page refresh to the next). To that end, I'm calling middleware that sets cache-control headers to avoid caching. However, said cache-control headers also apply when fetching static resources. For obvious performance reasons, this is undesired behavior. We def want to cache static resources. My question is this... Is there a way to set diff response headers for static resources vs rendered html? I tried passing the setHeaders option to the express.static middleware, but the thread hangs, presumably because we are trying to set the same response header twice. Any help is greatly appreciated!
Edit: adding environment information -
I'm on Express 4 and Node 4.4
Edit: adding example code. This is the relevant bit from app.js that aggressively avoids caching HTML in browser.
app.use(express.static(config.static.public));
// ...Stuff
app.use(function (req, res, next) {
// Don't cache html
res.set('Cache-Control', 'no-cache, private, no-store, must-revalidate, '
+ 'max-stale=0, post-check=0, pre-check=0');
res.set('Expires', 'Fri, 31 Dec 1998 12:00:00 GMT');
next();
});
app.use(express.static("static", {maxage : 0}))
more info
Maybe clear all ready cached files in the browser before testing.

cors+s3+browser cache+chrome extension

Yes. this is a complex question. i will try to nake it brief.
My website fetches resources from s3.
I also have an extension that needs to prefetch that s3 file when someone does a google query, so later when they go on my site ,the resource is cached.
At this point I should probably stress that I'm not doing anything malicious. just a matter of user experience.
My problem is. that making an ajax request to s3 fron the extension (either from content-script or background) doesn't send an origin header.
This means that the resource is downloaded and cached without an allow origin header. s3
doesnt add that allow-origin:* if theres no origin in the request. so later, on my site it fails due to missing allow-origin header in cached file :-(
Any ideas on a better way to prefetch to browser cache?
Is there a way to force the ajax request to send an origin? Any origin?
Since I have an allow-origin:* on my s3 bucket, I think any origin will do accept null.
Thanks
Edit: Ended up using one of Rob W's solutions. You are awesome.
Let me comment on each of the options he suggested:
Not to add the host premissions on my manifest - clever idea but wouldn't work for me since I have a content script which runs on any website, so I must use a catch-all wildcard, and I don't think there is an "exclude" permission option.
I tried it, it issues a null origin, which as expected ends up in S3 sending the allow-origin:* header as required. this means I don't get that "allow-origin header missing" error, however the file is then not served from cache. I guess for it to be actually served from cache in chrome this has to be exactly the same origin. so that was very close but not enough.
third option is a charm. And it is the simplest. I didn't know I was able to manipulate the origin header. So I do that and set the exact origin of my website - And it works. The file is cached and later served from cache. I must stress that i had to add a Url filter to only apply this to requests going out to my s3 bucket, otherwise I expect this would wreak havoc on the user's browser.
Thanks. Well done
You've got three options:
Do not add the host permission for S3 to your manifest file. Then the extension will not have the blanket permission to access the resource, and an Origin request header will be sent with the request.
Use a non-extension frame to perform the AJAX request. For example, the following method will result in a cross-origin GET request with Origin: null.
function prefetchWithOrigin(url) {
var html = '<script>(' + function(url) {
var x = new XMLHttpRequest();
x.open('GET', url);
x.onloadend = function() {
parent.postMessage('done', '*');
};
x.send();
} + ')(' + JSON.stringify(url) + ');</script>';
var f = document.createElement('iframe');
f.src = 'data:text/html,' + encodeURIComponent(html);
(document.body || document.documentElement).appendChild(f);
window.addEventListener('message', function listener(event) {
// Remove frame upon completion
if (event.source === f.contentWindow) {
window.removeEventListener('message', listener);
f.remove();
}
});
}
Use the chrome.webRequest.onBeforeSendHeaders event to manually append the Origin header.

Firefox sends triple request for the first time load of a page

I am new to NodeJS.
I was practicing the below sample code.
var http = require('http');
var count = 0;
http.createServer(function(request,response){
console.log("request hit "+ ++count);
response.writeHeader(200,{"content-type":"text/plain"});
response.write("Hello World");
console.log("Request served", count);
response.end();
}).listen(8050);
console.log("Server running on 8050");
The output in firefox for the first hit.
request hit 1
Request served 1
request hit 2
Request served 2
request hit 3
Request served 3
On reloading the page, single response is received in Firefox.
Can anyone tell me the reason for it?
My guess is first request, firefox will get the page. One the next request(s) firefox will get the favicon.ico. Next time page is refreshed, firefox has already cached favicon.ico and it won't request for it again, so it will just get the page only.
Second request is probably GET /favicon.ico. Third could be initiated by some plugin. You can figure that out by printing request.method and request.url.
If you take a look to the urls...
server running on 8085>
Count: 1
GET
/
Count: 2
GET
/favicon.ico
Count: 3
GET
/favicon.ico
So seens to be like firefox is asking for /favicon.ico twice and in the second request firefox already cached this url.
The behaviour is similar in google chrome, but chrome just do one request for get / and get /favicon.ico so in chrome you have 2 request.
Is interesting why firefox is doing 2 request for favicon.ico maybe if the first request to /favico.icon fails it try it again.
Seems to be a known issue issue
Thanks for all. With these answers I will include mine which I found.
Firefox sends two request for favicon. The reason is that, Firefox functionality is to consider the favicon of chrome browser too.
I found this in "about.config" page of firefox. In that page, I can see "browser.chrome.favicons" and "browse.shell.shortcutsFavicons". On disabling two favicons, i can now get only one request all the time.

Express request is called twice

To learn node.js I'm creating a small app that get some rss feeds stored in mongoDB, process them and create a single feed (ordered by date) from these ones.
It parses a list of ~50 rss feeds, with ~1000 blog items, so it's quite long to parse the whole, so I put the following req.connection.setTimeout(60*1000); to get a long enough time out to fetch and parse all the feeds.
Everything runs quite fine, but the request is called twice. (I checked with wireshark, I don't think it's about favicon here).
I really don't get it.
You can test yourself here : http://mighty-springs-9162.herokuapp.com/feed/mde/20 (it should create a rss feed with the last 20 articles about "mde").
The code is here: https://github.com/xseignard/rss-unify
And if we focus on the interesting bits :
I have a route defined like this : app.get('/feed/:name/:size?', topics.getFeed);
And the topics.getFeed is like this :
function getFeed(req, res) {
// 1 minute timeout to get enough time for the request to be processed
req.connection.setTimeout(60*1000);
var name = req.params.name;
var callback = function(err, topic) {
// if the topic has been found
if (topic) {
// aggregate the corresponding feeds
rssAggregator.aggregate(topic, function(err, rssFeed) {
if (err) {
res.status(500).send({error: 'Error while creating feed'});
}
else {
res.send(rssFeed);
}
},
req);
}
else {
res.status(404).send({error: 'Topic not found'});
}};
// look for the topic in the db
findTopicByName(name, callback);
}
So nothing fancy, but still, this getFeed function is called twice.
What's wrong there? Any idea?
This annoyed me for a long time. It's most likely the Firebug extension which is sending a duplicate of each GET request in the background. Try turning off Firebug to make sure that's not the issue.
I faced the same issue while using Google Cloud Functions Framework (which uses express to handle requests) on my local machine. Each fetch request (in browser console and within web page) made resulted in two requests to the server. The issue was related to CORS (because I was using different ports), Chrome made a OPTIONS method call before the actual call. Since OPTIONS method was not necessary in my code, I used an if-statement to return an empty response.
if(req.method == "OPTIONS"){
res.set('Access-Control-Allow-Origin', '*');
res.set('Access-Control-Allow-Headers', 'Content-Type');
res.status(204).send('');
}
Spent nearly 3hrs banging my head. Thanks to user105279's answer for hinting this.
If you have favicon on your site, remove it and try again. If your problem resolved, refactor your favicon url
I'm doing more or less the same thing now, and noticed the same thing.
I'm testing my server by entering the api address in chrome like this:
http://127.0.0.1:1337/links/1
my Node.js server is then responding with a json object depending on the id.
I set up a console log in the get method and noticed that when I change the id in the address bar of chrome it sends a request (before hitting enter to actually send the request) and the server accepts another request after I actually hit enter. This happens with and without having the chrome dev console open.
IE 11 doesn't seem to work in the same way but I don't have Firefox installed right now.
Hope that helps someone even if this was a kind of old thread :)
/J
I am to fix with listen.setTimeout and axios.defaults.timeout = 36000000
Node js
var timeout = require('connect-timeout'); //express v4
//in cors putting options response code for 200 and pre flight to false
app.use(cors({ preflightContinue: false, optionsSuccessStatus: 200 }));
//to put this middleaware in final of middleawares
app.use(timeout(36000000)); //10min
app.use((req, res, next) => {
if (!req.timedout) next();
});
var listen = app.listen(3333, () => console.log('running'));
listen.setTimeout(36000000); //10min
React
import axios from 'axios';
axios.defaults.timeout = 36000000;//10min
After of 2 days trying
you might have to increase the timeout even more. I haven't seen the express source but it just sounds on timeout, it retries.
Ensure you give res.send(); The axios call expects a value from the server and hence sends back a call request after 120 seconds.
I had the same issue doing this with Express 4. I believe it has to do with how it resolves request params. The solution is to ensure your params are resolved by for example checking them in an if block:
app.get('/:conversation', (req, res) => {
let url = req.params.conversation;
//Only handle request when params have resolved
if (url) {
res.redirect(301, 'http://'+ url + '.com')
}
})
In my case, my Axios POST requests were received twice by Express, the first one without body, the second one with the correct payload. The same request sent from Postman only received once correctly. It turned out that Express was run on a different port so my requests were cross origin. This caused Chrome to sent a preflight OPTION method request to the same url (the POST url) and my app.all routing in Express processed that one too.
app.all('/api/:cmd', require('./api.js'));
Separating POST from OPTIONS solved the issue:
app.post('/api/:cmd', require('./api.js'));
app.options('/', (req, res) => res.send());
I met the same problem. Then I tried to add return, it didn't work. But it works when I add return res.redirect('/path');
I had the same problem. Then I opened the Chrome dev tools and found out that the favicon.ico was requested from my Express.js application. I needed to fix the way how I registered the middleware.
Screenshot of Chrome dev tools
I also had double requests. In my case it was the forwarding from http to https protocol. You can check if that's the case by looking comparing
req.headers['x-forwarded-proto']
It will either be 'http' or 'https'.
I could fix my issue simply by adjusting the order in which my middlewares trigger.

Resources