Why does http.get take so long in NodeJS? - node.js

Question

Using NodeJS and http.get, I am trying to see if a website uses a redirect. I tried a few URLs which all worked great. However, when I ran the code with washingtonpost.com it took over 5 seconds. In my browser the website works just fine. What could be the issue?
console.time("Done. Script executed in");
const http = require("http");
function checkRedirectHttp(input){
return new Promise((resolve) => {
http.get(input, {method: 'HEAD'}, (res) => { resolve([res.headers.location, res.statusCode]) })
.on('error', (e) => { throw {Error: `Cannot reach website ${input}`} });
});
};
checkRedirectHttp("http://www.washingtonpost.com/").then(result => {
console.log(result);
console.timeEnd("Done. Script executed in");
})
Output:
[
'http://www.washingtonpost.com/gdpr-consent/?next_url=https%3a%2f%2fwww.washingtonpost.com%2f',
302
]
Done. Script executed in: 8.101s

Answer 1

I ran your code, enhanced it some and slowly added back the actual headers that are sent from my browser when I go to the same link in the browser. When I changed the request to a "GET" (no longer a "HEAD") and added the following headers from my browser:
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-US,en;q=0.9",
"cookie": "a very long cookie here"
then the response went from 9 seconds to 71ms.
So, apparently the server doesn't like the HEAD request and doesn't like that a bunch of headers it expects to be there are missing. Probably, it is detecting that this isn't a real browser and it's either analyzing something for 8 seconds or it's just purposely delaying a response to a "fake client".
Also, if you use the http://www.washingtonpost.com URL instead of https://www.washingtonpost.com, it redirects to https every time for me. So, you may as well just start with the https:// form of the URL.