How to get HTML source of HTTPS website in Node - node.js

Question

I have this following code snippet which works with Google, but I noticed that trying to reach websites like Amazon which force HTTPS will throw an error 301 (permanently moved). I think the problem may be that I’m using the http package, but the HTTPS package confuses me. If anyone could help me out, that would be stupendous.
var vars = {
host: “www.google.com”,
port: 80,
path: “/index.html”
}
http.get(vars, function(res) {
console.log(res.statusCode);
res.setEncoding(“utf8”);
res.on(“data”, function(data) {
console.log(data);
}
})

Answer 1

You can just use https.get(). But, for https, you have to use a different port (443). I prefer to just pass in the URL and let the library handle the default port for me:
const https = require('https');
https.get("https://www.google.com/index.html", function(res) {
console.log(res.statusCode);
res.setEncoding('utf8');
res.on('data', function(data) {
console.log(data);
});
}).on('error', function(err) {
console.log(err);
});
This may return the data in multiple data events so if you want all the data, you'd have to manually combine all the data.
Personally, I prefer to use a higher level library that is promise-based and makes lots of things simpler:
const got = require('got');
got("https://www.google.com/index.html").then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
Among many other features, the got() library will automatically collect the whole response for you, uses promises, will follow redirects, will automatically parse JSON results, will check the status and provide an error for 4xx and 5xx statuses, supports lots of authentication means, etc... It's just easier to use than the plain http/https libraries.