Get URL Contents in Node.js with Express - node.js

Question

How would I go about downloading the contents of a URL in Node when using the Express framework? Basically, I need to complete the Facebook authentication flow, but I can't do this without GETing their OAuth Token URL.
Normally, in PHP, I'd use Curl, but what is the Node equivalent?

Answer 1

var options = {
host: 'www.google.com',
port: 80,
path: '/index.html'
};
http.get(options, function(res) {
console.log("Got response: " + res.statusCode);
}).on('error', function(e) {
console.log("Got error: " + e.message);
});
http://nodejs.org/docs/v0.4.11/api/http.html#http.get

Answer 2

The problem that you will front is: some webpage loads its contents using JavaScript. Thus, you needs a package, like After-Load which simulates browser's behavior, then gives you the HTML content of that URL .
var afterLoad = require('after-load');
afterLoad('https://google.com', function(html){
console.log(html);
});

Answer 3

Using http way requires way more lines of code for just a simple html page .
Here's an efficient way : Use request
var request = require("request");
request({uri: "http://www.sitepoint.com"},
function(error, response, body) {
console.log(body);
});
});
Here is the doc for request : https://github.com/request/request
2nd Method using fetch with promises :
fetch('https://sitepoint.com')
.then(resp=> resp.text()).then(body => console.log(body)) ;

Answer 4

Using http module:
const http = require('http');
http.get('http://localhost/', (res) => {
let rawHtml = '';
res.on('data', (chunk) => { rawHtml += chunk; });
res.on('end', () => {
try {
console.log(rawHtml);
} catch (e) {
console.error(e.message);
}
});
});
rawHtml - complete html of the page.
I just simplified example from official docs.

Answer 5

using Axios is much simpler
const axios = require("axios").default
const response = axios.get("https://google.com")
console.log(response.data)
or
const axios = require("axios").default
const response = axios.get("https://google.com").then((response)=>{
console.log(response.data)
})
for full docs, you can head over Axios Github