how can a path and a host be completely different in nodejs - node.js

I'm doing research in proxies in nodejs. I came across something that blew my mind. In one of the options for a http.request connection, the source code showed this as the options object
const options = {
port: 1337,
host: '127.0.0.1',
method: 'CONNECT',
path: 'www.google.com:80'
};
This was a part of a far bigger code which was the whole tunneling system. But can someone just explain how the options above work? The whole code is below
const http = require('http');
const net = require('net');
const { URL } = require('url');
// Create an HTTP tunneling proxy
const proxy = http.createServer((req, res) => {
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.end('okay');
});
proxy.on('connect', (req, clientSocket, head) => {
// Connect to an origin server
const { port, hostname } = new URL(`http://${req.url}`);
const serverSocket = net.connect(port || 80, hostname, () => {
clientSocket.write('HTTP/1.1 200 Connection Established\r\n' +
'Proxy-agent: Node.js-Proxy\r\n' +
'\r\n');
serverSocket.write(head);
serverSocket.pipe(clientSocket);
clientSocket.pipe(serverSocket);
});
});
// Now that proxy is running
proxy.listen(1337, '127.0.0.1', () => {
// Make a request to a tunneling proxy
const options = {
port: 1337,
host: '127.0.0.1',
method: 'CONNECT',
path: 'www.google.com:80'
};
const req = http.request(options);
req.end();
req.on('connect', (res, socket, head) => {
console.log('got connected!');
// Make a request over an HTTP tunnel
socket.write('GET / HTTP/1.1\r\n' +
'Host: www.google.com:80\r\n' +
'Connection: close\r\n' +
'\r\n');
socket.on('data', (chunk) => {
console.log(chunk.toString());
});
socket.on('end', () => {
proxy.close();
});
});
});
Source: https://nodejs.org/api/http.html#http_event_connect

You probably have never used a network that requires you to configure a HTTP proxy. Most networks these days configure their firewall to allow HTTP traffic. This means most people these days have never needed to use a HTTP proxy to access the web.
A long-long time ago when I first started using the internet (around 1994) a lot of networks don't allow transparent internet access. Your PC does not have any connection to the outside world. But sysadmins would install a HTTP proxy that you can connect to. Your PC would only have access to the LAN (which the proxy is a part of) and only the HTTP proxy would have access to the internet.
Here's an example of how you'd configure Windows to use a HTTP proxy:
If you configure your PC as above, then when you connect to www.google.com your browser would connect to the host proxy.example.com on port 8080 and then request it to fetch data from www.google.com.
As for why it calls the requested resource path it's because it is sent in the "path" part of the packet.
For example, a normal GET request for getting this page looks something like this:
GET /questions/60498963 HTTP/1.1
Host: stackoverflow.com
And the string after GET and before protocol version is normally called the path:
.---------- this is normally called
| the "path"
v
GET /questions/60498963 HTTP/1.1
Host: stackoverflow.com
When making a proxy request the HTTP header looks like this:
CONNECT stackoverflow.com/questions/60498963 HTTP/1.1
So the url including the domain name is sent to the proxy in the part of the packet usually used to send file path.
Note that all this has nothing to do with Node.js. This is just basic networking (no programming languages involved).

Related

How to use two chained socks proxies for a request in nodejs

I am trying to achieve an experimental setup in nodejs as illustrated below:
https-server --> local socks5 proxy (Tor) --> my external socks5 proxy --> webserver.
The existing https-server was written in nodejs and it intercepts requests from the clients firefox browser, modifies the headers, and uses the request module to fetch the requested url provided by the client.
I would like the https request to tunnel through Tor, then through my external proxy (necessary for data collection for my experiments), and then to the webserver.
I found that the socks module has a feature for chaining proxies but it has no Agent to send the https request. The code below works with the chain of proxies and with http and not https.
const SocksClient = require('socks').SocksClient;
const options = {
destination: {
host: 'ip-api.com', // host names are supported with SOCKS v4a and SOCKS v5.
port: 80
},
command: 'connect', // Only the connect command is supported when chaining proxies.
proxies: [ // The chain order is the order in the proxies array, meaning the last proxy will establish a connection to the destination.
{
ipaddress: '127.0.0.1', // ipv4, ipv6, or hostname
port: 9050,
type: 5
},
{
ipaddress: 'my external proxy ip', // ipv4, ipv6, or hostname
port: 1080,
type: 5
}
]
};
var socket = SocksClient.createConnectionChain(options)
.then(info => {
//console.log(info.socket);
console.log(info.socket.remoteAddress);
info.socket.write('GET /json HTTP/1.1\nHost: ip-api.com\n\n');
info.socket.on('data', (data) => {
console.log(data.toString()); // ip-api.com sees that the last proxy in the chain is connected to it.
});
})
.catch(err => {
// Handle errors
console.log(err);
});
Also, there is a module called socks5-https-client that is able to send https requests over a single socks5 proxy (see code below). It uses the request module and the socks5-https-client as an agent, a solution I would prefer. Unfortunately, it does not support proxy chaining (more than 1 proxy).
var Agent = require('socks5-https-client/lib/Agent');
request({
url: 'https://encrypted.google.com/',
strictSSL: true,
agentClass: Agent,
agentOptions: {
socksHost: 'my-tor-proxy-host', // Defaults to 'localhost'.
socksPort: 9050, // Defaults to 1080.
// Optional credentials that I don't need
//socksUsername: 'proxyuser',
//socksPassword: 'p#ssw0rd',
}
}, function(err, res) {
console.log(err || res.body);
});
I am a beginner in node programming but these are my general thoughts to a solution. An https agent can be added to the existing socks module -- the module already supports chaining of proxies -- such that it can be parsed to the request module as an agent, just like in the second code. Alternatively, the socks5-https-client can be modified to have a function like the createConnectionChain() in the socks module to support multiple proxies.
I will be grateful if anyone can assist me with a solution or materials that I can follow to get this done. Thanks

How to connect to an HTTPS website via a HTTP Proxy in Node.js

I created a local HTTP Proxy server for the test, which works fine for my Firefox and Chrome browsers, however, when I tried the following code, the server simply replied a 400 error.
const http = require("http");
const req = http.request({
host: "localhost",
port: 8001,
method: "CONNECT",
path: "sfnjs.com:443"
})
req.end();
req.on("error", err => {
console.log(err);
}).on('connect', (res, socket, head) => {
// Make a request over an HTTP tunnel
socket.write('GET / HTTP/1.1\r\n' +
'Host: sfnjs.com:443\r\n' +
'Connection: close\r\n' +
'\r\n');
socket.on('data', (chunk) => {
console.log(chunk.toString());
});
});
And the server responded
HTTP/1.1 400 Bad Request
Server: nginx/1.16.1
Date: Mon, 22 Jun 2020 03:00:49 GMT
Content-Type: text/html
Content-Length: 255
Connection: close
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.16.1</center>
</body>
</html>
And I wasn't able to use the https module to connect the proxy server since the proxy server only accepts HTTP messages. If I trying so, for example:
const https = require("https");
const req = https.request({
host: "localhost",
port: 8001,
method: "CONNECT",
path: "sfnjs.com:443"
})
The proxy server would simply close the connection and throws an error like this:
Error: Client network socket disconnected before secure TLS connection was established
at connResetException (internal/errors.js:570:14)
at TLSSocket.onConnectEnd (_tls_wrap.js:1361:19)
at Object.onceWrapper (events.js:312:28)
at TLSSocket.emit (events.js:228:7)
at endReadableNT (_stream_readable.js:1185:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21) {
code: 'ECONNRESET',
path: null,
host: 'localhost',
port: 8001,
localAddress: undefined
}
However, if I configure this proxy in Firefox or Chrome and visit the target website, it will work very well, so how does the browser do this and how can I implement it in a Node.js application?
And I wasn't able to use the https module to connect the proxy server since the proxy server only accepts HTTP messages
It sounds like you are using the https module at the wrong place. You should be using the http module to connect to the proxy then the https module where you now are using your own custom code: socket.write('GET / HTTP/1.1\r\n' ...
req.on("error", err => {
console.log(err);
}).on('connect', (res, socket, head) => {
// Make a request over an HTTP tunnel
const req2 = https.request({ // HTTPS
host: 'sfnjs.com:443',
method: 'GET',
createConnection: () => { return socket }, // tunnel
path: '/'
}, res2 => {
res2.on('data', (chunk) => {
console.log(chunk.toString());
});
});
req2.end()
});
Note that the code above is untested but should illustrate the general idea.

How to skip specifying port in Node.js http.request

I am using node.js to connect to NetSuite's REST api. Issue is, I dont know what is port number in NetSuite.. so I want to skip specifying port number. When I dont provide a port number, it takes 80 (default) which is not correct. I have tried port 443 as its https but its not correct port for NetSuite.
I am getting error "getaddrinfo ENOTFOUND". Here is my code.
exports.helloGET = function helloGET (req, res) {
//remove extra node
var json_build = '{"__type" : "salesorder", "status":"Approved"}';
console.log(json_build);
//call NS rest api
var options = {
host: 'rest.na2.netsuite.com',
path: '/app/site/hosting/restlet.nl?script=55&deploy=1',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Keep-alive': 'timeout=30',
'Authorization' : 'NLAuth nlauth_account="4678710", nlauth_email="someemail#domain.com", nlauth_signature="password", nlauth_role="3"'
}
};
var http = require('http');
var post_req = http.request(options, function(res) {
console.log('STATUS: ' + res.statusCode);
console.log('HEADERS: ' + JSON.stringify(res.headers));
res.on('data', function (chunk) {
console.log('BODY: ' + chunk);
});
res.on('end',function(){
console.log('response ended');
});
});
post_req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
//post data
post_req.write(json_build);
post_req.end();
};
It goes inside "post_req.on error".
The error ENOTFOUND typically means that the host you are specifying cannot be found in DNS so that may actually be where your error is. Several possible reasons for this here: What ENOTFOUND error in an http request, happening time to time, means?.
You have to know the port in order to connect. You cannot skip it. A TCP connection requires a destination IP address AND a port number. You can't make a TCP connection (which is what's under an HTTP connection) without the port number.
As you appear to know, the default port for HTTP is port 80 and the default port for HTTPS is 443. With http.request(), it will be port 80 unless you specify the port. With https.request() (note different https module), then it will be 443 if you don't specify the port.
So, the only answer here is for you to find out what port the service is running on that you want to connect to. If it isn't a well-known port known in advance, then there needs to be a way to query what port it's running on from some other known service or configuration data.
Also, if you're trying to connection to an HTTPS service, you need to use the https module, not the http module.
You’re actually not connecting to the correct host.
Use the getdatacenters API to look up the RESTlet host for your account.

How can I use an https proxy with node.js https/request Client?

I need to send my client HTTPS requests through an intranet proxy to a server.
I use both https and request+global-tunnel and neither solutions seem to work.
The similar code with 'http' works. Is there other settings I missed?
The code failed with an error:
REQUEST:
problem with request: tunneling socket could not be established, cause=socket hang up
HTTPS:
events.js:72
throw er; // Unhandled 'error' event
^
Error: socket hang up
at SecurePair.error (tls.js:1011:23)
at EncryptedStream.CryptoStream._done (tls.js:703:22)
at CleartextStream.read [as _read] (tls.js:499:24)
The code is the simple https test.
var http = require("https");
var options = {
host: "proxy.myplace.com",
port: 912,
path: "https://www.google.com",
headers: {
Host: "www.google.com"
}
};
http.get(options, function(res) {
console.log(res);
res.pipe(process.stdout);
});
You probably want to establish a TLS encrypted connection between your node app and target destination through a proxy.
In order to do this you need to send a CONNECT request with the target destination host name and port. The proxy will create a TCP connection to the target host and then simply forwards packs between you and the target destination.
I highly recommend using the request client. This package simplifies the process and handling of making HTTP/S requests.
Example code using request client:
var request = require('request');
request({
url: 'https://www.google.com',
proxy: 'http://97.77.104.22:3128'
}, function (error, response, body) {
if (error) {
console.log(error);
} else {
console.log(response);
}
});
Example code using no external dependencies:
var http = require('http'),
tls = require('tls');
var req = http.request({
host: '97.77.104.22',
port: 3128,
method: 'CONNECT',
path: 'twitter.com:443'
});
req.on('connect', function (res, socket, head) {
var tlsConnection = tls.connect({
host: 'twitter.com',
socket: socket
}, function () {
tlsConnection.write('GET / HTTP/1.1\r\nHost: twitter.com\r\n\r\n');
});
tlsConnection.on('data', function (data) {
console.log(data.toString());
});
});
req.end();

nodejs make a GET call to an https site

i am using nodejs v0.10.28 on ubuntu 12LTS,
am trying to make a request to https enable site behind a web proxy using this test code:
var http = require("http");
var options = {
host: "mycompany.proxy.host",
port: 8088,
//host: "https://login.launchpad.net",
//port: 443,
//path: "/",
headers: {
Host: "login.launchpad.net"
}
};
http.get(options, function(res) {
console.log(res);
res.pipe(process.stdout);
}).on('error', function(e) {
console.log(e);
});
but am not sure what are the correct option params to use to make a successful HTTP GET
You should be using node https instead of http module for making https requests. Otherwise the server will most likely reset your connection, and an ECONNRESET error will be caught by the error handler.

Resources