Here is a simple nodejs script for using node as an https client to interact with the outside world. How do you modify it so you don't get "ECONNREFUSED" errors when you run it behind a corporate proxy like http://10.10.10.10:8080 ?
var https = require('https');
var options = {
hostname: 'encrypted.google.com',
port: 443,
path: '/',
method: 'GET'
};
var req = https.request(options, function(res) {
console.log("statusCode: ", res.statusCode);
console.log("headers: ", res.headers);
res.on('data', function(d) {
process.stdout.write(d);
});
});
req.end();
req.on('error', function(e) {
console.error(e);
});
----Update---
I figured out how to with the npm 'request' package (https://github.com/mikeal/request). Still wondering how to do this with vanilla node libraries.
var request = require('request').defaults({proxy:'http://my.proxy.server:8080', agent:false});
request('https://www.google.com', function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body) // Print the google web page.
}
})
Normally its best to handle proxying at the system level - configure your operating system to use the proxy not the script. However this may also fail, as most corporate proxies don't have all the ports open.
A universal solution would be to have a server (EC2 or Rackspace instance) to act as your own proxy and have it listen for ssh on port 443.Port 443 is open in corporate firewalls for https.
At which point you can either tunnel all you local network traffic to the server, or ssh to the server and run the script there.
If you were to make scripts proxy specific, they wouldn't be very portable, and not production hardened as they'd break when the proxy configuration changed.
Related
I'm doing research in proxies in nodejs. I came across something that blew my mind. In one of the options for a http.request connection, the source code showed this as the options object
const options = {
port: 1337,
host: '127.0.0.1',
method: 'CONNECT',
path: 'www.google.com:80'
};
This was a part of a far bigger code which was the whole tunneling system. But can someone just explain how the options above work? The whole code is below
const http = require('http');
const net = require('net');
const { URL } = require('url');
// Create an HTTP tunneling proxy
const proxy = http.createServer((req, res) => {
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.end('okay');
});
proxy.on('connect', (req, clientSocket, head) => {
// Connect to an origin server
const { port, hostname } = new URL(`http://${req.url}`);
const serverSocket = net.connect(port || 80, hostname, () => {
clientSocket.write('HTTP/1.1 200 Connection Established\r\n' +
'Proxy-agent: Node.js-Proxy\r\n' +
'\r\n');
serverSocket.write(head);
serverSocket.pipe(clientSocket);
clientSocket.pipe(serverSocket);
});
});
// Now that proxy is running
proxy.listen(1337, '127.0.0.1', () => {
// Make a request to a tunneling proxy
const options = {
port: 1337,
host: '127.0.0.1',
method: 'CONNECT',
path: 'www.google.com:80'
};
const req = http.request(options);
req.end();
req.on('connect', (res, socket, head) => {
console.log('got connected!');
// Make a request over an HTTP tunnel
socket.write('GET / HTTP/1.1\r\n' +
'Host: www.google.com:80\r\n' +
'Connection: close\r\n' +
'\r\n');
socket.on('data', (chunk) => {
console.log(chunk.toString());
});
socket.on('end', () => {
proxy.close();
});
});
});
Source: https://nodejs.org/api/http.html#http_event_connect
You probably have never used a network that requires you to configure a HTTP proxy. Most networks these days configure their firewall to allow HTTP traffic. This means most people these days have never needed to use a HTTP proxy to access the web.
A long-long time ago when I first started using the internet (around 1994) a lot of networks don't allow transparent internet access. Your PC does not have any connection to the outside world. But sysadmins would install a HTTP proxy that you can connect to. Your PC would only have access to the LAN (which the proxy is a part of) and only the HTTP proxy would have access to the internet.
Here's an example of how you'd configure Windows to use a HTTP proxy:
If you configure your PC as above, then when you connect to www.google.com your browser would connect to the host proxy.example.com on port 8080 and then request it to fetch data from www.google.com.
As for why it calls the requested resource path it's because it is sent in the "path" part of the packet.
For example, a normal GET request for getting this page looks something like this:
GET /questions/60498963 HTTP/1.1
Host: stackoverflow.com
And the string after GET and before protocol version is normally called the path:
.---------- this is normally called
| the "path"
v
GET /questions/60498963 HTTP/1.1
Host: stackoverflow.com
When making a proxy request the HTTP header looks like this:
CONNECT stackoverflow.com/questions/60498963 HTTP/1.1
So the url including the domain name is sent to the proxy in the part of the packet usually used to send file path.
Note that all this has nothing to do with Node.js. This is just basic networking (no programming languages involved).
I have some back-end services which can be called only through an API Gateway server. the back-end services are in Spring rest service and API GateWay is a Node Server .these two servers are running locally on different ports(backend:8080 , node :3000).
How can i make a request to back-end service from my node Server ?
If they both expose rest API you can make use of the inbuilt- http module for communication
require('http');
var options = {
host: 'www.google.com',
port: 80,
path: '/index.html'
};
http.get(options, function(res) {
console.log("Got response: " + res.statusCode);
res.on("data", function(chunk) {
console.log("BODY: " + chunk);
});
}).on('error', function(e) {
console.log("Got error: " + e.message);
});
But i would recommend using libraries like superagent or axios
At a high level, I am trying to use Quota Guard Static to talk to an IP limited API from a Heroku app from a node.js application. The API has its own node.js client implementation, but underneath the covers it's just an HTTP[S] api. The library uses superagent and superagent-proxy underneath the covers to do the actual HTTP[S] requests.
In node 0.10, all worked fine. In node 0.12, I see errors like:
Error: write EPROTO 140735203734288:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:782:
at exports._errnoException (util.js:746:11)
at WriteWrap.afterWrite (net.js:775:14)
In io.js 2.02 I see:
Error: write EPROTO at Object.exports._errnoException (util.js:844:11)
I tried globally using SSLv3 as shown in this answer, but it seemed to have no effect.
The proxy url is specified as an http URL with port 9293. This answer suggested using port 443, but since the proxy provider is external to me, I cannot change it.
How might I get the proxied request to work in node 0.12?
Tim here from QuotaGuard. This seems to be an issue manifesting itself in the https-proxy-agent used by superagent-proxy for HTTPS requests causing the request to be made to the secure endpoint on the wrong port.
This is a simple example that should connect to Google on port 443.
var url = require('url');
var https = require('https');
var HttpsProxyAgent = require('https-proxy-agent');
// HTTP/HTTPS proxy to connect to
var proxy = process.env.QUOTAGUARDSTATIC_URL;
console.log('using proxy server %j', proxy);
// HTTPS endpoint for the proxy to connect to
var endpoint = process.argv[2] || 'https://www.google.com/';
console.log('attempting to GET %j', endpoint);
var opts = url.parse(endpoint);
// create an instance of the `HttpsProxyAgent` class with the proxy server information
var agent = new HttpsProxyAgent(proxy);
opts.agent = agent;
https.get(opts, function (res) {
console.log('"response" event!', res.headers);
res.pipe(process.stdout);
});
The actual request is being made on port 80 so Google is rejecting it. Here are the HTTP headers:
["Proxy-Authorization: Basic Xgat28sa78saBZZ \r\n", "Host: www.google.com:80\r\n", "Connection: close\r\n"]
The same example on a patched version correctly connects to port 443 and works:
https://github.com/TooTallNate/node-https-proxy-agent/compare/master...timrwilliams:master
I suspect something has changed upstream which is causing the wrong port to be passed to https-proxy-agent but this type of problem is more appropriately discussed on Github issues.
A quick fix would be switching to use the request library instead:
var request = require('request');
var options = {
proxy: process.env.QUOTAGUARDSTATIC_URL,
url: 'https://www.google.com/',
headers: {
'User-Agent': 'node.js'
}
};
function callback(error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body);
}
}
request(options, callback);
I am working on setting up an aws instance to listen for http POST's. When running the server and client both as local host, everything seems to work fine. However, when running trying to make a post with the client to an aws instance with the server running, I am getting a connect ECONNREFUSED error.
The aws instance (ubuntu server) that I am using has both ports 80 and 8080 open to all ip addresses. I am using the pm2 module to keep the server running. Although using pm2 is giving the same error.
Server Setup: (aws instance terminal)
$ sudo apt-get install git
$ curl -sL https://deb.nodesource.com/setup | sudo bash -
$ sudo apt-get install -y nodejs
$ sudo npm install pm2 -g --unsafe-perm
Using node to start server:
$ node nodeServerTest.js
Using pm2 to start server:
$ pm2 start nodeServerTest.js --name "nodeServerTest" -i max
Server Code:
// nodeServerTest.js
var http = require("http");
function startServer(port, ip) {
// requestListener handles incoming requests
function requestListener(request, response) {
if (request.method == 'POST') {
var body = '';
request.on('data', function (data) {
body += data;
// destroys connection if too long
if (body.length > 1e6) {
request.connection.destroy();
}
});
request.on('end', function() {
// checks for json, otherwise destroys connection
try {
var POST = JSON.parse(body);
console.log('valid post:')
console.log(POST)
response.end('post accepted');
}
catch(err) {
console.log('bad post, ending connection')
response.connection.destroy();
}
});
}
else {
response.end();
}
};
// creates the eventEmitter
var server = http.createServer(requestListener);
// beigns listening on specified port
server.listen(port, ip);
// logs when someone attempts to connect
server.on('connection', function (stream) {
console.log('someone connected!');
});
// notifies when server has started
console.log('Server running at http://'+ip+':'+port);
}
startServer(8080, '127.0.0.1');
Client Code: (I change the host field to the ip address when attempting to post to the aws instance)
// nodeClientTest.js
function httpPost() {
var http = require("http");
var postData = JSON.stringify({
'msg': 'Hello World!'
});
var options = {
host: "localhost", // using correct public ip
port: 8080,
method: 'POST',
path: '/',
headers: {
'Content-Type': 'application/json',
'Content-Length': postData.length
}
};
var req = http.request(options, function(res) {
console.log('STATUS: ' + res.statusCode);
console.log('HEADERS: ' + JSON.stringify(res.headers));
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log('BODY: ' + chunk);
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
// write data to request body
req.write(postData);
req.end();
}
httpPost();
Thank you for any help, if more information is needed please leave a comment
You are binding your server to the loopback interface here:
startServer(8080, '127.0.0.1');
..which makes is available only locally. To allow access via your eth0 interface, so from other network hosts replace this line with:
startServer(8080);
..to listen on port 8080 TCP on all interfaces, according to Node.js manual.
PS Please remember that your private IP on AWS EC2 != public IP address. You can check both by running ec2metadata command on your EC2 host.
An explanation, as OP requested:
each TCP server socket has to be bound to a specific network interface. This is because you make you connect your client TCP socket to a combination of IP:port and the IP is bound to the interface.
loopback interface with a 127.0.0.1 IP address is as special virtual interface available only from the localhost, targeting ifself (hence the name),
So when you run your server binding it to loopback the only way you could have only made the request to it would be initiating the connection from that host itself, for example with telnet like this: telnet 127.0.0.1 8080.
You could have bind the server to the actual IP of eth0 interface but this is inpractical, especially on EC2 servers where private IPs change.
That's why I proposed the simpler, universal syntax. A side effect is that this way your server listens on both loopback and eth0 but it only helps, for example when you want to separate your own, local traffic from the rest of the traffic by based on the interface used.
It seems like your server is only binding to the local address of the server (127.0.0.1) and not listening to connections outside of the server.
Please try to change the listening address of the server to 0.0.0.0
startServer(8080, '0.0.0.0');
I would also advise you to check the express web framework for your server.
i am using nodejs v0.10.28 on ubuntu 12LTS,
am trying to make a request to https enable site behind a web proxy using this test code:
var http = require("http");
var options = {
host: "mycompany.proxy.host",
port: 8088,
//host: "https://login.launchpad.net",
//port: 443,
//path: "/",
headers: {
Host: "login.launchpad.net"
}
};
http.get(options, function(res) {
console.log(res);
res.pipe(process.stdout);
}).on('error', function(e) {
console.log(e);
});
but am not sure what are the correct option params to use to make a successful HTTP GET
You should be using node https instead of http module for making https requests. Otherwise the server will most likely reset your connection, and an ECONNRESET error will be caught by the error handler.