The initial problem is nginx proxy_pass with a socks5 proxy?
I decided to proxy_pass to localhost with node js server. So then I need to somehow convert http/https requests to socksv5.
I have very simple node js server let's it handle simple get and post requests like
const express = require('express');
const app = express();
app.get('/', function(req,res) {
// need to forward via socks5
});
app.post('/', function(req,res) {
// need to forward via socks5
});
app.listen(3000,() => {
console.log("Started on PORT 3000");
})
There are some libraries like https://www.npmjs.com/package/socks or https://github.com/mattcg/socks5-https-client but it can only do get requests and also without handle 301 redirect response.
It seems that it's very easy and simple task but I still can not find the turnkey solution. Is there some or at least some good libraries to convert post and get (put, delete) to socks5?
Related
I am trying to create and express server that reverse proxies to multiple applications. What I am running in to is that when I go to one of the routes the request never makes it to the server. Here are some code snippets of what I am doing:
let app = express();
app.use('/app2', express.static(__dirname + '/build', {
setHeaders: (res, req, path) => {
console.log(path);
metrics.httpRequestDurationMicroseconds // handles http_request_duration_ms Duration of HTTP requests in ms
.labels(req, path, res.statusCode)
.observe(10)
}
}));
When I go to localhost:5000/app2 it loads fine. Here is my proxy server:
proxyApp.use('/app2', proxy('http://localhost:5000/app2' ) );
proxyApp.use('/', proxy('http://my-site.com/'));
I am running it on port 5001. When i go to localhost:5001 my-site.com loads as expected. When I go to localhost:5001/app2 I get nothing and i see no traffic on the server.
For a little more context, I originally had app2 being served at / instead of /app2 and then I was able to make my proxy server load app2. But when I changed app2 to serve static content at /app2 it started breaking.
Anyone have any ideas on how to make this work or what is going on? It looks like the proxy wants to always send requests to / instead of /app2 no matter what I put in.
I am using express-http-proxy:1.4.0 and express:4.16.4
Any help is appreciated.
Try to use proxyReqPathResolver option
app.use('/app2', proxy("http://localhost:5000", {
proxyReqPathResolver: function (req) {
return "/app2"
},
}
I have a node express server responding to http traffic:
const http = require("http");
const express = require("express");
const app = express();
const server = http.createServer(app);
app.use(function(req,res,next){
console.log(`logging: req: ${util.inspect(req)}`);
next();
});
and all that works fine. I'd like to have a program on my node server inject emulated http traffic into the express stack, without a network connection. I can't just magic up a (req,res) pair and call a middleware function like the one in app.use above, because I don't have a next to give it, and my req and res will not be the ones next passes on to the next middleware in the stack.
Edit: What I actually have is a websocket connection sending data packets of a different format, different data contents from http traffic that can also carry the same information. I can take those websocket packets and build from those a request that is in the same format that the http traffic uses. I would like to pass that transformed request through the express http middleware stack and have it processed in the same way. Going all the way back to create an http request having just dealt with a ws request seems a bit far.
What's the simplest way to emulate some traffic, please? Can I call a function on app? Call some express middleware, or write a middleware of my own to inject traffic? Call a function on server?
Thanks!
Emulation traffic by calling some Express.js internal functions isn't the right way. Much easier is to trigger the server by HTTP request from the same process
const http = require('http');
const util = require('util');
const express = require('express');
const app = express();
const server = http.createServer(app);
app.use(function(req, res, next) {
console.log(`logging: req: ${util.inspect(req)}`);
next();
});
const port = 8081;
server.listen(port);
http.request({ port }).end();
From your question
I'd like to have a program on my node server inject emulated http traffic into the express stack, without a network connection
Can you clarify, why without a network connection?
A few things:
You need to make an endpoint
You need to host your server somewhere
You need something to send requests to your server
Express provides you a way to receive requests (req, res) (might be from a browser, might not be), perform some operations, and return responses (req, res) to the requester.
The expression
app.use(function(req,res,next){
console.log(`logging: req: ${util.inspect(req)}`);
next();
});
is actually a middleware function. This will take every request to your server and change the request object created by express into a string, and print it in your server log.
If you want a testable endpoint, you would add this to the bottom of the snippet you posted
app.get('/test', function (req, res) {
res.json({success:true})
})
This tells your app to allow GET requests at the endpoint /test
Next you're going to need to host your express server somewhere you can send requests to it. Your local machine (localhost) is a good place to do that. That way, you don't need an internet connection.
Pick a port you want to host the server on, and then it will be reachable at http://localhost:<Your Port>.
Something like this will host a server on http://localhost:3000. Add this below the route we declared above:
server.listen(3000, function() {
console.log('Server running on port 3000');
});
Finally, you'll need a way to send requests to the server on localhost. Postman is a great tool for testing express routes.
I would suggest installing Postman and using that to emulate http traffic.
Once your server is running, open postman and send a GET request to your server by entering the server address and port, and hitting the blue send button (You'll be sending the request to http://localhost:3000/test).
Here's an image of what postman should look like if all goes well
You should also see your middleware fire and print out the request object in your terminal.
Good Luck!
Basically what happened was we have an app server that is running express and routes to a bunch of SPAs. This was great but then we wanted to have an app that runs its own node/express script (ghost). I can't figure out how to set the route /ghost to go to ./webapps/ghost/index.js
Is this just not possible?
You need to redirect incoming requests to the ghost express instance. I have done so in my personal site by adding a /blog route to my primary express instance and forwarding any request to it to the ghost expresss instance. Check it out here: https://github.com/evanshortiss/evanshortiss.com/blob/master/server.js
The basic gist is that you do the following:
app.use('/blog', function(req, res, next) {
// Forward this request on...
return next();
}, ghostServer.rootApp); //...but we forward it to a different express instance
If you're running both as separate processes then you could use Apache or nginx to just redirect the requests. If you absolutely must use an express application to forward requests then try the node-http-proxy module.
If you need to proxy from express you could do this using the http-proxy module by Nodejitsu:
var proxy = require('http-proxy').createProxyServer({});
app.use('/blog', function (req, res) {
// You may need to edit req.url (or similar) to strip the /blog part of the url or ghost might not recognise it
proxy.web(req, res, {
target: 'http://127.0.0.1:'+GHOST_PORT
});
});
I wrote an express app as an HTTP proxy, to intercept and analyse some of the network traffic. The parts of traffic my app is interested in are all HTTP, however I still want my app to proxy HTTPS so users can use it without extra setting.
My express app is created with a HTTP server. When testing, I changed the proxy setting in Chrome with SwitchyOmega, to proxy HTTPS connections with HTTP. HTTP works well, But my express app couldn't get these proxy requests for HTTPS.
So I wrote a simple TCP proxy to check on them, and find that they're like this:
CONNECT HOSTNAME:443 HTTP/1.1
Host: HOSTNAME
Proxy-Connection: keep-alive
User-Agent: MY_AGENT
ENCRYPTED HTTPS
I believe these requests are HTTP, but why express isn't receiving them?
For sure if I change the browser proxy setting to ignore HTTPS, the app works well. But I do want to know if there is any workaround that I can use to proxy all protocols with HTTP and only one port.
THX.
UPDATE- code from my express app
app.use('*', function (req, res, next) {
// print all the request the app receive
console.log('received:', req.url)
})
app.use(bodyParser.text({type: '*/*'}))
app.use(cookieParser())
app.use(logger('dev'))
app.use(express.static(path.join(__dirname, 'public')))
// serve web pages for my app, only the request targeting my server
// is handled here(right IP and port), proxy request gets handled after this.
app.use('/', internalRoute)
// analyse the part I want
app.use('/END_POINT_I_WANT', myRoute)
// handle proxy requests
app.use('*', function (req, res, next) {
// proxy the request here
})
The problem is, my first middleware, which is used to display all the requests the app receive, can't catch the HTTPS proxy requests wrapped in HTTP described above. And of course the middleware I used as proxy can't catch them either.
UPDATE-tried node-http-prxoy, no luck
var httpProxy = require('http-proxy')
, http = require('http')
, fs = require('fs')
var options = {target: 'http://127.0.0.1:8099'}
, proxy = httpProxy.createServer(options)
http.createServer(function (req, res) {
console.log(req.url)
proxy.web(req, res)
}).listen(5050)
With the above code, and browser setting to proxy all protocols with HTTP, it works the same as my express app. HTTPS proxy requests gets ERR_EMPTY_RESPONSE, and nothing on the console.
With the below options, it seems that I have to change the proxy protocol to HTTPS, which I'd rather not use, at least for now. And I get ERR_PROXY_CERTIFICATE_INVALID for my self-signed certs...
var options = { secure: true
, target: 'http://127.0.0.1:8099'
, ssl: { key: fs.readFileSync('cert/key.pem', 'utf8')
, cert: fs.readFileSync('cert/server.crt', 'utf8')
}
}
UPDATE- pin point the problem to the 'connect' event listener
Through some searching, I found this post helpful.
It pointed out that the http server doesn't have a listener for the connect event. I tried the code in the post, works. But as the last comment of that post mentioned, my app serves as a proxy in order to get the data, it then proxy the request to another proxy in order to go over the GreatFireWall.
The process is like : BROWSER -> MY_APP -> ANOTHER_PROXY -> TARGET.
Without the ANOTHER_PROXY, which is an HTTP proxy, it works well for both HTTP and HTTPS. However I failed to chain them all up. The ANOTHER_PROXY I use supports HTTPS over HTTP.
It's hard to see what might be wrong, since you haven't posted any code.
However, if you just want to create a simple proxy that supports HTTP and HTTPS, i think that you should consider using a module like node-http-proxy.
Their readme has example code for the most common scenarios, and it sounds like it will support your needs fine.
I have Moto Adverts application in angularjs and nodejs.
Angularjs-client-side is running on Apache HTTP Server (localhost:8000) but nodejs-server-side is runnning as node.js http server (localhost:3000).
Piece of client-side code (angularjs):
var motoAdsServices = angular.module('motoAdsServices', ['ngResource']);
motoAdsServices.factory('Brand', ['$resource', function($resource) {
return $resource('http://localhost\\:3000/api/:id', {}, {
query: {
method: 'GET',
params: {
id: 'brands'
},
isArray: true
}
});
}]);
Piece of server-side code (nodejs):
var express = require('express');
var path = require('path');
var http = require('http');
var brands = require('./routes/brands');
var app = express();
var allowCrossDomain = function(req, res, next) {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "X-Requested-With");
next();
};
app.configure(function() {
app.set('port', process.env.PORT || 3000);
app.use(express.logger('dev')); /* 'default', 'short', 'tiny', 'dev' */
app.use(express.bodyParser()),
app.use(allowCrossDomain);
app.use(express.static(path.join(__dirname, 'public')));
});
app.get('/api/brands', brands.findAll);
http.createServer(app).listen(app.get('port'), function() {
console.log("Express server listening on port " + app.get('port'));
});
My questions are:
What I should do to run client-side and server-side on the same server.
a) On Apache HTTP Server (localhost:8000).
b) On Node.js self http server on (localhost:3000).
What architecture will be the best for production use - two independent servers for client-side and server-side or only one?
Is it good practise to use Cross-origin resource sharing (CORS) on server-side (if I should hava two independed servers)?
What I should do to not hard code address http://localhost:3000/api/brands to server-side (best practise)?
Node.js
One server will be more maintainable. To optimize you can cache static files with nginx later if you need to. (Just search 'nginx Node.js static' but it will work fine without that if you have light traffic and not a ton of static files). I would not use Apache for anything.
Here is a sample nginx config:
server {
listen 80;
server_name myserver.net;
root /mnt/app;
index index.html index.htm;
location /static/ {
try_files $uri $uri/ =404;
}
location /api/ {
proxy_pass http://127.0.0.1:8080;
}
}
You won't need CORS with one.
The ports will be the same so that isn't an issue.
I believe this is the most popular architecture for apache nodejs angularjs.
(A) in the figure.
I recommend for you to serve all files including static files via nodejs server as I wrote in my figure.
On the other hand, you could use node server only for dynamic contents and use apache to serve static files including your client side codes if you like.
But if you do so, you need apache server ALWAYS even when you develop your application. I feel that will be troublesome.
(B) in the figure.
You can serve your client side codes including other static files by locating them in public directory.
If you decide to serve all files from nodejs server, you can develop without apache and avoid to write specific port number in your code.
I think your application code should not be conscious about the port number you will use.
I hope this could be answer for your all questions.
What I should do to run client-side and server-side on the same server. a) On Apache HTTP Server (localhost:8000). b) On Node.js self http server on (localhost:3000).
Ans : you don't need to run nodejs as self hosted.instead run nodejs through Apache server and use fusion passenger too. fusion passenger will take care of running your node application in background forever. Personally I prefer Nginx + fusion for my nodejs applications.
What architecture will be the best for production use - two independent servers for client-side and server-side or only one?
Ans : I don't understand what you mean by having two servers one for client and one for server-side. Keep your client and server code on single server.
Is it good practice to use Cross-origin resource sharing (CORS) on server-side (if I should have two independent servers)?
Ans : if your server and your client are under same domain then you don't need to worry about CORS but if in future you want to expose your API to any of your client apps, then you will need to do CORS configurations.
What I should do to not hard code address http://localhost:3000/api/brands to server-side (best practice)?
Ans : I use constant to declare my base path and then do the DI in my services and factory that make the API Calls.
motoAdsServices.constant("API", {
"endpoint": "http://localhost:8080",
})
motoAdsServices.factory('Brand', ['$resource','API', function($resource,API) {
return $resource(API.endpoint + '/api/:id', {}, {
query: {
method: 'GET',
params: {
id: 'brands'
},
isArray: true
}
});
}]);