I am trying to use node.js to setup a simple proxy server. The idea behind that is to get all web services calls made to one web service go through a node.js proxy in order to easily inspect and debug web service calls.
In order to do that, I am trying to use the following code to proxy the requests:
var
url = require('url'),
http = require('http'),
acceptor = http.createServer().listen(8008);
acceptor.on('request', function(request, response) {
console.log('request ' + request.url);
request.pause();
var options = url.parse(request.url);
options.headers = request.headers;
options.method = request.method;
options.agent = false;
var connector = http.request(options, function(serverResponse) {
serverResponse.pause();
response.writeHeader(serverResponse.statusCode, serverResponse.headers);
serverResponse.pipe(response);
serverResponse.resume();
});
request.pipe(connector);
request.resume();
});
But I can't figure out where to inspect / dump to file the response. With node-inspector, I was looking at the response object at line: serverResponse.pipe(response); but the body of the response is not yet available.
I found the following question node.js proxied request body but it is written in CoffeeScript.
The idea is write your own 'data' handler and don't use pipe().
You cannot eavesdrop on the data once you piped the stream.
Related
I have a sample http server .
I have a post API which in some scenario have to route to a third party https-server.
Third party server also exposes a post API.
I dont want to send a redirect to client and do this silently on http server.
My application is built using express.
I have tried using request module and tried to use pipe like this .. but request is timing out.
let request = require('request');
console.log(`vishal going here for 300 `);
var pipe = req.pipe(request.post('https url here'));
var response = [];
pipe.on('data', function (chunk) {
response.push(chunk);
});
pipe.on('end', function () {
var res2 = Buffer.concat(response);
console.log(res2);
res.send(res2);
});
Not sure whats missing.
Also how to pass the body and hears to https server request
There is a node server hosted and from application we also make a request to some external api over http. This external service can process 10request/sec. Application is behind Nginx which has timeout 30 secs.
Now let say we put load of 10k request on nodejs app server. Since we do have dependency on external api which can process max 10*30 request in 30 secs. Only 300 request will be served and remaining will be terminated by Nginx. But still this 10k request got queued into https.globalAgent.requests queue and keep running. There is no way to specify sockettimeout or limit the size of the request queue. Further call to application will eventually be queued up for external service and later will be terminated by Nginx.
So questions are :
Is there any way we can set socketTimeOut?
Is there any way we can limit the size of the queue?.
Any workaround ?
Sample Code
var http = require('http');
var https = require('https');
var Q = require('q');
var file = require('fs');
var request = require('request');
http.globalAgent.maxSockets = 10;
https.globalAgent.maxSockets = 10;
https.globalAgent.keepAlive=true;
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
var st = new Date();
var rr = request(
{url:'https://amazon.com',timeout:100000,time:true},
function(err,resp,body){
var et = new Date();
// console.log( resp && resp.timings.socket,st-et,err);
console.log(https.globalAgent)
res.end('ok');
}
);
}).listen(9898);
I'll suggest you to use a rate limiting library like https://github.com/jhurliman/node-rate-limiter
In case of an exceded limit, you can answer immediately to the requests, with a 429 error code (https://httpstatuses.com/429) or a ko error message
or you can wait until you have an available request slot
I'm reading through portions of the Twilio documentation (https://www.twilio.com/help/faq/why-does-my-twilio-number-respond-with-thanks-for-the-message-configure-your-numbers-sms-url-to-change-this-message) pertinent to SMS messaging responses, and am trying to build a node.js app which will allow me to respond to inbound SMS messages with programmatic responses.
I've been trying to emulate this SO post which deals with a similar problem (How can I respond to incoming Twilio calls and SMS messages using node.js?) and have the following code:
var AUTH_TOKEN = "*********************";
var twilio = require('twilio');
var express = require('express');
var http = require('http');
var app = express();
var bodyParser = require('body-parser');
app.use(bodyParser.urlencoded());
app.post('/welcome/sms/reply/', function(req, res) {
//Validate that this request really came from Twilio...
if (twilio.validateExpressRequest(req, AUTH_TOKEN)) {
var twiml = new twilio.TwimlResponse();
twiml.say('Hi! Thanks for checking out my app!');
res.type('text/xml');
res.send(twiml.toString());
}
else {
res.send('you are not twilio. Buzz off.');
}
});
http.createServer(app).listen(3000);
Calling the POST request /welcome/sms/reply through a REST client yields the else statement, and I'm not sure why since the AUTH_TOKEN is exactly what I have in my account dashboard.
Twilio developer evangelist here.
If you're trying to call your own endpoint there using a REST client and you are validating requests (twilio.validateExpressRequest) then you will need to construct your request the same as Twilio does. Crucially this includes a X-Twilio-Signature header, read more at that link for more details.
If you test your code with Twilio, it should work and be a valid request.
See this post for reference in incorporating ngrok + Express. https://www.twilio.com/blog/2015/09/monitoring-call-progress-events-with-node-js-and-express.html
I'm currently experimenting with NodeJS streams and had a deeper look into the http.ServerResponse object that's being created upon every http request of the http.createServer request handler.
What I'm now trying to do is having the exact same API of the http.ServerResponse object in a different process connected via another arbitrary method (for instance using Streams) and pipe all output of this object (including headers!) to the actual request, like the following:
-[http]-> server1 -[stream]-> server2 -[stream]-> server1 -[http]->
I've tried a couple of variants, like the following (only local) example:
var http = require('http');
var net = require('net');
var through = require('through');
var StreamWrap = require('_stream_wrap').StreamWrap;
http.createServer(function(req, _res) {
var resStream = through();
resStream.pipe(_res.socket);
resStream.__proto__.cork = function() {};
resStream.__proto__.uncork = function() {};
var resSocket = new StreamWrap(resStream);
var res = new http.ServerResponse(req);
res.assignSocket(resSocket);
res.writeHead(201, {'Content-Type': 'text/plain'});
res.write('hello');
res.end();
}).listen(8000, function() {
console.log("Server listening");
});
which should essentially send the raw HTTP message to the underlying socket (or not?), but for some reason I'm getting a Segmentation fault: 11 - I'm also not sure whether I'm using the StreamWrap object correctly. But I think that this is not an optimal solution as the ServerResponse object handles a lot of things with regards to the socket internally.
So what would be the best way to tackle this?
If I'm piping directly to the response object, this results only in writing to the HTTP body (see also this answer), and I'm loosing the header information. So would it be best to separate header and body and then transfer them separately? How do I know when the headers are being set final on the ServerResponse object, and when the data starts (apart from other options such as trailing headers)?
Another option would be to remotely call methods on the ServerResponse Object i.e. via dnode, but isn't there a better way to do this? I would really love to be able to use Express for instance on the remotely connected server, that's why I want to keep the http.ServerResponse API.
I'm glad for any input!
Thanks!
I want to print a message whenever the file that I am watching has changed. I am able to do that using console.log but I couldn't figure out how to do that using response.write or similar functions.
var counter = 0;
const
http = require('http'),
fs = require('fs'),
filename = process.argv[2];
var server = http.createServer(function(request, response) {
response.writeHead(200, { 'Content-Type' : 'text/plain' });
counter = counter + 1;
response.end('Hello client ' + Math.round(counter / 2));
});
server.on('listening', function() {
var watcher = fs.watch(filename, function(){
console.log('The file ' + filename + ' has just changed.');
});
});
server.listen(8080);
Also, the reason why I have done that Math.round(counter / 2) is because counter is increasing by 2 each time a client is connected. I was wondering why this is happening and if there is a better technique to resolve this.
For you to be able to do it using response.write it would need to be part of your server request handler function.
File events can occur independently of someone sending a request, so it's handling is independent to that of handling a request. Because of this, there is no associated request for you to write to.
If you want to keep track of all the file change events and then show it to the user whenever they do send a request, consider storing the information about changes in an object outside your handler functions and when the a request takes place, read that object to see if there have been changes and write a response based on it to the user.
If you want to inform an end user that the file has change, for example in a web browser, then you have a number of options, including polling the server or using websockets.
I would recommend you take a look at Server Sent Events
It is easy to implement and there are npm module out there to make it even easier in node.js. e.g. npm sse
You can try node module chokidar
https://github.com/paulmillr/chokidar
it is a gread module for file watching