I'm debugging calls made from my express app to another micro-service on my network. I'm receiving 401 errors and I need to get full raw http logs to give to my security team for analysis.
I'm looking for some advice on tracking HTTP calls from a micro-service I have deployed on Pivotal Cloud Foundry. I've been doing some research and ran across tools like Zipkin and OpenTracing etc.. but those appear to be more about debugging latency and probably do not show HTTP logs. I've also tried using Morgan/Winston modules but they do not track internal calls. Morgan is currently what I'm using to log out the basic HTTP codes but it doesn't pick up on my calls from inside my app either, just the ones made to the app itself from the browser. I need to get the full raw HTTP request to assist the security team. I'm using the default logging output with morgan (STDOUT). I've console logged the headers to see the headers but would like to get them out in a slightly more readable format.
To log internal HTTP request sent from a Node.js server, you can create a Proxy Node.js server and log all requests there using Morgan.
First, define 3 constants (or read from your project config file):
// The real API endpoint, such as "another micro-service" in your network
const API = http://<real_server>
// Proxy Node.js server running on localhost
const LOGGER_ENDPOINT=http://localhost:3010
// Flag, decide whether logger is enabled.
const ENABLE_LOGGER=true
Second, When your Node.js server is launched, start the logger server at the same time if ENABLE_LOGGER is true. The logger server only do one thing: log the request and forward it to the real API server using request module. You can use Morgan to provide more readable format.
const request = require('request');
const morgan = require('morgan')(':method :url :status Cookie: :req[Cookie] :res[content-length] - :response-time ms');
...
if (ENABLE_LOGGER && LOGGER_ENDPOINT) {
let loggerPort = 3010;
const logger = http.createServer((req, res) => {
morgan(req, res, () => {
req.pipe(request(API + req.url)).pipe(res);
});
});
logger.listen(loggerPort);
}
Third, in your Node.js server, send API request to logger server when ENABLE_LOGGER is true, and send API directly to the real server when ENABLE_LOGGER is false.
let app = express(); // assume Express is used, but this strategy can be easily applied to other Node.js web framework.
...
let API_Endpoint = ENABLE_LOGGER ? LOGGER_ENDPOINT : API;
app.set('API', API_Endpoint);
...
// When HTTP request is sent internally
request(app.get('API') + '/some-url')...
Related
Currently, I am developing a simple project, which uses strong-soap module and expressjs. To create a soap server, I have to use http module of NodeJs, using express for soap module causes errors (wsdl file content can't be seen in browser). And i declare my routes and its functions by help of ExpressJS. My simple codebase is similar to the given below.
index.js
const app = require('express')();
const http = require('http');
var MyServiceObject = { /* ...some methods which exist in wsdl file */ };
var xml = require('fs').readFileSync('myWsdlFile.wsdl');
let server = http.createServer(function(request,response) {
response.end("404: Not Found: " + request.url);
});
server.listen(8000);
soap.listen(server, '/wsdl', MyServiceObject, xml);
/*########################### SOME ROUTES ############################################*/
app.listen(8002, (req, res) => {
console.log('App is listening on port 8002');
});
I am concerning about security, so i have a long question:
I'm not able to apply some authorization processes on HTTP Object in my code. How i can apply authorization on http? Is leaving http object as seen in code block, causes some security problems? Must i apply some authorization processes on http object? And i am using strong-soap server in this project. Must i apply some authorization processes on strong-soap object also. I can apply authorization processes on Express.js. Is applying authorization processes on express object (app) is sufficient for security?
Thanks in advance.
You can go with the soap package (https://www.npmjs.com/package/soap), you will get more flexibility to work with. Also, you can install the soap client (https://www.soapui.org/downloads/soapui/) to test services before implementing them with Node.js. It will help you to understand the request and response of each service.
I have a node express server responding to http traffic:
const http = require("http");
const express = require("express");
const app = express();
const server = http.createServer(app);
app.use(function(req,res,next){
console.log(`logging: req: ${util.inspect(req)}`);
next();
});
and all that works fine. I'd like to have a program on my node server inject emulated http traffic into the express stack, without a network connection. I can't just magic up a (req,res) pair and call a middleware function like the one in app.use above, because I don't have a next to give it, and my req and res will not be the ones next passes on to the next middleware in the stack.
Edit: What I actually have is a websocket connection sending data packets of a different format, different data contents from http traffic that can also carry the same information. I can take those websocket packets and build from those a request that is in the same format that the http traffic uses. I would like to pass that transformed request through the express http middleware stack and have it processed in the same way. Going all the way back to create an http request having just dealt with a ws request seems a bit far.
What's the simplest way to emulate some traffic, please? Can I call a function on app? Call some express middleware, or write a middleware of my own to inject traffic? Call a function on server?
Thanks!
Emulation traffic by calling some Express.js internal functions isn't the right way. Much easier is to trigger the server by HTTP request from the same process
const http = require('http');
const util = require('util');
const express = require('express');
const app = express();
const server = http.createServer(app);
app.use(function(req, res, next) {
console.log(`logging: req: ${util.inspect(req)}`);
next();
});
const port = 8081;
server.listen(port);
http.request({ port }).end();
From your question
I'd like to have a program on my node server inject emulated http traffic into the express stack, without a network connection
Can you clarify, why without a network connection?
A few things:
You need to make an endpoint
You need to host your server somewhere
You need something to send requests to your server
Express provides you a way to receive requests (req, res) (might be from a browser, might not be), perform some operations, and return responses (req, res) to the requester.
The expression
app.use(function(req,res,next){
console.log(`logging: req: ${util.inspect(req)}`);
next();
});
is actually a middleware function. This will take every request to your server and change the request object created by express into a string, and print it in your server log.
If you want a testable endpoint, you would add this to the bottom of the snippet you posted
app.get('/test', function (req, res) {
res.json({success:true})
})
This tells your app to allow GET requests at the endpoint /test
Next you're going to need to host your express server somewhere you can send requests to it. Your local machine (localhost) is a good place to do that. That way, you don't need an internet connection.
Pick a port you want to host the server on, and then it will be reachable at http://localhost:<Your Port>.
Something like this will host a server on http://localhost:3000. Add this below the route we declared above:
server.listen(3000, function() {
console.log('Server running on port 3000');
});
Finally, you'll need a way to send requests to the server on localhost. Postman is a great tool for testing express routes.
I would suggest installing Postman and using that to emulate http traffic.
Once your server is running, open postman and send a GET request to your server by entering the server address and port, and hitting the blue send button (You'll be sending the request to http://localhost:3000/test).
Here's an image of what postman should look like if all goes well
You should also see your middleware fire and print out the request object in your terminal.
Good Luck!
My Node.js folders hirarchy looks like the next image:
My folders hirarchy
While app.js it's the Node.js main file, routes it's the Node.js routes and src it's the client public html files.
This is the code in app.js:
var express = require('express');
var app = express();
var server = require('http').createServer(app);
global.io = require('socket.io').listen(server);
var compression = require('compression');
var helmet = require('helmet');
var session = require('express-session');
var bodyParser = require('body-parser');
app.use(bodyParser.json()); // support json encoded bodies
app.use(bodyParser.urlencoded({ extended: true })); // support encoded bodies
app.use(express.static(__dirname + '/src'));
app.use(helmet());
app.use(compression());
app.use('/rides', require('./routes/ridesServer'));
app.use('/user', require('./routes/userServer'));
app.use('/offers', require('./routes/offersServer'));
app.use('/notifications', require('./routes/notificationsServer'));
server.listen("8080", function() {
console.log("Connected to db and listening on port 8080");
});
This is another API in routes/userServer.js file:
router.post('/verifytoken', function(req, res, next) {
// some functions here
});
And this another HTTP REQUEST I am doing from client side, in page: ride.js:
$.ajax({
method: "POST",
headers: {
"Content-Type": "application/json"
},
url: "user/verifytoken",
data: JSON.stringify(something),
success: function(response) {
// some code
},
error: function(error) {
// some code
}
});
As you can see, client files and Node.js server files are on the same server, and Node.js serves those static files via this command:
app.use(express.static(__dirname + '/src'));
I think, that it should be avoided, and there is a better way!
If you are a Node.js expert and familier with best practices, please, tell me if the next method of working is correct, if it does not, please correct me:
I thought about putting static files on public_html directory
and Node.js files in server directory which is under public_html directory.
Then run pm2 start app.js --watch or, node app.js on the app.js which is located in server directory, and not in public_html.
In result, index.html file will be served as just as another static file without any relation to Node.js server, and Node.js will be in its own folder, not dealing with any kind of client the side.
In other words, seperate Node.js and static files and put Node.js files as a sub directory and not main directory.
Then the HTTP REQUEST will be looking like this:
$.ajax({
method: "POST",
headers: {
"Content-Type": "application/json"
},
url: "server/user/verifytoken",
data: JSON.stringify(something),
success: function(response) {
// some code
},
error: function(error) {
// some code
}
});
Please note that I have added SERVER directory.
Furthermore, I can exchange the
url: "server/user/verifytoken",
to an IP from a remote app (like Ionic):
url: "123.123.123.123:443/server/user/verifytoken",
And then my HTTP REQUESTS will be served via HTTPS (because I am sending for port 443), I can create multiple apps on the same server and I have no struggles with any Node.js express static folders.
What do you think?
Thanks!
First let me say I'm not an expert. But I have 3 years of continuous development of Node.js based solutions.
In the past I have created solutions mixing client side code and server side code on the same project and it has work. At least for a while. But in the long run is a bad idea for many possible reasons. Some of them are:
Client side code and server side code may require different processes to produce working code. For example client side code may require trans compiling from ES6 to more compatible ES5 using something as gulp or Webpack. This is normally not the case for server side code because the runtime is more targeted.
Mixing client side code and an API server may prevent you from horizontally scaling one of them without the other.
This is like a mono repo. And having a mono repo without a CI process tailor for this scenario may produce very long development times.
What we currently do at my work is as follow:
Create a separated API server project. This way you can concentrate on developing a good API while working on this specific project. Let cross-cutting concerns (like authentication) outside the API server.
Create a separated project for your client side code (SPA maybe). Set your dev environment to proxy API requests to a running API server (may be running locally).
Create a separated project for the deployment of the entire solution. This project will put together the serving of the client code, proxying requests to the API and implementing cross-cutting concerns like authentication, etc.
Having your code separated in this way makes easy developing each pieces and fast evolution. But it may introduce some complexities:
This multi-project structure require you to be able to trigger testing the hole product each time one of the project changes.
It surface the need of integration testing
Some other considerations are:
API server and Website server may run on the same machine but in different ports.
You may secure your API server using SSL (on node using the standard https module) but notice that in all cases you need another actor in front of the API server (a website proxying requests to the actual API server of a API gateway that implement cross-cutting concerns like authentication, rate limiting, etc). In the past I pose the same question you have made yourself regarding the apropriate of using SSL in this scenario and the answer is here. My answer is: depends on the deployment conditions.
I did a Node.js meetup presentation and was unable to answer this question. It is still bothering me.
Suppose I have a legacy http application or an Express.js application. It is a function, of the form
function legacy_app(request, response) {
// Handle the request.
}
Suppose I adopt Hapi.js for new versions of my application. But I have lots of debugged legacy or upstream code which I wish to integrate into the Hapi application. For example, a legacy vhost will run the legacy version, or it is accessible inside a /legacy namespace in the URL.
What is the best way to do this?
Wrapping existing HTTP node server dispatch function for use as a hapi handler is probably ok but you must add to your hapi_wrap function (at the end):
reply.close(false);
so that hapi can finish handling the request without messing with you legacy logic (https://github.com/spumko/hapi/blob/master/docs/Reference.md#replycloseoptions).
Wrapping Express handler/middleware is much more complicated because you are probably relying on some other middleware (e.g. body parser, cookie parse, session, etc.) and using some of the Express decorator that are not part of node (e.g. res.send(), res.json(), etc.).
The only way I can think to do this is manually. Just directly break the advice in the documentation: pull out the raw request and response objects and pass them to the legacy handler.
// An application built with http core.
var http = require('http')
var legacy_server = http.createServer(legacy_handler)
function legacy_handler(request, response) {
response.end('I am a standard handler\n')
}
// An express application.
var express = require('express')
var express_app = express()
express_app.get('*', function(request, response) {
response.send('I am an Express app\n')
})
// A Hapi application.
var Hapi = require('hapi')
var server = new Hapi.Server(8080, "0.0.0.0")
server.route({path:'/', method:'*', handler:hapi_handler})
function hapi_handler(request, reply) {
reply('I am a Hapi handler\n')
}
// Okay, great. Now suppose I want to hook the legacy application into the
// newer Hapi application, for example under a vhost or a /deprecated namespace.
server.route({path:'/legacy', method:'*', handler:hapi_wrap(legacy_handler)})
server.route({path:'/express', method:'*', handler:hapi_wrap(express_app)})
// Convert a legacy http handler into a Hapi handler.
function hapi_wrap(handler) {
return hapi_handler
function hapi_handler(request, reply) {
var req = request.raw.req
var res = request.raw.res
reply.close(false)
handler(req, res)
}
}
legacy_server.listen(8081)
express_app.listen(8082)
server.start()
This seems to work, although I would love if somebody who knew Hapi well could confirm that it is bug-free.
$ # Hit the Hapi application
$ curl localhost:8080/
I am a Hapi handler
$ # Hit the http application
$ curl localhost:8081/
I am a standard handler
$ # Hit the Express application
$ curl localhost:8082/
I am an Express app
$ # Hit the http application hosted by Hapi
$ curl localhost:8080/legacy
I am a standard handler
$ # Hit the Express application hosted by Hapi
$ curl localhost:8080/express
I am an Express app
I have recently have gotten into AngularJS and I love it. For an upcoming project I am looking to use the MEAN stack (MongoDB, Express, Angular, Node). I'm pretty familiar with Angular and I have a modest understanding of the purposes of MongoDB and Node in the stack. However, I don't really understand what the purpose of Express.js is. Is it essential to the MEAN stack? What would you compare it to in a traditional MySQL, PHP, javascript app? What does it do that the other three components can't do?
Also, if someone wants to give their own take on how the four parts of the stack work together, that'd be great.
MongoDB = database
Express.js = back-end web framework
Angular = front-end framework
Node = back-end platform / web framework
Basically, what Express does is that it enables you to easily create web applications by providing a slightly simpler interface for creating your request endpoints, handling cookies, etc. than vanilla Node. You could drop it out of the equation, but then you'd have to do a lot more work in whipping up your web-application. Node itself could do everything express is doing (express is implemented with node), but express just wraps it up in a nicer package.
I would compare Express to some PHP web framework in the stack you describe, something like slim.
You can think of Express as a utility belt for creating web applications with Node.js. It provides functions for pretty much everything you need to do to build a web server. If you were to write the same functionality with vanilla Node.js, you would have to write significantly more code. Here are a couple examples of what Express does:
REST routes are made simple with things like
app.get('/user/:id', function(req, res){ /* req.params('id') is avail */ });
A middleware system that allows you plug in different synchronous functions that do different things with a request or response, ie. authentication or adding properties
app.use(function(req,res,next){ req.timestamp = new Date(); next(); });
Functions for parsing the body of POST requests
Cross site scripting prevention tools
Automatic HTTP header handling
app.get('/', function(req,res){ res.json({object: 'something'}); });
Generally speaking, Sinatra is to Ruby as Express is to Node.js. I know it's not a PHP example, but I don't know much about PHP frameworks.
Express handles things like cookies, parsing the request body, forming the response and handling routes.
It also is the part of the application that listens to a socket to handle incoming requests.
A simple example from express github
var express = require('express');
var app = express();
app.get('/', function(req, res){
res.send('Hello World');
});
app.listen(3000);
Shows the creation of the express server, creating a route app.get('/'... and opening the port to listen for incoming http requests on.
Express allows you to manage http request easily compared to vanilla js.
you need to the following to make a get request
const Http = new XMLHttpRequest();
const url='https://jsonplaceholder.typicode.com/posts';
Http.open("GET", url);
Http.send();
Http.onreadystatechange=(e)=>{
console.log(Http.responseText)
}
In express, you require express and use it and make http requests
const express = require("express")
const app =express();
app.get("url",callback function);
Express in a Node.js based framework which simplifies writing Server-side Code and Logic.
Adds a lot of utility features and offers additional functionality, and in general, makes things easier.
Express is middleware-based : It basically funnels incoming requests through a chain of middlewares (of steps) where we can do something with the request, read some data from it, manipulate it, check if the user is authenticated or basically send back a response immediately.
This middlewares chain allows us to write very structured code
Express is a nodejs Framework build upon the top of Http module with more usable and better functionalities like easy way to handle routes.
eg: Using HTTP
var http = require('http');
//create a server object:
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/html'}); // http header
var url = req.url;
if(url ==='/about'){
res.write('<h1>about us page<h1>'); //write a response
res.end(); //end the response
}else if(url ==='/contact'){
res.write('<h1>contact us page<h1>'); //write a response
res.end(); //end the response
}else{
res.write('<h1>Hello World!<h1>'); //write a response
res.end(); //end the response
}
}).listen(3000, function(){
console.log("server start at port 3000"); //the server object listens on port 3000
});
using Express:
var express = require('express');
var app = express();
app.get('/about',function(req,res)=>{
res.write('<h1>about us page<h1>'); //write a response
res.end();
})