So I understand that Node.js Connect works like a stack through which it runs, starting from the top and going to the bottom. From the Connect introduction by its author at http://howtonode.org/connect-it it shows an example like
var Connect = require('connect');
module.exports = Connect.createServer(
require('./log-it')(),
require('./serve-js')()
);
The article reads
Every request enters the onion at the outside and traverses layer by
layer till it hits something that handles it and generates a response.
In Connect terms, these are called filters and providers. Once a layer
provides a response, the path happens in reverse.
I'm particulary curious about "Once a layer provides a response, the path happens in reverse". How does that happen? Every middleware gets called again, but in reverse order?
No, they don't get called again in reverse, but each middleware has a chance to monkey-patch the request methods and hijack them. It's not ideal.
// basic logger example
module.exports = function () {
return function logger(req, res, next) {
var writeHead = res.writeHead;
res.writeHead = function (code, headers) {
console.log(req.method, req.url, code);
res.writeHead = writeHead;
return res.writeHead(code, headers);
};
next();
};
};
Now, this code has issues because writeHead isn't the only way to set the status code, so it won't catch all requests. But that is the basic way that middleware can catch events on the way out.
Related
My current setup involves a Node.js web application using Express.js.
I am using DataDog's dd-tracer to measure the time Node.js spends for particular method invocations as part of my APM solution.
I would like to know if it is possible to measure the portion of time an incoming HTTP request is busy sending data back to the client as HTTP response body.
Are there any pitfalls or inaccuracies involved when trying to do this kind of instrumentation?
Does anybody know why this is not measured by APM client libraries by default?
I would like to know if it is possible to measure the portion of time an incoming HTTP request is busy sending data back to the client as HTTP response body.
You could wrap calls to res.write manually to create additional spans in the request trace. I would only recommend this if there are not many calls to the method within a request, and otherwise I would recommend to capture just a metric instead.
Alternatively, profiling might be an option that would give you a lot more information about exactly what is taking time within the res.write calls.
I look for a "global" solution which can be integrated into a Nest.js application without instrumenting each call to res.write manually.
As described above, you can simply wrap res.write directly at the start of every request. Using the tracer, this can be achieved like this:
res.write = tracer.wrap('http.write', res.write)
This should be done before any other middleware has the chance to write data.
Example middleware:
app.use((req, res) => {
res.write = tracer.wrap('http.write', res.write)
})
Are there any pitfalls or inaccuracies involved when trying to do this kind of instrumentation?
Nothing major that I can think of.
Does anybody know why this is not measured by APM client libraries by default?
The main issue for doing this out of the box is that creating a span for every call to res.write may be expensive if there are too many calls. If you think it would make sense to have an option to do this out of the box, we can definitely consider adding that.
Hope this helps!
It depends if you want to have the response time for each of the calls or if you want to gather statistics about the response time.
For the first, to get the response time in the header of the response for each request, you can use response-time package: https://github.com/expressjs/response-time
This will add to the response header a value (by default X-Response-Time). That will have the the elapsed time from when a request enters the middleware to when the headers are written out.
var express = require('express')
var responseTime = require('response-time')
var app = express()
app.use(responseTime())
app.get('/', function (req, res) {
res.send('hello, world!')
})
If you want a more complete solution and gather statistics that include the response time you can use the
express-node-metrics package
https://www.npmjs.com/package/express-node-metrics
var metricsMiddleware = require('express-node-metrics').middleware;
app.use(metricsMiddleware);
app.get('/users', function(req, res, next) {
//Do Something
})
app.listen(3000);
You can expose and access this statistics like this:
'use strict'
var express = require("express");
var router = express.Router();
var metrics = require('express-node-metrics').metrics;
router.get('/', function (req, res) {
res.send(metrics.getAll(req.query.reset));
});
router.get('/process', function (req, res) {
res.send(metrics.processMetrics(req.query.reset));
});
router.get('/internal', function (req, res) {
res.send(metrics.internalMetrics(req.query.reset));
});
router.get('/api', function (req, res) {
res.send(metrics.apiMetrics(req.query.reset));
});
First of all I state that I don't know dd-tracer, but I can try to provide a way to get the requested time, then it's up to the developer to use it as needed.
The main inaccuracy coming to my mind is that every OS has its own TCP stack and writing on a TCP socket is a buffered operation: for response bodies smaller than OS TCP stack buffer we are probably going to measure a time close to 0; the result we have is moreover influenced by the Node.js event loop load. The larger the response body becomes the more the event loop load related time becomes negligible. So, if we want to measure the write time for all request only to have a single point, but we'll do our analysis only for long time requests, I think the measurement will be quite accurate.
Another possible source of inaccuracy is how the request handlers write their output: if a request handler writes part of the body, then performs a long time operation to compute last part of the body, then writes missing part of the body, the measured time is influenced by the long time computing operation; we should take care that all request handlers write headers and body all at once.
My solution proposal (which works only if the server do not implements keep alive) is to add a middleware like this.
app.use((req, res, next) => {
let start;
const { write } = res.socket;
// Wrap only first write call
// Do not use arrow function to get access to arguments
res.socket.write = function() {
// Immediately restore write property to not wrap next calls
res.socket.write = write;
// Take the start time
start = new Date().getTime();
// Actually call first write
write.apply(res.socket, arguments);
};
res.socket.on("close", () => {
// Take the elapsed time in result
const result = new Date().getTime() - start;
// Handle the result as needed
console.log("elapsed", result);
});
next();
});
Hope this helps.
You can start a timer before res.end and then any code after res.end should run after it is finished so stop the timer after the res.end function. Don't quote me on that tho.
I'm trying to write a simple Swagger API that will allow me to sync a couple of systems on demand. The syncing is one way, so basically the end goal will be to send a request to both system, see what's new/changed/removed on the origin, then update the destination. I've been trying to do this using node.js instead of Java, to which I'm more used to, as a learning experience, but I'm really having a hard time figuring out a key issue due to the async nature.
As a test, I constructed a simple Express node.js app on IntelliJ, where in one of the routes I'm calling a function exported from a different file and trying to get the response back. Unfortunately, this isn't working so well. What I've done is this:
getit.js - (this goes to the Ron Swanson generator to get a quote)
const rp = require('request-promise');
async function dorequest() {
const response = await rp(uri);
return Promise.resolve(response);
};
module.exports = {dorequest}
In the route I've done this:
var getit = require ('./getit.js');
/* GET users listing. */
router.get('/', function(req, res, next) {
var ret = getit.dorequest();
res.send(ret);
console.log('res out' + ret);
});
What I get in the console is
res out[object Promise]
and the response is of course empty.
What am I doing wrong? I've been playing with this for a week now, tried various methods, but I keep getting similar results. I'm obviously missing something out, and would appreciate some help.
Thanks!
Object is empty because it was written on the console before the Promise is resolved. You have to wait until Promise is resolved and then send the response back so try to change your code like this:
var getit = require ('./getit.js');
/* GET users listing. */
router.get('/', function(req, res, next) {
getit.dorequest().then(function(data) {
console.log('res out' + data);
res.send(data);
});
});
Since you are using async/await approach all you need to do is to place await before getit.dorequest();
so this line will look like var ret = await getit.dorequest();
Based upon the voted answer events.js:130 throw TypeError('listener must be a function')
I am trying to implement Logging wherein we save every action in DB. To cater this, I was planning to listen to finish or close event of response stream in app.js
I understand that I need to put this code in Express frameworks root folder.Can you please specify where exactly so that I can access listen to this event and access requested URL as well.
Below is the code snippet
app.use(function (req, res,next) {
function afterResponse (req,res,next){
console.log("Response finished from the path "+ req.url);
}
res.on('finish', afterResponse(req,res,next));
});
While the 'finish' event emitted by the response object is not in the documentation, it does exist. However, as an undocumented feature, you may want to avoid it since it could be removed anytime. Without more information about exactly what you are trying to accomplish, it's hard to say whether there might be a better alternative, although I suspect there is. With that caveat, here is the answer to your question.
If you want to capture every response, you need to define a middleware function near the top of your app, before any response has been sent (order matters in Express 4). The example you have above is too complicated; the 'finish' event listener doesn't take parameters. Instead, you can simply do this:
// This example uses ES2015 syntax
app.use((req, res, next) => {
res.on('finish', () => {
console.log(`Response finished from the path ${req.url}`);
});
next();
});
The example above adds the listener before any request is sent, but the listener will only be called after the request is sent. Make sure the listener itself takes no parameters; if you pass req as a parameter as in your example, it will be undefined, which is exactly what you don't want. Just define your litsener in the middleware function itself, and you can access req or res as much as you like.
Alternatively, if you don't want the listener to be redefined on every request, you can access req through the response object itself, which is available as the this context within the listener:
// This example uses ES2015 syntax
app.use((req, res, next) => {
res.on('finish', onFinish);
next();
});
function onFinish() {
console.log(`Response finished from the path ${this.req.url}`);
}
Unable to find out the issue in following script, what i want to achieve with the script is to have a node log server that would listen to post requests with log title and log details as query parameters, write to a file and then throw back as json on get request.
Problem:
It constantly shows loader sometime and gives the required log sometime.
Note:
The process spawning is done to update the browser during the logging, if someone has better solution, plz suggest
Post Call:
http://127.0.0.1:8081/log?title="test"&detail="test detail"
Code:
var express = require("express");
var spawn = require('child_process').spawn;
var fs = require("fs");
var srv = express();
var outputFilename = '/tmp/my.json';
function getParamsObject(context) {
var params = {};
for (var propt_params in context.params) {
params[propt_params] = context.params[propt_params];
//define(params, propt_params, context.params[propt_params]);
}
for (var propt_body in context.body) {
params[propt_body] = context.body[propt_body];
//define(params, propt_body, context.body[propt_body]);
}
for (var propt_query in context.query) {
params[propt_query] = context.query[propt_query];
//define(params, propt_query, context.query[propt_query]);
}
return params;
}
srv.get("/", function(req, res) {
res.send("Hello World From Index\n");
});
srv.get("/Main", function(req, res) {
res.send("Hello World From Main\n");
});
srv.get("/ReadFile", function(req, res) {
fs.readFile("example_one.txt", function(err, data) {
if(err) throw err;
res.send(data.toString());
});
});
srv.get("/ReadFileJSON", function(req, res) {
fs.readFile("example_one.txt", function(err, data) {
if(err) throw err;
res.setHeader("content-type", "application/json");
res.send(new Parser().parse(data.toString()));
});
});
srv.post("/log", function(req, res) {
var input = getParamsObject(req);
if(input.detail) {
var myData = {
Date: (new Date()).toString(),
Title: input.title,
Detail: input.detail
}
fs.writeFile(outputFilename, JSON.stringify(myData, null, 4), function(err) {
if(err) {
console.log(err);
}
});
}
res.setHeader("content-type", "application/json");
res.send({message:"Saved"});
});
srv.get("/log", function(req, res) {
var child = spawn('tail', ['-f', outputFilename]);
child.stdout.pipe(res);
res.on('end', function() {
child.kill();
});
});
srv.listen(8081);
console.log('Server running on port 8081.');
To clarify the question...
You want some requests to write to a log file.
You want to effectively do a log tail over HTTP, and are currently doing that by spawning tail in a child process.
This isn't all that effective.
Problem: It constantly shows loader sometime and gives the required log sometime.
Web browsers buffer data. You're sending the data, sure, but the browser isn't going to display it until a minimum buffer size is reached. And then, there are rules for what will display when all the markup (or just text in this case) hasn't loaded yet. Basically, you can't stream a response to the client and reliably expect the client to do anything with it until it is done streaming. Since you're tailing a log, that puts you in a bad predicament.
What you must do is find a different way to send that data to the client. This is a good candidate for web sockets. You can create a persistent connection between the client and the server and then handle the data immediately rather than worrying about a client buffer. Since you are using Node.js already, I suggest looking into Socket.IO as it provides a quick way to get up and running with web sockets, and long-polling JSON (among others) as a fallback in case web sockets aren't available on the current browser.
Next, there is no need to spawn another process to read a file in the same way tail does. As Trott has pointed out, there is an NPM package for doing exactly what you need: https://github.com/lucagrulla/node-tail Just set up an event handler for the line event, and then fire a line event on the web socket so that your JavaScript client receives it and displays it to the user immediately.
There are a couple of things that seem to stand out as unnecessary complications that may be the source of your problem.
First, the spawn seems unnecessary. It appears you want to open a file for reading and get updated any time something gets added to the file. You can do this in Node with fs.watch(), fs.watchFile(), or the node-tail module. This may be more robust than using spawn() to create a child process.
Second (and less likely to be the source of the problem, I think), you seem to be using query string parameters on a POST request. While not invalid, this is unusual. Usually, if you are using the POST method, you send the data via post, as part of the body of the request. If using the GET method, data is sent as a query string. If you are not using the body to send data, switch to GET.
In my restify server I have a route that I want to add a filter to. The way this would work in Express (this does work in Express) is the response.on would add my handler as the callback. Then before the response is returned to the requestor my handler would be called. This is not the behavior I am seeing in Restify. My guess is I am not registering my handler to the event machine properly.
restifyServer.get({ path: "/api/v1/readings", version: ["1.0.0"]}, Reading.getReadings);
I add a handler:
function zipkinTracing(request, response, next){
// handle the callback request and record the trace
var trace = zipkin.getTraceFromRequest(request, "ids-api", {address: config.externalIP, port: 80});
if (trace){
zipkin.sendAnnotationReceive(trace);
zipkin.sendAnnotation(trace, 'request.headers',JSON.stringify(request.headers));
trace.parentSpanId = trace.spanId;
zipkin.sendAnnotation(trace, 'http.uri', request.url);
var queryObj = urlLib.parse(request.url, true).query;
Object.keys(queryObj).forEach( function(key){
zipkin.sendAnnotation(trace, key, queryObj[key]);
});
request.zipkinTrace = trace;
}
response.on('after', function(request, response, route, error){
var t = request.zipkinTrace;
if (t) {
zipkin.sendAnnotation(t, 'http.response.code',response.code);
zipkin.sendAnnotationSend(t);
}
});
return next();
}
Now my routes look like this:
restifyServer.use(zipkinTracing);
restifyServer.get({ path: "/api/v1/readings", version: ["1.0.0"]}, Reading.getReadings);
The problem is the response.on('after', is never fired. What am I doing wrong here?
Reading.getReadings looks like this:
makeHttpQuery(request, response, function(error, message) {
....
response.send(message['data']['data']);
return next();
}
I was able to resolve this issue. In order to create a true filter I registered the handler function to the response.on('finish', handler) as you would do in Express. The alternate solution would be to place
restifyServer.on('after', function(request, response){
var t = request.zipkinTrace;
if (t) {
zipkin.sendAnnotation(t, 'http.response.code',response.code);
zipkin.sendAnnotationSend(t);
}
});
after the route to be filtered. I prefer the response.on('finish', handler). Any other solutions or interesting ways to create a filter for requests and responses?
You can also add multiple handlers to the request. One handler will finish and pass on the the next one, much like a filter.
See: http://restify.com/#routing