I'm struggling to pass response time data at the application-level to every route in my Express app. I've Googled and Googled but everything comes back suggesting one throw a new Date() into API calls, which is gross, or the Express response-time package (including many results from here on SO). While I'm sure it's great, I don't understand what purpose it serves other than adding a header seeing as how I can get response times in my browser's dev tools.
What I want to do is access all response time data from the server into the view, on every route.
Express already provides some request data, but not all of it. I just can't seem to access all responses. My terminal looks like this when I load up a basic page, even though I have 6 images on the page besides the one CSS file.
GET / 200 64.439 ms - 1663
GET /styles/index.css 200 36.582 ms - 2035
If I use the response-time package, I can't seem to access the 'X-Response-time' header. req.headers only seems to return a subset of all headers, similar to the Express traffic output mentioned above.
Maybe I'm just dense, but even the docs of the response-time package mention how to configure it with Express, but I still don't understand what it's supposed to be adding or how I would access it outside of my console.
Create a new middleware that records the response time of a request and makes this available to your own function fn. The fn argument will be invoked as fn(req, res, time), where time is a number in milliseconds.
Couldn't you just do this:
var responseTime = require('response-time')
app.use(responseTime(function(req, res, time) {
res.header('X-Response-Time', time);
}));
Now every route below this will have a response time header on it.
The response-time package will add the X-Response-Time header at the same time as you send the request:
app.use(responseTime());
app.get('/', function(req, res) {
console.log(res.get('X-Response-Time')); // undefined
res.send('Hello');
console.log(res.get('X-Response-Time')); // 3.720ms
});
It does this by listening out for when headers are about to be written (e.g. when a request is about to be sent), appending the response time just before (using the on-headers package on npm).
It sounds like what you want to do is record the response time, then set it into res.locals so you can render it in your view.
I've played around for a while trying to use the on-headers package to set the response time inside res.locals, but I think it must be too late by then to write to the response... Maybe this is the right path? Sorry I don't have more time right now to play around :(
Related
I am using AXIOS in a web client in order to POST a file to the express backend as an upload. Since both the file's size and the client user's bandwidth is variable, it may take a certain amount of time for the POST request to finish. On the backend some logic applies and the request is promptly rejected.
The problem is that the client receives the response only after the request is finished, which can be several seconds.
I have already tested that it is not the backend's fault, as the behavior is the same when POSTing to any arbitrary post-enabled url in the web, regardless of the technology. Here is an (over)simplified example of the case.
Here's the post action. Notice the commended request to the arbitrary post-enabled url. It behaves exactly the same:
try{
console.log("posting....")
const res = await axios.post("http://localhost:4000/upload", formData)
// const res = await axios.post("https://github.com/logout", formData)
console.log("result:")
console.log(res)
}catch(err){
console.error(err)
}
And the demo express backend route:
app.post("/upload", (req, res) => {
console.log("Rejecting...")
res.status(403).send()
console.log("Rejected.")
return
})
For testing purposes I choose a 3.7Mb file, and throttle down my browsers bandwidth to the Fast 3G preset.
The backend immediately outputs:
Rejecting...
Rejected.
Whereas the request is pending for about 43 seconds before returning the 403 error:
Am I missing something obvious here? It is such a common functionality, it makes me doubt that this is the correct way to be handled. And if it really is, do we have any information on whether express's thread is active during that time, or is it just a client inconvenience?
Thanks in advance!
I believe you could just use res.status(403) rather than res.status(403).send() .
You could also try using res.status(403).end() and I am not sure why you should use a return statement in the router part.
It seems that first sending the response headers and then manually destroying the request does the trick:
app.post("/upload", (req, res) => {
console.log("Rejecting...")
res.status(403).send("Some message")
return req.destroy()
})
The AXIOS request stays pending until just the current chunk is uploaded, and then immediately results in the correct status and message. In the throttled down fast 3g example, pending time went down from 43s to 900ms.
Also, this solution emerged through trial and error, so it can possibly not be the best practice.
I would still be interested in an AXIOS oriented solution, if one exists.
I want to see in Node Exprees enviroment all the headers that are sent to the client
But when i see do this:
app.get("/", (req, res) => {
console.log(res.getHeaders());
});
i only see this :
At the time you're looking at the outgoing headers, those are the only ones that have been added so far. The rest will be added by the code that sends the actual response or by other middleware.
If you want to see all the headers that were eventually added to the response before it was sent, you can monitor the finish event and THEN look at the headers:
app.use(function(req, res, next) {
res.on('finish', () => {
console.log(`request url = ${req.originalUrl}`);
console.log(res.getHeaders());
});
next();
});
This will sometimes not include the date, content-type or content-length headers unless they are specifically set by the sending code. This is because if these are not set by the sending code, then the HTTP library adds these headers at a lower level as it is sending the headers and thus res.getHeaders() does not retrieve them and never knows about them.
Edit: I overlooked your first screenshot... Are you using any middleware? It looks like you're using the CORS middleware, at least - which is why you are showing more headers than the defaults..
It looks like Node/Express sends a Content-Length header when it can..
The Date header is mandatory per the HTTP spec, but it looks like you can change that behavior in Node/Express - I'm guessing by default Node/Express sets that value to true.
I did test setting res.sendDate = false and the date header was not sent, so it looks like that header is set by default for you, most likely as the last step in the response?
With res.sendDate = false;
Without setting res.sendDate (aka the default):
All in all, I'm assuming the headers you don't see when you console.log(res.getHeaders()) are set by Node/Express by default..
I wasn't able to find anything in the docs about default response headers (outside of the Date header), but it's possible I overlooked something. The Express docs don't have anything on it as they just use the built in http module from Node.
I am overall clueless about how and why you set up a node.js app, and how any of the app.use functions work - the tutorials on it don't explain the why of anything.
Anyway, I have socket.io, res.locals and index.js set up like so in the app.js root file.
const sockets = require('./models/socket')(io)
app.use(function (req, res, next) {
res.locals.user_id = req.session.user_id;
next();
});
const routes = require('./routes/index');
app.use('/', routes);
I'd like to be able to access res.locals in the socket.js model, like I can in index.js found in the routes folder.
I can't guess how to go about doing this. If anybody is able to explain how and why I can or can't that would be a bonus. Thanks!
Welcome to Expressjs, there are a few fundamentals you should probably research before going any further, they'll help solve some of your confusion. I'll give a brief explanation of them but I suggest you do further research. I'll then answer your actual question at the end.
Middleware and app.use
Expressjs is built upon an idea that everything is just "middleware". Middleware is a function which runs as part of a request chain. A request chain is essentially a single client request, which then goes through a chain of a number of middleware functions until it either reaches the end of the chain, exits early by returning a response to the client, or errors.
Express middleware is a function which takes the following three arguments.
req (request) - Representing the request made by a client to your
server.
res (response) - Representing the response you will return to
the client.
next - A way of telling express that your current
middleware function is done, and it should now call the next piece of
middleware. This can either be called "empty" as next(); or with an
error next(new Error());. If it is called empty, it will trigger
the next piece of middleware, if it is called with an error then it
will call the first piece of error middleware. If next is not called at the
end of a piece of middleware, then the request is deemed finished and the
response object is sent to the user.
app.use is a way of setting middleware, this means it will run for every request (unless next() is either not called by the previous piece of middleware for some reason, or it's called with an error). This middleware will run for any HTTP request type (GET, POST, PUT, DELETE, etc).
app.use can take multiple arguments, the important ones for beginners to learn are: app.use(func) and app.use(path, func). The former sets "global" middleware which runs no matter what endpoint (url path) the client requests, the latter (with a specific path) is run only if that specific path is hit. I.e. app.use('/hello', (req, res, next) => { res.send('world'); }); will return "world" when the endpoint "/hello" is hit, but not if the client requests "/hi". Where as app.use((req, res, next) => { res.send('world'); }); would return "world" when you hit any endpoint.
There are more complex things you can do with this, but that's the basics of attaching middleware to your application. The order they are attached to the application, is the order in which they will run.
One more thing, this will blow your mind, an express application made with the standard const app = express() can also be used as middleware. This means you can create several express applications, and then mount them using app.use to a single express application. This is pretty advanced, but does allow you to do some really great things with Express.
Why can you not access res.locals in socket.io? (The real question)
Within your middleware handler, you are setting up a res.locals.use_id property. This only lives with that individual request, you can pass it around as long as the request is alive by passing it into other functions, but outside of that request it doesn't exist. res is literally the response object that tells Express how to respond to the clients request, you can set properties of it during the request but once that HTTP request has ended it's gone.
Socket.io is a way of handling web socket requests, not standard HTTP requests. Thus, in a standard express HTTP request you will not be able to hand off the connection to anything with socket.io, because the connection is a single short lived HTTP request. Likewise, you won't be able to do the same the other way.
If you wish to find the users id in a socket.io request, you'll have to do this within the socket.io request itself.
Right now, you're entering a piece of middleware for an Express.js request, you are then calling next() which runs the next piece of express middleware, at no point does it cross over into Socket.io realms. This is often confused by tutorials because Socket.io can handle requests across the same port as Express is listening on, but the two are not crossed over. So you will need to write separate middleware for both Express.js requests chains, and socket.io request chains. There are ways of writing this code once and then writing an adapter to use it across both platforms, but that's not what you've tried to do here.
I would suggest you look at doing just nodejs and express for a time before taking on socket.io as well, otherwise you're trying to learn a whole heap of technologies all at once is quite a lot to try and take on board all at once.
I'm setting up a site that posts to a remote server. The user performs some steps on this remote server and when the user is done. The server issues a get request with a bunch of query parameters to my server.
The thing is that this request never arrives at the controller method it is supposed to.
I have a custom middleware that intercepts all requests and i am doing some logging in there, i can see there every time this request arrives to my server but express doesn't seem to match it to the controller.
However if i go into my browser and do the request from there with the exact same path and query string it works fine.
I thought maybe the remote server was using a proxy so i enabled trust proxy in express but that didn't make any difference.
I have tried changing the route path and that didn't make a difference.
I don't know what code would be helpful since it is pretty standard express code. I have tried putting the route before all the middleware and the request still bypassed the controller and was logged by my logging middleware.
I'm completely baffled, anyone got any idea what could be causing this?
EDIT:
Here is some of the code I'm using. This is not the exact setup I'm trying but I have tried to make it work this way to rule out my routing logic and this example has the exact same problem. I tried putting the app.get above the middleware and the request from the external server still bypassed the app.get and went directly to the middleware.
app.use(function(req, res, next) {
console.log('path', req.method, req.path, req.query);
});
app.get('/payments/success', function (req, res) {
console.log('SUCCESS', req.query);
res.status(200).send();
});
Here is the result of the middleware logging a request arriving from the remote server(i removed the variables logged from the request query because it is sensitive information, they are returned just the way they are supposed to):
path GET /payment/success {
variable: 'variable",
}
If anyone needs any more information from the request object i will provide it.
To learn node.js I'm creating a small app that get some rss feeds stored in mongoDB, process them and create a single feed (ordered by date) from these ones.
It parses a list of ~50 rss feeds, with ~1000 blog items, so it's quite long to parse the whole, so I put the following req.connection.setTimeout(60*1000); to get a long enough time out to fetch and parse all the feeds.
Everything runs quite fine, but the request is called twice. (I checked with wireshark, I don't think it's about favicon here).
I really don't get it.
You can test yourself here : http://mighty-springs-9162.herokuapp.com/feed/mde/20 (it should create a rss feed with the last 20 articles about "mde").
The code is here: https://github.com/xseignard/rss-unify
And if we focus on the interesting bits :
I have a route defined like this : app.get('/feed/:name/:size?', topics.getFeed);
And the topics.getFeed is like this :
function getFeed(req, res) {
// 1 minute timeout to get enough time for the request to be processed
req.connection.setTimeout(60*1000);
var name = req.params.name;
var callback = function(err, topic) {
// if the topic has been found
if (topic) {
// aggregate the corresponding feeds
rssAggregator.aggregate(topic, function(err, rssFeed) {
if (err) {
res.status(500).send({error: 'Error while creating feed'});
}
else {
res.send(rssFeed);
}
},
req);
}
else {
res.status(404).send({error: 'Topic not found'});
}};
// look for the topic in the db
findTopicByName(name, callback);
}
So nothing fancy, but still, this getFeed function is called twice.
What's wrong there? Any idea?
This annoyed me for a long time. It's most likely the Firebug extension which is sending a duplicate of each GET request in the background. Try turning off Firebug to make sure that's not the issue.
I faced the same issue while using Google Cloud Functions Framework (which uses express to handle requests) on my local machine. Each fetch request (in browser console and within web page) made resulted in two requests to the server. The issue was related to CORS (because I was using different ports), Chrome made a OPTIONS method call before the actual call. Since OPTIONS method was not necessary in my code, I used an if-statement to return an empty response.
if(req.method == "OPTIONS"){
res.set('Access-Control-Allow-Origin', '*');
res.set('Access-Control-Allow-Headers', 'Content-Type');
res.status(204).send('');
}
Spent nearly 3hrs banging my head. Thanks to user105279's answer for hinting this.
If you have favicon on your site, remove it and try again. If your problem resolved, refactor your favicon url
I'm doing more or less the same thing now, and noticed the same thing.
I'm testing my server by entering the api address in chrome like this:
http://127.0.0.1:1337/links/1
my Node.js server is then responding with a json object depending on the id.
I set up a console log in the get method and noticed that when I change the id in the address bar of chrome it sends a request (before hitting enter to actually send the request) and the server accepts another request after I actually hit enter. This happens with and without having the chrome dev console open.
IE 11 doesn't seem to work in the same way but I don't have Firefox installed right now.
Hope that helps someone even if this was a kind of old thread :)
/J
I am to fix with listen.setTimeout and axios.defaults.timeout = 36000000
Node js
var timeout = require('connect-timeout'); //express v4
//in cors putting options response code for 200 and pre flight to false
app.use(cors({ preflightContinue: false, optionsSuccessStatus: 200 }));
//to put this middleaware in final of middleawares
app.use(timeout(36000000)); //10min
app.use((req, res, next) => {
if (!req.timedout) next();
});
var listen = app.listen(3333, () => console.log('running'));
listen.setTimeout(36000000); //10min
React
import axios from 'axios';
axios.defaults.timeout = 36000000;//10min
After of 2 days trying
you might have to increase the timeout even more. I haven't seen the express source but it just sounds on timeout, it retries.
Ensure you give res.send(); The axios call expects a value from the server and hence sends back a call request after 120 seconds.
I had the same issue doing this with Express 4. I believe it has to do with how it resolves request params. The solution is to ensure your params are resolved by for example checking them in an if block:
app.get('/:conversation', (req, res) => {
let url = req.params.conversation;
//Only handle request when params have resolved
if (url) {
res.redirect(301, 'http://'+ url + '.com')
}
})
In my case, my Axios POST requests were received twice by Express, the first one without body, the second one with the correct payload. The same request sent from Postman only received once correctly. It turned out that Express was run on a different port so my requests were cross origin. This caused Chrome to sent a preflight OPTION method request to the same url (the POST url) and my app.all routing in Express processed that one too.
app.all('/api/:cmd', require('./api.js'));
Separating POST from OPTIONS solved the issue:
app.post('/api/:cmd', require('./api.js'));
app.options('/', (req, res) => res.send());
I met the same problem. Then I tried to add return, it didn't work. But it works when I add return res.redirect('/path');
I had the same problem. Then I opened the Chrome dev tools and found out that the favicon.ico was requested from my Express.js application. I needed to fix the way how I registered the middleware.
Screenshot of Chrome dev tools
I also had double requests. In my case it was the forwarding from http to https protocol. You can check if that's the case by looking comparing
req.headers['x-forwarded-proto']
It will either be 'http' or 'https'.
I could fix my issue simply by adjusting the order in which my middlewares trigger.