I have a very limited knowledge about node and nob-blocking IO so forgive me if my question is too naive.
In order to return needed information in response body, I need to
Make a call to 3rd party API
Wait for response
Add some modifications and return JSON response with the information I got from API.
My question is.. how can I wait for response? Or is it possible to send the information to the client only when I received response from API (as far as I know, connection should be bidirectional in this case which means I won't be able to do so using HTTP).
And yet another question. If one request waits for response from API, does this mean than other users will be forced to wait too (since node is single-threaded) until I increase numbers of threads/processes from 1 to N?
You pass a callback to the function which calls the service. If the service is a database, for example:
db.connect(host, callback);
And somewhere else in the code:
var callback = function(err, dbObject) {
// The connection was made, it's safe to handle the code here
console.log(dbObject.status);
res.json(jsonObject, 200)
};
Or you can use anonymous functions, so:
db.connect(host, function(err, dbObject) {
// The connection was made, it's safe to handle the code here
console.log(dbObject.status);
res.json(jsonObject, 200)
});
Between the call and the callback, node handles other clients / connections freely, "non-blocking".
This type of situation is exactly what node was designed to solve. Once you receive the request from your client, you can make a http request, which should take a callback parameter. This will call your callback function when the request is done, but node can do other work (including serving other clients) while you are waiting for the response. Once the request is done, you can have your code return the response to the client that is still waiting.
The amount of memory and CPU used by the node process will increase as additional clients connect to it, but only one process is needed to handle many simultaneous clients.
Node focuses on doing slow I/O asynchronously, so that the application code can start a task, and then have code start executing again after the I/O has completed.
An typical example might make it clear. We make a call to the FB API. When we get a response, we modify it and then send JSON to the user.
var express = require('express');
var fb = require('facebook-js');
app.get('/user', function(req, res){
fb.apiCall('GET', '/me/', {access_token: access_token}, function(error, response, body){ // access FB API
// when FB responds this part of the code will execute
if (error){
throw new Error('Error getting user information');
}
body.platform = 'Facebook' // modify the Facebook response, available as JSON in body
res.json(body); // send the response to client
});
});
Related
I have a security issue that someone is trying to call random APIs that are not supported on our server but are frequently used for administrators API in general. and I set this code below to handle 404 to not respond to this attack
url-not-found-handler.js
'use strict';
module.exports = function () {
//4XX - URLs not found
return ((req, res, next) => {
});
};
what happens to client is that it waits until the server responds but I want to know if this will affect the performance of my express.js server also what happens behind the scene in the server without res.send() or res.end() ?
According to the documentation of res.end().
Ends the response process. This method actually comes from Node core,
specifically the response.end() method of http.ServerResponse.
And then response.end
This method signals to the server that all of the response headers and
body have been sent; that server should consider this message
complete. The method, response.end(), MUST be called on each response.
If you leave your request hanging, the httpserver will surely keep data about it. Which means that if you let hang many requests, your memory will grow and reduce your server performance.
About the client, he's going to have to wait until he got a request timeout.
The best to do having a bad request is to immediately reject the request, which is freeing the memory allowed for the request.
You cannot prevent bad requests (maybe have a firewall blocking requests from certains IP address?). Best you can do is to handle them as fast as possible.
i will share the code directly
app.get('/ListBooks', function (req, res) {
console.log("Function called");
//internally calls another URL and sends its response to browser
request({
url: 'someURLinRESTServer',
method: 'POST',
json: MyJsonData
}, function (error, response, body) {
if (error) {
console.log("/Call Failed ->" + error);
res.status(200).send('Failed');
} else {
console.log("/Call got Response");
console.log(response.statusCode, body);
res.send(body); res.end();
}
})
now when the browser generates a request on http://localhost/ListBooks
my node console shows the first message "Function called" and waits for internal REST URL Response
the real problem occurs only when the REST SERVER is down
then if i try to call http://localhost/ListBooks from another browser tab the Node server console doesnt show any changes and only after the repsonse of previous function REST CALL call it displays console message of second function call on app.get('/ListBooks'
i thought node js makes async functions bt here i dnt want functions to wait likes this for multiple instance calls
or is it just a delay in printing message and each function call executes separately .Plz clarify ...
If this is only occurring when the REST server is down (as your comment indicates), then that's just a function of how long your calls to request() take to fail. And, each separate call to request() goes through its own cycle of trying to connect and then eventually timing out. If both are timing out, then you will issue request1, then request2, then some timeout amount of time will pass and request1 will fail and then request2 will fail shortly after it. This has nothing to do with how express handles multiple requests and everything to do with how the calls to your REST server behave.
You can set the timeout option for request() if you want to shorten how long it will wait for a response, but you do need to make sure you don't shorten it so much that a busy REST server that just takes a little while to actually respond gets timed out.
or is it just a delay in printing message and each function call
executes separately
Each call is acting completely separately. There is no serialization of these responses by node.js or by Express. The appearance of serialization is just because they both take the same amount of time to fail with a timeout so they will fail one after the other.
Currently still understanding the logic flow of a typical web application. In PHP, since its not async, typically we will have to wait for database insert/update to be completed before returning response to user.
But since in NodeJs we can have Async database query with callback and return response to user even before the callback is being called by database.
In these kind of situation, assuming validation of the data was already done, do typical nodeJs application wait for the callback before response or always assume the query is successful and return response to user?
I'm not entirely sure what you're asking but you should take a look at these explanations of async database queries:
Async explanation
Callback explanation
If I interpreted your question correctly you are asking when a node application would return a response, right? Typically you return a response to the user (like updating the client view) within the callback. By providing a callback you can make sure that the response is received and successful before moving on in that thread. Here is a bit of code to illustrate:
functionThatTakesForever(argument, function(error, response){ //this is the callback
if(error)
console.log(error);
else{
\\do something with response to return the response to the user
}
});
In an app that I was working, I encountered "headers sent already error" if I test using concurrency and parallel request methods.
ultimately I resolved the problem using !response.headersSent but my question is why am I forced to use it? is node caching similar requests and reuses them for the next repeated call.
if(request.headers.accept == "application/json") {
if(!response.headersSent) {response.writeHead(200, {'Content-Type': 'application/json'})}
response.end(JSON.stringify({result:{authToken:data.authToken}}));
}
Edit
var express = require('express');
var app = express();
var server = app.listen(process.env.PORT || 3000, function () {
console.log('Example app listening at http://%s:%s', server.address().address, server.address().port);
});
Edit 2:
Another problem is while testing using mocha, super agent and while the tests in progress if I just send another request through postman on the side, one of the tests in mocha end with a timeout error. These steps I'm taking to ensure the code is production ready for simultaneous, parallel requests? please advise on what measures I can take to ensure node/code works under stress.
Edit 3:
app.use(function(request, response, next){
request.id = Math.random();
next();
});
OK, in an attempt to capture what solved this for you via all our conversation in comments, I will attempt to summarize here:
The message "headers sent already error" is nearly always caused by improper async handling which causes the code to call methods on the response object in a wrong sequence. The most common case is non-async code that ends the request and then an async operation that ends some time later that then tries to use the request (but there are other ways to misuse it too).
Each request and response object is uniquely created at the time each individual HTTP request arrives at the node/express server. They are not cached or reused.
Because of asynchronous operations in the processing of a request, there may be more than one request/response object in use at any given time. Code that is processing these must not store these objects in any sort of single global variable because multiple ones can be in the state of processing at once. Because node is single threaded, code will only be running on any given request at any given moment, but as soon as that code hits an async operation (and thus has nothing to do until the async operation is done), another request could start running. So multiple requests can easily be "in flight" at the same time.
If you have a system where you need to keep track of multiple requests at once, you can coin a request id and attach it to each new request. One way to do that is with a few lines of express middleware that is early in the middleware stack that just adds a unique id property to each new request.
One simple way of coining a unique id is to just use a monotonically increasing counter.
I think it's a generally applicable question, but this is what I'm trying to do: small Node.JS app that receives a POST request from GroupMe whenever a message is posted in a group to trigger and fulfills a completely separate action. They state that the callback URL will receive an HTTP POST request from us every time a message is published to that group. It doesn't make much of a difference to me if I need to respond to the POST, but I was curious if it's even necessary or if it had any negative implications.
var server = http.createServer(function(request, response) {
doStuff(request);
});
It really depends what you're doing, but generally if no response is sent then applications or users which expect there to be one will wait until the connection times out, which may take up to a minute or longer, get a time out error and assume your service doesn't work.
You don't have to send any text, just a HTTP response code will do. Have doStuff() return whether it was successful and send the appropriate code based on it:
var server = http.createServer(function(request, response) {
response.send(doStuff(request) ? 200 : 400);
});
200 means OK, 400 means Bad Request
What I know in the field of HTTP request and response, You should at least say response.end().
If you won't say response.end client will be waiting for response. So better to say a response.end after tour task.
var server = http.createServer(function(request, response) {
doStuff(request);
response.end(); //It tells browser/client that server has finished it's work on this request.
});