Lets say I have the following Express/Connect middleware:
return function(req, res, next) {
mongoose.connect(url, options);
var Config = mongoose.model('Config', mongoose.Schema({
field1: Boolean,
field2: []
}));
Config.findOne({}, function (err, doc) {
if(!err) {
if(someCondition)
// some logic: send response and interrupt middleware chain
res.end('Some response');
else
next();
}
}
};
The issue is that database call is asynchronous. So middleware function exits before any logic executed.
The task is simple: read the configuration from mongoDB and if some field value = 'something" send response, otherwise - continue middleware chain.
So, I have 2 questions at the moment:
Is there any way to make asynchronous calls inside middleware?
If not, is there any workaround? (AFAIK, there is no way to execute db call synchronously)
So middleware function exits before any logic executed.
Yes, but this is totally normal and 100% OK for node.js asynchronous code. That's how asynchronous code works. When the DB call completes, control flow resumes and you can either send a response or call next() at that time. This is fine.
Is there any way to make asynchronous calls inside middleware?
Yes, and the way you are doing it is fine (minus some minor caveats listed below).
If not, is there any workaround?
No workaround necessary. This is how node.js works by design. All network I/O in node.js is asynchronous.
A few minor notes
Make sure you either send a response OR call next() in every possible branch of your database callback function logic: error, someCondition is true, and someCondition is false. If any of your logic can end up in a branch that doesn't respond to the request and doesn't call next() the request will just hang until it fails due to a timeout. Your snippet above is missing error handling, which you need to do.
Move your mongoose connect call to application startup time. Don't connect on each request.
Move your mongoose model definition out to a separate module Config.js
Related
Environment: Node.js, Express
Question: When the end of a route is hit does all asynchronous code that has not yet completed stop? I have been experimenting for several hours and this seems to be true but is there a definitive answer?
Example: In the simplified example below when the / route is hit getInfoAndSaveIt is called. This function has a fetch call that will be slow followed by a database save that will also be slow. In the route the program will certainly reach the line res.render('homepage'); before the function has completed. In this case does homepage render and all async code in the function that has not yet completed stop? Or alternately does the async code continue in the background even though the end of the route has been hit?
app.get('/', function(req, res) {
getInfoAndSaveIt();
res.render('homepage');
});
function getInfoAndSaveIt() {
let randomData = fetch('https://www.example.com/data.txt');
User.updateOne({ data: randomData }, { id: 'abc123' });
}
With Node.js, Express when the end of a route is hit does all asynchronous code that has not yet completed stop?
No. Asynchronous code has a mind of its own and has no direct connection to a particular route. For example, imagine you were using some asynchronous code to do some logging. You send the response and then you call some asynchronous code to log some data. That logging will continue on just fine. It has no direct connection to the route. It's just Javascript running in the interpreter.
As for your example:
In this case does homepage render and all async code in the function that has not yet completed stop?
Yes, the homepage will render. Your async code will finish on its own.
Or alternately does the async code continue in the background even though the end of the route has been hit?
Yes, the async code continues on its own.
Try this simple example:
app.get('/', function(req, res) {
setTimeout(() => {
console.log("timer fired");
}, 2000);
res.render('homepage');
console.log("done rendering page");
});
When you hit this route, you will see this in the server logs:
done rendering page
timer fired
As evidence that an asynchronous operation continues after the route is done.
Keep in mind that processing a route in node.js is just another asynchronous operation. It's not some magic process thing or thread thing. It's just an asynchronous operation triggered by the incoming http request. So, there's no magic properties that would go find all asynchronous operations started by this and kill them somehow. That's not how node.js works. It's the opposite. Once you start them, they have a mind of their own and will complete regardless of what else is doing on in the process unless the process crashes or you specifically call process.exit().
I have seen some questions about sending response immediately and run CPU intensive tasks.
My case is my node application depends on third party service responses so the process flow is
Node receives request and authenticates with third-party service
Send response to user after authentication
Do some tasks that needs responses from third party service
Save the results to database
In my case there is no CPU intensive tasks and no need to give results of additional tasks to the user but node needs to wait for responses from third-party service. I have to do multiple req/res to/from the third-party service after the authentication to complete the task.
How can I achieve this situation?
I have seen some workarounds with child_process, nextTick and setTimeOut.
Ultimately I want to send response immediately to user and do tasks related to that user.
Thanks in advance.
elsewhere in your code
function do_some_tasks() { //... }
// route function
(req, res) => {
// call some async task
do_some_tasks()
// if the above is doing some asynchronous task, next function should be called immediately without waiting, question is is it so?
res.send()
}
// if your do_some_tasks() is synchronous func, the you can do
// this function call will be put to queue and executed asynchronously
setImmediate(() => {
do_some_tasks()
})
// this will be called in the current iteration
res.send(something)
Just writing a very general code block here:
var do_some_tasks = (req, tp_response) => {
third_party_tasks(args, (err, result)=<{
//save to DB
});
}
var your_request_handler = (req,res) => {
third_party_auth(args, (tp_response)=>{
res.send();
//just do your tasks here
do_some_tasks(req, tp_response);
});
}
Currently still understanding the logic flow of a typical web application. In PHP, since its not async, typically we will have to wait for database insert/update to be completed before returning response to user.
But since in NodeJs we can have Async database query with callback and return response to user even before the callback is being called by database.
In these kind of situation, assuming validation of the data was already done, do typical nodeJs application wait for the callback before response or always assume the query is successful and return response to user?
I'm not entirely sure what you're asking but you should take a look at these explanations of async database queries:
Async explanation
Callback explanation
If I interpreted your question correctly you are asking when a node application would return a response, right? Typically you return a response to the user (like updating the client view) within the callback. By providing a callback you can make sure that the response is received and successful before moving on in that thread. Here is a bit of code to illustrate:
functionThatTakesForever(argument, function(error, response){ //this is the callback
if(error)
console.log(error);
else{
\\do something with response to return the response to the user
}
});
iOS application perform request for sending messages to users. I want to return result to application, and after that send push notification to users, and I don't want to wait until notifications were pushed successfully or not.
app.post("/message", function(req, res, next) {
User.sendMessages(query, options, function(err, results) {
res.json(results);
sendPushNotifications();
});
});
How can I do this?
That's how it works.
Keep in mind everything that happens in node is in a single thread, unlike other back-end languages you might be used to.
Requests, jobs, everything happens in that single thread. Unless, of course, you use cluster or something like that.
I have a very limited knowledge about node and nob-blocking IO so forgive me if my question is too naive.
In order to return needed information in response body, I need to
Make a call to 3rd party API
Wait for response
Add some modifications and return JSON response with the information I got from API.
My question is.. how can I wait for response? Or is it possible to send the information to the client only when I received response from API (as far as I know, connection should be bidirectional in this case which means I won't be able to do so using HTTP).
And yet another question. If one request waits for response from API, does this mean than other users will be forced to wait too (since node is single-threaded) until I increase numbers of threads/processes from 1 to N?
You pass a callback to the function which calls the service. If the service is a database, for example:
db.connect(host, callback);
And somewhere else in the code:
var callback = function(err, dbObject) {
// The connection was made, it's safe to handle the code here
console.log(dbObject.status);
res.json(jsonObject, 200)
};
Or you can use anonymous functions, so:
db.connect(host, function(err, dbObject) {
// The connection was made, it's safe to handle the code here
console.log(dbObject.status);
res.json(jsonObject, 200)
});
Between the call and the callback, node handles other clients / connections freely, "non-blocking".
This type of situation is exactly what node was designed to solve. Once you receive the request from your client, you can make a http request, which should take a callback parameter. This will call your callback function when the request is done, but node can do other work (including serving other clients) while you are waiting for the response. Once the request is done, you can have your code return the response to the client that is still waiting.
The amount of memory and CPU used by the node process will increase as additional clients connect to it, but only one process is needed to handle many simultaneous clients.
Node focuses on doing slow I/O asynchronously, so that the application code can start a task, and then have code start executing again after the I/O has completed.
An typical example might make it clear. We make a call to the FB API. When we get a response, we modify it and then send JSON to the user.
var express = require('express');
var fb = require('facebook-js');
app.get('/user', function(req, res){
fb.apiCall('GET', '/me/', {access_token: access_token}, function(error, response, body){ // access FB API
// when FB responds this part of the code will execute
if (error){
throw new Error('Error getting user information');
}
body.platform = 'Facebook' // modify the Facebook response, available as JSON in body
res.json(body); // send the response to client
});
});