I have an API POST route where I receive data from a client and upload the data to another service. This upload is done inside of the post request (async) and takes awhile. The client wants to know their post req was received prior to the async (create project function) is finished. How can I send without ending the POST? (res.send stops, res.write doesn't send it out)
I thought about making an http request back to their server as soon as this POST route is hit. . .
app.post('/v0/projects', function postProjects(req, res, next) {
console.log('POST notice to me');
// *** HERE, I want to send client message
// This is the async function
createProject(req.body, function (projectResponse) {
projectResponse.on('data', function (data) {
parseString(data.toString('ascii'), function (err, result) {
res.message = result;
});
});
projectResponse.on('end', function () {
if (res.message.error) {
console.log('MY ERROR: ' + JSON.stringify(res.message.error));
next(new Error(res));
} else {
// *** HERE is where they finally receive a message
res.status(200).send(res.message);
}
});
projectResponse.on('error', function (err) {
res.status(500).send(err.message);
});
});
});
The internal system requires that this createProject function is called in the POST request (needs to exist and have something uploaded or else it doesn't exist) -- otherwise I'd call it later.
Thank you!
I think you can't send first response that post request received and send another when internal job i.e. createProject has finished no matter success or fail.
But possibly, you can try:
createProject(payload, callback); // i am async will let you know when done! & it will push payload.jobId in doneJobs
Possibility 1, If actual job response is not required:
app.post('/v0/projects', function (req, res, next) {
// call any async job(s) here
createProject(req.body);
res.send('Hey Client! I have received post request, stay tuned!');
next();
});
});
Possibility 2, If actual job response is required, try maintaining queue:
var q = []; // try option 3 if this is not making sense
var jobsDone = []; // this will be updated by `createProject` callback
app.post('/v0/projects', function (req, res, next) {
// call async job and push it to queue
let randomId = randomId(); // generates random but unique id depending on requests received
q.push({jobId: randomId });
req.body.jobId = randomId;
createProject(req.body);
res.send('Hey Client! I have received post request, stay tuned!');
next();
});
});
// hit this api after sometime to know whether job is done or not
app.get('/v0/status/:jobId', function (req, res, next) {
// check if job is done
// based on checks if done then remove from **q** or retry or whatever is needed
let result = jobsDone.indexOf(req.params.jobId) > -1 ? 'Done' : 'Still Processing';
res.send(result);
next();
});
});
Possibility 3, redis can be used instead of in-memory queue in possibility 2.
P.S. There are other options available as well to achieve the desired results but above mentioned are possible ones.
Related
I want to be able to exit execution of a post route when an event is sent from the client-side. I'm using socket.io but I'm not sure it can do what I want. I am using the uploads route to process a file, but if the user deletes the file, I want the app.post execution to end, similar to either a res.end() or return statement.
My app in the front-end receives a file from the user and immediately is sent to the post route for processing. If the user deletes the file and uploads a new one, the previous post route is still going. I want to make sure the previous one was terminated, cancelled, etc.
I'm currently using socket.io to communicate front-end to back-end.
How can I achieve this?
app.post('/uploads', async (req, res) => {
// async func1
// async func2
// if we receive an event from the front end while processing here, how can I exit the post route?
// async func3
});
You can add UUID for each request you make and return it to the front-end. The request will be resolved with the 202 ACCEPTED status code meaning the request was accepted and being handled but the HTTP request will be resolved.
Now you can implement a resourceManagerServeic that will allow APIs (http or ws) to change the state of a resource (like canceling it).
app.post('/uploads', async (req, res) => {
const resourceUuid = resourceManagerServeic.createResource();
res.status(202); // ACCEPTED
res.send({ uuid: resourceUuid });
// start besnise logic
await function1();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function2();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function3();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
});
app.del('/uploads/:resourceUuid', async (req, res) => {
resourceManagerServeic.cancle(req.params.resourceUuid);
res.end() // handle response
});
I guess that your are using Express. Take a look at express-async-handler
You can invoke it
const asyncHandler = require('express-async-handler')
app.post('/upload', asyncHandler(async(req, res) => {
await firstfunc()
await secondfunc()
}))
I have built , using express() , a variety of methods. for simplicity let's I assume I built 2 POST() functions and I want to be able to use them by themselves and also to concatenate them via middleware for combine usage.
app.post('/create_obj_1' , function (req,res) {
//create Object_type_1
// send Object_type_1 via EXTERNAL API to somewhere
res.json({{ "statusCode": 200, "message": "OK" }
}
app.post('/create_obj_2' , function (req,res) {
//create Object_type_2
// send Object_type_2 via EXTERNAL API to somewhere
res.json({{ "statusCode": 200, "message": "OK" }
}
I want to have a new POST() that can invoke both of the other 2 (but still support stand alone invoking of the original 2
I think it's possible via middleware but I am not sure how - this is how I thought the new POST() should look like -
app.post('/create_obj_all' , function (req,res) {
//I want to invoke the create_obj_1 & create_obj_2 , check all OK, and finish
res.json({{ "statusCode": 200, "message": "OK" }
}
I am not sure how to approach the middleware usage in such case.
On top - how can I connect them to use one each other res? let's say the EXTERNAL API returns some value from obj_1 creation which I want to use in obj_2 post() function..
a Pseudo code of my attempt to use request() inside the middlware_1 -
var middle_1 = function (req, res, next) {
req.middle_1_output = {
statusCode : 404,
message : "fail_1"
}
var options = {
method: 'PUT', url: `EXTERNAL_API`, headers:
{
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
request(options, function (error, response, body) {
if (error) throw new Error(error);
// CODE THAT DO SOMETHING AND GET INFORMATION
// OLD WAY OF res.send here , to allow using in post.POST() was - res.status(200).send(body);
//res.status(200).send(body);
req.middle_1_output.statusCode = 200;
req.middle_1_output.message = "hello world";
});
next(); // trigger next middleware
}
Given the current example, I don't think you can do it unless you tweak the middlewares for the first two routes a bit:
var middleware1 = function(req, res, next) {
//create Object_type_1
// send Object_type_1 via EXTERNAL API to somewhere
next(); // calling next() triggers the next middleware
};
var middleware2 = function(req, res, next) {
//create Object_type_2
// send Object_type_2 via EXTERNAL API to somewhere
next(); // calling next() triggers the next middleware
};
/**
* This middleware is only used to send success response
*/
var response_success = function(req, res) {
res.json({ "statusCode": 200, "message": "OK" });
}
app.post('/create_obj_1', middleware1, response_success);
app.post('/create_obj_2', middleware2, response_success);
app.post('/create_obj_all', middleware1, middleware2, response_success);
Note that this is a very simplistic solution that I made from your example. The actual implementation will depend on what input each middleware is expecting and what output they generate. Also unlike here, there may also be different middlewares for sending the response.
2nd Part Addressing the second part of your question, if I have got you correctly you want to pass the output from middleware1 to middleware2. You can simply attach the output to the req object before calling next();. Like so:
var middleware1 = function(req, res, next) {
// do something
some_external_api_call(function(error, data) {
if (error) {
// handle the error yourself or call next(error);
} else {
req.middleware1_output = data; // set the output of the external api call into a property of req
next();
}
});
};
var middleware2 = function(req, res, next) {
// check to see if the middleware1_output has been set
// (meaning that the middleware has been called from /create_obj_all )
if (req.middleware1_output) {
// do something with the data
} else {
// handle scenario when /create_obj_2 is called by itself
}
next(); // calling next() triggers the next middleware
};
Notice how you have to account for both scenarios where middleware2 is called from POST /create_obj_all or directly from POST /create_obj_2.
3rd Part You should call next from within the callback. See my above example. This is due to the asynchronous/non-blocking nature of javascript.
function middleware(req, res, next) {
// do something
call_1st_external_api(some_options, function(error, data) {
// executed after call_1st_external_api completes
req.output_of_1st_external_api = data; // store the data of this api call for access from next middleware
next(); // calls the next middleware
// nothing here will be executed as next has already been called
});
// anything here will be executed before call_1st_external_api is completed
next(); // this will call the next middleware before call_1st_external_api completes
}
To handle two external APIs in the same middlewares you have to nest them (or use async or promises):
function middleware(req, res, next) {
// do something
call_1st_external_api(some_options, function(error1, data1) {
// executed after call_1st_external_api completes
req.output_of_1st_external_api = data1; // store the data of this api call for access from next middleware
// executed after call_2nd_external_api completes
call_2nd_external_api(some_options, function(error2, data2) {
req.output_of_2nd_external_api = data2; // store the data of this api call for access from next middleware
next();
});
// anything here will be executed before call_2nd_external_api is completed
});
// anything here will be executed before call_1st_external_api is completed
}
You have to handle all the errors above like I've shown in the 2nd Part which I have not shown in the above example for the sake of simplicity.
I am using 'multiparty' parser to read and process multipart-form-data in my application. My app middleware reads the uploaded file contents, parses it on-the-fly, and if successful, sends to next middleware. If not, i have to abort the request and return error.
/**
* Middleware to read n parse uploaded multi-part form data
* for campaign creation.
* After successful parse, sends to next layer.
*
**/
var mp = require('multiparty');
module.exports = function(req, res, next) {
console.log("File processor at work...");
var form = new mp.Form();
form.on('part', function(part) {
if (!part.filename) { part.resume();}
else {
part.setEncoding('utf8');
part.on('readable', function() {
readFile(req, res, part);
});
part.on('error', function (e) {
console.log("Am i reached??");
//return res.status(400).json(e);
form.emit('error', e);
})
}
});
form.on('error', function(err) {
console.log("Error while processing upload.", 3);
res.status(400).json(err);
});
form.on('close', function() {
console.log("Reading finished. Forwarding to next layer...");
return next();
});
form.parse(req);
}
In case of any errors in during parsing (i.e. readFile()), I want to return HTTP error without having to consume the remaining part buffer. I am sure there could be a decent way to do this, but I am not getting it right.
I tried throwing exception from within my readFile(), trying to catch it in form.on('part', ...). Even though I was able to catch the exception, it didn't abort the flow. A return() from form.on(part,..) would would return from this evt handler function, but not from the outer function.
As per the nodejs streams documentation, I tried emitting error event from within readFile() and handle the error in part.on('error', ...). This also gives the same behaviour, and does not end the processing.
What am I missing here? Is there a proper way to tell the stream that I don't want to process it any further..?
I would like to get some help with the following problem. I'm writing my bsc thesis, and this small part of code would be responsible for registering a user. (I'm new at nodejs actually). I'm using express and mongoose for this too.
I would like to process the request data, and check for some errors, first I would like to check if all fields exist, secondly if someone already registered with this e-mail address.
Based on the errors (or on success), I would like to send different responses. If a field is missing, then a 400 Bad request, if a user exists, then 409 Conflict, and 200 OK, if everything is ok. But I would only like to do the callback if there are no errors, but I'm kinda stuck here... I get the error Can't set headers after they are sent, which is obvious actually, because JS continues processing the code even if a response is set.
app.post('/register', function (req, res) {
var user = new User(req.body);
checkErrors(req, res, user, registerUser);
});
var registerUser = function(req, res, user){
user.save(function(err, user){
if (err) return console.log(err);
});
res.sendStatus(200);
};
var checkErrors = function(req, res, user, callback){
var properties = [ 'firstName', 'lastName', 'email', 'password', 'dateOfBirth' ];
for(var i = 0; i < properties.length; i++){
if(!req.body.hasOwnProperty(properties[i])){
res.status(400).send('field ' + properties[i] + ' not found');
}
}
var criteria = {
email: req.body.email
};
User.find(criteria).exec(function(err, user){
if(user.length > 0){
res.status(409).send('user already exists');
}
});
callback(req, res, user);
};
I think the problem is in the for loop in checkErrors. Since you call res.status(400).send() within the loop, you can end up calling it multiple times, which will trigger an error after the first call since a response will already have been sent back to the client.
Inside the loop, you can instead add missing fields to an array, then check the length of the array to see if you should respond with a 400 or continue. That way, you will only call res.status(400).send() one time.
For example:
...
var missingFields = [];
for(var i = 0; i < properties.length; i++){
if(!req.body.hasOwnProperty(properties[i])){
missingFields.push(properties[i]);
}
}
if(missingFields.length > 0) {
return res.status(400).send({"missingFields" : missingFields});
}
...
In general, I advise that you put return in front of each res.send() call, to ensure that no others are accidentally called later on.
An example of this is:
User.find(criteria).exec(function(err, user){
// We put return here in case you later add conditionals that are not
// mutually exclusive, since execution will continue past the
// res.status() call without return
if(user.length > 0){
return res.status(409).send('user already exists');
}
// Note that we also put this function call within the block of the
// User.find() callback, since it should not execute until
// User.find() completes and we can check for existing users.
return callback(req, res, user);
});
You probably noticed that I moved callback(req, res, user). If we leave callback(req, res, user) outside the body of the User.find() callback, it is possible that it will be executed before User.find() is completed. This is one of the gotchas of asynchronous programming with Node.js. Callback functions signal when a task is completed, so execution can be done "out of order" in relation to your source code if you don't wrap operations that you want to be sequential within callbacks.
On a side note, in the function registerUser, if user.save fails then the client will never know, since the function sends a 200 status code for any request. This happens for the same reason I mentioned above: because res.sendStatus(200) is not wrapped inside the user.save callback function, it may run before the save operation has completed. If an error occurs during a save, you should tell the client, probably with a 500 status code. For example:
var registerUser = function(req, res, user){
user.save(function(err, user){
if (err) {
console.error(err);
return res.status(500).send(err);
}
return res.sendStatus(201);
});
};
Your call to registerUser() is defined after the route and would be undefined since it's not a hoisted function.
Your use of scope in the closures isn't correct. For your specific error, it's because you're running res.send() in a loop when it's only supposed to be called once per request (hence already sent headers a.k.a. response already sent). You should be returning from the function directly after calling res.send() as well.
Background
Yes, there are a lot of different Node.js logging library winston, bunyan and console.log. It's easy to log down the information of the specific request when it has called and when and what information would be in response.
The problem
The problem begins with the sub function calls. When under one request your calling multiple functions which also uses the same logging, how would you pass the request meta - data to these log calls (function parameters seems to be one possible way but these are really messy) ?
Example
Small visual for coders:
// Middleware to set some request based information
app.use(function (req, res, next) {
req.rid = 'Random generated request id for tracking sub queries';
});
app.get('/', function (req, rest) {
async.series({
'users': async.apply(db.users.find),
'posts': async.apply(db.posts.find),
}, function (err, dbRes) {
console.log('API call made ', req.rid)
res.end(dbRes);
});
});
// Now the database functions are in other file but we also need to track down the request id in there
(db.js)
module.exports = {
users: {
find: function () {
console.log('Calling users listing ', req.rid); // ERROR this is not possible to access, not in this scope
// Make query and return result
}
},
posts: {
find: function () {
console.log('Calling post listing ', req.rid); // ERROR this is not possible to access, not in this scope
// Make query and return result
}
}
};
You can log your requests with simple conf in your app.js with;
app.use(function(req, res, next){
console.log('%s %s', req.method, req.url);
next();
});
However, you need to provide logs for specific functions in your controller.