I am trying to understand a case I am running into with my Node/Express application.
Let's say I have the following which is a function called when a specific route is hit:
async function routeThatDoesStuff(req, res, next) {
doAsyncStuff()
res.status(200).json({ message: 'Completed' })
}
In this case doAsyncStuff() does database operations on resources that are not critical to send back to the user so theoretically we don't need to await it. However, it seems that this operation does not actually complete unless I put the await in front.
My guess is that this has to do with the event loop etc in Node. That potentially because the route is completing before the doAsyncStuff() completes, the function doesn't actually complete because Node terminated it prematurely.
My big question is how Node handles async children function execution when parent functions have already completed?
An example to show that this should work:
const sleep = (milliseconds, cb) => {
return new Promise(resolve => setTimeout(() => {
resolve(cb())
}, milliseconds))
}
async function routeThatDoesStuff(req, res, next) {
sleep(5000, () => console.log('done'))
res.status(200).json({ message: 'Completed' })
}
JSON response is shown immediately, callback is executed after 5 seconds even though the response has been sent.
Related
I want to be able to exit execution of a post route when an event is sent from the client-side. I'm using socket.io but I'm not sure it can do what I want. I am using the uploads route to process a file, but if the user deletes the file, I want the app.post execution to end, similar to either a res.end() or return statement.
My app in the front-end receives a file from the user and immediately is sent to the post route for processing. If the user deletes the file and uploads a new one, the previous post route is still going. I want to make sure the previous one was terminated, cancelled, etc.
I'm currently using socket.io to communicate front-end to back-end.
How can I achieve this?
app.post('/uploads', async (req, res) => {
// async func1
// async func2
// if we receive an event from the front end while processing here, how can I exit the post route?
// async func3
});
You can add UUID for each request you make and return it to the front-end. The request will be resolved with the 202 ACCEPTED status code meaning the request was accepted and being handled but the HTTP request will be resolved.
Now you can implement a resourceManagerServeic that will allow APIs (http or ws) to change the state of a resource (like canceling it).
app.post('/uploads', async (req, res) => {
const resourceUuid = resourceManagerServeic.createResource();
res.status(202); // ACCEPTED
res.send({ uuid: resourceUuid });
// start besnise logic
await function1();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function2();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function3();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
});
app.del('/uploads/:resourceUuid', async (req, res) => {
resourceManagerServeic.cancle(req.params.resourceUuid);
res.end() // handle response
});
I guess that your are using Express. Take a look at express-async-handler
You can invoke it
const asyncHandler = require('express-async-handler')
app.post('/upload', asyncHandler(async(req, res) => {
await firstfunc()
await secondfunc()
}))
I am writing the middleware for API endpoints in my app that respond to webhooks from other applications, and am relatively new to Koa, so am not completely familiar with its patterns.
I would like to structure my middleware as follows:
exports.updateReceived = async (ctx, next) => {
// Respond to server issuing the webhook
ctx.res.body = "ok";
ctx.res.statusCode = 200;
// Grab what we need from the request
const { headers, state, request } = ctx;
const { body } = request;
// Do some async work
const { example } = await doSomeAsyncWork(ctx);
// Prepare a database query
const query = { aValue: anId };
// Run the DB query
const result = await Thing.findOne(query);
// Add data to the request
state.thing = result;
// Move on...
return next();
};
However, this does not appear to be working, as an error in any of my async methods can cause the route to error out.
My goal is for this endpoint to always respond "yep, ok" (immediately), meaning it is simply up to the application to handle any error states.
I have researched this fairly well, and have come across this pattern:
app.use(async ctx => {
db.fetch() // Assuming a Promise is returned
.then(() => { ... })
.catch(err => {
log(err)
})
// status 200 will be returned regardless of if db.fetch() resolves or rejects.
ctx.status = 200
})
However, this does not meet my needs as the middleware makes no use of next, so it is not really a useful pattern, so far as I can tell.
Could someone tell me what I am overlooking?
next() invokes the downstream middleware chain and returns a promise that resolves after all downstream middleware/handlers have finished.
That means you can simply implement your own upstream error handler that catches any errors and always ensures a 200 OK response.
const Koa = require('koa')
const app = new Koa()
app.use(async (ctx, next) => {
next().catch((err) => {
// Print error for our own records / debugging
console.error(err)
// But ensure that outgoing response is always a smile
ctx.status = 200
ctx.body = ':)'
})
})
app.use(async (ctx) => {
// Do your webhook / biz logic here, but for demonstration
// let's make it always throw an error. Thus upstream next()
// will be a rejected promise.
throw new Error('this middleware will always bubble up a rejected promise')
})
app.listen(3000, () => {
console.log('listening on 3000')
})
Note: We are not awaiting next(), so we can end the request immediately. However the next handler in the chain will still have the opportunity to process the logic
app.use((ctx, next) => {
next()
ctx.status = 200
})
app.use( async ctx =>{
db.fetch()
.then(() => { ... })
.catch(err => log(err))
}
}
Just to divert the solution in a different side, You could consider adding your work to some kind of MessageQueue and then let another process do that task for you. Basically asynchrously but you will still be important. This kind of pattern suits for your requirement.
There are many messaging system availble like AWS SQS which you could consider. This way your api will be very light weight and it will do thing which it needs to and send a command to your messaging system to do extra stuff. You are basically separting your core logic and the doing things in background which scales very nicely as well.
I'm quite new to Nodejs. In the following code I am getting json data from an API.
let data_json = ''; // global variable
app.get('/', (req, res) => {
request('http://my-api.com/data-export.json', (error, response, body) => {
data_json = JSON.parse(body);
console.log( data_json ); // data prints successfully
});
console.log(data_json, 'Data Test - outside request code'); // no data is printed
})
data_json is my global variable and I assign the data returned by the request function. Within that function the json data prints just fine. But I try printing the same data outside the request function and nothing prints out.
What mistake am I making?
Instead of waiting for request to resolve (get data from your API), Node.js will execute the code outside, and it will print nothing because there is still nothing at the moment of execution, and only after node gets data from your api (which will take a few milliseconds) will it execute the code inside the request. This is because nodejs is asynchronous and non-blocking language, meaning it will not block or halt the code until your api returns data, it will just keep going and finish later when it gets the response.
It's a good practice to do all of the data manipulation you want inside the callback function, unfortunately you can't rely on on the structure you have.
Here's an example of your code, just commented out the order of operations:
let data_json = ''; // global variable
app.get('/', (req, res) => {
//NodeJS STARTS executing this code
request('http://my-api.com/data-export.json', (error, response, body) => {
//NodeJS executes this code last, after the data is loaded from the server
data_json = JSON.parse(body);
console.log( data_json );
//You should do all of your data_json manipluation here
//Eg saving stuff to the database, processing data, just usual logic ya know
});
//NodeJS executes this code 2nd, before your server responds with data
//Because it doesn't want to block the entire code until it gets a response
console.log(data_json, 'Data Test - outside request code');
})
So let's say you want to make another request with the data from the first request - you will have to do something like this:
request('https://your-api.com/export-data.json', (err, res, body) => {
request('https://your-api.com/2nd-endpoint.json', (err, res, body) => {
//Process data and repeat
})
})
As you can see, that pattern can become very messy very quickly - this is called a callback hell, so to avoid having a lot of nested requests, there is a syntactic sugar to make this code look far more fancy and maintainable, it's called Async/Await pattern. Here's how it works:
let data_json = ''
app.get('/', async (req,res) => {
try{
let response = await request('https://your-api.com/endpoint')
data_json = response.body
} catch(error) {
//Handle error how you see fit
}
console.log(data_json) //It will work
})
This code does the same thing as the one you have, but the difference is that you can make as many await request(...) as you want one after another, and no nesting.
The only difference is that you have to declare that your function is asynchronous async (req, res) => {...} and that all of the let var = await request(...) need to be nested inside try-catch block. This is so you can catch your errors. You can have all of your requests inside catch block if you think that's necessary.
Hopefully this helped a bit :)
The console.log occurs before your request, check out ways to get asynchronous data: callback, promises or async-await. Nodejs APIs are async(most of them) so outer console.log will be executed before request API call completes.
let data_json = ''; // global variable
app.get('/', (req, res) => {
let pr = new Promise(function(resolve, reject) {
request('http://my-api.com/data-export.json', (error, response, body) => {
if (error) {
reject(error)
} else {
data_json = JSON.parse(body);
console.log(data_json); // data prints successfully
resolve(data_json)
}
});
})
pr.then(function(data) {
// data also will have data_json
// handle response here
console.log(data_json); // data prints successfully
}).catch(function(err) {
// handle error here
})
})
If you don't want to create a promise wrapper, you can use request-promise-native (uses native Promises) created by the Request module team.
Learn callbacks, promises and of course async-await.
I'm using react, electron, nodejs, asyncjs redux and thunk.
I wrote the following code which is supposed to download a list of files and write it to disk. In my code when the user presses a button i call this actionCreator:
export function downloadList(pack) {
return (dispatch, getState) => {
const { downloadManager } = getState();
async.each(downloadManager.downloadQueue[pack].libs, async (url, callback) => {
const filename = url.split('/').pop().split('#')[0].split('?')[0];
await downloadFile(url, `dl/${filename}`);
callback();
}, (err) => {
if (err) {
console.log('A file failed to process');
} else {
dispatch({
type: DOWNLOAD_COMPLETED,
packName: pack
});
}
});
};
}
async function downloadFile(url, path) {
const file = fs.createWriteStream(path);
const request = https.get(url, (response) => {
response.pipe(file);
file.on('finish', () => {
file.close();
});
}).on('error', (err) => { // Handle errors
fs.unlink(path); // Delete the file async. (But we don't check the result)
});
}
It does what it's supposed to do but while it does that, it blocks the entire UI. I really can't understand why it's happening since if i use an
setTimeout
with 3000ms delay inside the async.each it doesn't block the UI.
Another strange behaviour is that if i use the eachLimit function of asyncJS it just downloads me the limit of files, so if i want to download 100 files but i set eachLimit to 10 parallel, it just downloads the first 10 files and then stops. Can you enlight me about this?
I wanted to use axios to download files since it doesn't need to know if the urls are http or https but i can't find any resource on using axios with stream responsetype
I can answer the first part. Pretty much every existent implementation of JavaScript runs on one thread. This means that the runtime is concurrent, but not parallel, i.e. the runtime does one and exactly one thing at a time. This means that if there is a function call that takes a while, it will block everything else. Therefore, something in the downloadList function is blocking the event loop. However, if you use setTimeout, then the downloadList function will be pushed onto the message queue, which will unblock the event and allow the UI to be rendered. For more information on the event loop check out this video
I'm looking for a solution to waiting for an event to happen before sending a HTTP response.
Use Case
The idea is I call a function in one of my routes: zwave.connect("/dev/ttyACM5"); This function return immediately.
But there exists 2 events that notice about if it succeed or fail to connect the device:
zwave.on('driver ready', function(){...});
zwave.on('driver failed', function(){...});
In my route, I would like to know if the device succeed or fail to connect before sending the HTTP response.
My "solution"
When an event happen, I save the event in a database:
zwave.on('driver ready', function(){
//In the database, save the fact the event happened, here it's event "CONNECTED"
});
In my route, execute the connect function and wait for the event to
appear in the database:
router.get('/', function(request, response, next) {
zwave.connect("/dev/ttyACM5");
waitForEvent("CONNECTED", 5, null, function(){
response.redirect(/connected);
});
});
// The function use to wait for the event
waitForEvent: function(eventType, nbCallMax, nbCall, callback){
if(nbCall == null) nbCall = 1;
if(nbCallMax == null) nbCallMax = 1;
// Looking for event to happen (return true if event happened, false otherwise
event = findEventInDataBase(eventType);
if(event){
waitForEvent(eventType, nbCallMax, nbCall, callback);
}else{
setTimeout(waitForEvent(eventType, callback, nbCallMax, (nbCall+1)), 1500);
}
}
I don't think it is a good practice because it iterates calls over the database.
So what are your opinions/suggestions about it?
I've gone ahead and added the asynchronous and control-flow tags to your question because at the core of it, that is what you're asking about. (As an aside, if you're not using ES6 you should be able to translate the code below back to ES5.)
TL;DR
There are a lot of ways to handle async control flow in JavaScript (see also: What is the best control flow module for node.js?). You are looking for a structured way to handle it—likely Promises or the Reactive Extensions for JavaScript (a.k.a RxJS).
Example using a Promise
From MDN:
The Promise object is used for asynchronous computations. A Promise represents a value which may be available now, or in the future, or never.
The async computation in your case is the computation of a boolean value describing the success or failure to connect to the device. To do so, you can wrap the call to connect in a Promise object like so:
const p = new Promise((resolve) => {
// This assumes that the events are mutually exclusive
zwave.connect('/dev/ttyACM5');
zwave.on('driver ready', () => resolve(true));
zwave.on('driver failed', () => resolve(false));
});
Once you have a Promise representing the state of the connection, you can attach functions to its "future" value:
// Inside your route file
const p = /* ... */;
router.get('/', function(request, response, next) {
p.then(successful => {
if (successful) {
response.redirect('/connected');
}
else {
response.redirect('/failure');
}
});
});
You can learn more about Promises on MDN, or by reading one of many other resources on the topic (e.g. You're Missing the Point of Promises).
Have you tried this? From the look of it, your zwave probably have already implemented an EventEmmiter, you just need to attach a listener to it
router.get('/', function(request, response, next) {
zwave.connect("/dev/ttyACM5");
zwave.once('driver ready', function(){
response.redirect(/connected);
});
});
There is a npm sync module also. which is used for synchronize the process of executing the query.
When you want to run parallel queries in synchronous way then node restrict to do that because it never wait for response. and sync module is much perfect for that kind of solution.
Sample code
/*require sync module*/
var Sync = require('sync');
app.get('/',function(req,res,next){
story.find().exec(function(err,data){
var sync_function_data = find_user.sync(null, {name: "sanjeev"});
res.send({story:data,user:sync_function_data});
});
});
/*****sync function defined here *******/
function find_user(req_json, callback) {
process.nextTick(function () {
users.find(req_json,function (err,data)
{
if (!err) {
callback(null, data);
} else {
callback(null, err);
}
});
});
}
reference link: https://www.npmjs.com/package/sync