I made my express application with the async+await.
However it took so long time to finish this task.
async.parallel(
{
cars: function (callback) {
Car.findById(req.params.id).exec(callback);
},
},
async (err, results) => {
if (err) {
return next(err);
} else {
var paramsDel = { Bucket: "test-s3-may", Key: results.cars.image_id };
s3.deleteObject(paramsDel, function (err, file) {
if (err) {
return next(err);
}
s3.upload(params, function (err, data) {
if (err) {
return next(err);
}
console.log("go to the next step"); //lineA
});
});
var updated_car = await Car.findByIdAndUpdate(req.params.id, car, {});
res.redirect(updated_car.url);
}
}
);
When I proceed this code, lineA(console.log("go to the next step) is displayed instantly.
After that, I have to wait almost 1-2 minutes for redirect commands.
Is it the problem of await? I'm sure that my internet speed is fast enough to upload any file.
You are assuming that the issue lies with the upload process, but following you method signature:
Car.findByIdAndUpdate
it seems rather to be local data storage based update call.
await is nothing but a syntactic sugar around Promise(s) with which you can write plain imperative statements in the old fashioned way and don't have to chain you completion callback into #then.
Thereby, awaited statement call will still cause the function to be paused in a non-blocking way until the underlying promise finishes (either fulfills or rejects).
In other words, the runtime is not blocked and further requests can be processed but the res.redirect(updated_car.url); won't be executed until the Car.findByIdAndUpdate function is resolved which thus needs to be profiled based on your storage system.
Related
I have a basic CRUD application using html forms, nodejs/express and mongodb. I have been learning about synchronous vs asynchronous code via callbacks, promises, and async/await and to my understanding for a crud application you would want the operations to be asynchronous so multiple users can do the operations at the same time. I am trying to implement aync/await with my express crud operations and am not sure if they are executing synchronously or asynchronously.
Here is my update function, which allows a user to type in the _id of the blog they want to change, then type in a new title and new body for the blog and submit it. In its current state, to my knowledge it is executing synchronously:
app.post('/update', (req, res) => {
const oldValue = { _id: new mongodb.ObjectId(String(req.body.previousValue)) }
const newValues = { $set: { blogTitle: req.body.newValue, blogBody: req.body.newValue2 } }
db.collection("miscData").updateOne(oldValue, newValues, function (err, result) {
if (err) throw err;
console.log("1 document updated");
res.redirect('/')
});
})
The way in which I was going to change this to asynchronous was this way:
app.post('/update', async (req, res) => {
const oldValue = { _id: new mongodb.ObjectId(String(req.body.previousValue)) }
const newValues = { $set: { blogTitle: req.body.newValue, blogBody: req.body.newValue2 } }
await db.collection("miscData").updateOne(oldValue, newValues, function (err, result) {
if (err) throw err;
console.log("1 document updated");
res.redirect('/')
});
})
Both blocks of code work, however I am not sure if the second block of code is doing what I am intending it to do, which is allow a user to update a blog without blocking the call stack, or if the second block of code would only make sense if I was running more functions after the await. Does this achieve the intended purpose, if not how could/should I do that?
db.collection(...).updateOne is always asynchronous, so you need not worry that a long-running database operation might block your application. There are two ways how you can obtain the asynchronous result:
With a callback function
db.collection(...).updateOne(oldValues, newValues, function(err, result) {...});
console.log("This happens synchronously");
The callback function with the two parameters (err, result) will be called asynchronously, after the database operation has completed (and after the console.log). Either err contains a database error message or result contains the database result.
With promises
try {
var result = await db.collection(...).updateOne(oldValues, newValues);
// Do something with result
} catch(err) {
// Do something with err
}
console.log("This happens asynchronously");
The updateOne function without a callback function as third parameter returns a promise that must be awaited. The statements that do something with result will be executed asynchronously, after the database operation has successfully completed. If a database error occurs, the statements in the catch block are executed instead. In either case (success or error), the console.log is only executed afterwards.
(If updateOne does not have a two-parameter version, you can write
var result = await util.promisify(db.collection(...).updateOne)(oldValues, newValues);
using util.promisify.)
Your second code snippet contains a mixture of both ways (third parameter plus await), which does not make sense.
I am trying to return jsonObj below and it is not returning. Help would be appreciated. Here fileName is the path of the file which I need to read. And again, I need to return jsonObj. So I can access it outside fs.readFile function.``
Here is the code
return fs.readFile(fileName, 'utf-8', (err, fileContent) => {
if(err) {
console.log(err); // Do something to handle the error or just throw it
throw new Error(err);
}
let jsonObj = csvjson.toObject(fileContent);
console.log(jsonObj)
return jsonObj
});
In Node you don't return from a callback function, that gets discarded and ignored. You must either use Promises, possibly in conjunction with async/await, or you must accept a callback function of your own.
This is the founding principle of asynchronous code:
// The fs.readFile() function executes immediately
fs.readFile(..., (err, ...) => {
// This code runs in the distant future, like imagine ten years from now
// from the perspective of the computer.
});
// JavaScript continues here immediately.
The readFile function does not return anything in particular, the inner function has not yet been run. It's also worth noting that throwing exceptions in there is pointless, don't do it.
To chain:
function getData(fileName, cb) {
fs.readFileName(fileName, 'utf-8', (err, fileContent) => {
if (err) {
// Back-propagate the error
return cb(err);
}
// Engage the callback function with the data
cb(null, csvjson.toObject(fileContent));
}
}
Note that this code is a lot less convoluted and messy if you use promises instead.
var urlArr = {
//ex url_for_site0 = 'https://www.google.com'
url_for_site0,
url_for_site1,
url_for_site2,
url_for_site3,
...
url_for_site50
};
urlArr.forEach(function(url, index) {
request(url, function(err, res, body) {
if(err) console.log(index+err);
else console.log(index+" success");
});
});
I got different unordered results and errors everytime I execute my app.
Example:
1 error : socket hang out
21 error : socket hang out
17 error : socket hang out
1 error : socket hang out
19 error : socket hang out
...(omission)
5 success
15 success
45 success
50 success
11 success
37 success
Everytime I get the results, they are in a different order.
Is this becasue I called too much request simultaneously?
When I request one by one, there's no error.
Example:
request(url_for_site0)
and restart program
request(url_for_site1)
and restart program
request(url_for_site2)
...
NodeJS events are all handled in a single pool and has a non-blocking nature. You can refer to the illustration below.
It happened to me once when I try to call multiple SQL queries. When I did it using C#, there is no problem at all. However, NodeJS gave me a similar behaviour to yours.
I am not sure if this is the best solution for the problem. However, here is how I fixed my problem with my SQL calls. I used the async waterfall function so that the whole process becomes synchronous. Each function will be run one by one with its return value piped to the next function. So, you can even do more stuffs. The usage of this library is not very staightforward, you can refer to this link to better help you understand how async waterfall works, then suit it to fit your solution.
https://gist.github.com/dineshsprabu/e6c1cf8f2ca100a8f5ae
Here is how I visualize your solution will roughly looks like:
var async = require('async');
async.waterfall(
[
function(callback) {
function_urlArr(url, index, function (returnVal) {
//Do something with the returnVal
callback(null, returnVal);
});
},
function(returnVal, callback) {
//the returnVal from first function gets passed here synchronously
function_urlArr(url2, index2, function (returnVal) {
//Do something with the returnVal
callback(null, returnVal);
});
},
function(returnVal, callback) {
//and so on ...
}
],
function (err) {
//console.log(err);
});
//define your function and enable callback
//you will need to include an extra third argument to receive the callback
function urlArr(url, index, callback) {
//your code
return callback(returnValue)
}
This is happening because of non-blocking nature of javascript.
If you want to make it happen one by one in order, you can use Async functions.
Socket hung up error may be because of the url you hit didn't respond anything after accepting the request.
You might have issue with non-blocking nature of loop forEach.
You can combine Promise and aysnc/await to make it blocking. Here is one way of handling it.
const request = require('request');
let urlArr = [
'https://localhost:9090',
'https://www.google.com',
'https://www.ebay.com',
'https://www.amazon.com',
];
//Creating promise for the requests.
let fetchPromise = function(url) {
return new Promise((resolve, reject) => {
request(url, (err, res, body) => {
if (err)
reject(Error(url + ' cannot be fetched'));
else
resolve(body);
});
}
);
};
//creating a blocking function
let fetchAllData = async function(urls) {
for (url of urls) { //using modern for loop instead for forEach
try {
data = await fetchPromise(url); // waiting until promise is resolved.
console.log('Recieved :' + data.length + 'bytes from ' + url);
} catch(e) {
console.log('Error :' + e); // catching error in case promise is rejected
}
}
};
//calling the function
fetchAllData(urlArr);
/*
// In case you want to wait until all promises are resolved.
// Then use Promise.all, however it will fail if any of the promise is rejected.
// One way to handle it would be to modify function fetchPromise such that it
// always resolves.
Promise
.all(urlArr.map(url => fetchPromise(url)))
.then(data => console.log(data))
.catch(err => console.log(err));
*/
I hope it helps.
Currently i have a dashboard that lists a bunch of records in a table. users can select 1 record and hit execute and i send a AJAX POST request to my routes middleware which executes 3 functions inside async.waterfall and returns a 200 response back to my client if everything works correctly. this async waterfall usually takes about 40-55 seconds to finish executing (fn_1,fn_2 and fn_3) and works perfectly fine.
router.post('/url', function(req, res, next) {
try {
async.waterfall([
fn_1,
fn_2,
fn_3
], function (err, body) {
res.writeHead(200, {'Content-Type': 'application/json'});
res.end(JSON.stringify({"error":err, "result":body}));
});
function fn_1(callback) {
callback(null, response);
}
function fn_1(result, callback) {
callback(err, result);
}
function fn_2(result, callback) {
callback(null, result);
}
}
catch (err){
console.log(err)
}
});
But, If i were to give provision for users to select MULTIPLE records and send that as an array back to my route middleware. how can i execute multiple async.waterfall methods for each item in the array in parallel
i can run a loop and execute the waterfall inside the loop but it again will wait for each item to complete and only then start the next iteration. this is not what i want.
is this doable in node / express . whats the easiest way to achieve this ? or are there modules/plugins that can help solve this case ?
Here is an abbreviated version of your code and how it could be changed to suit your needs. If none of your calls need data from any of the other calls, you can just run them in parallel with promises and use Promise.all to capture the result.
function fn_1(callback) {
// See function fn_2 for structure
}
function fn_1(result, callback) {
// See function fn_2 for structure
}
function fn_2(result, callback) {
return new Promise(resolve, reject => {
resolve(result)
})
.then(d => {
// Instead of callbacks, use a "then"
// block/statement.
//
// Do something with D here.
})
}
Promise.all([fn_1(), fn_2(), fn_3()])
.then(v => {
// Do somthing with v;
})
.catch(e => {
// Do something with e
})
I tend to advocate the use of native Promises over libraries like async, however, since your already using async...
You can use parallel and map each item in the array to a waterfall handler e.g.
async.parallel(
myArray.map(val => cb => async.waterfall(fn_1, fn_2, fn_3, cb)
, (err, results) => {
// return consolidated response
})
You would need to rework your waterfall handlers to not send a response but instead just propagate any errors.
It should also be noted that parallel is only useful if you are infact running I/O bound code, if the code is anything like your example then you won't really gain anything from using parallel over something like async.each
Where are people getting cb() from, is this a Node thing or vanilla JS thing?
For example:
Managing Node.js Callback Hell with Promises, Generators and Other Approaches
they're using cb() to I guess callback and return an error or a value or both in some cases depending on what the callback function sig is?
cb in the context you're describing it is how a vanilla callback function is passed into a (typically) asynchronous function, which is a common pattern in node.js (it's sometimes labelled next, but you can call it bananas if you so desire - it's just an argument).
Typically the first argument is an error object (often false - if all went as planned) and subsequent arguments are data of some form.
For example:
function myAsyncFunction(arg1, arg2, cb) {
// async things
cb(false, { data: 123 });
}
then using this function:
myAsyncFunction(10, 99, function onComplete(error, data) {
if (!error) {
// hooray, everything went as planned
} else {
// disaster - retry / respond with an error etc
}
});
Promises are an alternative to this design pattern where you would return a Promise object from myAsyncFunction
For example:
function myAsyncFunction2(arg1, arg2) {
return new Promise(function resolution(resolve, reject, {
// async things
resolve({ data: 123 });
});
}
then using this function:
myAsyncFunction2(10, 99)
.then(function onSuccess(data) {
// success - send a 200 code etc
})
.catch(function onError(error) {
// oh noes - 500
});
They're basically the same thing, just written slightly differently. Promises aren't supported especially widely in a native form, but if put through a transpiler (I'd recommend babel) during a build step they should perform reliably enough in a browser too.
Callbacks will always work in a browser with no shimming / transpilation.
node.js has lots of asynchronous operations that take a completion callback as an argument. This is very common in various node.js APIs.
The node.js convention for this callback is that the first argument passed to the callback is an error code. A falsey value for this first argument means that there is no error.
For example:
fs.readFile("test.txt", function(err, data) {
if (!err) {
console.log("file data is: " + data);
}
});
A function you create yourself may also define it's own callback in order to communicate the end of one or more asynchronous operations.
function getData(id, cb) {
var fname = "datafile-" + id + ".txt";
fs.readFile(fname, function(err, data) {
if (err) {
cb(err);
} else if (data.slice(0, 6) !== "Header"){
// proper header not found at beginning of file data
cb(new Error("Invalid header"));
} else {
cb(0, data);
}
});
}
// usage:
getData(29, function(err, data) {
if (!err) {
console.log(data);
}
});
From the Vanilla JS, you can declare a function and pass throuw parameters a declaration of another function, that can called async
https://developer.mozilla.org/en-US/docs/Glossary/Callback_function