So I am not sure how to convert a function that I have that is built synchronously but uses asynchronous calls.
do_thing = () => {
var N = new Thing(); //sync
N.do_that(); // calls fs.readFile() and does some stuff with data :async
this.array.push(N); // sync but does not affect anything
this.save(); // calls fs.writeFile() but needs info from N that is not created yet as N.do_that() is not done
}
I am not sure how to make it so when N.do_that() is done it then calls this.save(). I do not want to use fs.readFileSync() or fs.writeFileSync(). I would like to know how to something like:
N.do_that().then( this.save() );
Ok I figured it out. Inside my N.do_that(); I added a callback, something like so:
do_that = (callback) => {
fs.readFile("file", (err, fd) => {
// do stuff with fd
callback();
return;
});
}
then rather than calling :
N.do_that();
array.push(N);
this.save();
I did
N.do_that(() => {
array.push(N);
this.save();
};
You could use a 3rd-party, open source tool that already gives you Promisified versions to help! One Google search revealed: https://www.npmjs.com/package/fs-promise
Otherwise, you can promisify the method yourself using tips from https://stackoverflow.com/a/34642827/3814251.
(Alternatively, use callbacks, which you've posted in your own answer.)
Related
can someone explain me how should we know that when we should use a callback?
like in the code given here as a link
snip of code is given here
we see that in readFile method inside fetchAll(cb), we used callback denoted by (cb) to read the content, parse it and stringify it and whatever, but in readFile method of save(), there was no need to use (cb). So how can we know when to use the callback?
Its simple. just be aware what is the nature of the methods you are using.
readFile is async so it will need a callback. What it is basically saying is "hey, I am going to read the file you asked, but while I read it you can do other stuff instead of waiting for me and when I am finished, I will come back to you". In your readFile method in save() you still pass a callback:
(err, fileContent) => {
// do stuff
}
Callback are used to handle async code so we can continue work, while something else is happening and we do not want to stop and wait for it.
const fs=require('fs')
const path=require('path')
module.exports=class Prroduct{
constructor(title,imgurl,description,price){
this.title=title
this.imgurl=imgurl
this.description=description
this.price=price
}
save(){
const p=path.join(__dirname,'../','data','products.json')
fs.readFile(p,(err,fileContent)=>{
let products=[]
if(!err){
products=JSON.parse(fileContent)
}
products.push(this)
fs.writeFile(p,JSON.stringify(products),(err)=>{
console.log(err)
})
})
}
static fetchAll(cb){
const p=path.join(__dirname,'../','data','products.json')
fs.readFile(p,(err,fileContent)=>{
if(err){
cb([])
}
cb(JSON.parse(fileContent))
})
}
}
Can I make a synchronous method into asynchronous by using promise?
For example reading a file synchronously (yes there is fs.readFile which has callback):
// Synchronous read
var data = fs.readFileSync('input.txt');
Should I do this:
function readFileAsync(){
return new Promise((resolve, reject) => {
try {
resolve(fs.readFileSync('input.txt'));
} catch(err) {
reject(err);
}
})
}
or use async/await:
function async readFileAsync(){
try {
let result = await fs.readFileSync('input.txt');
return result;
} catch(err) {
return err;
}
})
}
TL;DR NO, pure synchronous functions are not promisifiable in order to avoid blockage
No. For a method to be promisifiable it needs to be already asynchronous, i.e. return immediately, and also use callbacks upon finish.
For example:
function loop1000() {
for (let i = 0; i < 1000; ++i) {}
}
Is not promisifiable because it does not return immediately and does not use callbacks. But
function loop1000(err, callback) {
process.nextTick(() => {
for (let i = 0; i < 1000; ++i) { }
callback();
});
}
Is promisifiable as
function loop1000promisified() {
return new Promise((resolve, reject) => loop1000(resolve));
}
BUT all those approaches are going to block on the loop anyway. The original version blocks immediately and the one using process.nextTick() will block on the next processor tick. Making the application unresponsive for the duration of the loop.
If you wanted to make loop1000() asynchronous friendly you could rewrite it as:
function loop1000(err, callback) {
const segmentDuration = 10;
const loopEnd = 1000;
let i = 0;
function computeSegment() {
for (let segment = 0;
segment < segmentDuration && i < loopEnd;
++segment, ++i) { }
if (i == loopEnd) {
callback();
return;
}
process.nextTick(computeSegment);
}
computeSegment();
}
So instead of a longer blocking time it would have several smaller blockings. Then the promisified version loop1000promisified() could make some sense.
disclaimer: code typed directly on SO w/o any test.
Can I make a synchronous method into asynchronous by using promise?
No.
Can I make a synchronous method into asynchronous at all?
No. That's why promises don't help here. You need to use the natively asynchronous counterpart, i.e. fs.readFile instead of fs.readFileSync in your case.
Regarding your alternatives, you probably should do neither. But if you absolutely need a synchronous function that returns a fulfilled or rejected promise (instead of throwing exceptions), you can do
function readFileSync(){
return new Promise(resolve => {
resolve(fs.readFileSync('input.txt'))
});
}
or
async function readFileSync() {
return fs.readFileSync('input.txt');
}
I would re-phrase the the other to answers from "No" to "Not Really".
First a point of clarification: In NodeJS, everything is asynchronous, except your code. Specifically, one bit of your code will never run in parallel with another bit of your code -- but the NodeJS runtime may manage other tasks (namely IO) at the same time your code is being executed.
The beauty of functions like fs.readFile is that the IO happens in parallel with your code. For example:
fs.readFile("some/file",
function(err,data){console.log("done reading file (or failed)")});
do.some("work");
The second line of code will be executed while NodeJS is busily reading the file into memory. The problem with fs.readFileSync is that when you call it, NodeJS stops evaluating your code (all if it!) until the IO is done (i.e. the file has been read into memory, in this case). So if you mean to ask "can you take a blocking (presumably IO) function and make it non-blocking using promises?", the answer is definitely "no".
Can you use promises to control the order in which a blocking function is called? Of course. Promises are just a fancy way of declaring the order in which call backs are called -- but everything you can do with a promise, you can do with setImmediate() (albeit with a lot less clarity and a lot more effort).
I would disagree slightly with the others who say you should never promisify your function. There ARE cases when you want to promisify a function. For example a legacy code base that uses native processes and similar, where no callbacks and no promises were used, but you can assume the function is async and will execute within certain time.
Instead of writing a ton of setTimeout() callbacks you want to use promises.
This is how I do it for the testing purposes. Check the Ph library, especially the promisify function, and check how it is used to set up the mocha test in before function.
// Initial state
var foo = 1;
var xml = "";
// Promise helper library
var Ph = (function(){
return {
delay: function (milis){
var milis = milis || 200;
return function(){
return new Promise(function(resolve, reject){
setTimeout(function(){
resolve();
}, milis)
})
}
},
promisify: function(syncFunc){
return new Promise(function(resolve, reject){
syncFunc();
resolve();
})
}
}
}());
// 'Synchronous' functions to promisify
function setXML(){
console.log("setting XML");
xml = "<bar>";
}
function setVars(){
console.log("setting Vars");
foo = 2;
}
// Test setup
before(function(done) {
this.timeout(0);
Promise.resolve()
.then(promisify(setXML))
.then(Ph.delay(3000))
.then(Ph.promisify(setVars))
.then(Ph.delay(3000))
.then(function(){
done();
})
});
// Test assertions
describe("Async setup", function(done){
it("should have XML set", function(done){
expect(xml).to.be.not.equal("");
done();
});
it("should have foo not equal 1.", function(done){
expect(foo).to.be.not.equal(1);
done();
});
it("should have foo equal to 2.", function(done){
expect(foo).to.be.equal(2);
done();
});
});
To make it work in IE, I use Promise CDN:
<script src="https://cdnjs.cloudflare.com/ajax/libs/es6-promise/4.1.1/es6-promise.auto.min.js"></script>
Folks,
I have the following function, and am wondering whats the correct way to call the callback() only when the database operation completes on all items:
function mapSomething (callback) {
_.each(someArray, function (item) {
dao.dosomething(item.foo, function (err, account) {
item.email = account.email;
});
});
callback();
},
What I need is to iterate over someArray and do a database call for each element. After replacing all items in the array, I need to only then call the callback. Ofcourse the callback is in the incorrect place right now
Thanks!
The way you currently have it, callback is executed before any of the (async) tasks finish.
The async module has an each() that allows for a final callback:
var async = require('async');
// ...
function mapSomething (callback) {
async.each(someArray, function(item, cb) {
dao.dosomething(item.foo, function(err, account) {
if (err)
return cb(err);
item.email = account.email;
cb();
});
}, callback);
}
This will not wait for all your database calls to be done before calling callback(). It will launch all the database calls at once in parallel (I'm assuming that's what dao.dosomething() is). And, then immediately call callback() before any of the database calls have finished.
You have several choices to solve the problem.
You can use promises (by promisifying the database call) and then use Promise.all() to wait for all the database calls to be done.
You can use the async library to manage the coordination for you.
You can keep track of when each one is done yourself and when the last one is done, call your callback.
I would recommend options 1. or 2. Personally, I prefer to use promises and since you're interfacing with a database, this is probably not the only place you're making database calls, so I'd promisify the interface (bluebird will do that for you in one function call) and then use promises.
Here's what a promise solution could look like:
var Promise = require('bluebird');
// make promise version of your DB function
// ideally, you'd promisify the whole DB API with .promisifyAll()
var dosomething = Promise.promisify(dao.dosomething, dao);
function mapSomething (callback, errCallback) {
Promise.all(_.map(someArray, function(item) {
return dosomething(item.foo).then(function (account) {
item.email = account.email;
});
}).then(callback, errCallback);
}
This assumes you want to run all the DB calls in parallel and then call the callback when they are all done.
FYI, here's a link to how Bluebird promisify's existing APIs. I use this mechanism for all async file I/O in node and it saves a ton of coding time and makes error handling much more sane. Async callbacks are a nightmare for proper error handling, especially if exceptions can be thrown from async callbacks.
P.S. You may actually want your mapSomething() function to just return a promise itself so the caller is then responsible for specifying their own .then() handler and it allows the caller to use the returned promise for their own synchronization with other things (e.g. it's just more flexible that way).
function mapSomething() {
return Promise.all(_.map(someArray, function(item) {
return dosomething(item.foo).then(function (account) {
item.email = account.email;
});
})
}
mapSomething.then(mapSucessHandler, mapErrorHandler);
I haven't tried Bluebird's .map() myself, but once you've promisified the database call, I think it would simplify it a bit more like this:
function mapSomething() {
return Promise.map(someArray, function(item) {
return dosomething(item.foo).then(function (account) {
item.email = account.email;
});
})
}
mapSomething.then(mapSucessHandler, mapErrorHandler);
I have the following example code - the first part can result in an async call or not - either way it should continue. I cannot put the rest of the code within the async callback since it needs to run when condition is false. So how to do this?
if(condition) {
someAsyncRequest(function(error, result)) {
//do something then continue
}
}
//do this next whether condition is true or not
I assume that putting the code to come after in a function might be the way to go and call that function within the async call above or on an else call if condition is false - but is there an alternative way that doesn't require me breaking it up in functions?
A library I've used in Node quite often is Async (https://github.com/caolan/async). Last I checked this also has support for the browser so you should be able to npm / concat / minify this in your distribution. If you're using this on server-side only you should consider https://github.com/continuationlabs/insync, which is a slightly improved version of Async, with some of the browser support removed.
One of the common patterns I use when using conditional async calls is populate an array with the functions I want to use in order and pass that to async.waterfall.
I've included an example below.
var tasks = [];
if (conditionOne) {
tasks.push(functionOne);
}
if (conditionTwo) {
tasks.push(functionTwo);
}
if (conditionThree) {
tasks.push(functionThree);
}
async.waterfall(tasks, function (err, result) {
// do something with the result.
// if any functions in the task throws an error, this function is
// immediately called with err == <that error>
});
var functionOne = function(callback) {
// do something
// callback(null, some_result);
};
var functionTwo = function(previousResult, callback) {
// do something with previous result if needed
// callback(null, previousResult, some_result);
};
var functionThree = function(previousResult, callback) {
// do something with previous result if needed
// callback(null, some_result);
};
Of course you could use promises instead. In either case I like to avoid nesting callbacks by using async or promises.
Some of the things you can avoid by NOT using nested callbacks are variable collision, hoisting bugs, "marching" to the right > > > >, hard to read code, etc.
Just declare some other function to be run whenever you need it :
var otherFunc = function() {
//do this next whether condition is true or not
}
if(condition) {
someAsyncRequest(function(error, result)) {
//do something then continue
otherFunc();
}
} else {
otherFunc();
}
Just an another way to do it, this is how I abstracted the pattern. There might be some libraries (promises?) that handle the same thing.
function conditional(condition, conditional_fun, callback) {
if(condition)
return conditional_fun(callback);
return callback();
}
And then at the code you can write
conditional(something === undefined,
function(callback) {
fetch_that_something_async(function() {
callback();
});
},
function() {
/// ... This is where your code would continue
});
I would recommend using clojurescript which has an awesome core-async library which makes life super easy when dealing with async calls.
In your case, you would write something like this:
(go
(when condition
(<! (someAsyncRequest)))
(otherCodeToHappenWhetherConditionIsTrueOrNot))
Note the go macro which will cause the body to run asynchronously, and the <! function which will block until the async function will return. Due to the <! function being inside the when condition, it will only block if the condition is true.
I'm using Mongoose with Node.js and have the following code that will call the callback after all the save() calls has finished. However, I feel that this is a very dirty way of doing it and would like to see the proper way to get this done.
function setup(callback) {
// Clear the DB and load fixtures
Account.remove({}, addFixtureData);
function addFixtureData() {
// Load the fixtures
fs.readFile('./fixtures/account.json', 'utf8', function(err, data) {
if (err) { throw err; }
var jsonData = JSON.parse(data);
var count = 0;
jsonData.forEach(function(json) {
count++;
var account = new Account(json);
account.save(function(err) {
if (err) { throw err; }
if (--count == 0 && callback) callback();
});
});
});
}
}
You can clean up the code a bit by using a library like async or Step.
Also, I've written a small module that handles loading fixtures for you, so you just do:
var fixtures = require('./mongoose-fixtures');
fixtures.load('./fixtures/account.json', function(err) {
//Fixtures loaded, you're ready to go
};
Github:
https://github.com/powmedia/mongoose-fixtures
It will also load a directory of fixture files, or objects.
I did a talk about common asyncronous patterns (serial and parallel) and ways to solve them:
https://github.com/masylum/i-love-async
I hope its useful.
I've recently created simpler abstraction called wait.for to call async functions in sync mode (based on Fibers). It's at an early stage but works. It is at:
https://github.com/luciotato/waitfor
Using wait.for, you can call any standard nodejs async function, as if it were a sync function, without blocking node's event loop. You can code sequentially when you need it.
using wait.for your code will be:
//in a fiber
function setup(callback) {
// Clear the DB and load fixtures
wait.for(Account.remove,{});
// Load the fixtures
var data = wait.for(fs.readFile,'./fixtures/account.json', 'utf8');
var jsonData = JSON.parse(data);
jsonData.forEach(function(json) {
var account = new Account(json);
wait.forMethod(account,'save');
}
callback();
}
That's actually the proper way of doing it, more or less. What you're doing there is a parallel loop. You can abstract it into it's own "async parallel foreach" function if you want (and many do), but that's really the only way of doing a parallel loop.
Depending on what you intended, one thing that could be done differently is the error handling. Because you're throwing, if there's a single error, that callback will never get executed (count won't be decremented). So it might be better to do:
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
And handle the error in the callback. It's better node-convention-wise.
I would also change another thing to save you the trouble of incrementing count on every iteration:
var jsonData = JSON.parse(data)
, count = jsonData.length;
jsonData.forEach(function(json) {
var account = new Account(json);
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
});
If you are already using underscore.js anywhere in your project, you can leverage the after method. You need to know how many async calls will be out there in advance, but aside from that it's a pretty elegant solution.