I have a list of tasks that are initiated by an async function. I end up with a promise of an array of streams. I need to merge these streams into a single stream and return that to Gulp. The following is what I came up with. It's kind of clunky. Is there a better way to do this?
return es.readable(function(count, callback) {
var dest = this;
promise.then(function(streams) {
es.merge.apply(null, streams)
.on('end', dest.emit.bind(dest, 'end'))
.on('data', dest.emit.bind(dest, 'data'))
.on('error', dest.emit.bind(dest, 'error'));
});
});
es.merge returns Readable stream.
No reason to create one more with es.readable.
I have a long time without node.js experience, but think I write anything helpful :)
// async way
function merged(promise, next) {
promise.then(function(streams) {
next(null, es.merge.apply(null, streams));
}, next);
}
// like a promise
function merged(promise) {
return {
then: function(ok, fail) {
promise.then(function(streams) {
ok(es.merge.apply(null, streams));
}, fail);
}
};
}
Usage:
merged(promise, function(err, stream){stream.pipe(anyWritable);}); // async
merged(promise).then(function(stream){stream.pipe(anyWritable);}); // then
Related
I'm struggling right now with "async".
I'm starting with NestJs Framework and i need run a bunch of task on parallel.
Before, using Nodejs + Express only, i have been useing something like this
...
...
.get('/some/path', function(){
async.parallel({
todayBalance: async.apply(function (cb) {
return SomeFunction(params, cb);
}),
currentWeek: async.apply(function (cb) {
return SomeFunction2(params, cb);
}),
...
...
// more tasks
},
function (err, results) {
# some code for logic
# here i get my result and gather data and RETURN
});
});
Nowadays, using NestJs framework, i have got something like this
myservice.ts
This is a service created for doing this.
// Service Definitions
async someFunction(userBusinessId: string): Promise<any> {
// i tried to use same strategy from Async
// but nothing accuring
async.parallel({
todayBalance: async.apply(function (cb) {
return SomeFunction(params, cb);
}),
currentWeek: async.apply(function (cb) {
return SomeFunction2(params, cb);
}),
...
...
// more tasks
},
function (err, results) {
# some code for logic
# here i get my result and gather data and RETURN
# DOESNT RETURN, NEVER EVER GETS HERE
});
}
Any idea what's wrong?
Thanks for your support!
First off I made two basics mistake
1. I Forgot use cb Function
using Nodejs + Express we just made something like this
In this case I use mysql
const SomeFunction= (userBusinessId, cb) => {
cnx.connection.query(`some pretty query`,
cb // <===== use Callback here
);
};
now using Nestjs i tried(badly results) to made something like this. Ignoring cb
const SomeFunction= (userBusinessId, cb) => {
const data = await getManager().query(`some pretty query`);
return data; // <===== WRONG INSTEAD USE cb Function, using like this parallel function will never triggered
};
Instead return single data, we must trigger the cb function overloading with result
Documentation
const SomeFunction= (userBusinessId, cb) => {
...
cb(null, data); // EXECUTE LIKE THIS, this will resume the pipe. null param suppose no error
};
2. Try to return the function service value inside the Async callback
Even if you try to do something like
async list():Promise<any>{
async.parallel({
...
// more tasks
},
function (err, results) {
return data;
});
}
OR
async list():Promise<any>{
const data = await async.parallel({
...
// more tasks
},
function (err, results) {
return data;
});
return data;
}
this funtion with always return undefined. even if you remove the Funtion Type Promise<any>
For avoid this you must return a Promise like
async list():Promise<any>{
return new Promise( (resolver, reject) => {
async.parallel({
...
// more tasks
},
function (err, results) {
if(err)
reject(err);
...
resolver(results);
});
} )
}
Hope this help you too!
I have two arrays. I would like to iterate the arrays using async.foreach.
But when I do so, only the second array is getting executed.How to execute both.
Below is my code:
var _sizesArray1 = [_2000px, _1400px]
var _sizesArray2 = [_800px, _400px, _200px, _100px]
async.forEachOf(_sizesArray1, function(value, key, callback) {
async.waterfall([
function download(next){
//code
},
function convert(response, next) {
//code
},
function process(response, next) {
gm(response).command('convert')
.resize(_sizesArray[key].width,_sizesArray[key].width)
.gravity('Center')
.extent(_sizesArray[key].width,_sizesArray[key].width)
.quality('20')
.toBuffer(
'JPG', function(err,
buffer) {
if (err) {
next(err);
} else {
console.timeEnd(
"processImage array1"
);
next(null,
buffer,
key);
}
});
}
});
async.forEachOf(_sizesArray2, function(value, key, callback) {
async.waterfall([
function download1(next){
//code
},
function convert2(response, next) {
//code
},
function process3(response, next) {
//code
}
});
In my code, only array2 is getting invoked.Why don't first one get executed.
Is there any mistake in my code. Can somebody help resolve this.
How about this simple technique:
var allSizes = _sizesArray1.concat(_sizesArray2);
async.foreach(allSizes, function(value, key, next) {
// do whatever you like with each item in the concatenated arrays
// once you are done move to the next item
next()
})
Updated based on comments
Version 1, based on callbacks (welcome callback hell):
function asyncIterate(array, onEach, onDone) {
async.forEach(array, (val, index, onNext) => {
onEach(val, key, function onDecoratedNext() {
// tell the async.forEach it's ready for the next item
onNext();
// but if it got to the end,
// then mark the completion of the whole iteration
if (index === array.length) {
onDone();
}
});
});
}
and implement it like:
function doAsyncStuffWithEachItem(val, index, cb) {
// do async stuff with each item
// make sure to mark the completion of the async operation
cb();
}
asyncIterate(
_sizesArray1,
doAsyncStuffWithEachItem,
function onDone() {
// once the 1st iteration finished, time to move to the 2nd array
asyncIterate(
_sizesArray2,
doAsyncStuffWithEachItem,
function onDone2() {
// great we are done
// What if we have _sizesArray3, 4, 5 ... ?
// Well, we can continue to nest more callback here
// but in that way we'll soon end up with callback hell
// and that's a big NoNo
}
)
}
);
Version 2, based on Promises:
To avoid callback hell, luckily we can use Promises. Something like this should do it:
const promisifiedAsyncIterate = (array, onEach) =>
new Promise((resolve) => {
asyncIterate(array, onEach, resolve);
});
and use it like:
promisifiedAsyncIterate(_sizeArray1, doAsyncStuffWithEachItem)
.then(() => promisifiedAsyncIterate(_sizeArray2, doAsyncStuffWithEachItem))
.then(() => promisifiedAsyncIterate(_sizeArray3, doAsyncStuffWithEachItem))
.then(() => promisifiedAsyncIterate(_sizeArray4, doAsyncStuffWithEachItem))
It could be abstracted and cleaned up even more, or even made completely dynamic – say you have an array of sizeArrays that you pass in to your function, but I think this is enough for now :). Hope it helps.
Let's say I have 3 files.
index.js makes a call to the backend like this
$.post('/test/', data, function(response) {
//handle success here
})
routes.js handles the route like this
app.post('/test/', function(req, res){
item.getItems(function(response){
res.json(response);
});
});
items.js is the model which accesses the database and makes a POST request for each item
function getItems(callback) {
database.query('SELECT * from items', function(result){
result.forEach(function(item){
request.post('/api/', item, function(req, res) {
//finished posting item
});
});
});
//callback here doesnt wait for calls to finish
}
where/when should I call the callback passed to getItems() to handle a success/failure in index.js?
Because your request.post() operations are async, you have to use some method of keeping track of when they are all done and then you can call your callback. There are multiple ways of doing that. I'll outline a few:
Manually Keeping Track of Count of Request Operations
function getItems(callback) {
database.query('SELECT * from items', function(result){
var remaining = result.length;
result.forEach(function(item){
request.post('/api/', item, function(err, res) {
--remaining;
//finished posting item
if (remaining === 0) {
callback();
}
});
});
});
}
The main problem with doing this manually is that propagating error in nested async operations is difficult when you get to actually figuring out how you're going to handle errors. This is much easier in the other methods shown here.
Using Promises
// load Bluebird promise library
var Promise = require('bluebird');
// promisify async operations
Promise.promisifyAll(request);
function queryAsync(query) {
return new Promise(function(resolve, reject) {
// this needs proper error handling from the database query
database.query('SELECT * from items', function(result){
resolve(result);
});
});
}
function getItems(callback) {
return queryAsync('SELECT * from items').then(function(result) {
return Promise.map(result, function(item) {
return request.postAsync('/api/', item);
});
});
}
getItems.then(function(results) {
// success here
}, function(err) {
// error here
})
It seems strange that you're making an API request in your server-side code, unless this is some sort of middle tier code that interacts with the API... but you're interacting with a database, so I'm still confused on why you can't just do a database insert, or have a bulk insert API call?
Anyway, if you must do it the way you're asking, I've done this in the past with a recursive method that trims down the result array... I really don't know if this is a good approach, so I'd like to hear any feedback. Something like this:
function recursiveResult(result, successfulResults, callback) {
var item = result.shift();
// if item is undefined, then we've hit the end of the array, so we'll call the original callback
if (item !== undefined) {
console.log(item, result);
// do the POST in here, and in its callback, call recursiveResult(result, successfulResults, callback);
successfulResults.push(item);
return recursiveResult(result, successfulResults, callback);
}
// make sure callback is defined, otherwise, server will crash
else if (callback) {
return callback(successfulResults);
}
else {
// log error... callback undefined
}
}
function getItems(callback) {
var successfulResults = [];
var result = [1, 2, 3, 4];
recursiveResult(result, successfulResults, callback);
}
console.log('starting');
getItems(function(finalResult) {
console.log('done', finalResult);
});
How can I avoid using a recursion like structure when I got several streams to open and I have to get an absolute end event to finish the logic.
var someArray = ['file1', 'file2', 'file3'];
someArray.forEach(function( file ) {
fs
.createReadStream( file )
.pipe( /* do some stuff */ )
.on('data', function( usageInfo ) {
// done?
});
}
I got several files I have to pipe through tp some processes. How can I setup an event that tells me when all of them are done?
Currently what I'm getting is each end event individually.
I can absolutely start each stream at the same time. I just need to somehow collect the end?
I could invoke a function call for each end event and count it... that sounds hacky though?...
I feel like there is a way to do this with promises but I don't know how.
I feel like there is a way to do this with promises but I don't know how.
Yes, there is. As promises do represent asynchronous values, you'd get a promise for the end of one stream:
var p = new Promise(function(resolve, reject) {
fs.createReadStream(file)
.on('error', reject)
.pipe(/* do some stuff */)
.on('end', resolve)
.on('error', reject); // call reject(err) when something goes wrong
});
It could be used like p.then(functio(usageInfo) { console.log("stream ended"); }).
Now if you create multiple promises, one for filename in your array, all the streams will run in parallel and resolve their respective promise when done. You then can use Promise.all to collect - read "await" - all the results from each of them into a new promise for an array of results.
var promises = ['file1', 'file2', 'file3'].map(function(file) {
return new Promise(function(resolve, reject) {
…
});
});
Promise.all(promises).then(function(usageInfos) {
console.log("all of them done", usageInfos),
}, function(err) {
console.error("(At least) one of them failed", err);
});
Use a counter:
var someArray = ['file1', 'file2', 'file3'];
var still_processing = someArray.length;
someArray.forEach(function( file ) {
fs.createReadStream( file )
.pipe( /* do some stuff */ )
.on('end', function() {
still_processing--;
if (!still_processing) {
// done
}
});
}
This is the basic mechanism. This control flow pattern is encapsulated by the async.parallel() function in async.js:
var someArray = ['file1', 'file2', 'file3'];
var streams_to_process = [];
someArray.forEach(function( file ) {
streams_to_process.push(function(callback) {
var result = "";
fs.createReadStream( file )
.pipe( /* do some stuff */ )
.on('end', function() {
callback(null, result);
});
});
});
async.parallel(streams_to_process, function(err, results) {
// all done
});
Internally, async.parallel uses a counter captured in a closure to keep track of when all async processes (in this case the 'end' events) are done.
There are other libraries for this. Most promise libraries for example provide an .all() method that works the same way - internally keeping track of a counter value and firing the .then() callback when all is done.
I'm using the Q library and async library in nodejs.
Here's an example of my code:
async.each(items, cb, function(item) {
saveItem.then(function(doc) {
cb();
});
}, function() {
});
saveItem is a promise. When I run this, I always get cb is undefined, I guess then() doesn't have access. Any ideas how to work around this?
Your issue doesn't lie with promises, it lies with your usage of async.
async.each(items, handler, finalCallback) applies handler to every item of the items array. The handler function is asynchronous, i.e. it is handed a callback, that it must call when it has finished its work. When all handlers are done, the final callback is called.
Here's how you'd fix your current issue:
var handler = function (item, cb) {
saveItem(item)
.then(
function () { // all is well!
cb();
},
function (err) { // something bad happened!
cb(err);
}
);
}
var finalCallback = function (err, results) {
// ...
}
async.each(items, handler, finalCallback);
However, you don't need to use async for this particular piece of code: promises alone fill this job quite nicely, especially with Q.all():
// Create an array of promises
var promises = items.map(saveItem);
// Wait for all promises to be resolved
Q.all(promises)
.then(
function () { // all is well!
cb();
},
function (err) { // something bad happened!
cb(err);
}
)