How to create parameterized and reusable gulp tasks - node.js

Is there a way to make this generic to the point where I can have one copy of it and pass the config item, and file list into it rather than duplicating it for every file/config combination?
I'd love to have something like
gulp.task('foo_test', function (cb) {
run_tests(files.foo_list, config.fooCoverage);
cb();
}
Note on potential oddities in the code
I'm using lazypipe and gulp-load-plugins full file here
// test the server functions and collect coverage data
gulp.task('api_test', function (cb) {
gulp.src(files.api_files)
.pipe(istanbulPre())
.on('end', function () {
gulp.src(files.api_test_files)
.pipe(mochaTask())
.pipe(istanbulAPI())
.on('end', cb);
});
});
var istanbulAPI = lazypipe()
.pipe(plugins.istanbul.writeReports, config.apiCoverage);
config = {
apiCoverage: {
reporters: ['json'],
reportOpts: {
json: {
dir: 'coverage',
file: 'coverage-api.json'
}
}
},

Gulp is just JavaScript.
You can write plain old regular functions, just as you normally would:
function run_tests(srcFiles, srcTestFiles, coverageConfig, cb) {
var istanbul = lazypipe()
.pipe(plugins.istanbul.writeReports, coverageConfig);
gulp.src(srcFiles)
.pipe(istanbulPre())
.on('end', function () {
gulp.src(srcTestFiles)
.pipe(mochaTask())
.pipe(istanbul())
.on('end', cb);
});
}
gulp.task('unit_test', function (cb) {
run_tests(files.lib_files, files.unit_test_files, config.unitCoverage, cb);
});
gulp.task('api_test', function (cb) {
run_tests(files.api_files, files.api_test_files, config.apiCoverage, cb);
});
Note that the callback cb is just another parameter that is passed to the run_tests function. If it was called immediately after calling run_tests that would signal task completion to gulp before the asynchronous code in run_tests has actually finished.

Lazypipe was a solution (and there are other alternatives) but since Gulp 4 these do not seem to work anymore. Gulp 4 does not pass the stream to pipe functions. Yet the gulp.src(...) function returns a stream.
Also a nice feature of gulp 4 is that functions with Promises can also be a task.
So in the end I came up with this solution that worked for me. With my gulpfile.js looking something like this:
const {
src,
dest,
series,
parallel
} = require('gulp');
// other gulp packages omitted in this example...
const myReusableJsParser = (sources, destination) => {
return src(sources)
.pipe(stripComments({...}))
.pipe(obfuscator({compact:true}))
.pipe(...) //etc
.pipe(dest(destination));
};
const parseScriptsX = () => {
return myReusableJsParser('./js/x/**/*.js', './dist/js/x/');
}
const parseScriptsY = () => {
return myReusableJsParser('./js/y/**/*.js', './dist/js/y/');
}
// more
...
const cleanUp = () => new Promise((resolve, reject) => {
try {
deleteFiles('./dist/').then(resolve).catch(reject);
} catch(err) {
reject(err);
}
});
// export
module.exports = {
default: series(cleanUp, paralell(parseScriptsX, parseScriptsY), paralell(...)),
...,
clean: cleanUp
};

Related

Nestjs Framework: Run parallel task using Async

I'm struggling right now with "async".
I'm starting with NestJs Framework and i need run a bunch of task on parallel.
Before, using Nodejs + Express only, i have been useing something like this
...
...
.get('/some/path', function(){
async.parallel({
todayBalance: async.apply(function (cb) {
return SomeFunction(params, cb);
}),
currentWeek: async.apply(function (cb) {
return SomeFunction2(params, cb);
}),
...
...
// more tasks
},
function (err, results) {
# some code for logic
# here i get my result and gather data and RETURN
});
});
Nowadays, using NestJs framework, i have got something like this
myservice.ts
This is a service created for doing this.
// Service Definitions
async someFunction(userBusinessId: string): Promise<any> {
// i tried to use same strategy from Async
// but nothing accuring
async.parallel({
todayBalance: async.apply(function (cb) {
return SomeFunction(params, cb);
}),
currentWeek: async.apply(function (cb) {
return SomeFunction2(params, cb);
}),
...
...
// more tasks
},
function (err, results) {
# some code for logic
# here i get my result and gather data and RETURN
# DOESNT RETURN, NEVER EVER GETS HERE
});
}
Any idea what's wrong?
Thanks for your support!
First off I made two basics mistake
1. I Forgot use cb Function
using Nodejs + Express we just made something like this
In this case I use mysql
const SomeFunction= (userBusinessId, cb) => {
cnx.connection.query(`some pretty query`,
cb // <===== use Callback here
);
};
now using Nestjs i tried(badly results) to made something like this. Ignoring cb
const SomeFunction= (userBusinessId, cb) => {
const data = await getManager().query(`some pretty query`);
return data; // <===== WRONG INSTEAD USE cb Function, using like this parallel function will never triggered
};
Instead return single data, we must trigger the cb function overloading with result
Documentation
const SomeFunction= (userBusinessId, cb) => {
...
cb(null, data); // EXECUTE LIKE THIS, this will resume the pipe. null param suppose no error
};
2. Try to return the function service value inside the Async callback
Even if you try to do something like
async list():Promise<any>{
async.parallel({
...
// more tasks
},
function (err, results) {
return data;
});
}
OR
async list():Promise<any>{
const data = await async.parallel({
...
// more tasks
},
function (err, results) {
return data;
});
return data;
}
this funtion with always return undefined. even if you remove the Funtion Type Promise<any>
For avoid this you must return a Promise like
async list():Promise<any>{
return new Promise( (resolver, reject) => {
async.parallel({
...
// more tasks
},
function (err, results) {
if(err)
reject(err);
...
resolver(results);
});
} )
}
Hope this help you too!

Nodejs debug errors in production

I have a nodejs script running in production
I unlikely (once in thousand times) get errors like this:
TypeError: value is out of bounds
at checkInt (buffer.js:1009:11)
at Buffer.writeUInt16LE (buffer.js:1067:5)
at Object.foo.bar (/fake/path/name.js:123:1);
at Object.foo.bar2 (/fake/path/name2.js:123:1);
at Object.foo.bar3 (/fake/path/name3.js:123:1);
Causing the production server to crash...
Great I have a stacktrace! But I want to know what it's current data is for each call or all the data?
What are some great tools or code to use to for error logging(with it's current data) on production code?
I highly recommend in using either Winston or Bunyan. The selection of npm package is a decision of your application.
You can benchmark the available npm packages by going through the stats in the available in npm pages. The stats are basically the following.
downloads in the last day
downloads in the last week
downloads in the last month
open issues and open pull requests.
Having higher number of downloads recently will indicate that there is a great support for the module you are using in the long run. So that is important.
Both Winstan and Bunyan being best logging npm packages in the market, the main difference is that, Winstan is really awesome and flexible for normal logging purposes. Of course Winstan offers a great deal of logging capabilities. But how ever, to make use of these capabilities, some effort need to put on compared to Bunyan.
Bunyan on the other hand is specially support the fact of "analysing logs". So basically Bunyan is for logs processing. So if you want to analyse your logs, log files it is highly recommend to use Bunyan. Tweaking logs with Bunyan is fairly easy vis-a-vis to Winstan.
I did a thorough comparison between Bunyan and Winstan. Please do check the link below to view how the Winstan and Bunyan can use depending on the scope, use-case and necessity of the logging to the Node application.
link : https://docs.google.com/document/d/1pD9PLyxlcHVxxOvserNLO9tAz-QA_Co-xo6cWLhLghc/edit?usp=sharing
Also in the Production environment, make sure to use the logging levels wisely. The mostly used logging levels in production environment are:
error
info
debug
Yoou can use Winston or Pino
With winston you can load many modules for log where you want and maybe store logs online. I never use pino, but i have read good things about it.
Set env variables for choose where output your log, for example you wont to show output on stdout only if you are in develop and store online only if the app is in production.
A good way to handle the asynchronous functions in node.js is by using the decofun debug tool.
The main feature of it is to parse the code and the names of the anonymous functions according to their context.
You can deanonymise any anonymous function by running it with deco filename.js
A simple example would be as mentioned in the documentation
function gravy() {
return function returnedᅠfromᅠgravyᅠㅣlineᅠ2 () {
return {
prop: function asᅠpropertyᅠpropᅠㅣlineᅠ4 () {
setTimeout(function passedᅠintoᅠsetTimeoutᅠㅣlineᅠ5 () {
console.trace('Getting a trace...');
}, 10)
}
}
}
}
Trace: Getting a trace...
at passedᅠintoᅠsetTimeoutᅠㅣlineᅠ5 [as _onTimeout] (/home/ubuntu/workspace/node_modules/decofun/examples/loadable/index.js:6:22)
at Timer.listOnTimeout (timers.js:92:15)
Since it comes with embedded cute-stack library which normalises the path to the current directory
By applying the command deco examples/loadable --cute table
the output would be shown as
The best thing i like about it is that it transforms the functions based upon their calls to the original one as seen in the example from this
function one (a, cb) {
}
one('blah', function () {
})
function two () {
return function () { }
}
function three () {
return {
shoe: function () {}
}
}
function four () {
return function () {
return function () {
}
}
}
function five () {
return function () {
return function () {
return function () {
foo('blue', function () {
})
}
}
}
}
var six = function () {
}
var seven = function (err, cb) {
return function () {
cb(function () {
})
}
}
var o = {};
o.eight = function (cb) { }
o.eight(function () { })
o.eight.nine = function () {}
o.eight.nine(function () { })
var o2;
o2 = function () { }
;(function () {}())
!function () { }()
function toodeep () {
return function () {
return function () {
return function () {
return function () {
return function () {
return function () {
return function () {
return function () {
return function () {
return function () {
}
}
}
}
}
}
}
}
}
}
}
into this
function one (a, cb) {
}
one('blah', function passedᅠintoᅠoneᅠㅣlineᅠ6 () {
})
function two () {
return function returnedᅠfromᅠtwoᅠㅣlineᅠ11 () { }
}
function three () {
return {
shoe: function asᅠpropertyᅠshoeᅠㅣlineᅠ17 () {}
}
}
function four () {
return function returnedᅠfromᅠfourᅠㅣlineᅠ22 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠfourᅠᐳᅠㅣlineᅠ23 () {
}
}
}
function five () {
return function returnedᅠfromᅠfiveᅠㅣlineᅠ30 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠfiveᅠᐳᅠㅣlineᅠ31 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠfiveᅠᐳᅠᐳᅠㅣlineᅠ32 () {
foo('blue', function passedᅠintoᅠfooᅠㅣlineᅠ33 () {
})
}
}
}
}
var six = function asᅠvarᅠsixᅠㅣlineᅠ42 () {
}
var seven = function asᅠvarᅠsevenᅠㅣlineᅠ47 (err, cb) {
return function returnedᅠfromᅠᐸᅠasᅠvarᅠsevenᅠᐳᅠㅣlineᅠ49 () {
cb(function passedᅠintoᅠcbᅠㅣlineᅠ50 () {
})
}
}
var o = {};
o.eight = function asᅠpropertyᅠeightᅠㅣlineᅠ58 (cb) { }
o.eight(function passedᅠintoᅠoːeightᅠㅣlineᅠ61 () { })
o.eight.nine = function asᅠpropertyᅠnineᅠㅣlineᅠ63 () {}
o.eight.nine(function passedᅠintoᅠeightːnineᅠㅣlineᅠ64 () { })
var o2;
o2 = function asᅠvarᅠo2ᅠㅣlineᅠ68 () { }
;(function IIFEᅠㅣlineᅠ71 () {}())
!function IIFEᅠㅣlineᅠ73 () { }()
function toodeep () {
return function returnedᅠfromᅠtoodeepᅠㅣlineᅠ78 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠtoodeepᅠᐳᅠㅣlineᅠ79 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠtoodeepᅠᐳᅠᐳᅠㅣlineᅠ80 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠtoodeepᅠᐳᅠᐳᅠᐳᅠㅣlineᅠ82 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠtoodeepᅠᐳᅠᐳᅠᐳᅠᐳᅠㅣlineᅠ83 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠtoodeepᅠᐳᅠᐳᅠᐳᅠᐳᅠᐳᅠㅣlineᅠ84 () {
return function returnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠᐸᅠreturnedᅠfromᅠtoodeepᅠᐳᅠᐳᅠᐳᅠᐳᅠᐳᅠᐳᅠㅣlineᅠ86 () {
return function () {
return function () {
return function () {
}
}
}
}
}
}
}
}
}
}
}
Hope this might help a bit! Cheers!
I am using pm2 which is a process manager for node.js and also rollbar error reporting. I think you should define some metrics for the part of your code which this error comes from.
For any uncaughtException the server will stop in order to make the server keep on running even when there is an uncaught exception what i have done is created a separate collection for storing error, save error once an uncaught exception occurs and returns.
Collection
var ErrorSchema = new mongoose.Schema({
err_Message:{type:String},
err_Stack:{type:String},
date:{type:Date}
});
Controller
process.on('uncaughtException', function (err) {
console.log(err);
console.error((new Date).toUTCString() + ' uncaughtException:', err.message);
console.error(err.stack);
var newError = new Error;
newError.err_Message = err.message;
newError.err_Stack = err.stack;
newError.date = moment();
newError.save(function(saveErr,errData){
if(!saveErr)
console.log('New Error is saved');
else
console.log('Error in saving error');
});
//process.exit(1)
});
The above methods stores the uncaught exception in the Error collection and the process/server does not stops.
Hope this helps.

How can I stub a Promise such that my test can be run synchronously?

I am trying to unit test a module by stubbing one of its dependencies, in this case the UserManager
A simplified version of the module is as follows:
// CodeHandler
module.exports = function(UserManager) {
return {
oAuthCallback: function(req, res) {
var incomingCode = req.query.code;
var clientKey = req.query.key;
UserManager.saveCode(clientKey, incomingCode)
.then(function(){
res.redirect('https://test.tes');
}).catch(function(err){
res.redirect('back');
}
);
}
};
};
I'm stubbing the UserManager's saveCode function which returns a Promise such that it returns a resolved Promise, but when I assert that res.redirect has been called, alas at the time of the assertion res.redirect has not yet been called.
A simplified version of the unit test is:
// test
describe('CodeHandler', function() {
var req = {
query: {
code: 'test-code',
key: 'test-state'
}
};
var res = {
redirect: function() {}
};
var expectedUrl = 'https://test.tes';
var ch;
beforeEach(function() {
sinon.stub(UserManager, 'saveCode').returns(
new RSVP.Promise(function(resolve, reject){
resolve();
})
);
sinon.stub(res, 'redirect');
ch = CodeHandler(UserManager);
});
afterEach(function() {
UserManager.saveCode.restore();
res.redirect.restore();
});
it('redirects to the expected URL', function(){
ch.oAuthCallback(req, res);
assert(res.redirect.calledWith(expectedUrl));
})
});
How can I properly stub the promise such that the method under test behaves synchronously?
I've worked out a solution using sinon-stub-promise.
describe('CodeHandler', function() {
var req = {
query: {
code: 'test-code',
key: 'test-state'
}
};
var ch;
var promise;
var res = {
redirect: function() {}
};
beforeEach(function() {
promise = sinon.stub(UserManager, 'saveCode').returnsPromise();
ch = CodeHandler(UserManager);
sinon.stub(res, 'redirect');
});
afterEach(function() {
UserManager.saveCode.restore();
res.redirect.restore();
});
describe('can save code', function() {
var expectedUrl = 'https://test.tes';
beforeEach(function() {
promise.resolves();
});
it('redirects to the expected URL', function(){
ch.oAuthCallback(req, res);
assert(res.redirect.calledWith(expectedUrl));
});
});
describe('can not save code', function() {
var expectedUrl = 'back';
beforeEach(function() {
promise.rejects();
});
it('redirects to the expected URL', function(){
ch.oAuthCallback(req, res);
assert(res.redirect.calledWith(expectedUrl));
})
})
});
This works perfectly.
Well, the easiest thing would be not to stub it to run synchronously at all since that might change execution order and use Mocha's built in promises support (or jasmine-as-promised if using jasmine).
The reason is there can be cases like:
somePromise.then(function(){
doB();
});
doA();
If you cause promises to resolve synchronously the execution order - and thus output of the program changes, making the test worthless.
On the contrary, you can use the test syntax:
describe("the test", () => { // use arrow functions, node has them and they're short
it("does something", () => {
return methodThatReturnsPromise().then(x => {
// assert things about x, throws will be rejections here
// which will cause a test failure, so can use `assert`
});
});
});
You can use the even lighter arrow syntax for single lines which makes the test even less verbose:
describe("the test", () => { // use arrow functions, node has them and they're short
it("does something", () =>
methodThatReturnsPromise().then(x => {
// assert things about x, throws will be rejections here
// which will cause a test failure, so can use `assert`
});
);
});
In RSVP, you can't set the scheduler as far as I know so it's quite impossible to test things synchronously anyway, other libraries like bluebird let you do it at your own risk, but even in libraries that let you do it it's probably not the best idea.

Yeoman generator: how to run async command after all files copied

I'm writing a yeoman generator.
I need to run some shell script after all files copied.
The generator is being called as a child generator so it should wait till script finished.
The script is some command file being run via spawn:
that.spawnCommand('createdb.cmd');
As the script depends on files created by the generator it cannot run right inside generator's methods as all copy/template action are async and have not executed yet:
MyGenerator.prototype.serverApp = function serverApp() {
if (this.useLocalDb) {
this.copy('App_Data/createdb.cmd', 'App_Data/createdb.cmd');
// here I cannot run spawn with createdb.cmd as it doesn't exist
}
}
So the only place I found where I can run spawn is the 'end' event handler:
var MyGenerator = module.exports = function MyGenerator (args, options, config) {
this.on('end', function () {
if (that.useLocalDb) {
that.spawnCommand('createdb.cmd')
}
}
}
The script runs successfully but the generator finishes earlier than the child process. I need to tell Yeoman to wait for my child process.
Something like this:
this.on('end', function (done) {
this.spawnCommand('createdb.cmd')
.on('close', function () {
done();
});
}.bind(this));
But 'end' handler doesn't have the argument with 'dine' callback.
How to do this?
UPDATE:
thanks to #SimonBoudrias I got it working.
The full working code is below.
BTW: end method is described in the docs
var MyGenerator = module.exports = yeoman.generators.Base.extend({
constructor: function (args, options, config) {
yeoman.generators.Base.apply(this, arguments);
this.appName = this.options.appName;
},
prompting : function () {
// asking user
},
writing : function () {
// copying files
},
end: function () {
var that = this;
if (this.useLocalDb) {
var done = this.async();
process.chdir('App_Data');
this.spawnCommand('createdb.cmd').on('close', function () {
that._sayGoodbay();
done();
});
process.chdir('..');
} else {
this._sayGoodbay();
}
},
_sayGoodbay: funciton () {
// final words to user
}
});
Never trigger any action in the end event. This event is to be used by implementors, not generator themselves.
In your case:
module.exports = generators.Base({
end: function () {
var done = this.async();
this.spawnCommand('createdb.cmd').on('close', done);
}
});

Processing a method synchronously in Node.js

I've inherited some Node.js code and I need to add some functionality. However, I'm not sure syntactically how to accomplish my goal due to the asynchronous nature of Node. Currently, I have a function defined like this:
return {
myEntryPoint: function(req, res) {
var results = getResults();
res.send(200, results);
}
};
Several pieces are calling this function already. Inside of it, I'm calling a function called getResults which is defined like this:
var getResults = function() {
var results = [];
async.series([
function(callback) {
// add to results
},
function(callback) {
// add to results
}
]);
return results;
};
My problem is, I need to wait until all of the functions inside of the async.series call are made before I return the results. How do I do this?
You could change it to add a callback. Series has an optional callback to run once all the functions have completed within.
var getResults = function(finished) {
var results = [];
async.series([
function(callback) {
// add to results
},
function(callback) {
// add to results
}
], function() {
//Called when series is finished
finished(results);
});
};
And to get the results,
return {
myEntryPoint: function(req, res) {
getResults(function(results) {
res.send(200, results);
});
}
};
The usual way to handle this is to take a callback that uses the argument in question.
results = getResults(args);
someFunction(results);
Is equivalent to
function getResults(args, callback) {
var results;
//do stuff and populate results.
callback(results);
}
getResults(args, someFunction);
Except you don't have the issue of trying to wait for async stuff to happen, which you can't do, without weird logic.

Resources