I have a gulp task that I would like to run on multiple sets of files. My problem is pretty much similar to what is described here except that I define my sets of files in an extra config.
What I've come up with so far looks like the following:
config.json
{
"files": {
"mainScript": [
"mainFileA.js",
"mainFileB.js"
],
"extraAdminScript": [
"extraFileA.js",
"extraFileB.js"
]
}
}
gulpfile.js
var config = require ('./config.json');
...
gulp.task('scripts', function() {
var features = [],
dest = (argv.production ? config.basePath.compile : config.basePath.build) + '/scripts/';
for(var feature in config.files) {
if(config.files.hasOwnProperty(feature)) {
features.push(gulp.src(config.files[feature])
.pipe(plumper({
errorHandler: onError
}))
.pipe(jshint(config.jshintOptions))
.pipe(jshint.reporter('jshint-stylish'))
.pipe(sourcemaps.init())
.pipe(concat(feature + '.js'))
.pipe(gulpif(argv.production, uglify()))
.pipe(sourcemaps.write('.'))
.pipe(gulp.dest(dest))
);
}
}
return mergeStream(features);
});
My problem is that this doesn't seem to work. The streams are not combine or at least nothing really happens. Some while ago others ran into a similar problem, see here, but even though it should have been fixed it's not working for me.
By the way I've also tested merging the streams in this way:
return es.merge(features)
return es.merge.apply(null, features)
And if I just run the task on a single set of files it works fine.
Motivation
The reason why I want to do this is that at some point concatenating and minifying ALL scripts into one final file doesn't make sense when the sheer number of files is too large. Also, sometimes there is no need to load everything at once. For example all scripts related to an admin interface doesn't need to be load by every visitor.
Related
I'm trying to split out a "prebid" file , so as I can have seperate files from the "bidders", analytics client, bidder settings and some other bits. I've basically made my original file the main.js and have split out some of the code into different files such as
var pbjs = pbjs || {};
pbjs.que = pbjs.que || [];
pbjs.que.push(function() {
pbjs.addAdUnits(adUnits);
requirejs(['bidder_settings']);
requirejs(['pbjs_config']);
pbjs.requestBids({
bidsBackHandler: initAdserver,
timeout: PREBID_TIMEOUT
});
});
i'm trying to call in the files within the orginal file, so as it pulls those bits in, it sometimes works but other times it doesnt seem to load certain bits, any clue what i'm doing wrong/ is there a way to make sure the file loads the "modules/ seperate file" in sequencence down the page?
This sounds like an asynchronous race condition on your page where requirejs is not loading your modules by the time Prebid needs them to finish the auction. According to the [requirejs docs] (https://requirejs.org/docs/api.html#jsfiles) you should be using a callback to then run code that will require your loaded modules.
example:
var pbjs = pbjs || {};
pbjs.que = pbjs.que || [];
requirejs(['bidder_settings', 'pbjs_config', ],
function (bidder_settings, pbjs_config) {
pbjs.que.push(function () {
pbjs.addAdUnits(adUnits);
requirejs(['bidder_settings']);
requirejs(['pbjs_config']);
// do what you need with modules here
pbjs.requestBids({
bidsBackHandler: initAdserver, // make sure you utilize disableInitialLoad
timeout: PREBID_TIMEOUT
});
});
}
);
I need to alter a stream of files to contain a different base folder name. I thought the gulp-rename plugin would allow for this, but it only seems to replace the glob portion.
Example:
gulp.task("test", function() {
gulp.src("bower_components/**/*", { base: "bower_components", read:false })
.pipe($.rename(function (p) { p.dirname = "X/" + p.dirname; }))
.pipe($.print());
});
outputs:
[gulp] bower_components\X\jquery\test\data\offset\scroll.html
[gulp] bower_components\X\jquery\test\data\offset\static.html
[gulp] bower_components\X\jquery\test\data\offset\table.html
...
I want
[gulp] X\jquery\test\data\offset\scroll.html
[gulp] X\jquery\test\data\offset\static.html
[gulp] X\jquery\test\data\offset\table.html
...
Is there a way to do this with gulp-replace, or some other plugin?
I believe you could do this with gulp-tap to get a hold of the the file instances and alter properties on them before they get printed or use it to print them.
Out of curiosity what are you aiming to do?
Hope that helps!
EDIT-1::
The following is a slightly modified version of the example in the gulp-tap documentation which may work for your use case.
gulp.src("src/**/*.{coffee,js}")
.pipe(tap(function(file, t) {
file.path = 'X/' + file.path;
}))
.pipe($.print())
.pipe(gulp.dest('build'));
EDIT-2::
This is a common task I have set up in my projects for handling external scripts (note; I am using gulp-load-plugins hence invoking my plugins with plugins.<NAME>);
gulp.task('vendor:scripts:publish', function() {
return gulp.src(sources.vendor.js)
.pipe(plugins.plumber())
.pipe(plugins.concat('vendor.js'))
.pipe(gulp.dest(destinations.js))
.pipe(plugins.uglify())
.pipe(plugins.rename(pluginOpts.rename))
.pipe(gulp.dest(destinations.js));
});
destinations and sources are two variables that I have defined in a config file for my gulpfile.
But for clarity, sources.vendor.js points at an array much like the following;
js: [
'src/vendor/jquery/dist/jquery.js',
'src/vendor/lodash/lodash.js',
'src/vendor/backbone/backbone.js'
],
The reason my folder is named vendor and not bower_components is because I've made use of a .bowerrc file to point my bower installation at a different folder.
In addition if you have discrete scripts that you may not want to include all of the time you can look to make use of gulp-utils and gulp-filter to filter out certain scripts when an option is passed or not passed when gulp is invoked on the CLI.
For example; having gulp vendor:scripts:publish include all scripts but gulp vendor:scripts:publish --release omitting discrete scripts.
This then requires modifying your task to declare a filter that is piped in based on an option flag being picked up by gulp-utils.
var isRelease = (plugins.utils.env.release) ? true: false;
gulp.task('vendor:scripts:publish', function() {
var discreteFilter = plugins.filter([
'**/*.js',
'!**/discrete.min.js'
]);
return gulp.src(sources.vendor.js)
.pipe(plugins.plumber())
.pipe(isRelease ? discreteFilter: plugins.utils.noop())
.pipe(plugins.concat('vendor.js'))
.pipe(gulp.dest(destinations.js))
.pipe(plugins.uglify())
.pipe(plugins.rename(pluginOpts.rename))
.pipe(gulp.dest(destinations.js));
});
Hope that helps you out!
Using Express with Node.js, we might do something like this:
app.use('api/:controller/:action/:id', function(req,res,next){
var controller = req.params.controller;
var action = req.params.action;
var route = require('./routes/' + controller + '/' + action);
route(req,res,next);
}
now this is all fine and well, except there is at least one problem: the route file is dynamically loaded at runtime if this file has not been 'require'd yet. Which means it's a little bit slower at least.
Does someone have a script that recurses through a directory and pre-loads/pre-requires all the .js files when a server first starts up?
I have a similar problem for the front-end as well, using RequireJS. The solution seems to be to write a bash script that writes out all the .js filepaths in a directory and its subdirectories to a text file. then when the server starts up, it reads that text file and requires all the files in the directory that are listed in the text file. Is that the best way to do it?
If you can use io.js, it can preload modules using command-line -r or --require:
iojs -r <module_name> server.js
I created an NPM module that does this for the front-end, doing it for Node.js / CommonJS is another story.
https://www.npmjs.com/package/requirejs-metagen
you can use it like so:
var grm = require('requirejs-metagen'); //you can use with Gulp
var controllersOpts = {
inputFolder: './public/static/app/js/controllers/all',
appendThisToDependencies: 'app/js/controllers/',
appendThisToReturnedItems: '',
eliminateSharedFolder: true,
output: './public/static/app/js/meta/allControllers.js'
};
grm(controllersOpts,function(err){
//handle errors your own way
});
it generates a corresponding AMD/RequireJS module like so:
define(
[
"app/js/controllers/all/jobs",
"app/js/controllers/all/users"
],
function(){
return {
"jobs": arguments[0],
"users": arguments[1]
}
});
you can also require subdirectories and all that stuff like so:
var allViewsOpts = {
inputFolder: './public/static/app/js/jsx',
appendThisToDependencies: 'app/js/',
appendThisToReturnedItems: '',
eliminateSharedFolder: true,
output: './public/static/app/js/meta/allViews.js'
}
grm(allViewsOpts );
which generates output like so:
define([
"app/js/jsx/BaseView",
"app/js/jsx/reactComponents/FluxCart",
"app/js/jsx/reactComponents/FluxCartApp",
"app/js/jsx/reactComponents/FluxProduct",
"app/js/jsx/reactComponents/Item",
"app/js/jsx/reactComponents/Job",
"app/js/jsx/reactComponents/JobsList",
"app/js/jsx/reactComponents/listView",
"app/js/jsx/reactComponents/Picture",
"app/js/jsx/reactComponents/PictureList",
"app/js/jsx/reactComponents/RealTimeSearchView",
"app/js/jsx/reactComponents/Service",
"app/js/jsx/reactComponents/ServiceChooser",
"app/js/jsx/reactComponents/todoList",
"app/js/jsx/relViews/getAll/getAll",
"app/js/jsx/relViews/jobs/jobsView",
"app/js/jsx/standardViews/dashboardView",
"app/js/jsx/standardViews/overviewView",
"app/js/jsx/standardViews/pictureView",
"app/js/jsx/standardViews/portalView",
"app/js/jsx/standardViews/registeredUsersView",
"app/js/jsx/standardViews/userProfileView"
],
function(){
return {
"BaseView": arguments[0],
"reactComponents/FluxCart": arguments[1],
"reactComponents/FluxCartApp": arguments[2],
"reactComponents/FluxProduct": arguments[3],
"reactComponents/Item": arguments[4],
"reactComponents/Job": arguments[5],
"reactComponents/JobsList": arguments[6],
"reactComponents/listView": arguments[7],
"reactComponents/Picture": arguments[9],
"reactComponents/PictureList": arguments[10],
"reactComponents/RealTimeSearchView": arguments[11],
"reactComponents/Service": arguments[12],
"reactComponents/ServiceChooser": arguments[13],
"relViews/getAll/getAll": arguments[14],
"relViews/jobs/jobsView": arguments[15],
"standardViews/dashboardView": arguments[16],
"standardViews/overviewView": arguments[17],
"standardViews/pictureView": arguments[18],
"standardViews/portalView": arguments[19],
"standardViews/registeredUsersView": arguments[20],
"standardViews/userProfileView": arguments[21]
}
});
I need to update the library so it returns the stream so you can handle when it completes, otherwise it works great.
I've got an initConfig with this code in it:
grunt.initConfig({
pkg: grunt.file.readJSON('package.json'),
watch: {
options: {
spawn: false
},
coffee: {
files: [
'src/**/*.coffee'
],
tasks: ['coffee', 'coffeelint', 'concat', 'qunit']
},
...
coffee: {
glob_to_multiple: {
expand: true,
flatten: false,
cwd: '.',
src: ['src/**/*.coffee'],
ext: '.js'
}
},
...
grunt.event.on('watch', function (action, filepath) {
if (grunt.file.isMatch("**/*.coffee", filepath)) {
grunt.config(['coffee', 'glob_to_multiple', 'src'], filepath);
}
});
This is supposed to compile only the .coffee files that have changed. This works pretty well. But I just noticed that if I modify multiple at once, it will output this:
Waiting...src\test\resources\app\js\FILE1.coffee
src\main\resources\app\js\FILE2.coffee
OK
>> File "src\test\resources\app\js\FILE1.coffee" changed.
>> File "src\main\resources\app\js\FILE2.coffee" changed.
Running "coffee:glob_to_multiple" (coffee) task
File src/main/resources/app/js/FILE2.js created.
...
As you can see, I've changed two files, but it's only running the tasks on "FILE2.js". How can I avoid this? I want it to run coffee:glob_to_multiple on FILE1 and FILE2, not just one of them.
NOTE: I'm pretty sure the documentation explains how to do this:
If you save multiple files simultaneously you may opt for a more robust method:
var changedFiles = Object.create(null);
var onChange = grunt.util._.debounce(function() {
grunt.config(['jshint', 'all'], Object.keys(changedFiles));
changedFiles = Object.create(null);
}, 200);
grunt.event.on('watch', function(action, filepath) {
changedFiles[filepath] = action;
onChange();
});
Following that documentation, I made this change to my code:
var changedFiles = Object.create(null);
var onChange = grunt.util._.debounce(function() {
grunt.config(['coffee', 'glob_to_multiple', 'src'], Object.keys(changedFiles));
changedFiles = Object.create(null);
}, 200);
grunt.event.on('watch', function(action, filepath) {
if (grunt.file.isMatch("**/*.coffee", filepath)) {
changedFiles[filepath] = action;
onChange();
}
});
And things worked exactly the way I want. But I'm not sure how this works. Could someone explain it to me?
It's a pretty sophisticated solution using Lo-Dash debounce ;-) (in a sec...)
Know that when you used your older code of:
grunt.config(['coffee', 'glob_to_multiple', 'src'], filepath);
Grunt is instructed to run the coffee task with the new file. The problem with this is that it's a synchronic process and so when another file is changed ( usually this happens in a matter of milliseconds) then Grunt Watch won't allow you to run another process until the debounceDelay has been reached.
The default debounceDelay is 500 ms, but this can be changed using options of the watch task. (read more About option.debounceDelay
Basically when you save multiple files, as you saw - only the first file saved is changed. In order to bypass this, a great utility for delaying (debouncing) function run is in the grunt.util._.debounce (Lo-Dash link to it is here
The function's parameters are:
_.debounce(func, wait, options)
So it takes in the function, how many ms to wait, and some options (that we don't need here).
When you call the debounce utility it will delay the execution of the function the waitTime and that way - when you save multiple files at once - all the calls will add up to a single function call after those 200ms time period.
That way - the most useful line here, besides the debounce util is the following:
changedFiles[filepath] = action;
Which will add the files to the (at first) empty object of changedFiles. Notice that after the debounce function has launched we reset the changedFiles obj so that the next call will contain only freshly changed files.
Amazing solution indeed ;-)
I'm using requirejs and AMD modules for my TypeScript project, with something like 20 different source files at the moment and likely to grow substantially. All of this works, but it's very slow to load all 20 files, so it would be better to have them minified. But because of how requirejs wants to load everything, it seems like it's going to require that I keep the modules in separate files - I don't think I can just take the generated module1.js and module2.js files and minify them into one file and then have requirejs load those without changing some code. (I could be wrong on this.)
The other way that I see to do this is to use the r.js file that requirejs provides to merge all the different files together in a way that still keeps requirejs happy. But r.js requires node.js, and I'd rather not introduce that as a dependency in my build process if there's any other way to do it.
So before I dive into this and try half a dozen different solutions - how are other folks approaching this with big projects?
What you could do is to implement a thin RequireJS shim to use in a minified build. Depending on how much of the RequireJS API you want to use, you could get by with very little. For simplicity you could also use named modules.
Say, while developing you use RequireJS to load your modules. When you want to make a minified build, you could simply include a simple loader in the minified file.
If you have files app.js, foo.js and bar.js as follows:
//from app.js
define("app", ["foo", "bar"], function(foo, bar) {
return {
run: function() { alert(foo + bar); }
}
});
//from foo.js
define("foo", [], function() {
return "Hello ";
});
//from bar.js
define("bar", [], function() {
return "World!";
});
And let's say you minify all those files together. At the top of the file you include the following shim:
//from your-require-shim.js
(function(exports) {
var modules = {};
var define = function(name, dependencies, func) {
modules[name] = {
name:name,
dependencies:dependencies,
func:func,
result:undefined
};
};
var require = function(name) {
var module = modules[name];
//if we have cached result -> return
if(module.result) { return module.result; }
var deps = [];
//resolve all dependencies
for(var i=0,len=module.dependencies.length;i<len;i++) {
var depName = module.dependencies[i];
var dep = modules[depName];
if(!dep.result) {
//resolve dependency
require(depName);
}
deps.push(dep.result);
}
module.result = module.func.apply(this, deps );
return module.result;
};
exports.require = require;
exports.define = define;
}(window));
And execute the module defined in app.js
require("app").run();
Like in this fiddle.
It's a crude PoC of course, but I'm sure you get the meaning.
If you are using ASP.NET MVC 4, you can make a bundle which will minify everything when you deploy to production in a set of files or in a folder. You'll find more info on bundles here.