In my gulp build I've made a task that runs after all compiling, uglifying and minification has occurred. This task simply copies everything from the src into the dest directory that hasn't been touched/processed by earlier tasks. The little issue I'm having is that this results in empty directories in the dest directory.
Is there a way to tell the gulp.src glob to only include files in the pattern matching (like providing the 'is_file' flag)?
Thanks.
Fixed it by adding a filter to the pipeline:
var es = require('event-stream');
var onlyDirs = function(es) {
return es.map(function(file, cb) {
if (file.stat.isFile()) {
return cb(null, file);
} else {
return cb();
}
});
};
// ...
var s = gulp.src(globs)
.pipe(onlyDirs(es))
.pipe(gulp.dest(folders.dest + '/' + module.folder));
// ...
I know I'm late to the party on this one, but for anyone else stumbling upon this question, there is another way to do this that seems pretty elegant in my eyes. I found it in this question
To exclude the empty folders I added { nodir: true } after the glob pattern.
Your general pattern could be such (using the variables from Nick's answer):
gulp.src(globs, { nodir: true })
.pipe(gulp.dest(folders.dest + '/' + module.folder));
Mine was as follows:
gulp.src(['src/**/*', '!src/scss/**/*.scss', '!src/js/**/*.js'], { nodir: true })
.pipe(gulp.dest('dev/'));
This selects all the files from the src directory that are not scss or js files, and does not copy any empty folders from those two directories either.
Related
Using archiver.js (for Node.js), I need to exclude images from a recursive (multi-subdir) archive. Here is my code:
const zip = archiver('zip', { zlib: { level: 9 } });
const output = await fs.createWriteStream(`backup/${fileName}.zip`);
res.setHeader('Content-disposition', `attachment; filename=${fileName}.zip`);
res.setHeader('Content-type', 'application/download');
output.on('close', function () {
res.download(`backup/${fileName}.zip`, `${fileName}.zip`);
});
output.on('end', function () {
res.download(`backup/${fileName}.zip`, `${fileName}.zip`);
});
zip.pipe(output);
zip.glob('**/*',
{
cwd: 'user_uploads',
ignore: ['*.jpg', '*.png', '*.webp', '*.bmp'],
},
{});
zip.finalize();
The problem is that it did not exclude the ignore files. How can I correct the syntax?
Archiver uses Readdir-Glob for globbing which uses minimatch to match.
The matching in Readdir-Glob (node-readdir-glob/index.js#L147) is done against the full filename including the pathname and it does not allow us to apply the option matchBase which will much just the basename of the full path.
In order for to make it work you have 2 options:
1. Make your glob to exclude the file extensions
You can just convert your glob expression to exclude all the file extensions you don't want to be in your archive file using the glob negation !(...) and it will include everything except what matches the negation expression:
zip.glob(
'**/!(*.jpg|*.png|*.webp|*.bmp)',
{
cwd: 'user_uploads',
},
{}
);
2. Make minimatch to work with full file pathname
To make minimatch to work without us being able to set the matchBase option, we have to include the matching directory glob for it to work:
zip.glob(
'**/*',
{
cwd: 'user_uploads',
ignore: ['**/*.jpg', '**/*.png', '**/*.webp', '**/*.bmp'],
},
{}
);
Behaviour
This behaviour of Readdir-Glob is a bit confusing regarding the ignore option:
Options
ignore: Glob pattern or Array of Glob patterns to exclude matches. If a file or a folder matches at least one of the provided patterns, it's not returned. It doesn't prevent files from folder content to be returned.
This means that igrore items have to be actual glob expressions that must include the whole path/file expression. When we specify *.jpg, it will match files only in the root directory and not the subdirectories. If we want to exclude JPG files deep into the directory tree, we have to do it using the include all directories pattern in addition with the file extension pattern which is **/*.jpg.
Exclude only in subdirectories
If you want to exclude some file extensions only inside specific subdirectories, you can add the subdirectory into the path with a negation pattern like this:
// The glob pattern '**/!(Subdir)/*.jpg' will exclude all JPG files,
// that are inside any 'Subdir/' subdirectory.
zip.glob(
'**/*',
{
cwd: 'user_uploads',
ignore: ['**/!(Subdir)/*.jpg'],
},
{}
);
The following code is working with this directory structure :
node-app
|
|_ upload
|_subdir1
|_subdir2
|_...
In the code __dirname is the node-app directory (node-app is the directory where your app resides). The code is an adaptation of the code on https://www.archiverjs.com/ in paragraph Quick Start
// require modules
const fs = require('fs');
const archiver = require('archiver');
// create a file to stream archive data to.
const output = fs.createWriteStream(__dirname + '/example.zip');
const archive = archiver('zip', {
zlib: { level: 9 } // Sets the compression level.
});
// listen for all archive data to be written
// 'close' event is fired only when a file descriptor is involved
output.on('close', function() {
console.log(archive.pointer() + ' total bytes');
console.log('archiver has been finalized and the output file descriptor has closed.');
});
// This event is fired when the data source is drained no matter what was the data source.
// It is not part of this library but rather from the NodeJS Stream API.
// #see: https://nodejs.org/api/stream.html#stream_event_end
output.on('end', function() {
console.log('Data has been drained');
});
// good practice to catch warnings (ie stat failures and other non-blocking errors)
archive.on('warning', function(err) {
if (err.code === 'ENOENT') {
// log warning
} else {
// throw error
throw err;
}
});
// good practice to catch this error explicitly
archive.on('error', function(err) {
throw err;
});
// pipe archive data to the file
archive.pipe(output);
archive.glob('**',
{
cwd: __dirname + '/upload',
ignore: ['*.png','*.jpg']}
);
// finalize the archive (ie we are done appending files but streams have to finish yet)
// 'close', 'end' or 'finish' may be fired right after calling this method so register to them beforehand
archive.finalize();
glob is an abbreviation for 'global' so you use wildcards like * in the filenames ( https://en.wikipedia.org/wiki/Glob_(programming) ). So one possible accurate wildcard expression is *.jpg, *.png,... depending on the file type you want to exclude. In general the asterisk wildcard * replaces an arbitrary number of literal characters or an empty string in in the context of file systems ( file and directory names , https://en.wikipedia.org/wiki/Wildcard_character)
See also node.js - Archiving folder using archiver generate an empty zip
Using Node, I create a folder and then have a file in that folder. I created a function to delete it, but it absolutely refuses to find the folder.
Here's my function:
function deleteFile(path) {
if( !fs.existsSync(path) ) {
setTimeout(deleteFile(path), 500)
} else {
fs.readdirSync(path).forEach(function(file){
var curPath = path + "/" + file;
if(fs.lstatSync(curPath).isDirectory()) { // recurse
deleteFolderRecursive(curPath);
} else { // delete file
fs.unlinkSync(curPath);
}
});
fs.rmdirSync(path);
}
}
It will continue to recurse until it hits maximum call stack and crash, but the folder exists LONG before that happens. As you can see, there exists both the folder and the file inside of it. Could someone please help me fix this?
If anyone else comes across this issue, I figured it out. When the folder is created, it gives the incorrect permissions. I used fs.chmod to change permissions beforehand, and that fixed it.
I put hours into this and can't really understand what's going on here. I try to concat, minify and create sourcemaps for my css files. I want my folder structure like this:
- Assets
----bundles
--------css
----css
----maps
--------css
I used lots of variations of the following code but can't get it working. For example when I use the following code:
gulp.src("./Assets/css/*.css", { base: "." })
.pipe(sourcemaps.init())
.pipe(concat("./Assets/bundles/css/"))
.pipe(cssmin())
.pipe(gulp.dest("."))
.pipe(sourcemaps.write("./Assets/bundles/maps"))
.pipe(gulp.dest("."));
This creates a folder structure as:
./Assets/bundles/maps/Assets/bundles/css/myFile.css.map
However, when I use
gulp.src("./Assets/css/*.css", { base: "." })
.pipe(sourcemaps.init())
.pipe(concat("./Assets/bundles/css/"))
.pipe(cssmin())
.pipe(gulp.dest("."))
.pipe(sourcemaps.write("./css"))
.pipe(gulp.dest("."));
This outputs to my root project folder like
./css/Assets/bundles/css/myFile.css.map
What's going on here? I tried to use gulp-rename but couldn't make it work too.
Note: Gulp version is 3.9.1
I ended up using it like this
gulp.src("./Assets/css/*.css", { base: "." })
.pipe(sourcemaps.init())
.pipe(concat("./Assets/bundles/css/"))
.pipe(cssmin())
.pipe(gulp.dest("."))
.pipe(sourcemaps.write("./Assets/bundles/maps/css", {
sourceMappingURL: function (file) { //This is to reference the correct paths for our map file..
return "../maps/css/" + path.basename(file.path) + ".map"; //require the path module..
}
}))
.pipe(rename(function (path) {
if (path.extname == ".map") { //Only change the paths of map files not the .min files
path.dirname = "./Assets/bundles/maps/css";
}
}))
.pipe(gulp.dest("."));
My project structure is the following:
MyApp
- hooks
- platforms
- android
- ios
- www
- js / css / templates..
- lib (including all bower components)
Right now, the www/lib directory is taking up 21,8 Mb. (I have a large set of bower components added to my project.)
When building each project, the entire www folder is copied to the platform/android (for instance) folder for build, including of course www/lib.
This leads to a very big build, as lots of files included into bower
components are useless for production.
Manually managing all bower dependencies is clearly not an option. So how do you guys manage to clean your project platform directory for build?
I was thinking about creating a hook for that but before writing lines of code in a language that i do not know (nodeJS), I was hoping for your return and advises.
According to Cordova workflow you can add a hook script that removes unnecessary files.
A detailed example of a cleanup script can be found here: https://www.thepolyglotdeveloper.com/2015/01/hooks-apache-cordova-mobile-applications/
But to give a quick step by step summary:
Add to the after_prepare hook folder (/hooks/after_prepare) a script (01_junk_cleanup.js - 01 to be run first, the rest whatever you want) and in the file specify the files and folders you want to delete. For example, here is how you can delete a test folder and relevant files just change to you lib directory and to the files there. Note that this example is a bit different from the example in the link i gave earlier so you might want to take a look there as well.
01_junk_cleanup.js:
#!/usr/bin/env node
var fs = require('fs');
var path = require('path');
var foldersToProcess = [
"js",
"css"
];
var foldersToDelete = [
"test"
];
var filesToDelete = [
"karmaOnBrowser.conf.js",
"karmaOnEmulators.conf.js",
"SpecRunner.html"
];
var iosPlatformsDir = "platforms/ios/www/";
var androidPlatformsDir = "platforms/android/assets/www/";
filesToDelete.forEach(function(file) {
var filePathIOS = iosPlatformsDir + file;
var filePathAndroid = androidPlatformsDir + file;
if(fs.existsSync(filePathIOS)){
fs.unlinkSync(filePathIOS);
};
if(fs.existsSync(filePathAndroid)){
fs.unlinkSync(filePathAndroid);
};
});
foldersToProcess.forEach(function(folder) {
processFiles(iosPlatformsDir + folder);
processFiles(androidPlatformsDir + folder);
});
foldersToDelete.forEach(function(folder) {
deleteFolderRecursive(iosPlatformsDir + folder);
deleteFolderRecursive(androidPlatformsDir + folder);
});
function deleteFolderRecursive(path){
if( fs.existsSync(path) ) {
fs.readdirSync(path).forEach(function(file,index){
var curPath = path + "/" + file;
if(fs.lstatSync(curPath).isDirectory()) { // recurse
deleteFolderRecursive(curPath);
} else { // delete file
fs.unlinkSync(curPath);
}
});
fs.rmdirSync(path);
}
}
function processFiles(dir) {
fs.readdir(dir, function(err, list) {
if(err) {
console.log('processFiles err: ' + err);
return;
}
list.forEach(function(file) {
file = dir + '/' + file;
fs.stat(file, function(err, stat) {
if(!stat.isDirectory()) {
switch(path.basename(file)) {
case ".DS_Store":
fs.unlink(file, function(error) {
console.log("Removed file " + file);
});
break;
case "Thumbs.db":
fs.unlink(file, function(error) {
console.log("Removed file " + file);
});
break;
default:
console.log("Skipping file " + file);
break;
}
}
});
});
});
}
Aside to above, A bit more obvious but I feel worth mentioning anyhow, After having the www/lib bloat as well I always try to keep the folder lean and add only libraries required for deployment, the other dev. dependencies such as jasmine I either hold in the 'node_modules' folder or 'bower_components' as I only install today through them.
Hope this helps,
Good luck
I think the best approach would be to do this:
Move the bower_components folder and your index.html file to the project root, outside the /www folder
Install gulp and gulp-usemin
Wrap all of the .js files and .css files from bower components in usemin <build:js> and <build:css> sections
Configure a task in your gulpfile to concatenate all those files into a lib.js and a lib.css file. Make sure that those two files as well as the rewritten index.html are output to the /www folder
Execute the gulp task before your next build, and each time you add a new bower component.
This will keep your /www folder tidy and only containing the files you need in your cordova build.
With Bower you need to use npm preen to remove unnecessary files
See my example using Gulp with Ionic Framework: https://github.com/jdnichollsc/Ionic-Starter-Template
Basically you can set your bower.json file to indicate the path which files you need, for example:
"preen": {
//... More libraries
"ionic-datepicker": [
"dist/*.js" //You only need these files
//Other files and folders will be deleted to reduce the size of your app
],
"ion-floating-menu": [
"dist/*" //Keep all the files (.html, .css, .js, etc) of the directory.
]
}
Regards, Nicholls
This is an improvement over this answer. I've applied it to my own project.
Move the bower_components folder to the project root, outside the www folder.
Rename index.html to _index.html. We will later make sure that Gulp automatically generates index.html.
Install gulp and gulp-useref.
Edit your _index.html so that it looks something like this:
<!-- build:js dist/js/vendor.js -->
<script src="../bower_components/ionic/release/js/ionic.bundle.min.js"></script>
<script src="../bower_components/ngstorage/ngStorage.min.js"></script>
<script src="../bower_components/ngCordova/dist/ng-cordova.min.js"></script>
<!-- endbuild -->
Configure your gulp watch task to build new index.html file in the www folder with the concatenated files.
var entrypoint = './www/_index.html';
gulp.task('watch', function() {
gulp.watch(entrypoint, ['concat-js-and-css']);
});
gulp.task('concat-js-and-css', function () {
return gulp.src(entrypoint)
.pipe(useref())
.pipe(rename(function (path) {
// rename _index.html to index.html
if (path.basename == '_index' && path.extname == '.html') {
path.basename = "index";
}
}))
.pipe(gulp.dest('./www'))
});
gulp.task('build', ['concat-js-and-css']);
When that task runs, your index.html file will contain just this:
<script src="dist/js/vendor.js"></script>
Edit your ionic.project file so that it looks like the following. This will make sure that gulp watch is run before ionic serve.
{
"watchPatterns": [
"www/_index.html",
],
"gulpStartupTasks": [
"watch"
]
}
OK so I am creating a gulp plugin to create a table of contents and imports for my sass setup.
I have the gulp task all setup like this
var sgc = require('./tasks/sass-generate-contents');
gulp.task('sass-generate-contents', function(){
gulp.src(config.tests + '/**/*.scss')
.pipe(sgc(config.tests + '/_main.scss'));
});
and then my plugin code
function sassGenerateContents(){
return through.obj(function(file, enc, cb){
if (file.isNull()) {
cb(null, file);
return;
}
if (file.isStream()) {
cb(new gutil.PluginError(PLUGIN_NAME, 'Streaming not supported'));
return;
}
console.log(file);
});
}
module.exports = sassGenerateContents;
in my file system (the test directory) are 3 x *.scss files
the console log in the plugin only ever returns the first.
I'm probably missing somehting really obvious here, but I thought the idea of **/* was to loop through the folder structure specified and pull out the files.
Can someone explain to me why it's only returning the first file?
Thanks
I was missing the callback function, which called back to itself. At the end of the function i needed to ad
return cb();