Asynchronous operation of FileTree files? - multithreading

Question

Is there a way I can easily make the processing of FileTree files in a smart way in gradle tasks? I basically need to wait for the execution of all files, much like what you can do with GPars, but how do I do this gradle with FileTree?
task compressJs(dependsOn: [copyJsToBuild]) << {
println 'Minifying JS'
fileTree {
from 'build/js'
include '**/*.js'
}.visit { element ->
if (element.file.isFile()) {
println "Minifying ${element.relativePath}"
ant.java(jar: "lib/yuicompressor-2.4.6.jar", fork: true) {
arg(value: "build/js/${element.relativePath}")
arg(value: "-o")
arg(value: "build/js/${element.relativePath}")
}
}
}
}
It would be lovely if I could do something like .visit{}.async(wait:true), but my googling turned up nothing. Is there a way I can easily make this multi-threaded? The processing of one element has no effect on the processing of any other element.

Answer 1

Before thinking about going multi-threaded, I'd try the following:
Run everything in the same JVM. Forking a new JVM for each input file is very inefficient.
Make the compressJs task incremental so that it only executes if some input file has changed since the previous run.
Run the minifier directly rather than via Ant (saves creation of a new class loader for each input file; not sure if it matters).
If this still leaves you unhappy with the performance, and you can't use a more performant minifier, you can still try to go multi-threaded. Gradle won't help you there (yet), but libraries like GPars or the Java Fork/Join framework will.

Answer 2

The GPars solution. Note that the compress() function could be modified to properly accept source dir/target dir/etc, but since all my names are consistent, I'm just using the one argument for now. I was able to cut my build time from 7.3s to 5.4s with only 3 files being minified. I've seen build times spiral out of control, so I'm always wary of performance with this kind of behavior.
import groovyx.gpars.GParsPool
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'org.codehaus.gpars:gpars:0.12'
}
}
def compress(String type) {
def elementsToMinify = []
fileTree {
from type
include "**/*.$type"
}.visit { element ->
if (element.file.isFile()) {
elementsToMinify << element
}
}
GParsPool.withPool(8) {
elementsToMinify.eachParallel { element ->
println "Minifying ${element.relativePath}"
def outputFileLocation = "build/$type/${element.relativePath}"
new File(outputFileLocation).parentFile.mkdirs()
ant.java(jar: "lib/yuicompressor-2.4.6.jar", fork: true) {
arg(value: "$type/${element.relativePath}")
arg(value: "-o")
arg(value: outputFileLocation)
}
}
}
}
task compressJs {
inputs.dir new File('js')
outputs.dir new File('build/js')
doLast {
compress('js')
}
}
task compressCss {
inputs.dir new File('css')
outputs.dir new File('build/css')
doLast {
compress('css')
}
}