I have come on to a project which has a Gradle script that builds tasks dynamically based on a list. This is so that when people fork the repository, they can just edit the list. It boils down to something like this:
def files = ['file1', 'file2', 'file3']
files.each { f ->
task (f, type: Exec) {
executable 'touch'
args file
}
}
What I would like to do is to be able to publish the outputs of the tasks. I ideally want to do this as multiple publishes. I gave it a stab with the below code snippet, but Gradle seems to really hate the fact that I'm using a java.lang.String.call() as a class name!
publishing {
publications {
files.each { f ->
f (IvyPublication) {
module f
artifact (file(f))
}
}
}
}
Is what I am trying to do possible, or should I completely change tack?
[edit] I think I can do this with IvyPublication.loadClass(f), but struggling with the details. Any help much appreciated
Aha! I have found the solution. It's not as exciting as using the class loader, but it does work:
publishing {
files.each { f ->
publications.create(f, IvyPublication) {
module f
artifact (file(f))
}
}
}
Related
I'm new to Gradle and Groovy.
I have something like this in my build.gradle file:
jacocoTestReport {
afterEvaluate {
classDirectories.setFrom(files(classDirectories.files.collect {
fileTree(dir: it, exclude: ['aaa/bbb.*', 'ccc/ddd/*',])
}))
}
}
Now, I need this in my custom plugin. But I'm not sure how to set these excludes to it.
I have something like this:
project.afterEvaluate { p ->
def reportTask = project.tasks.findByName('jacocoTestReport') as JacocoReport
reportTask.classDirectories.setFrom(reportTask.classDirectories.files.) // Now what?
}
And I'm stuck here. Can you help me sort this out, please?
EDIT1:
I tried to solve it using patternSet.
println '----------------BEFORE REPORT----------------------'
def patternSet = new PatternSet()
def filesAsString = reportTask.classDirectories.files.collect { file ->
file.path
}
patternSet.setIncludes(filesAsString)
patternSet.setExcludes(['aaa/bbb.*', 'ccc/ddd/*'])
def excludeFiles = reportTask.classDirectories.asFileTree.matching(patternSet).files
println filesAsString
println excludeFiles
def files = reportTask.classDirectories.asFileTree.minus(excludeFiles)
reportTask.classDirectories.setFrom(files)
println '----------------AFTER REPORT----------------------'
println reportTask.classDirectories.files
Result is not what I expected:
----------------BEFORE REPORT----------------------
[aaa\bbb\build\classes\java\main, aaa\bbb\build\classes\groovy\main, aaa\bbb\build\resources\main]
[]
----------------AFTER REPORT----------------------
[]
No exclude directories at all.
So, I somehow solved it.
def ex = ['aaa/bbb/*','ccc/ddd.*','eee/fff/*']
def ft = reportTask.classDirectories.asFileTree.matching {
ex.each {
exclude it
}
}
Don't ask me how Gradle wraps this since it's not a Groovy syntax, but it works.
My opinion is that Gradle somehow implements AntlrTask as a pattern matcher since it implements interface 'PatternFilterable' and documentation mentions Ant. I tried to solve it in pure Groovy using PatternSet which is another class implementing PatternFilterable but I haven't succeded.
But the simple exclude 'aaa/bbb/*' works OK. It doesn't take list so I had to do it using forEach cycle. Maybe there is more simple and elegant way and I'll try to play with it but this solution is good enough.
I would like feedback on the best practices for defining plugin tasks that depend on external state (i.e. defined in the build.gradle that referenced the plugin). I'm using extension objects and closures to defer accessing those settings until they're needed and available. I'm also interested in sharing state between tasks, e.g. configuring the outputs of one task to be the inputs of another.
The code uses "project.afterEvaluate" to define the tasks when the required settings have been configured through the extension object. This seems more complex than should be needed. If I move the code out of the "afterEvaluate", it gets compileFlag == null which isn't the external setting. If the code is changed again to use the << or doLast syntax, then it will get the external flag... but then it fails to work with type:Exec and other similarly helpful types.
I feel that I'm fighting Gradle in some ways, which means I don't understand better how to work well with it. The following is a simplified pseudo-code of what I'm using. This works but I'm looking to see if this can be simplified, or indeed what the best practices are. Also, the exception shouldn't be thrown unless the tasks are being executed.
apply plugin: MyPlugin
class MyPluginExtension {
String compileFlag = null
}
class MyPlugin implements Plugin<Project> {
void apply(Project project) {
project.extensions.create("myPluginConfig", MyPluginExtension)
project.afterEvaluate {
// Closure delays getting and checking flag until strictly needed
def compileFlag = {
if (project.myPluginConfig.compileFlag == null) {
throw new InvalidUserDataException(
"Must set compileFlag: myPluginConfig { compileFlag = '-flag' }")
}
return project.myPluginConfig.compileFlag
}
// Inputs for translateTask
def javaInputs = {
project.files(project.fileTree(
dir: project.projectDir, includes: ['**/*.java']))
}
// This is the output of the first task and input to the second
def translatedOutputs = {
project.files(javaInputs().collect { file ->
return file.path.replace('src/', 'build/dir/')
})
}
// Translates all java files into 'translatedOutputs'
project.tasks.create(name: 'translateTask', type:Exec) {
inputs.files javaInputs()
outputs.files translatedOutputs()
executable '/bin/echo'
inputs.files.each { file ->
args file.path
}
}
// Compiles 'translatedOutputs' to binary
project.tasks.create(name: 'compileTask', type:Exec, dependsOn: 'translateTask') {
inputs.files translatedOutputs()
outputs.file project.file(project.buildDir.path + '/compiledBinary')
executable '/bin/echo'
args compileFlag()
translatedOutputs().each { file ->
args file.path
}
}
}
}
}
I'd look at this problem another way. It seems like what you want to put in your extension is really owned by each of your tasks. If you had something that was a "global" plugin configuration option, would it be treated as an input necessarily?
Another way of doing this would have been to use your own SourceSets and wire those into your custom tasks. That's not quite easy enough yet, IMO. We're still pulling together the JVM and native representations of sources.
I'd recommend extracting your Exec tasks as custom tasks with a #TaskAction that does the heavy lifting (even if it just calls project.exec {}). You can then annotate your inputs with #Input, #InputFiles, etc and your outputs with #OutputFiles, #OutputDirectory, etc. Those annotations will help auto-wire your dependencies and inputs/outputs (I think that's where some of the fighting is coming from).
Another thing that you're missing is if the compileFlag effects the final output, you'd want to detect changes to it and force a rebuild (but not a re-translate).
I simplified the body of the plugin class by using the Groovy .with method.
I'm not completely happy with this (I think the translatedFiles could be done differently), but I hope it shows you some of the best practices. I made this a working example (as long as you have a src/something.java) by implementing the translate as a copy/rename and the compile as something that just creates an 'executable' file (contents is just the list of the inputs). I've also left your extension class in place to demonstrate the "global" plug-in config. Also take a look at what happens with compileFlag is not set (I wish the error was a little better).
The translateTask isn't going to be incremental (although, I think you could probably figure out a way to do that). So you'd probably need to delete the output directory each time. I wouldn't mix other output into that directory if you want to keep that simple.
HTH
apply plugin: 'base'
apply plugin: MyPlugin
class MyTranslateTask extends DefaultTask {
#InputFiles FileCollection srcFiles
#OutputDirectory File translatedDir
#TaskAction
public void translate() {
// println "toolhome is ${project.myPluginConfig.toolHome}"
// translate java files by renaming them
project.copy {
includeEmptyDirs = false
from(srcFiles)
into(translatedDir)
rename '(.+).java', '$1.m'
}
}
}
class MyCompileTask extends DefaultTask {
#Input String compileFlag
#InputFiles FileCollection translatedFiles
#OutputDirectory File outputDir
#TaskAction
public void compile() {
// write inputs to the executable file
project.file("$outputDir/executable") << "${project.myPluginConfig.toolHome} $compileFlag ${translatedFiles.collect { it.path }}"
}
}
class MyPluginExtension {
File toolHome = new File("/some/sane/default")
}
class MyPlugin implements Plugin<Project> {
void apply(Project project) {
project.with {
extensions.create("myPluginConfig", MyPluginExtension)
tasks.create(name: 'translateTask', type: MyTranslateTask) {
description = "Translates all java files into translatedDir"
srcFiles = fileTree(dir: projectDir, includes: [ '**/*.java' ])
translatedDir = file("${buildDir}/dir")
}
tasks.create(name: 'compileTask', type: MyCompileTask) {
description = "Compiles translated files into outputDir"
translatedFiles = fileTree(tasks.translateTask.outputs.files.singleFile) {
includes [ '**/*.m' ]
builtBy tasks.translateTask
}
outputDir = file("${buildDir}/compiledBinary")
}
}
}
}
myPluginConfig {
toolHome = file("/some/custom/path")
}
compileTask {
compileFlag = '-flag'
}
I am experimenting with Gradle to build a few jars, rather than maintain a list of classes that hold EJBs so that I can deploy them separately I thought it might be neat to scan the classes when making the jar.
Rather than load the classes and use reflection to get the annotations I thought it may be simpler to scan the classes with asm, hence the chuncky ClassReader in one of the tasks.
I don't think this is the issue so can be ignored, basically I have 2 tasks that I use to define the contents of the jars, both report that different content is going into them via the eachFile print out, however when I look in the publish repository location both files and associated sha1 are identical.
Either Gradle is broken or, more likely, I've done something crazy but can't see what it is, can anyone help?
By the way if I disable the publish of either of the jar files the one that does get created is correct so I think it's something wrong with the publish rather than the jarring up, but could be wrong.
// ASM is used to interpret the class files, this avoids having to load all classes in the vm and use reflection
import org.objectweb.asm.*
task ejbJar(type: Jar) {
//outputs.upToDateWhen { false }
from "${project.buildDir}/classes/main"
eachFile { println "EJB server: ${name}" }
include getEjbClassFiles(project.buildDir)
}
task clientEjbJar(type: Jar) {
//outputs.upToDateWhen { false }
from "${project.buildDir}/classes/main/com/company/core/versioner"
eachFile { println "Client EJB ${name}" }
include '**/*'
}
artifacts {
archives clientEjbJar
archives ejbJar
}
String[] getEjbClassFiles(base) {
def includedFiles = []
def baseDir = project.file("${base}/classes/main")
def parentPath = baseDir.toPath()
if (baseDir.isDirectory()) {
baseDir.eachFileRecurse(groovy.io.FileType.FILES) { file ->
if(file.name.endsWith('.class')) {
//get hold of annotations in there --- org.objectweb.asm.Opcodes.ASM4
def reader = new ClassReader(file.bytes).accept(
new ClassVisitor(Opcodes.ASM4) {
public AnnotationVisitor visitAnnotation(String desc, boolean visible) {
if(desc.equals("Ljavax/ejb/Stateless;") ||
desc.equals("Ljavax/ejb/Stateful;")) {
includedFiles += parentPath.relativize(file.toPath())
}
return null //no interest in actually visiting the annotation values
}
},
ClassReader.SKIP_DEBUG | ClassReader.EXPAND_FRAMES | ClassReader.SKIP_FRAMES | ClassReader.SKIP_CODE
)
}
}
}
return includedFiles
}
publishing {
publications {
mypub(IvyPublication) {
artifact(ejbJar) {
name 'ejb'
}
artifact(clientEjbJar) {
name 'client-ejb'
}
}
}
repositories {
ivy {
name 'personal'
url "${ant['developer.repository']}/"
layout 'pattern', {
artifact "[organisation]/[module]/[artifact]/[revision]/[type]/[artifact]-[revision].[ext]"
ivy "[organisation]/[module]/[type]/[revision]/[type]/[type]-[revision].[ext]"
}
}
}
}
I did break the thing down into a simpler form as I thought it may be a Gradle bug.
The simplified form was:
apply plugin: 'java'
apply plugin: 'ivy-publish'
task bigJar(type: Jar) {
from "${rootDir}/src/main/resources"
include '**/*'
}
task smallJar(type: Jar) {
from "${rootDir}/src/main/resources/A/B"
include '**/*'
}
group 'ICantBeEmpty'
artifacts {
archives bigJar
archives smallJar
}
publishing {
publications {
mypub(IvyPublication) {
artifact(bigJar) { name 'biggie' }
artifact(smallJar) { name 'smallie' }
}
repositories {
ivy {
name 'personal'
url "c:/temp/gradletest"
layout 'pattern', {
artifact "[organisation]/[module]/[artifact]/[revision]/[type]/[artifact]-[revision].[ext]"
ivy "[organisation]/[module]/[type]/[revision]/[type]/[type]-[revision].[ext]"
}
}
}
}
}
This results in 2 files in c:/temp/gradletest/ICantBeEmpty/report-bug/biggie/unspecified/biggie-unspecified.jar and c:/temp/gradletest/ICantBeEmpty/report-bug/smallie/unspecified/smallie-unspecified.jar
Both of these files are identical, however I think I know why see my later answer.
Whilst looking at some configurations I noticed some odd behaviour that led me to a resolution of this issue, and it is a Gradle bug.
In my build I had a scratch task doing
configurations.archives.artifacts.each { println it }
This gave me 5 different lines output, however changing it to this
configurations.archives.artifacts.each { println it.file }
produced the same filename 5 times.
It turns out this is related to my issue, although the artifacts are there as separate entities the name used to uniquely identify them was the same so the same file was always chosen during a publish. The name of the artifacts is given by ${baseName}-${appendix}-${version}-${classifier}.${extension} by default in the java plugin. This means that if neither appendix or classifier is specified then the artifact will have the same name.
I tested this using the above sample code by adding an appendix name
task bigJar(type: Jar) {
appendix = 'big'
from "${rootDir}/src/main/resources"
include '**/*'
}
task smallJar(type: Jar) {
appendix = 'small'
from "${rootDir}/src/main/resources/A/B"
include '**/*'
}
Using this rather than the code from the question produces 2 different jars.
It's not a complete answer but is a good enough work around, if I add a new publication definition I can publish the artifacts that I want to to the location that I want, the only downside is that it will create another gradle task which isn't ideal.
publications {
mypub(IvyPublication) {
artifact(ejbJar) {
name 'ejb'
}
}
newpub(IvyPublication) {
artifact(clientEjbJar) {
name 'client-ejb'
}
}
}
The above answer works in the short term, however does reveal yet another short coming in the Gradle world enter link description here
Not sure Gradle is all it could be at the moment, and so far no one has answered my questions so maybe it's not that actively developed!!
I'm no expert in this part of Gradle, but the functionality you are using is marked as "incubating"; you are using the new publishing feature which might or might not be complete. Perhaps you should use the old way of doing things. You also seem to be mixing both ways by using the artifacts closure.
I have a gradle project that has java applications as well as android applications.
root/
build.gradle
settings.gradle
java1/
java2/
android1/
android2/
java3/
etc.
What is the best practice for structuring my build script? I am a total gradle novice and am migrating the project from maven to gradle.
I wanted to do something instead of
configure(subprojects) {}
to apply plugins and other specific things.
such as
configure(java1, java2, java3) { // java specifics }
configure(android1, android2) { // android specifics }
I am probably approaching this from the wrong way.
More explicitly I need to apply the plugin java only for the java projects and the android plugin for the android projects.
configure(subprojects.findAll {it.name == "java1" || it.name == "java2"}) {
Under the filtering section in the guide
Hope this helps someone else out.
There are multiple ways, depending on what you want... Some examples:
// Configures just project java1
project(":java1") { ... }
// Configures projects java1 and java2
["java1","java2"].each { name ->
project(":$name") { ... }
}
You can use normal groovy to find/iterate over all the projects.
Another option:
configure([ project(':sub1'), project(':sub2'), ... ]) {
...
}
The shortest and easiest option:
configure(subprojects.findAll()) {
if (it.name.equals('MY_PROJECT')) {
...
} else {
...
}
}
Another approach...
In the settings.gradle you do define your projects like this:
gradle.ext.javaProjects=[]
gradle.ext.androidProjects=[]
javaProject('java1')
javaProject('java2')
javaProject('java3')
androidProject('android1')
androidProject('android2')
def javaProject(String name) {
gradle.ext.javaProjects.add(name)
include ":$name"
}
def androidProject(String name) {
gradle.ext.androidProjects.add(name)
include ":$name"
}
Now you can reference those in your root build.gradle:
def javaProjects = subprojects.findAll {gradle.ext.javaProjects.contains(it.name)};
def androidProjects = subprojects.findAll {gradle.ext.javaProjects.contains(it.name)};
configure(javaProjects) {
...
}
configure(androidProjects) {
...
}
Maybe thats overkill... but i usually have the project definition method in my settings.gradle anyway. E.g. if you want to put all java projects under a java folder you could define things like this:
def javaProject(String name) {
gradle.ext.javaProjects.add(name)
include ":$name"
project(":$name").projectDir = file('java/'+ name)
}
Is there a way I can easily make the processing of FileTree files in a smart way in gradle tasks? I basically need to wait for the execution of all files, much like what you can do with GPars, but how do I do this gradle with FileTree?
task compressJs(dependsOn: [copyJsToBuild]) << {
println 'Minifying JS'
fileTree {
from 'build/js'
include '**/*.js'
}.visit { element ->
if (element.file.isFile()) {
println "Minifying ${element.relativePath}"
ant.java(jar: "lib/yuicompressor-2.4.6.jar", fork: true) {
arg(value: "build/js/${element.relativePath}")
arg(value: "-o")
arg(value: "build/js/${element.relativePath}")
}
}
}
}
It would be lovely if I could do something like .visit{}.async(wait:true), but my googling turned up nothing. Is there a way I can easily make this multi-threaded? The processing of one element has no effect on the processing of any other element.
Before thinking about going multi-threaded, I'd try the following:
Run everything in the same JVM. Forking a new JVM for each input file is very inefficient.
Make the compressJs task incremental so that it only executes if some input file has changed since the previous run.
Run the minifier directly rather than via Ant (saves creation of a new class loader for each input file; not sure if it matters).
If this still leaves you unhappy with the performance, and you can't use a more performant minifier, you can still try to go multi-threaded. Gradle won't help you there (yet), but libraries like GPars or the Java Fork/Join framework will.
The GPars solution. Note that the compress() function could be modified to properly accept source dir/target dir/etc, but since all my names are consistent, I'm just using the one argument for now. I was able to cut my build time from 7.3s to 5.4s with only 3 files being minified. I've seen build times spiral out of control, so I'm always wary of performance with this kind of behavior.
import groovyx.gpars.GParsPool
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'org.codehaus.gpars:gpars:0.12'
}
}
def compress(String type) {
def elementsToMinify = []
fileTree {
from type
include "**/*.$type"
}.visit { element ->
if (element.file.isFile()) {
elementsToMinify << element
}
}
GParsPool.withPool(8) {
elementsToMinify.eachParallel { element ->
println "Minifying ${element.relativePath}"
def outputFileLocation = "build/$type/${element.relativePath}"
new File(outputFileLocation).parentFile.mkdirs()
ant.java(jar: "lib/yuicompressor-2.4.6.jar", fork: true) {
arg(value: "$type/${element.relativePath}")
arg(value: "-o")
arg(value: outputFileLocation)
}
}
}
}
task compressJs {
inputs.dir new File('js')
outputs.dir new File('build/js')
doLast {
compress('js')
}
}
task compressCss {
inputs.dir new File('css')
outputs.dir new File('build/css')
doLast {
compress('css')
}
}