How Can I Write A Dependency List On The Fly - groovy

I occasionally get code drops from a third party as a zip file of jars that I need to use as dependencies.
I want to unzip the jars into a flatDir repository and update a dependencies.gradle file automatically. I can do the unzip just fine, the writing to the file is the tricky bit.
Lets give an example using open source libraries:
If the dependencies.gradle originally looked like this:
repositories {
flatDir {
dirs 'lib'
}
}
dependencies {
compile 'commons-codec:1.2'
compile 'commons-logging:1.2'
compile 'log4j:1.2.17'
}
Then a Zip file arrived containing commons-codec-1.10.jar, commons-logging-1.2.jar and guava-18.0.jar. The dependencies.gradle would have to be updated to:
repositories {
flatDir {
dirs 'lib'
}
}
dependencies {
compile 'commons-codec:1.10'
compile 'commons-logging:1.2'
compile 'log4j:1.2.17'
compile 'guava:18.0'
}
I want to have specific versions and not just glob the whole flatDir as there might be different versions of the same library in the flatDir.
I can't use normal filtering as the # symbol wont be in the file.
The challenge is to recognise when a dependency already exists and needs updating, when a dependency already exists and doesn't need updating and when a dependency doesn't exist and needs adding.
I am using a Copy task with a zipTree to unzip so my thought is to use eachFile with a closure to do the work. However, I'm not quite sure how to do the steps outlined in the paragraph above.

Related

rust libraries with cargo (rlib)

I am trying to create a library in rust to be used with rust executables. In C you can just create your .a or .so (or .lib or .dll on windows) and use tools like CMake to link everything, however rust does not seem to have this kind of infrastructure?
It is possible to make an executable with cargo (cargo new ) and create a library by adding the --lib flag (cargo new --lib), but then how would you use the resulting .rlib file (from the library cargo project)? I managed to link the .rlib file as follows:
rustc main.rs --extern foo=libfoo.rlib
and that works beautifully, though, I am not interested in writing a thousand rustc commands to build the final executable (which depends on the .rlib) if there is cargo that can do that for you. I tried working with a build script (which works perfectly for any C library, static or dynamic), but if I try it with the .rlib file, cargo says that it cannot find "foo" (-lfoo), the build script:
fn main() {
println!("cargo:rustc-link-search=.");
println!("cargo:rustc-link-lib=foo");
}
I tried replacing the path (search) to different directories (whilst also moving the .rlib file to the correct directory), also tried different combinations of libfoo, libfoo.rlib, ... (note that for the C libaries, foo is sufficient).
So my question really is: How can you create a rust library for private use, and how do you use it with a rust executable in a proper way, avoiding manual rustc commands? Are there tools that do this? Am I missing something in the build script? Perhaps there exists something like CMake for rust?
I suppose it is possible to just create a C interface over the rust code and compile another C project as that does work with cargo.
I do NOT want to publish the code to crates.io as I want this library strictly for private use.
Cargo does not support using pre-compiled .rlibs. Cargo is designed to compile programs fully from source (not counting native libraries).
How can you create a rust library for private use … I do NOT want to publish the code to crates.io as I want this library strictly for private use.
To use a private library, you write a dependency using a path or git dependency (or use a private package registry, but that's more work to set up).
[dependencies]
my-lib-a = { path = "../my-lib-a/" }
my-lib-b = { git = "https://my-git-host.example/my-lib-b", branch = "stable" }
Your private library is now compiled exactly like a “public” one.

Fat JAR with Kotlin and Apache Spark 2.3

Im using gradle to build my project mixing Kotlin and Apache Spark, but as soon I declare the spark dependency, the Fat JAR I generate gets non working. Otherwise it will work. The source code not even import anything from Spark
buildscript {
ext.kotlin_version = '1.2.40'
repositories {
mavenCentral()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
apply plugin: 'kotlin'
repositories {
mavenCentral()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
// uncomment this line to get main class 'myownpkg.SparkApplicationKt'
// not found error
// compile 'org.apache.spark:spark-core_2.11:2.3.0'
}
jar {
manifest { attributes 'Main-Class': 'myownpkg.SparkApplicationKt' }
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
}
After some experimentation I realized the generated JAR was having there a lot of duplicated files, causing that runtime not finding the .class required.
It was triggered after enable the Spark because it is the dependency causing snowball of other dependencies having the same file paths under META-INF folder
exclude 'META-INF/*'
That line made the trick to avoid duplicates but still wil have a META-INF folder in final JAR
The main reason is because you aren't creating the "FatJar" artifact with necessary dependencies. The compile dir in configuration only contain compiled source code.
From the maven central you need at least the 50 compile dependencies that spark-core require. Have you consider using the shadow plugin ?
Take a look at this thread on gradle discuss.

What files in a Cargo project should be in my .gitignore?

I created a "hello world" Rust app using cargo new. When I executed git status it showed a bunch of files:
A rust/welcomec/Cargo.lock
A rust/welcomec/Cargo.toml
A rust/welcomec/src/main.rs
A rust/welcomec/target/debug/.cargo-lock
A rust/welcomec/target/debug/.fingerprint/welcomec-2d68725c8fae6fd1/bin-welcome-2d68725c8fae6fd1
A rust/welcomec/target/debug/.fingerprint/welcomec-2d68725c8fae6fd1/bin-welcome-2d68725c8fae6fd1.json
A rust/welcomec/target/debug/.fingerprint/welcomec-2d68725c8fae6fd1/dep-bin-welcome-2d68725c8fae6fd1
A rust/welcomec/target/debug/deps/welcome-2d68725c8fae6fd1
A rust/welcomec/target/debug/welcome
A rust/welcomec/target/debug/welcome.d
Can I safely ignore any of these files and/or directories?
Summary
.gitignore for library crates
# Generated files
/target/
# The library shouldn't decide about the exact versions of
# its dependencies, but let the downstream crate decide.
Cargo.lock
.gitignore for executable crates
# Generated files
/target/
Details
You need one or two entries in your .gitignore, depending on what kind of crate you're building. The target/ folder can be ignore completely regardless of crate type; it only contains generated files (e.g. compile artifacts).
The Cargo.lock file should be included in the repository if you're writing an executable, and should be ignored if you're writing a library. You can read more about this in the FAQ. To quote the most important part:
The purpose of a Cargo.lock is to describe the state of the world at the time of a successful build. [...]
This property is most desirable from applications and projects which are at the very end of the dependency chain (binaries). As a result, it is recommended that all binaries check in their Cargo.lock.
For libraries the situation is somewhat different. [...] If a library ends up being used transitively by several dependencies, it’s likely that just a single copy of the library is desired (based on semver compatibility). If all libraries were to check in their Cargo.lock, then multiple copies of the library would be used, and perhaps even a version conflict.
Also, please note that cargo new and cargo init automatically generates a .gitignore file in the project, unless the parameter --vcs none is passed.
You can take some inspiration from GitHub's gitignore for Rust. At the time of writing, the file goes as follows:
# Generated by Cargo
# will have compiled files and executables
debug/
target/
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
Cargo.lock
# These are backup files generated by rustfmt
**/*.rs.bk
# MSVC Windows builds of rustc generate these, which store debugging information
*.pdb

Android Studio: including AAR library from a library project

In my Android Studio project I have two subprojects/modules: an Android application (App1) and an Android library project (LibraryProject1). App1 depends on LibraryProject1. So far so good.
However, LibraryProject1, in turn, needs to import an AAR library to work properly.
So my Configuration is as follows:
App1 includes LibraryProject1
LibraryProject1 includes dependency.aar
Now, to include dependecy.aar I use the method detailed here:
How to manually include external aar package using new Gradle Android Build System
So basically in my build.gradle for LibraryProject1 I have:
repositories {
flatDir {
dirs 'libs'
}
}
dependencies {
compile (name:'dependency', ext:'aar') //my AAR dependency
compile fileTree(dir: 'libs', include: ['*.jar'])
compile 'com.android.support:appcompat-v7:22.1.1'
}
Obviously, I put my dependency.aar file in the libs directory of LibraryProject1
However, this doesn't work. It seems that the repository added by LibraryProject1 is completely ignored and the local "libs" folder is not included as a repository, causing compilation to fail.
If I add the repository from the App1's build.gradle it works, but I don't want to do that, it's LibraryProject1 that needs the AAR file, not App1.
How can I do this??
Well, I found a way, but since it's very "hacky" I'll leave the question open, in case anyone comes up with a better, "proper" solution.
Basically the problem is that the flatDir repository is ignored at compilation time if included from LibraryProject1's build.gradle script, so what I do is I use App1's build.gradle to "inject" the flatDir repository in LibraryProject1. Something like this:
//App1 build.gradle
dependencies {
//get libraryproject1 Project object
Project p = project(':libraryproject1')
//inject repository
repositories{
flatDir {
dirs p.projectDir.absolutePath + '/libs'
}
}
//include libraryproject1
compile p
}
This effectively allows LibraryProject1 to include the external AAR library without having App1 include it. It's hacky but it works. Note that you still have to put:
//LibraryProject1 build.gradle
repositories {
flatDir {
dirs './libs'
}
}
inside LibraryProject1's build.gradle otherwise, even if the project itself would compile fine, the IDE wouldn't recognize the types included in the AAR library. Note that the ./ in the path also seems to be important, without it the IDE still doesn't recognized the types.
I faced to the same issue, and I figure out it by putting all libraries on that depends LibraryProject1 in LibraryProject1/libs as a .jar.
I think that aar library cannot be linked to another aar library.
Hope that help you,
Best regards

My Gradle project depends on commons-io 2.4, but Gradle puts $GRADLE_HOME/commons-io-1.4.jar into the classpath, causing failures

I've been iterating on features in my first Gradle plugin. I determined early on that I needed commons-io, so I added a dependency on commons-io 2.4, being the latest version.
This has been going well for a while, with the build working from the command line, and no errors in Eclipse.
I just started trying to integrate some code that uses "FileUtils.write(File,String)". I didn't need that method before. I got everything un-redded in Eclipse, and then I tried a command line build.
This failed with errors like the following:
... error: cannot find symbol
FileUtils.write(serviceLoaderFile,
^
symbol: method write(File,String)
location: class FileUtils
This confused me. I went to the failing lines in Eclipse, and no issues were indicated. I navigated into the "write()" method, and it looked fine to me. I then ran my command-line build with "--debug" to get some clues.
When I found the "javac" line, I found that "$GRADLE_HOME\lib\commons-io-1.4.jar" (where "GRADLE_HOME" is just my Gradle 2.3 distribution) was in the classpath BEFORE my dependency jar. I then inspected the code in the 1.4 jar, and I determined that the "FileUtils" class in that version didn't have a "write" method.
What am I supposed to do about this?
Update:
I suppose it's likely my "dependencies" block would be useful, which is this:
dependencies {
compile ("org.codehaus.groovy:groovy-all:2.3.9")
compile gradleApi()
compile "org.opendaylight.yangtools:yang-parser-impl:0.7.0-SNAPSHOT"
compile "org.opendaylight.yangtools:binding-java-api-generator:0.7.0-SNAPSHOT"
compile "org.opendaylight.yangtools:binding-generator-api:0.7.0-SNAPSHOT"
compile "org.opendaylight.yangtools:binding-generator-impl:0.7.0-SNAPSHOT"
compile "org.opendaylight.controller:yang-jmx-generator:0.3.0-SNAPSHOT"
compile "commons-io:commons-io:2.4"
testCompile("org.spockframework:spock-core:1.0-groovy-2.3") {
exclude group: "org.codehaus.groovy"
}
I tried commenting out the "gradleApi" reference, and that had no effect. I also tried adding an "exclude" for commons-io associated with the "groovy-all" reference, but that also didn't appear to make any difference.
}
You probably added gradleApi() under dependencies {} block - see the docs. The problem is that it ships all dependencies gradle requires - including commons-io in version 1.4 - see the extract below:
[opal#opal-mac-2]~/.gvm/gradle/current/lib % pwd
/Users/opal/.gvm/gradle/current/lib
[opal#opal-mac-2]~/.gvm/gradle/current/lib % ll commons-io-1.4.jar
-rw-rw-r-- 1 opal staff 109043 23 gru 13:17 commons-io-1.4.jar
[opal#opal-mac-2]~/.gvm/gradle/current/lib %
You probably added version 2.4 separately and that's why conflict occurred. You can also run
gradle dependencies
to view the full dependency tree and verify the problem.
There's no possibility to exclude a transitive dependency from gradleApi().
The solution to this required adding the following block to the "sourceSets" block:
main {
compileClasspath = configurations.compile.minus files("$gradle.gradleHomeDir/lib/commons-io-1.4.jar")
}
This is pretty simple, but I wish there was a cleaner solution for this. I'm not sure what that would look like.

Resources