I am currently working on a project in Spark 2.1.0 and I need to import a library on which Spark itself already depends. In particular, I want org.roaringbitmap:RoaringBitmap:0.7.42 to replace org.roaringbitmap:RoaringBitmap:0.5.11 (the library on which both org.apache.spark:spark-core_2.11:2.1.0.cloudera1 and org.apache.spark:spark-sql_2.11:2.1.0.cloudera1 depend on).
My dependencies in build.gradle are the following
dependencies {
compile 'org.apache.spark:spark-core_2.11:2.1.0.cloudera1'
runtime ('org.apache.spark:spark-core_2.11:2.1.0.cloudera1') {
exclude group: 'org.roaringbitmap'
}
compile 'org.apache.spark:spark-sql_2.11:2.1.0.cloudera1'
runtime ('org.apache.spark:spark-sql_2.11:2.1.0.cloudera1') {
exclude group: 'org.roaringbitmap'
}
compile 'org.roaringbitmap:RoaringBitmap:0.7.42'
implementation 'org.roaringbitmap:RoaringBitmap'
constraints {
implementation('org.roaringbitmap:RoaringBitmap:0.7.42') {
because 'because of transitive dependency'
}
}
}
The output of gradle -q dependencyInsight --dependency org.roaringbitmap shows that dependency has been updated
org.roaringbitmap:RoaringBitmap -> 0.7.42
variant "default+runtime" [
org.gradle.status = release (not requested)
Requested attributes not found in the selected variant:
org.gradle.usage = java-api
]
\--- compileClasspath
org.roaringbitmap:RoaringBitmap:0.5.11 -> 0.7.42
variant "default+runtime" [
org.gradle.status = release (not requested)
Requested attributes not found in the selected variant:
org.gradle.usage = java-api
]
\--- org.apache.spark:spark-core_2.11:2.1.0.cloudera1
+--- compileClasspath
+--- org.apache.spark:spark-sql_2.11:2.1.0.cloudera1
| \--- compileClasspath
\--- org.apache.spark:spark-catalyst_2.11:2.1.0.cloudera1
\--- org.apache.spark:spark-sql_2.11:2.1.0.cloudera1 (*)
Unfortunately, when I run my application with spark2-submit the actual version of the runtime dependency is org.roaringbitmap:RoaringBitmap:0.5.11.
How can I force my application to use the desired version of RoaringBitmap?
I believe CDH provided libraries takes precedence over your libraries anyway.
You could check this using the next piece of code in spark2-shell:
import java.lang.ClassLoader
val cl = ClassLoader.getSystemClassLoader
cl.asInstanceOf[java.net.URLClassLoader].getURLs.foreach(println)
Generally i use shade plugin to overcome it.
Spark has an option to prioritize the user class path over its own. Classpath resolution between spark uber jar and spark-submit --jars when similar classes exist in both
Most likely you should also look into shading.
Related
I'm trying to run a pitest report on a gradle + kotlin project, but I get the following error:
Exception in thread "main" org.pitest.help.PitHelpError: No mutations found. This probably means there is an issue with either the supplied classpath or filters.
See http://pitest.org for more details.
at org.pitest.mutationtest.tooling.MutationCoverage.checkMutationsFound(MutationCoverage.java:352)
at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:132)
at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:123)
at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:54)
at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:98)
at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:45)
I tried everything that I found on google but still not working for me:
This is my build.gradle config
plugins {
id 'groovy-gradle-plugin'
id 'info.solidsoft.pitest' version '1.7.4'
}
repositories {
maven { url "https://plugins.gradle.org/m2/" }
gradlePluginPortal()
}
dependencies {
implementation 'org.jetbrains.kotlin:kotlin-gradle-plugin:1.4.20'
implementation 'com.github.jengelman.gradle.plugins:shadow:6.1.0'
}
pitest {
targetClasses = ['com.project.root.to.test.with.pitest.src*'] //by default
"${project.group}.*"
pitestVersion = '1.7.4' //not needed when a default PIT version should be used
threads = 4
outputFormats = ['XML', 'HTML']
timestampedReports = false
}
I tried this targetClasses in a different ways:
targetClasses = ['com.project.root.to.test.with.pitest.src.*'] //by default
targetClasses = ['com/project/root/to/test/with/pitest/src*'] //by default
Can someone help me, please?
You look to be trying to supply pitest with a source folder
com.project.root.to.test.with.pitest.src.
Pitest works against the compiled bytecode, not the source files. It expects
a glob that matches against the package.
com.example.*
I've experienced this same issue today. You'll need to make sure all references to pitest use the same version 1.7.4. This includes
plugin: id 'info.solidsoft.pitest' version '1.7.4'
pitestVersion: pitestVersion.set('1.7.4')
dependency: testCompile
'info.solidsoft.gradle.pitest:gradle-pitest-plugin:1.7.4'
Which out changing all references, then it will break.
I have 2 micronaut (groovy ) projects , called project A and project B
Project B has controllers and services ( package com.service , com.controller )
but I only created jar from package com.service
the code in com.service package has #Singleton annotation and #Scheduled
and I has enabled annotation processing as link (https://docs.micronaut.io/latest/guide/index.html#ideaSetup) to both projects
please see my gradle code below to generate JAR file ( the output file is project-b-libs-0.x.jar)
task createLibraryJar(type: Jar) {
baseName( getArchivesBaseName() + "-libs")
from sourceSets.main.output
includeEmptyDirs = false
include '**/service/**/*.class'
}
Then I added proejct-b-libs-0.x.jar to Project A
The gradle's dependencies are below
dependencies {
annotationProcessor "io.micronaut:micronaut-inject-java"
implementation("io.micronaut:micronaut-validation")
implementation("io.micronaut.groovy:micronaut-runtime-groovy")
implementation("javax.annotation:javax.annotation-api")
implementation("io.micronaut:micronaut-http-client")
runtimeOnly("ch.qos.logback:logback-classic")
compileOnly files('libs/project-b-libs-0.4.jar')
compile "io.micronaut:micronaut-inject"
}
Finally after I tried call #Inject Object from the class in JAR file, it showed error on run time
Caused by: io.micronaut.context.exceptions.BeanContextException: Error
loading bean [com.service.TestService]:
com/service/StripePaymentService
Project A has com.service.TestService to call com.service.StripePaymentService which is in JAR file
Sorry for my English and Thank you to trying to understand me
compile files('libs/project-b-libs-0.4.jar')
I just use compile , not compileOnly , and can not use jar file from gralde script above when I extract jar file , some mincronaut's stuff is missing
so I try another gradle task's script as below ( change from include to exclude )
task createLibraryJar(type: Jar) {
baseName( getArchivesBaseName() + "-libs")
from sourceSets.main.output
includeEmptyDirs = false
exclude '**/controller/**/*.class'
}
and it works because the micronaut's stuff still in JAR file, I only exclude unnecessary classes from my jar file
MyApp build.gradle looks like this
apply plugin: 'holoeverywhere-app'
apply plugin: 'com.android.application'
configurations.all {
resolutionStrategy.failOnVersionConflict()
}
android {
.... blah blah nothing important
}
holoeverywhere {
library {
version = '2.1.0'
}
support {
version = 'inherit'
}
addons {
preferences
}
}
dependencies {
compile fileTree(dir: 'libs', include: '*.jar')
compile project(':google-play-services_lib')
compile project(':FacebookSDK')
compile('com.android.support:support-v4:21.0.0') { force = true }
}
My dependencies look like this
+--- project :google-play-services_lib
+--- project :CustomLib |
| \--- project :FacebookSDK
| \--- com.android.support:support-v4:21.0.0
| \--- com.android.support:support-annotations:21.0.0
+--- com.android.support:support-v4:21.0.0 (*)
+--- org.holoeverywhere:library:2.1.0
| +--- com.nineoldandroids:library:2.4.0
| \--- com.android.support:support-v4:18.0.3 -> 21.0.0 (*)
\--- org.holoeverywhere:addon-preferences:2.1.0
\--- org.holoeverywhere:library:2.1.0 (*)
The situation is
MyApp depends on HoloEverywhere but also requires android.support-v4:19.1.0 or higher(v4:21.0.0)
HoloEverywhere 2.1.0 requires android.support-v4:18.0.3 only (it's the custom version made for HE)
FacebookSDK requires android.support-v4: any version
The problem is
If I don't use force = true, the build breaks due to version conflict of v4:18.0.3 & v4:21.0.0
If I use force = true, HE will then rely on v4:21.0.0 and break at runtime(I dont know an option for HE plugin to force it to use v4:18.0.3)
I tried manually attaching HE as shown in wiki, but the maven repo url is broken, even the github repo url("https://raw.github.com/Prototik/HoloEverywhere/repo") is broken
Please help me with configuring dependencies, I've already lost 3 days trying to make this work.
As a side question, HE 2.1.0 uses ActionBarCompat instead of ActionBarSherlock, so my MenuItem imports now come from android.view instead of ABS or any other support library, I wonder how it'll work for older devices
I am using the war plugin to generate a simple .WAR file for my project in gradle. I'd like to know how to configure gradle so that I can create 4 different .WAR files with different dependencies.
I've configured the dependency compile configuration with the jars that are needed to go into the distribution. None of the code in the src depends on a couple of these jars but I would like to know how to configure the project to create
a standard.WAR file that contains all of the jars in the dependency graph (Even though they aren't used - that is OK - I am testing something)
another standard-qas-only.WAR file that only contains the qas.jar
another standard-qas-log4j.WAR file that contains qas.jar and log4j
What tasks do i configure to have the artifact generated use a particular dependency configuration?
FYI: The only jar that is required for compilation is qas.jar in this case.
My example below creates a war file that only includes one jar but i'd like to have 5 different .war files generated with different jars.
build.gradle
apply plugin: 'java'
apply plugin: 'war'
dependencies {
compile files('/lib/qas.jar','/lib/axis1-1.4.jar','/lib/axis2-kernel-1.3.jar','/lib/dom4j-1.6.1.jar','/lib/log4j-1.2.14.jar')
providedCompile files('/lib/j2ee-1.4.03.jar')
}
war {
classpath = ['/lib/qas.jar']
}
task dist(dependsOn: 'war') << {
copy {
from war.archivePath
into "dist/"
}
}
I got a bit confused on how many WAR distributions you are actually trying to build. You can easily modify it to create additional WAR files. Here's one approach to make this happen:
task createStandardWar(type: War, dependsOn: classes) {
baseName = 'standard'
destinationDir = file("$buildDir/dist")
}
task createStandardWarQasOnly(type: War, dependsOn: classes) {
baseName = 'standard-qas-only'
destinationDir = file("$buildDir/dist")
classpath = war.classpath.minus(files('/lib/axis1-1.4.jar','/lib/axis2-kernel-1.3.jar','/lib/dom4j-1.6.1.jar','/lib/log4j-1.2.14.jar'))
}
task createStandardWarQasAndLog4J(type: War, dependsOn: classes) {
baseName = 'standard-qas-log4j'
destinationDir = file("$buildDir/dist")
classpath = war.classpath.minus(files('/lib/axis1-1.4.jar','/lib/axis2-kernel-1.3.jar','/lib/dom4j-1.6.1.jar'))
}
task createDists(dependsOn: [createStandardWar, createStandardWarQasOnly, createStandardWarQasAndLog4J])
This build script excerpt creates three different WAR files by declaring enhanced tasks of type War. It assumes that you still want to have your compiled source files under WEB-INF/classes within the WAR files so I didn't remove it from the classpath. The distributions end up in the directory build/dist. The task createDists creates all of them.
When running groovyc in a Windows env, I am running into issues due to the length of the classpath, in my situation. I would like to work around this by creating a pathing jar, and then put that jar on the cp. How can I create a pathing jar w/ all of the classpath entries specified automatically in gradle and then add that jar to the cp?
Here is a tested solution:
task pathingJar(type: Jar) {
appendix = "pathing"
doFirst {
manifest {
attributes "Class-Path": configurations.compile.files.join(" ")
}
}
}
compileGroovy {
dependsOn(pathingJar)
classpath = files(pathingJar.archivePath)
}
Depending on your exact requirements, you might have to tweak this a bit. For example, if you have tests written in Groovy, you will also need a pathing Jar for the test compile class path. In this case you'll need to repeat above configuration as follows:
task testPathingJar(type: Jar) {
appendix = "testPathing"
doFirst {
manifest {
attributes "Class-Path": configurations.testCompile.files.join(" ")
}
}
}
compileTestGroovy {
dependsOn(testPathingJar)
classpath = files(testPathingJar.archivePath)
}
I finally got the "pathing jar" idea to work. I consider this to be a permanent workaround. This could be considered a solution if it is made part of gradle itself.
The original pathing jar code was provided by Peter, but it didn't work. The problem: classpath elements referenced in the pathing jar must be relative to the location of the pathing jar. So, this appears to work for me.
task pathingJar(type: Jar , dependsOn: 'cleanPathingJar') {
/**
* If the gradle_user_home env var has been set to
* C:\ on a Win7 machine, we may not have permission to write the jar to
* this directory, so we will write it to the caches subdir instead.
* This assumes a caches subdir containing the jars
* will always exist.
*/
gradleUserHome = new File(gradle.getGradleUserHomeDir(), "caches")
relativeClasspathEntries = configurations.compile.files.collect {
new File(gradleUserHome.getAbsolutePath()).toURI().
relativize(new File(it.getAbsolutePath()).toURI()).getPath()
}
appendix = "pathing"
destinationDir = gradleUserHome
doFirst {
manifest {
attributes "Class-Path": relativeClasspathEntries.join(" ")
}
}
}
compileGroovy {
dependsOn(pathingJar)
classpath = files(pathingJar.archivePath)
}
This is what helped me:
"The filename or extension is too long error" using gradle
In other words: use the com.github.ManifestClasspath plugin.
The other solutions did not work for me because the actual project main class ended up no being included in the classpath at execution time.