I am new to spark. I am trying to build spark with gradle where I am using as follows:
compile group: 'org.apache.spark', name: 'spark-core_2.10', version: '2.0.1'
compile group: 'org.apache.spark', name: 'spark-sql_2.10', version: '2.0.1'
I am getting the following error:
error: object UserDefinedFunction is not a member of package org.apache.spark.sql
[ant:scalac] import org.apache.spark.sql.UserDefinedFunction._
[ant:scalac] ^
[ant:scalac] error: bad symbolic reference. A signature in HiveContext.class refers to type Logging
[ant:scalac] in package org.apache.spark which is not available.
While the same code is building perfectly when I use jar 1.6 in my gradle build:
compile group: 'org.apache.spark', name: 'spark-core_2.10', version: '1.6.0'
compile group: 'org.apache.spark', name: 'spark-sql_2.10', version: '1.6.0'
I extract the jar 2.0.1 and found that the function UserDefinedFunction is not available, while it is available in 1.6.
Please advice. How can I solve this issue and upgrade my codebase to jar 2.0.1?
Thanks.
It is expected. In Spark 2.0.0 use org.apache.spark.sql.expressions.UserDefinedFunction:
SPARK-13296
https://github.com/apache/spark/commit/354d4c24be892271bd9a9eab6ceedfbc5d671c9c
Related
Using #grab for the first time (new to groovy)
I understand it is meant to work "out the box."
However, when I add this to my class:
#Grab(group='commons-lang', module='commons-lang', version='2.4')
I get the following compilation error:
Caused by: java.lang.NoClassDefFoundError: org/apache/ivy/plugins/resolver/DependencyResolver
Groovy version is Groovy Version: 3.0.4 JVM: 11.0.1 Vendor: Oracle Corporation OS: Mac OS X
you have missing dependency or library ivy-2.4.0.jar
this library is a part of groovy-all artifact. check groovy-all.pom to see all groovy dependencies/features
so, you could setup dependency to groovy-all artifact in your project
or to a separate ivy-2.4.0.jar artifact if you don't want to include all groovy features into your project
Try adding this to gradle.build
configurations {
ivy
}
dependencies {
ivy "org.apache.ivy:ivy:2.4.0"
...
}
tasks.withType(GroovyCompile) {
groovyClasspath += configurations.ivy
}
I'm new to Liferay. For my first project, I need to create a module which uses a third party library.
This library was developed by a university and is not used very often, so it's not on any maven repo or anything. Therefore I copied it into my Liferay project and I'm trying to figure out how to solve the dependency issues.
I read countless thread entries and blogs, but I'm still a little bit confused.
I hope someone can point me in the right direction to fix my problem.
build.gradle of the module
dependencies {
compileOnly group: "com.liferay.portal", name: "com.liferay.portal.kernel", version: "2.0.0"
compileOnly group: "com.liferay.portal", name: "com.liferay.util.taglib", version: "2.0.0"
compileOnly group: "javax.portlet", name: "portlet-api", version: "2.0"
compileOnly group: "javax.servlet", name: "javax.servlet-api", version: "3.0.1"
compileOnly group: "jstl", name: "jstl", version: "1.2"
compileOnly group: "org.osgi", name: "osgi.cmpn", version: "6.0.0"
compile files('lib/openBIS-API-V3-16.05.7-r1522065804.jar')
}
settings.gradle
buildscript {
dependencies {
classpath group: "com.liferay", name: "com.liferay.gradle.plugins.workspace", version: "1.5.0"
classpath group: "net.saliman", name: "gradle-properties-plugin", version: "1.4.6"
}
repositories {
maven {
url "https://cdn.lfrs.sl/repository.liferay.com/nexus/content/groups/public"
}
}
}
apply plugin: "net.saliman.properties"
apply plugin: "com.liferay.workspace"
bnd.bnd
Bundle-Name: my-dbdisplayer-project
Bundle-SymbolicName: de.mpi.prototype.dbdisplayer
Bundle-Version: 1.0.0
Export-Package: \
de.mpi.prototype.dbdisplayer.constants,\
Conditional-Package: \
ch.*
Bundle-Classpath:\
.,\
lib/openBIS-API-V3-16.05.7-r1522065804.jar
-includeresource:\
lib/openBIS-API-V3-16.05.7-r1522065804.jar
Terminal output when I deploy the module with blade deploy
BUILD SUCCESSFUL
Total time: 1.768 secs
stop 505
update 505 file:/home/liferay/MPI/liferay-prototype/modules/my-dbdisplayer-project/build/libs/de.mpi.prototype.dbdisplayer-1.0.0.jar
start 505
org.osgi.framework.BundleException: Could not resolve module: de.mpi.prototype.dbdisplayer [505]
Unresolved requirement: Import-Package: ch.ethz.sis.openbis.generic.asapi.v3
Updated bundle 505
Instead of using -includeresource and configuring Bundle-ClassPath yourself, consider using the configuration compileInclude in your gradle.properties.
As many pointed out, you probably need also to provide the library containing the package ch.ethz.sis.openbis.generic.asapi.v3. You probably don't want/cannot provide it as a module and want to include it in your bundle, the same way you include openBIS-API.jar. It would be something like this:
compileInclude files('lib/openBIS-API-V3-16.05.7-r1522065804.jar')
compileInclude files('lib/openBIS-asapi.jar')
or even:
compileInclude fileTree(dir: 'lib', include: '*.jar')
Of course you need this other openBIS-asapi.jar which contains the missing package. If this package is nowhere to be found, perhaps you could put this in your bnd.bnd:
Import-Package: *;resolution:=optional
EDIT
I just found in http://svnsis.ethz.ch/repos/cisd/openbis_api/trunk that the package is already in openbis_api, but this has a lot of dependencies. See:
http://svnsis.ethz.ch/repos/cisd/openbis_api/trunk/build.gradle
http://svnsis.ethz.ch/repos/cisd/common/trunk/build.gradle
compileInclude should copy and configure all dependencies for you, but I foresee more classloading problems...
EDIT 2
These two blog entries by Dave Nebinger explain how to use dependencies in Liferay modules
https://liferay.dev/blogs/-/blogs/osgi-module-dependencies
https://liferay.dev/blogs/-/blogs/gradle-compile-vs-compileonly-vs-compileinclude
By the way, in the second blog entry, I found something which does not seem to be correct. It is written there that packages your java code use from a compileOnly dependency will not be listed as Import-Package manifest entries, but I have a project with a compileOnly dependency for which said entry was generated.
Your bundle imports the package ch.ethz.sis.openbis.generic.asapi.v3. This is because that package is a dependency of the code inside the bundle.
You need to install a bundle that exports the package ch.ethz.sis.openbis.generic.asapi.v3.
I will leave an example of an uber jar for you here, with multiple libs inside.
In bnd.bnd
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Import-Package: *
Export-Package: javax.mail.*;version=1.5.6,\
javax.activation; version=1.1.1,\
com.sun.activation.*; version=1.1.1,\
com.sun.mail.*;version=1.5.6
-snapshot: ${tstamp}
-dsannotations: *
-check: all
-includeresource: lib/activation.jar=activation-1.1.1.jar,\
lib/javax.mail.jar=javax.mail-1.5.6.jar,\
lib/javax.mail-api.jar=javax.mail-api-1.5.6.jar
Bundle-ClassPath: ., lib/activation.jar, lib/javax.mail-api.jar, lib/javax.mail.jar
I have two dependencies that use a same library but with different versions and I get NoMethodError. I am not sure how to solve it using gradle or anything else? Any help would be great
group 'com.hello'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'idea'
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
compile 'org.slf4j:slf4j-log4j12:1.7.12'
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.0.1'
compile group: 'org.apache.spark', name: 'spark-streaming_2.11', version: '2.0.1'
compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: '2.0.1'
compile group:'org.apache.kafka', name: 'kafka-clients', version: '0.10.1.0'
compile group: 'com.datastax.spark', name: 'spark-cassandra-connector_2.11', version: '2.0.0-M3'
compile group: 'com.github.brainlag', name: 'nsq-client', version: '1.0.0.RC2'
compile group: 'com.google.code.gson', name: 'gson', version: '2.7'
}
task buildStreamingJar(type: Jar) {
if(project.hasProperty("jarname")) {
def fullyQualifiedPackageName = ""
def jarname = project.getProperty("jarname")
if (jarname.equalsIgnoreCase("SparkDriver1")) {
fullyQualifiedPackageName = "com.hello.streamprocessing.app.SparkDriver1"
}
if (jarname.equalsIgnoreCase("SparkDriver2")) {
fullyQualifiedPackageName = "com.hello.streamprocessing.app.SparkDriver2"
}
baseName = project.name + "-$jarname" + "-stream"
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
with jar
zip64 true
from sourceSets.test.output
manifest {
attributes 'Main-Class': fullyQualifiedPackageName
}
exclude 'META-INF/.RSA', 'META-INF/.SF', 'META-INF/*.DSA'
}
}
so what is happening here is that in my dependencies spark-core_2.11 uses
com.google.guava:14.0.1 and nsq-client uses com.google.guava:19.0
which is leading to the following error.
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
at com.github.brainlag.nsq.NSQProducer.(NSQProducer.java:22)
at com.hello.streamprocessing.app.SparkDriver2.main(SparkDriver2.java:37)
I need to use this build.gradle file to build a FAT Uber jar so I can use it with spark-submit
Basically you can't really solve this, if either spark-core or nsq-client is using features of guava that or not present in one version or the other. You can run gradlew dependencies which will show you how gradle resolves the version conflict on guava. By default it will choose version 19.0 as its the newer version. Your best chance is to use versions of your libraries that are dependent on the same version of guava. That is either to downgrade nsq-client or upgrade spark-core if possible.
You can exclude the transitive dependency to guava like this:
dependencies {
...
compile (group: 'com.github.brainlag', name: 'nsq-client', version: '1.0.0.RC2') {
exclude group: 'com.google.guava', module: 'guava'
}
...
}
However this will only work if nsq-client is not relying on features only present in version 19.0 of guava as mentioned above.
Sadly this is a common problem especially with the guava library as it is used in nearly every project and the developers don't care a lot about a stable API of their library.
Your best bet is probably to try excluding guava from spark-core and give it a try and, if it's not working, to exclude it from nsq-client and give it a try. But even if it seems to be working at a first glance odds are good (or actually bad in your case), that you will run into similar problems during runtime of your application.
If the above is not working, you could still fork nsq-client and replace the problematic parts with something compatible to guava 14.0.1. I don't really like this option but if the developers of nsq-client are nice they might accept your change...
One of the dependencies lib is not being downloaded. Use the latest lib of both. Use can as well use this:
repositories {
mavenCentral()
}
configurations {
compile
}
dependencies {
compile 'dependency repo1'
compile 'dependency repo2'
}
task libs(type: Sync) {
from configurations.compile
into "$buildDir/libs"
}
This will help download both dependencies.
I have the following Groovy test (not the real one, mind you) in a Gradle project (under src/test/groovy):
import javafx.scene.paint.Color
import org.junit.Test
class MyTest {
#Test
void test1() {
assert Color.AQUAMARINE != Color.BLUE
}
}
The build file is as simple as it gets and declares that I use Java 8 (so JavaFX is in the classpath, always):
apply plugin: 'groovy'
compileJava {
sourceCompatibility = 1.8
targetCompatibility = 1.8
}
...
It also has a test dependency on Groovy:
testCompile group: 'org.codehaus.groovy', name: 'groovy-all', version: '2.4.4'
Everything works fine, except when I refer to some JavaFX class in the Groovy test.
It fails with this error:
MyTest.groovy: 3: unable to resolve class javafx.scene.paint.Color
# line 3, column 1.
import javafx.scene.paint.Color
^
The Java code compiles fine and I can run my JavaFX application. Only the Groovy tests seem to not be able to see JavaFX.
I've already tried changing the compileJava block to compileGroovy, compileTestGroovy, also together with the compileJava block, so that the Groovy compiler sees JavaFX 8, but nothing helped.
How can I fix this?
Gradle version:
------------------------------------------------------------
Gradle 2.2
------------------------------------------------------------
Build time: 2014-11-10 13:31:44 UTC
Build number: none
Revision: aab8521f1fd9a3484cac18123a72bcfdeb7006ec
Groovy: 2.3.6
Ant: Apache Ant(TM) version 1.9.3 compiled on December 23 2013
JVM: 1.8.0_60 (Oracle Corporation 25.60-b23)
OS: Linux 3.16.0-38-generic amd64
Changed the Gradle version to 2.7 (latest as of today) and it all works.
If you have the same problem, just use Gradle's wrapper and set the Gradle version to the latest version (they move fast, so bugs will exist, but also get fixed quickly).
gradle wrapper --gradle-version 2.7
I am trying to run geb tests through gradle. I've these installed
java version "1.8.0_31"
Groovy Version: 2.4.0
Gradle 2.3
But i am getting this error when running the test.
Exception in thread "main" java.lang.NoClassDefFoundError: java/util/HashMap$Ent
ry
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2693)
Running a simple groovy script with above setup and the below Build.gradle is fine.
Build.gradle looks like this:
buildscript {
repositories {
jcenter()
}
}
apply plugin: 'java'
apply plugin: 'groovy'
repositories {
jcenter()
mavenCentral()
}
dependencies {
def seleniumVersion = "2.45.0"
def phantomJsVersion = '1.1.0'
// selenium drivers
compile "org.seleniumhq.selenium:selenium-firefox-driver:$seleniumVersion"
compile "org.seleniumhq.selenium:selenium-support:$seleniumVersion"
compile("com.github.detro.ghostdriver:phantomjsdriver:$phantomJsVersion") {
transitive = false
}
// geb
compile 'org.codehaus.geb:geb-core:0.7.2'
compile 'org.codehaus.geb:geb-spock:0.7.2'
// spock
compile 'org.spockframework:spock-core:0.6-groovy-1.8'
//junit
testCompile group: 'junit', name: 'junit', version: '4.7'
}
task runGebScript (dependsOn: 'classes', type: JavaExec) {
main = 'test'
classpath = sourceSets.main.runtimeClasspath
}
Can someone please help.
According to Gradle dependency report for the build you included, the Groovy version used for testCompile configuration is 1.8.5. Only groovy 2.x is JDK8 compatible. I would do as ataylor suggests and bump Spock version. If you use 1.0-groovy-2.4 then you'll be using Groovy 2.4.1. Also, I would suggest updating version of Geb while you're at it - the latest is 0.10.0. Your dependency versions are way out of date.
Something is trying to use the inner class HashMap.Entry, which no longer exists in Java 8. It's difficult to tell where without a complete stack trace.
However, you're using this version of spock: org.spockframework:spock-core:0.6-groovy-1.8. That version is not compatible with groovy 2.0+. Try updating the dependency to org.spockframework:spock-core:1.0-groovy-2.4.