My OS is OS X 10.11.6.
I'm running Spark 2.0, Zeppelin 0.6, Scala 2.11
When I run this code in Zeppelin I get an exception from Jackson.
When I run this code in spark-shell - no exception.
val filestream = ssc.textFileStream("/Users/davidlaxer/first-edition/ch06")
com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too old 2.5.3
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:56)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:651)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.streaming.StreamingContext.withNamedScope(StreamingContext.scala:273)
at org.apache.spark.streaming.StreamingContext.textFileStream(StreamingContext.scala:413)
... 51 elided
The dependency-reduced-pom.xml in spark-core shows version 2.6.5
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.5</version>
<scope>compile</scope>
</dependency>
The version of Maven is:
David-Laxers-MacBook-Pro:2.5.3 davidlaxer$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
Maven home: /opt/local/share/java/maven3
Java version: 1.8.0_05, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac"
The version of Java is:
David-Laxers-MacBook-Pro:2.5.3 davidlaxer$ java -version
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
Zeppelin was built with the following parameters:
$ mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pscala-2.11 -DskipTests
Generating the dependency tree in Zeppelin and in Spark showed all the jackson-databind references as 2.6.5.
$ mvn dependency:tree -Dverbose
shows [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:compile
I tried deleting ~/.m2/repository/com/fasterxml and rebuilding Spark and Zeppelin.
Here's my $PATH:
David-Laxers-MacBook-Pro:~ davidlaxer$ !e
echo $PATH
/Users/davidlaxer/.sdkman/candidates/springboot/current/bin:/Users/davidlaxer/.jenv/shims:/Users/davidlaxer/.jenv/bin:/Developer/NVIDIA/CUDA-6.5/bin:/usr/local/lib/erlang/erts-5.9.1/bin:/users/davidlaxer/anaconda/bin:/users/davidlaxer/anaconda/bin/bin:/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin:/opt/local/bin:/opt/local/sbin:/usr/local/lib/erlang/erts-5.9.1/bin:/users/davidlaxer/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/CrossPack-AVR/bin:/usr/local/git/bin:/usr/texbin::/opt/local/bin/scala/bin::/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/Users/davidlaxer/pig-0.13.0-src/bin:/Users/davidlaxer/pig-0.13.0-src/sbin:/Users/davidlaxer/Downloads/apache-maven-3.2.1/bin:/Users/davidlaxer/anaconda/bin/bin::/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/Users/davidlaxer/.rvm/bin:/Library/Frameworks/JRuby.framework/Versions/Current/bin:/usr/local/gradle-source-build/bin:/Users/davidlaxer/bin:/Users/davidlaxer/storm/bin:/usr/bin:/usr/sbin:/Users/davidlaxer/spark/bin:/Library/PostgreSQL/9.4/bin:/usr/local/mysql/bin
Any ideas why I'm picking up version 2.5.3?
Replacing the jackson jars (2.5.x version) in './zeppelin-zengine/target/lib/' and './zeppelin-server/target/lib/' with the following version will solve the problem.
jackson-databind-2.6.5.jar
jackson-core-2.6.5.jar
jackson-annotations-2.6.5.jar
you can use $SPARK_HOME/jars/ jackson-databind-2.6.5.jar jackson-core-2.6.5.jar jackson-annotations-2.6.5.jar to replace old version jars ofr $ZEPPELIN_HOME/lib
Related
Given the following code snippet:
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7.1')
def http = new groovyx.net.http.HTTPBuilder('http://www.codehaus.org'); // <<< THROWS EXCEPTION
println http
How can I fix the following error?
Caught: java.lang.NoClassDefFoundError: groovy/util/slurpersupport/GPathResult
java.lang.NoClassDefFoundError: groovy/util/slurpersupport/GPathResult
at java.desktop/com.sun.beans.introspect.MethodInfo.get(MethodInfo.java:70)
at java.desktop/com.sun.beans.introspect.ClassInfo.getMethods(ClassInfo.java:80)
at groovyx.net.http.ParserRegistry.<init>(ParserRegistry.java:87)
at groovyx.net.http.HTTPBuilder.<init>(HTTPBuilder.java:194)
at HttpBuilder.run(HttpBuilder.groovy:4)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.lang.ClassNotFoundException: groovy.util.slurpersupport.GPathResult
... 8 more
Versions of Groovy/Java/Gradle/Maven I'm using
I'm using Groovy 4.0 with JDK 17.0.2 as shown next:
groovy --version
Groovy Version: 4.0.0 JVM: 17.0.2 Vendor: Oracle Corporation OS: Windows 10
java -version
java version "17.0.2" 2022-01-18 LTS
Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)
mvn --version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: D:\p\apache-maven-3.8.4
Java version: 17.0.2, vendor: Oracle Corporation, runtime: D:\p\jdk-17.0.2
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
gradle --version
------------------------------------------------------------
Gradle 7.4
------------------------------------------------------------
Build time: 2022-02-08 09:58:38 UTC
Revision: f0d9291c04b90b59445041eaa75b2ee744162586
Kotlin: 1.5.31
Groovy: 3.0.9
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 17.0.2 (Oracle Corporation 17.0.2+8-LTS-86)
OS: Windows 10 10.0 amd64
Searching for solutions
I found this question: Groovy built-in REST/HTTP client? gave me a work around (not using http-builder).
Background on question
The UDEMY Groovy course https://www.udemy.com/course/apache-groovy/ provides some sample code during one of the lectures on a REST-based client in Groovy that uses the http-builder library.
The groovy and java version being used at the time of the course where
Java Version: 1.8.0_60
Groovy Version: 2.4.5
Gradle: 2.7
Maven: 3.3.3
Spring Boot: 3.0.M5
Spring Tool Suite (STS): 3.7.1.RELEASE-e4.5.1
On Windows 7 and Mac OS ?
i'm sure you have new version of groovy (for example 4.0.1) and http-builder library you are using is quite old.
starting from groovy 3.0 GPathResult class moved to another package: groovy.xml.slurpersupport.GPathResult
but according to error HTTPBuilder is looking for old package groovy.util.slurpersupport.GPathResult
option 1:
You have to downgrade groovy version
option 2: hacking
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7.1')
import groovyx.net.http.*
//define empty class with old name to prevent failure
this.getClass().getClassLoader().getParent().parseClass '''
package groovy.util.slurpersupport
class GPathResult{}
'''
def http = new HTTPBuilder('http://httpbin.org')
//redefine xml parser to use xml slurper from a new package
//you don't need this if you are not going to work with xml
http.parser['application/xml']={HttpResponseDecorator r->
return new groovy.xml.XmlSlurper().parse(r.entity.content)
}
http.get(path:'/xml',query:[a:123]){resp,body->
println "status: ${resp.statusLine}"
println groovy.xml.XmlUtil.serialize(body)
}
I red all other topics (maybe work for jdk 10) and still have problem to run javax.xml.bind on jdk 11.
My dependency:
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.4.0-b180830.0359</version> <!-- 2.2.12, 2.3.1 -->
</dependency>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-runtime</artifactId>
<version>2.4.0-b180830.0438</version> <!-- 2.3.1 -->
<!--<scope>compile</scope>-->
<!--<scope>runtime</scope>-->
</dependency>
<dependency>
<groupId>javax.activation</groupId>
<artifactId>javax.activation-api</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>javax.annotation-api</artifactId>
<version>1.3.2</version>
</dependency>
My IDE (newest intellij) still doesn't see import:
import javax.xml.bind.annotation.XmlAccessType;
Also my module-info.java doesnt see:
requires java.xml.bind;
And when i use maven i have an error:
[ERROR] ...ExceptionType.java:[6,22] package javax.xml.bind.annotation is not visible
(package javax.xml.bind.annotation is declared in the unnamed module, but module javax.xml.bind.annotation does not read it)
anybody run it on jdk 11 ?
edit:
1) App version with java module system:
i have similar problem to this:
https://youtrack.jetbrains.com/issue/IDEA-197956
but my error message is a little difrent
2) App version without java module system:
Maven compile correctly. Problem is that intellij dont see my imports:
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.bind.Unmarshaller;
and say: cannot resolve symbol.
Also at file > project structure > modules > dependencies intellij didnt add javax.xml.bind and org.glassfish.jaxb
Why?
I added this in pom:
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.4.0-b180830.0359</version> <!-- 2.2.8, 2.4.0-b180830.0359 -->
</dependency>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-runtime</artifactId>
<version>2.4.0-b180830.0438</version> <!-- 2.3.0, 2.4.0-b180830.0438 -->
</dependency>
JDK: openjdk11
Intellij version: IntelliJ IDEA 2018.2.4 (Ultimate Edition), Build #IU-182.4505.22, built on September 18, 2018, JRE: 1.8.0_152-release-1248-b8 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Maven:3.5.4
It seems to be a intellij bug.
Switched back to JDK 1.8 and it fixed this issue for me. How to switch in ubuntu: https://askubuntu.com/questions/740757/switch-between-multiple-java-versions
We managed to find the cause of the problem. In ide intellij I have a modules maven project and I use JLupin Platform Development Tool, so in *.iml files i dont have line:
'<'module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule = "true" type = "JAVA_MODULE" version = "4">
but i have for example:
'<'module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule = "true" type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE" version = "4">
During the development, the names of the modules were changed, then the idea asked whether to remove modules (because ide did not recognize:
type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE").
Although the chosen option was not, then ide automatically performed:
selecting the maven module as ignored
replace in the iml file from:
'<'module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule = "true" type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE" version = "4">
to:
'<'module type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE" version = "4">
For the above reason, the project did not correctly read the maven artifacts. After unifying the names and marking the project as not ignored, it works correctly.
The problem is not related to java packages
EDIT: its resolved as above
I built Spark 1.6 SNAPSHOT from sources with no issues:
$ mvn3 clean package -DskipTests.
I'm running:
OS X 10.10.5.
Java 1.8
Maven 3.3.3
Spark 1.6 SNAPSHOT
Scala 2.11.7
Zinc 0.3.5.3
Hadoop 3.0 SNAPSHOT
I added the following dependency to my pom.xml (to try to resolve the warning about native libraries):
<dependency>
<groupId>com.googlecode.netlib-java</groupId>
<artifactId>netlib</artifactId>
<version>1.1</version>
</dependency>
Environment variables:
HADOOP_INSTALL=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
HADOOP_CONF_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop
HADOOP_OPTS=-Djava.library.path=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native
CLASSPATH=/users/davidlaxer/trunk/core/src/test/java/:/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-dist-3.0.0-SNAPSHOT.jar:/Users/davidlaxer/clojure/target:/Users/davidlaxer/hadoop/lib/native:
SPARK_LIBRARY_PATH=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native
When I try to launch spark with: spark-shell I get the following error:
./spark-shell
Exception in thread "main" java.lang.IllegalStateException: Library directory '/Users/davidlaxer/spark/lib_managed/jars' does not exist.
at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:249)
at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:227)
at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:115)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:196)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:121)
at org.apache.spark.launcher.Main.main(Main.java:86)
I reverted to Spark 1.5 and didn't have the problem:
git clone git://github.com/apache/spark.git -b branch-1.5
mvn spring:boot run gives me this error below.
After running you jhipster for the first time on Mac OS X lion.
I have not changed any file at all, just using generated pom.
mvn 3.0.4 and 3.1.1 both tried, java 1.7
Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 11:22:22-0400)
Maven home: /Users/admin/installs/maven311
Java version: 1.7.0_60, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.7.5", arch: "x86_64", family: "mac"
Error log:
[INFO] ------------------------------------------------------------------------
[ERROR] No plugin found for prefix 'spring' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/Users/admin/.m2/repository), spring-snapshots (http://repo.spring.io/libs-snapshot), spring-milestones (http://repo.spring.io/milestone), central (http://repo.maven.apache.org/maven2)] -> [Help 1]
Try running with the following command instead:
mvn spring-boot:run
(note the dash between spring and boot and the colon between boot and run)
I am running gradle and have previously been running groovy 1.76. I have now updated to groovy on my local machine (groovy_home points to groovy 2.1.2 etc).
$ groovy -version
Groovy Version: 2.1.2 JVM: 1.7.0_17 Vendor: Oracle Corporation OS: Linux
However, when I am running gradle commands (gradle test, classes, etc) I believe it is not building against groovy 2.1.2 but is actually still building against 1.76. (The reason I believe this, is that when I execute the classes I keep getting this error Upgrading Groovy 1.7 - 2.1 Incompatability, which is related to changes made post 1.76)
Is there a way to confirm which version of groovy my gradle install is building against?
Also, can anyone confirm where I should be configuring the groovy version for gradle?
While trying to check the groovy version during gradle runtime, I found you can also print the Groovy version:
task version {
doLast {
println "Gradle version: " + project.getGradle().getGradleVersion()
println "Groovy version: " + GroovySystem.getVersion()
}
}
As examples:
$ ~/usr/gradle-1.8/bin/gradle -q version
Gradle version: 1.8
Groovy version: 1.8.6
$ ~/usr/gradle-2.1/bin/gradle -q version
Gradle version: 2.1
Groovy version: 2.3.6
Note.- GroovySystem.getVersion() is available since Groovy 1.6.9
Which Groovy library you are building against (and which Groovy compiler you are using) is determined by which Groovy library resides on the compile (or, in earlier Gradle versions, groovy) configuration. Typically a Groovy dependency is configured explicitly, but it may also be pulled in by transitive dependency management. (In case of a version conflict, the higher version wins by default. Which Groovy version(s) you have installed on your machine is irrelevant.) gradle dependencyInsight --configuration compile --dependency groovy should provide the answer.
Here is how a Groovy dependency is typically configured:
apply plugin: "groovy"
repositories {
mavenCentral() // or some other repository containing a Groovy library
}
dependencies {
// in Gradle 1.4 or earlier, replace 'compile' with 'groovy'
compile "org.codehaus.groovy:groovy-all:2.1.2"
}
in windows you can check it using: gradlew --v
------------------------------------------------------------
Gradle 6.2
------------------------------------------------------------
Build time: 2020-02-17 08:32:01 UTC
Revision: 61d3320259a1a0d31519bf208eb13741679a742f
Kotlin: 1.3.61
Groovy: 2.5.8
Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM: 15.0.2 (Oracle Corporation 15.0.2+7-27)
on linux the command will be : gradle --v
I'm using osx
gradle --v
------------------------------------------------------------
Gradle 7.5.1
------------------------------------------------------------
Build time: 2022-08-05 21:17:56 UTC
Revision: d1daa0cbf1a0103000b71484e1dbfe096e095918
Kotlin: 1.6.21
Groovy: 3.0.10
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 17.0.4.1 (Homebrew 17.0.4.1+1)
OS: Mac OS X 12.4 x86_64