I built Spark 1.6 SNAPSHOT from sources with no issues:
$ mvn3 clean package -DskipTests.
I'm running:
OS X 10.10.5.
Java 1.8
Maven 3.3.3
Spark 1.6 SNAPSHOT
Scala 2.11.7
Zinc 0.3.5.3
Hadoop 3.0 SNAPSHOT
I added the following dependency to my pom.xml (to try to resolve the warning about native libraries):
<dependency>
<groupId>com.googlecode.netlib-java</groupId>
<artifactId>netlib</artifactId>
<version>1.1</version>
</dependency>
Environment variables:
HADOOP_INSTALL=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
HADOOP_CONF_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop
HADOOP_OPTS=-Djava.library.path=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native
CLASSPATH=/users/davidlaxer/trunk/core/src/test/java/:/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-dist-3.0.0-SNAPSHOT.jar:/Users/davidlaxer/clojure/target:/Users/davidlaxer/hadoop/lib/native:
SPARK_LIBRARY_PATH=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native
When I try to launch spark with: spark-shell I get the following error:
./spark-shell
Exception in thread "main" java.lang.IllegalStateException: Library directory '/Users/davidlaxer/spark/lib_managed/jars' does not exist.
at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:249)
at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:227)
at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:115)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:196)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:121)
at org.apache.spark.launcher.Main.main(Main.java:86)
I reverted to Spark 1.5 and didn't have the problem:
git clone git://github.com/apache/spark.git -b branch-1.5
Related
I am building spark 2.3 scala code using maven , giving following error.
error: missing or invalid dependency detected while loading class file SparkSession.class.
This is snippet of pom file, please advise
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.0</version>
</dependency>
You might want to check your Java and Scala versions. They should be 1.6 or higher and 2.11 respectively. Or it could also be a mismatch with other dependencies like spark_sql.Make sure you have same version across all dependencies.
I am trying to run a Mahout project which I wrote in Eclipse with the Mahout and Hadoop libraries. It loads in a dataset and runs the FPGrowth algorithm. I set up the following Run configuration to run the project:
mvn exec:java -Dexec.mainClass=com.patternmatching.RecommendApp.TopPatternMatches
After running the program, I get the following error message:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I researched this issue, and realized that the Native hadoop libraries have to either be compiled, or downloaded from Apache (Hadoop "Unable to load native-hadoop library for your platform" warning) . I downloaded the libraries on a Cloudera Quickstart VM, on which I set up Mahout and Maven, along with my project package. After running it in cloudera, I get the same error. I also ran the Hadoop checknative -a command, which verifies that the Native libraries are available:
[root#quickstart /]# hadoop checknative -a
16/10/22 19:32:16 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/10/22 19:32:16 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
The output of the command verifies that the libraries are available, but are not being correctly loaded into the program or classpath. I am not sure how to configure Maven so that it loads in the Hadoop native libraries when running the program. This is the dependencies section of the Maven pom.xml file:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.0.0-alpha1</version>
</dependency>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.9</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
and the command I run to execute my Mahout java program is
mvn exec:java -Dexec.mainClass=com.patternmatching.RecommendApp.TopPatternMatches
How can I configure Maven to see these libraries so they are used in the program?
My OS is OS X 10.11.6.
I'm running Spark 2.0, Zeppelin 0.6, Scala 2.11
When I run this code in Zeppelin I get an exception from Jackson.
When I run this code in spark-shell - no exception.
val filestream = ssc.textFileStream("/Users/davidlaxer/first-edition/ch06")
com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too old 2.5.3
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:56)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:651)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.streaming.StreamingContext.withNamedScope(StreamingContext.scala:273)
at org.apache.spark.streaming.StreamingContext.textFileStream(StreamingContext.scala:413)
... 51 elided
The dependency-reduced-pom.xml in spark-core shows version 2.6.5
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.5</version>
<scope>compile</scope>
</dependency>
The version of Maven is:
David-Laxers-MacBook-Pro:2.5.3 davidlaxer$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
Maven home: /opt/local/share/java/maven3
Java version: 1.8.0_05, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac"
The version of Java is:
David-Laxers-MacBook-Pro:2.5.3 davidlaxer$ java -version
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
Zeppelin was built with the following parameters:
$ mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pscala-2.11 -DskipTests
Generating the dependency tree in Zeppelin and in Spark showed all the jackson-databind references as 2.6.5.
$ mvn dependency:tree -Dverbose
shows [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:compile
I tried deleting ~/.m2/repository/com/fasterxml and rebuilding Spark and Zeppelin.
Here's my $PATH:
David-Laxers-MacBook-Pro:~ davidlaxer$ !e
echo $PATH
/Users/davidlaxer/.sdkman/candidates/springboot/current/bin:/Users/davidlaxer/.jenv/shims:/Users/davidlaxer/.jenv/bin:/Developer/NVIDIA/CUDA-6.5/bin:/usr/local/lib/erlang/erts-5.9.1/bin:/users/davidlaxer/anaconda/bin:/users/davidlaxer/anaconda/bin/bin:/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin:/opt/local/bin:/opt/local/sbin:/usr/local/lib/erlang/erts-5.9.1/bin:/users/davidlaxer/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/CrossPack-AVR/bin:/usr/local/git/bin:/usr/texbin::/opt/local/bin/scala/bin::/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/Users/davidlaxer/pig-0.13.0-src/bin:/Users/davidlaxer/pig-0.13.0-src/sbin:/Users/davidlaxer/Downloads/apache-maven-3.2.1/bin:/Users/davidlaxer/anaconda/bin/bin::/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/Users/davidlaxer/.rvm/bin:/Library/Frameworks/JRuby.framework/Versions/Current/bin:/usr/local/gradle-source-build/bin:/Users/davidlaxer/bin:/Users/davidlaxer/storm/bin:/usr/bin:/usr/sbin:/Users/davidlaxer/spark/bin:/Library/PostgreSQL/9.4/bin:/usr/local/mysql/bin
Any ideas why I'm picking up version 2.5.3?
Replacing the jackson jars (2.5.x version) in './zeppelin-zengine/target/lib/' and './zeppelin-server/target/lib/' with the following version will solve the problem.
jackson-databind-2.6.5.jar
jackson-core-2.6.5.jar
jackson-annotations-2.6.5.jar
you can use $SPARK_HOME/jars/ jackson-databind-2.6.5.jar jackson-core-2.6.5.jar jackson-annotations-2.6.5.jar to replace old version jars ofr $ZEPPELIN_HOME/lib
Usage: java cucumber.api.cli.Main [options] [ [FILE|DIR][:LINE[:LINE]*] ]+
Options:
-g, --glue PATH Where glue code (step definitions and hooks) is loaded from.
-f, --format FORMAT[:PATH_OR_URL] How to format results. Goes to STDOUT unless PATH_OR_URL is specified.
Built-in FORMAT types: junit, html, pretty, progress, json.
FORMAT can also be a fully qualified class name.
-t, --tags TAG_EXPRESSION Only run scenarios tagged with tags matching TAG_EXPRESSION.
-n, --name REGEXP Only run scenarios whose names match REGEXP.
-d, --[no-]-dry-run Skip execution of glue code.
-m, --[no-]-monochrome Don't colour terminal output.
-s, --[no-]-strict Treat undefined and pending steps as errors.
--snippets Snippet name: underscore, camelcase
--dotcucumber PATH_OR_URL Where to write out runtime information. PATH_OR_URL can be a file system
path or a URL.
-v, --version Print version.
-h, --help You're looking at it.
Exception in thread "main" cucumber.runtime.CucumberException: Unknown option: --plugin
at cucumber.runtime.RuntimeOptions.parse(RuntimeOptions.java:119)
at cucumber.runtime.RuntimeOptions.<init>(RuntimeOptions.java:50)
at cucumber.runtime.RuntimeOptions.<init>(RuntimeOptions.java:44)
at cucumber.api.cli.Main.run(Main.java:20)
at cucumber.api.cli.Main.main(Main.java:16)
I am getting this error during running my feature file.
PoM dependency is given below and I am using 3.2.4 Spring version with cucumber veriosn 1.1.5
Looks like you are using a very old version of cucumber-jvm that is looking for
--format
instead of
--plugin
The latest cucumber-jvm usage text can be found here.
Get the latest cucumber-jvm from the Maven repository as described here or here.
Either its related with JAR library(ies) version mismatch or plugin(cucumber-eclipse-plugin) mismatch.
look Here: https://groups.google.com/forum/#!topic/cukes/1urjr3ASq78
I was also facing similar issue. The cucumber jars were old. So I updated it as below using maven repository
<properties>
<cucumber.version>6.10.3</cucumber.version>
</properties>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>${cucumber.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>${cucumber.version}</version>
<scope>test</scope>
</dependency>
NOTE: you can find the latest cucumber java in maven repository at this link Cucumber jars maven dependency
When I am building the Spark 1.6 source code for Hadoop version 2.6.0-cdh5.7.0 and Yarn with the sbt build and maven build i am getting below same error:
[error] (yarn/*:update) java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded.
IvyNode = org.codehaus.jackson#jackson-core-asl;1.8.3
I have added maven depencies
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.8.3</version>
</dependency>
and also I have added below dependency in plugins.sbt file inside project folder
libraryDependencies += "org.codehaus.jackson" % "jackson-core-asl" % "1.8.3" % "test" intransitive()
and I tried adding as
dependencyOverrides += "org.codehaus.jackson" % "jackson-core-asl" % "1.8.3" in plugins.sbt
But still error is not gone.
Please help on the same,
Thank you
Karim
I think this is the reason...
"All Codehaus services have now been terminated."