I am building spark 2.3 scala code using maven , giving following error.
error: missing or invalid dependency detected while loading class file SparkSession.class.
This is snippet of pom file, please advise
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.0</version>
</dependency>
You might want to check your Java and Scala versions. They should be 1.6 or higher and 2.11 respectively. Or it could also be a mismatch with other dependencies like spark_sql.Make sure you have same version across all dependencies.
Related
I red all other topics (maybe work for jdk 10) and still have problem to run javax.xml.bind on jdk 11.
My dependency:
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.4.0-b180830.0359</version> <!-- 2.2.12, 2.3.1 -->
</dependency>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-runtime</artifactId>
<version>2.4.0-b180830.0438</version> <!-- 2.3.1 -->
<!--<scope>compile</scope>-->
<!--<scope>runtime</scope>-->
</dependency>
<dependency>
<groupId>javax.activation</groupId>
<artifactId>javax.activation-api</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>javax.annotation-api</artifactId>
<version>1.3.2</version>
</dependency>
My IDE (newest intellij) still doesn't see import:
import javax.xml.bind.annotation.XmlAccessType;
Also my module-info.java doesnt see:
requires java.xml.bind;
And when i use maven i have an error:
[ERROR] ...ExceptionType.java:[6,22] package javax.xml.bind.annotation is not visible
(package javax.xml.bind.annotation is declared in the unnamed module, but module javax.xml.bind.annotation does not read it)
anybody run it on jdk 11 ?
edit:
1) App version with java module system:
i have similar problem to this:
https://youtrack.jetbrains.com/issue/IDEA-197956
but my error message is a little difrent
2) App version without java module system:
Maven compile correctly. Problem is that intellij dont see my imports:
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.bind.Unmarshaller;
and say: cannot resolve symbol.
Also at file > project structure > modules > dependencies intellij didnt add javax.xml.bind and org.glassfish.jaxb
Why?
I added this in pom:
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.4.0-b180830.0359</version> <!-- 2.2.8, 2.4.0-b180830.0359 -->
</dependency>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-runtime</artifactId>
<version>2.4.0-b180830.0438</version> <!-- 2.3.0, 2.4.0-b180830.0438 -->
</dependency>
JDK: openjdk11
Intellij version: IntelliJ IDEA 2018.2.4 (Ultimate Edition), Build #IU-182.4505.22, built on September 18, 2018, JRE: 1.8.0_152-release-1248-b8 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Maven:3.5.4
It seems to be a intellij bug.
Switched back to JDK 1.8 and it fixed this issue for me. How to switch in ubuntu: https://askubuntu.com/questions/740757/switch-between-multiple-java-versions
We managed to find the cause of the problem. In ide intellij I have a modules maven project and I use JLupin Platform Development Tool, so in *.iml files i dont have line:
'<'module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule = "true" type = "JAVA_MODULE" version = "4">
but i have for example:
'<'module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule = "true" type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE" version = "4">
During the development, the names of the modules were changed, then the idea asked whether to remove modules (because ide did not recognize:
type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE").
Although the chosen option was not, then ide automatically performed:
selecting the maven module as ignored
replace in the iml file from:
'<'module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule = "true" type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE" version = "4">
to:
'<'module type = "JLP_NATIVE_MICROSERVICE_IMPLEMENTATION_MODULE_TYPE" version = "4">
For the above reason, the project did not correctly read the maven artifacts. After unifying the names and marking the project as not ignored, it works correctly.
The problem is not related to java packages
EDIT: its resolved as above
Need to add spark connector over HBase where
Spark version: 2.3.1
HBase Version: 2.0.0
Getting Bellow Exception:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.deploy.SparkHadoopUtil.getCurrentUserCredentials()Lorg/apache/hadoop/security/Credentials;
at org.apache.hadoop.hbase.spark.HBaseContext.<init>(HBaseContext.scala:68)
at org.apache.hadoop.hbase.spark.JavaHBaseContext.<init>(JavaHBaseContext.scala:46)
at com.cloud.databaseroot.hbase.spark.JavaHBaseBulkPutExample.main(JavaHBaseBulkPutExample.java:60)
Snap from pom.xml:
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-spark</artifactId>
<version>2.0.0-alpha4</version>
<exclusions>
<exclusion>
<artifactId>jackson-module-scala_2.10</artifactId>
<groupId>com.fasterxml.jackson.module</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.17.Final</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.11</artifactId>
<version>2.8.8</version>
</dependency>
Let me know where am I getting wrong.
It seems that hbase-spark version 2.0.0-alpha4 is not compatible with Spark 2.3.1.
SparkHadoopUtil.getCurrentUserCredentials method is available in Spark version <= 2.2. Either downgrade Spark or build hbase-spark with Spark 2.3.1 which may require some code changes in it.
I am trying to run a Mahout project which I wrote in Eclipse with the Mahout and Hadoop libraries. It loads in a dataset and runs the FPGrowth algorithm. I set up the following Run configuration to run the project:
mvn exec:java -Dexec.mainClass=com.patternmatching.RecommendApp.TopPatternMatches
After running the program, I get the following error message:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I researched this issue, and realized that the Native hadoop libraries have to either be compiled, or downloaded from Apache (Hadoop "Unable to load native-hadoop library for your platform" warning) . I downloaded the libraries on a Cloudera Quickstart VM, on which I set up Mahout and Maven, along with my project package. After running it in cloudera, I get the same error. I also ran the Hadoop checknative -a command, which verifies that the Native libraries are available:
[root#quickstart /]# hadoop checknative -a
16/10/22 19:32:16 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/10/22 19:32:16 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
The output of the command verifies that the libraries are available, but are not being correctly loaded into the program or classpath. I am not sure how to configure Maven so that it loads in the Hadoop native libraries when running the program. This is the dependencies section of the Maven pom.xml file:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.0.0-alpha1</version>
</dependency>
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.9</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
and the command I run to execute my Mahout java program is
mvn exec:java -Dexec.mainClass=com.patternmatching.RecommendApp.TopPatternMatches
How can I configure Maven to see these libraries so they are used in the program?
I'm looking to minimize the size of my software distribution, and groovy-all.jar is by far the biggest JAR. Groovy is used for logback configuration[1]. On the bottom of the Groovy download page there's a section on the split Groovy distribution.
Which modules / JAR files does logback need to function properly? Is just groovy.jar sufficient?
[1] Yes, I realize I could configure logback with XML, eliminating the need for Groovy support. That is not my question.
I haven't found a source, but as of logback version 1.0.13 my tests show that groovy-jsr223 is needed as well. If I import only groovy in my pom.xml, logback complains about missing classes. The error message is
ERROR in ch.qos.logback.classic.LoggerContext[default] - Groovy classes are not available on the class path. ABORTING INITIALIZATION.
My dependency configuration that works is:
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.0.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy</artifactId>
<version>2.5.8</version>
<type>pom</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-jsr223</artifactId>
<version>2.5.8</version>
<type>pom</type>
<scope>test</scope>
</dependency>
I am currently unable to connect to my cassandra database using the datastax driver. I am getting the following error:
com.datastax.driver.core.TransportException: [/127.0.0.1] Unexpected exception triggered (java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.copyOf(Ljava/util/Collection;)Lcom/google/common/collect/ImmutableSet;)
at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:556)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
Caused by: java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.copyOf(Ljava/util/Collection;)Lcom/google/common/collect/ImmutableSet;
at com.datastax.driver.core.DataType.<clinit>(DataType.java:144)
at com.datastax.driver.core.Codec.<clinit>(Codec.java:31)
However, I have included the guava artefact in my pom.xml as follows:
<!-- Datastax driver -->
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>1.0.4</version>
</dependency>
<!-- Cassandra -->
<dependency>
<groupId>org.apache.cassandra</groupId>
<artifactId>cassandra-all</artifactId>
<version>1.2.9</version>
</dependency>
<!-- guava --<
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
Full pom.xml: http://pastebin.ubuntu.com/6358603/
Am I missing a dependency?
According to its POM, version 1.0.4 of cassandra-driver-core uses version 14.0.1 of Guava, not version 15.0. I'm guessing you are seeing a version clash. Even if that version difference is not the cause of this problem, it might cause other problems.
You do not usually need to include transitive dependencies in POMs, Maven takes care of them for you. Or does your own code use Guava itself?
Based on the advice of this question: no such method error: ImmutableList.copyOf()
I had to exclude the google collections jar:
<dependency>
<groupId>org.zkoss.zk</groupId>
<artifactId>zkspring-core</artifactId>
<version>3.1</version>
<exclusions>
<exclusion>
<groupId>com.google.collections</groupId>
<artifactId>google-collections</artifactId>
</exclusion>
</exclusions>
</dependency>