I am using Spark 2.2 on CDH 5.10 cluster with Scala 2.11.8. Everything was working fine but then I suddenly started getting this in the Driver code:
Exception in thread "main" java.lang.LinkageError: loader constraint violation: when resolving method
"org.apache.spark.streaming.StreamingContext$.getOrCreate(Ljava/lang/String;Lscala/Function0;Lorg/apache/hadoop/conf/Configuration;Z)Lorg/apache/spark/streaming/StreamingContext;"
the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader) of the current class, com/hp/hawkeye/driver/StreamingDriver$,
and the class loader (instance of sun/misc/Launcher$AppClassLoader)
for the method's defining class, org/apache/spark/streaming/StreamingContext$,
have different Class objects for the type scala/Function0 used in the signature
Any ideas how I can fix this?
Figured out the solution - there was a class loader conflict which was because of manually placing a dependency jar on the cluster. These helped :
rm -rf ~/.sbt
rm -rf ~/.ivy2/cache
Then restarted IDEA. Spark-submit on cluster was fine. But placing an extra dependent jar in lib (spark-avro-assembly-4.0.0-snapshot) brought back this issue. Somehow that jar which has a fix for Spark 2.2 with spark-avro 3.2 was creating the problem.
Related
We're trying to force an old vendor-provided Java-program (a JAR) to use log4j2, with the help of the log4j "bridge".
To do this, I:
Deleted all entries starting with org/apache/log4j/* from the vendor's JAR
Added the log4j-1.2-api-2.17.1.jar and the regular log4j-*-2.17.1.jar files to CLASSPATH
The program dies unable to find the org/apache/log4j/spi/RootLogger. Listing the contents of the bridge JAR, I find several classes under the org/apache/log4j/spi/, but the RootLogger is, indeed, missing.
How do I deal with this?
The newest version 2.17.2 of Log4j2 contains a couple of dozen of fixes to the Log4j 1.x bridge (cf. changelog).
Among them the org.apache.log4j.RootLogger class has been added to provide binary compatibility with Log4j 1.x.
Warning: many of the added classes are no-ops and just prevent ClassNotFoundExceptions as the one in your question.
I have spark job that is failing after the upgrade of the cdh from 5.5.4 which had spark 1.5.0 to cdh 5.13.0 which has spark 1.6.0
The job is running with the new spark dependencies but i see strange behavior for one spark job that:
1) sometimes it's oozie launcher marked as success and other as killed,
2) also for the spark job itself i see that is failing on the jackson databind.
2018-01-05 19:07:17,672 [Driver] ERROR
org.apache.spark.deploy.yarn.ApplicationMaster - User class threw
exception: java.lang.VerifyError: Bad type on operand stack Exception
Details: Location:
org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
#116: invokevirtual Reason:
Type 'com/codahale/metrics/json/MetricsModule' (current frame, stack[2]) is not assignable to 'com/fasterxml/jackson/databind/Module'
The error you are getting is a Java Bytecode Verification Error.
This happens right before the class can be loaded onto the JVM by the classloader.
The purpose of this step is to ensure that the code didn't come from a malicious compiler, but indeed follows the Java language rules.
Read more about it here: http://www.oracle.com/technetwork/java/security-136118.html
Now, to address your problem. This error is also thrown when your code finds different jars/classes at runtime than the ones which were used during compile time.
The MetricServlet class in the spark-core lib tries to instantiate an object of type MetricsModule which is packaged inside the metrics-json jar.
Then it tries to register this object (within it's 'ObjectMapper') as a generic Module object.
Note: MetricsModule extends from the Module class of jackson-databind jar.
So, in simple terms, an object of type MetricsModule is being type-casted to parent class Module.
However, the MetricsModule class in your environment is not loaded from the metrics-json Jar, but some other foreign Jar or third party library, where it extended a different Module parent class.
This Jar must have been compiled using some.other.package.Module class rather than the original com.fasterxml.jackson.databind.Module from jackson-databind.
E.g. Uber JAR for CosmosDB connector for Spark packages both MetricsModule and Module class. But the latter is packaged under "cosmosdb_connector_shaded.jackson.databind.Module" giving the exact same error -
"Type 'com/codahale/metrics/json/MetricsModule' (current frame,
stack[2]) is not assignable to
'com/fasterxml/jackson/databind/Module'"
To resolve this class conflict you need to find the JAR which actually loaded MetricsModule class. Use -verbose:class JVM option with your Spark Driver JVM to track this.
#sg1 explanation is accurate. For me, I fixed this error by adding the jars as part of spark.driver.extraClassPath instead of copying them in jars/ directory of spark. You can also try shading the particular dependency such as Jackson in your uber jar.
Since Spark already shipped the metrics-json jar, we can mark the scope as provided, which will resolve the conflicts.
<!-- Metrics -->
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>${metrics.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-json</artifactId>
<version>${metrics.version}</version>
<scope>provided</scope>
</dependency>
When JodaTime v. 2.5 library included into WAR file deployment fails in GF 5.0 with following error. It seems problem lies in hibernate-validator. FutureValidatorForReadableInstant is not loaded as ReadableInstant is not loaded by hibernate-validator bundle in GF5.0 while it is working GF 4.1.2.
Exception while loading the app : CDI deployment failure:Errornstantiating :org.hibernate.validator.cdi.internal.ValidationExtension
Caused by: java.lang.TypeNotPresentException: Type org.joda.time.ReadableInstant not present
at sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at sun.reflect.generics.repository.ClassRepository.getSuperclass(ClassRepository.java:90)
at java.lang.Class.getGenericSuperclass(Class.java:777)
at org.hibernate.validator.internal.util.TypeHelper.resolveTypeForClassAndHierarchy(TypeHelper.java:386)
at org.hibernate.validator.internal.util.TypeHelper.resolveTypes(TypeHelper.java:351)
at org.hibernate.validator.internal.util.TypeHelper.extractType(TypeHelper.java:327)
at org.hibernate.validator.internal.engine.constraintvalidation.ClassBasedValidatorDescriptor.(ClassBasedValidatorDescriptor.java:39)
at org.hibernate.validator.internal.engine.constraintvalidation.ConstraintValidatorDescriptor.forClass(ConstraintValidatorDescriptor.java:49)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.hibernate.validator.internal.metadata.core.ConstraintHelper.putConstraints(ConstraintHelper.java:686)
at org.hibernate.validator.internal.metadata.core.ConstraintHelper.(ConstraintHelper.java:386)
at org.hibernate.validator.internal.engine.ValidatorFactoryImpl.(ValidatorFactoryImpl.java:155)
at org.hibernate.validator.HibernateValidator.buildValidatorFactory(HibernateValidator.java:38)
at org.hibernate.validator.internal.engine.ConfigurationImpl.buildValidatorFactory(ConfigurationImpl.java:322)
at org.hibernate.validator.cdi.internal.ValidationExtension.(ValidationExtension.java:116)
... 78 more
I think it might be a bug in Hibernate Validator. I created https://hibernate.atlassian.net/browse/HV-1490 to track this issue.
When we try to detect Joda Time in ConstraintHelper, we use the LoadClass mechanism which might fall back to using the Thread Context class loader if it does not find the class within the ConstraintHelper/HV CL.
In your case, I think the ReadableInstant class is accessible in the TCCL but not in the HV CL.
When we try to get the type from the ConstraintValidator, we only have the HV CL and so it fails to find the class.
So I think we should fix that in HV but I'm not sure it will help you as, once fixed, the Joda Time constraint validators won't be enabled anyway.
What might not help is that the OSGi bundle of HV looks a bit outdated regarding the Joda Time dependency (https://github.com/javaee/glassfish/blob/master/nucleus/packager/external/bean-validator/pom.xml#L124). I'll see with the GlassFish team if they can update it.
I created a PR https://github.com/hibernate/hibernate-validator/pull/857 to fix the issue on our side. You might be able to get this branch, build it (it's just a mvn clean install) and upgrade the jars in your GlassFish. As mentioned above, it won't allow you to use constraints on Joda Time types but, at least, you should be able to start your app. I'm interested in your feedback if you can try that.
About the GF part, I emailed the GF team and will let you know if there is some progress on that front.
Update: GF 5.0.1 will contain a fix for this issue. If you upgrade your current GF to HV 6.0.3.Final we released today, you should be able to start your app (but won't be able to put constraints on Joda Time types, this would have to wait for GF 5.0.1).
I am getting this compilation error
HSSFWorkbook cannot be resolved to a type
when i am using this class in Selenium webdriver.I have added the respective jar, poi 3.9 jar, but still unable to resolve this compilation error.
Hi please update your apache poi to The latest stable release is Apache POI 3.14 from https://poi.apache.org/download.html and then import the Apache POI jars form
jars inside poi-3.14
jars inside lib
Look for "poi-3.17.jar"!!!
Download from "https://poi.apache.org/download.html".
Click the one Binary Distribution -> poi-bin-3.17-20170915.tar.gz
Unzip the file download and look for this "poi-3.17.jar".
Problem solved and errors disappeared.
Hope this answer your question!!!
The message indicates that the class could not be found on the classpath of your application. Did you make sure that the POI jar is added?
I am trying to generate Excel using Xssf API because its memory footprint is small.
It is working fine in my local machine which is having jdk1.7.
But when I try to run it on UNIX where java version is 1.6.0_75 it gives me the following error.
java.lang.NoClassDefFoundError: Could not initialize class org.apache.poi.openxml4j.opc.internal.marshallers.ZipPackagePropertiesMarshaller
I have following jars in my classpath
poi-3.11-20141221.jar
poi-excelant-3.11-20141221.jar
poi-ooxml-3.11-20141221.jar
poi-ooxml-schemas-3.11-20141221.jar
xmlbeans-2.6.0.jar
xercesImpl.jar
I have verified that poi-3.11-20141221.jar has the ZipPackagePropertiesMarshaller class.
Seems that some jar is missing.
Am I missing something?
I have found a solution to my own problem.
I replaced poi-3.11-20141221.jar with poi-ooxml-3.9.jar. That worked.
Java version 1.6.0_75 does not exists, I suppose you make a typo. The last update of Java 6 is the update 45 (6u45).
The class ZipPackagePropertiesMarshaller is loaded at run-time for sure. The exception NoClassDefFoundError occurs during the initialization phase; if the exception had been ClassNotFoundException, it would have been different...
The class ZipPackagePropertiesMarshaller is unaltered between the versions 3.11 and 3.9, but the class PackagePropertiesMarshaller extended by ZipPackagePropertiesMarshaller is changed: the main change regards the use of StAX in the newer version.
The distribution of StAX coming with Java 6, but the version of Java 6 update 18 (http://www.oracle.com/technetwork/java/javase/6u18-142093.html) introduces the StAX 1.2 API version.
Consider to use Java 6u18 or newer. This should solve your problem.
In the official FAQ there are some indications about a similar problem: https://poi.apache.org/faq.html#faq-N1017E.
Moreover, the workaround you found is not the best one, see the last FAQ of POI.