NoSuchMethodError? code is ok in idea, but wrong on cluster - apache-spark

Question

My code runs okay on my win7 idea64, but when i package the code, and run it on yarn cluster, it throws an expection:
java.lang.NoSuchMethodError: org.apache.commons.beanutils.PropertyUtilsBean.addBeanIntrospector(Lorg/apache/commons/beanutils/BeanIntrospector;)V
I checked the pom.xml, and find relative jars is already included
my pom.xml is like this:
<properties>
<scala.version>2.10.6</scala.version>
</properties>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.4.2</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-configuration2</artifactId>
<version>2.2</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.9.3</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<!--<scope>provided</scope>-->
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.0-cdh5.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.0-cdh5.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.6.0-cdh5.7.2</version>
<!--<scope>provided</scope>-->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>1.6.0-cdh5.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.2.0-cdh5.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>1.2.0-cdh5.7.2</version>
</dependency>
The line/part/block where the error is happening exactly is:
line43 val config = ConfigurationUtil("config.properties").config
The stack trace is:
19/05/22 10:30:11 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
19/05/22 10:30:12 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1557905260816_0073_000002
19/05/22 10:30:12 INFO spark.SecurityManager: Changing view acls to: yarn,yizheng
19/05/22 10:30:12 INFO spark.SecurityManager: Changing modify acls to: yarn,yizheng
19/05/22 10:30:12 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, yizheng); users with modify permissions: Set(yarn, yizheng)
19/05/22 10:30:12 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
19/05/22 10:30:12 INFO yarn.ApplicationMaster: Waiting for spark context initialization
19/05/22 10:30:12 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
19/05/22 10:30:12 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.commons.beanutils.PropertyUtilsBean.addBeanIntrospector(Lorg/apache/commons/beanutils/BeanIntrospector;)V
java.lang.NoSuchMethodError: org.apache.commons.beanutils.PropertyUtilsBean.addBeanIntrospector(Lorg/apache/commons/beanutils/BeanIntrospector;)V
at org.apache.commons.configuration2.beanutils.BeanHelper.initBeanUtilsBean(BeanHelper.java:631)
at org.apache.commons.configuration2.beanutils.BeanHelper.<clinit>(BeanHelper.java:89)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:259)
at com.sun.proxy.$Proxy11.<clinit>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
at org.apache.commons.configuration2.builder.fluent.Parameters.createParametersProxy(Parameters.java:307)
at org.apache.commons.configuration2.builder.fluent.Parameters.properties(Parameters.java:246)
at com.sunwada.utils.ConfigurationUtil$.<init>(ConfigurationUtil.scala:16)
at com.sunwada.utils.ConfigurationUtil$.<clinit>(ConfigurationUtil.scala)
at com.sunwada.sparkStreaming.SparkStreamKafkaHbaseSaveOffsetGson$.main(SparkStreamKafkaHbaseSaveOffsetGson.scala:43)
at com.sunwada.sparkStreaming.SparkStreamKafkaHbaseSaveOffsetGson.main(SparkStreamKafkaHbaseSaveOffsetGson.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
19/05/22 10:30:13 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: org.apache.commons.beanutils.PropertyUtilsBean.addBeanIntrospector(Lorg/apache/commons/beanutils/BeanIntrospector;)V)
19/05/22 10:30:22 ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.
19/05/22 10:30:22 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.NoSuchMethodError: org.apache.commons.beanutils.PropertyUtilsBean.addBeanIntrospector(Lorg/apache/commons/beanutils/BeanIntrospector;)V)
19/05/22 10:30:22 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1557905260816_0073
19/05/22 10:30:23 INFO util.ShutdownHookManager: Shutdown hook called
What can I do now?

Answer 1

AFAIK this is the culprit.
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.9.3</version>
<scope>compile</scope>
</dependency>
scope compile will be used for only compilation purpose.
you may need to change it to runtime
OR
It is an inconsistency between the dependencies at compile time and at runtime.

Answer 2

NoSuchMethodError mostly caused by version conflict.you can install dependency help plugin in your IDE solve it