How to solve NoMethodError that arises due to using a same library with two different versions using gradle or anything else? - apache-spark

Question

I have two dependencies that use a same library but with different versions and I get NoMethodError. I am not sure how to solve it using gradle or anything else? Any help would be great
group 'com.hello'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'idea'
repositories {
mavenCentral()
mavenLocal()
}
dependencies {
compile 'org.slf4j:slf4j-log4j12:1.7.12'
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.0.1'
compile group: 'org.apache.spark', name: 'spark-streaming_2.11', version: '2.0.1'
compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: '2.0.1'
compile group:'org.apache.kafka', name: 'kafka-clients', version: '0.10.1.0'
compile group: 'com.datastax.spark', name: 'spark-cassandra-connector_2.11', version: '2.0.0-M3'
compile group: 'com.github.brainlag', name: 'nsq-client', version: '1.0.0.RC2'
compile group: 'com.google.code.gson', name: 'gson', version: '2.7'
}
task buildStreamingJar(type: Jar) {
if(project.hasProperty("jarname")) {
def fullyQualifiedPackageName = ""
def jarname = project.getProperty("jarname")
if (jarname.equalsIgnoreCase("SparkDriver1")) {
fullyQualifiedPackageName = "com.hello.streamprocessing.app.SparkDriver1"
}
if (jarname.equalsIgnoreCase("SparkDriver2")) {
fullyQualifiedPackageName = "com.hello.streamprocessing.app.SparkDriver2"
}
baseName = project.name + "-$jarname" + "-stream"
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
with jar
zip64 true
from sourceSets.test.output
manifest {
attributes 'Main-Class': fullyQualifiedPackageName
}
exclude 'META-INF/.RSA', 'META-INF/.SF', 'META-INF/*.DSA'
}
}
so what is happening here is that in my dependencies spark-core_2.11 uses
com.google.guava:14.0.1 and nsq-client uses com.google.guava:19.0
which is leading to the following error.
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
at com.github.brainlag.nsq.NSQProducer.(NSQProducer.java:22)
at com.hello.streamprocessing.app.SparkDriver2.main(SparkDriver2.java:37)
I need to use this build.gradle file to build a FAT Uber jar so I can use it with spark-submit

Answer 1

Basically you can't really solve this, if either spark-core or nsq-client is using features of guava that or not present in one version or the other. You can run gradlew dependencies which will show you how gradle resolves the version conflict on guava. By default it will choose version 19.0 as its the newer version. Your best chance is to use versions of your libraries that are dependent on the same version of guava. That is either to downgrade nsq-client or upgrade spark-core if possible.
You can exclude the transitive dependency to guava like this:
dependencies {
...
compile (group: 'com.github.brainlag', name: 'nsq-client', version: '1.0.0.RC2') {
exclude group: 'com.google.guava', module: 'guava'
}
...
}
However this will only work if nsq-client is not relying on features only present in version 19.0 of guava as mentioned above.
Sadly this is a common problem especially with the guava library as it is used in nearly every project and the developers don't care a lot about a stable API of their library.
Your best bet is probably to try excluding guava from spark-core and give it a try and, if it's not working, to exclude it from nsq-client and give it a try. But even if it seems to be working at a first glance odds are good (or actually bad in your case), that you will run into similar problems during runtime of your application.
If the above is not working, you could still fork nsq-client and replace the problematic parts with something compatible to guava 14.0.1. I don't really like this option but if the developers of nsq-client are nice they might accept your change...

Answer 2

One of the dependencies lib is not being downloaded. Use the latest lib of both. Use can as well use this:
repositories {
mavenCentral()
}
configurations {
compile
}
dependencies {
compile 'dependency repo1'
compile 'dependency repo2'
}
task libs(type: Sync) {
from configurations.compile
into "$buildDir/libs"
}
This will help download both dependencies.