Fat JAR with Kotlin and Apache Spark 2.3 - apache-spark

Question

Im using gradle to build my project mixing Kotlin and Apache Spark, but as soon I declare the spark dependency, the Fat JAR I generate gets non working. Otherwise it will work. The source code not even import anything from Spark
buildscript {
ext.kotlin_version = '1.2.40'
repositories {
mavenCentral()
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
}
}
apply plugin: 'kotlin'
repositories {
mavenCentral()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
// uncomment this line to get main class 'myownpkg.SparkApplicationKt'
// not found error
// compile 'org.apache.spark:spark-core_2.11:2.3.0'
}
jar {
manifest { attributes 'Main-Class': 'myownpkg.SparkApplicationKt' }
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
}

Answer 1

After some experimentation I realized the generated JAR was having there a lot of duplicated files, causing that runtime not finding the .class required.
It was triggered after enable the Spark because it is the dependency causing snowball of other dependencies having the same file paths under META-INF folder
exclude 'META-INF/*'
That line made the trick to avoid duplicates but still wil have a META-INF folder in final JAR

Answer 2

The main reason is because you aren't creating the "FatJar" artifact with necessary dependencies. The compile dir in configuration only contain compiled source code.
From the maven central you need at least the 50 compile dependencies that spark-core require. Have you consider using the shadow plugin ?
Take a look at this thread on gradle discuss.