ClassNotFound when submit a Spring-boot-fat-jar to spark - apache-spark

Question

when I build a fat jar using spring-boot, its name is
sparker4m-0.0.1.jar
And I "spark-submit" it to spark cluster.
./bin/spark-submit --deploy-mode client --class org.springframework.boot.loader.JarLauncher ~/Desktop/sparker4m-0.0.1.jar --spark.master=spark://172.16.179.52:7077 --source_path=file:///usr/local/spark-2.3.1-bin-hadoop2.7/hdfs_mock/1530842877616 --out_path=file:///usr/local/spark-2.3.1-bin-hadoop2.7/hdfs_mock/153084287761_out --hadoop.home.dir=/usr/local/hadoop
You can see I use " --class org.springframework.boot.loader.JarLauncher", because it's the main-class of Spring-boot-jar, and the actual Main-class is test.sparker4m.Sparker4mApplication
here is my code:
#SpringBootApplication
public class Sparker4mApplication {
public static void main(String[] args) throws Exception{
ApplicationContext ctx =
SpringApplication.run(Sparker4mApplication.class, args);
SparkConf conf = new SparkConf().setAppName("Test")
.setMaster(happyDay.getVal("spark.master"))
.set("spark.driver.host", Inet4Address.getLocalHost().getHostAddress());
JavaSparkContext sc = new JavaSparkContext(conf);
.........
JavaPairRDD<String, String> transformed = rdd.mapToPair(new PairFunction<Tuple2<String, PortableDataStream>, String, String>() {
#Override
public Tuple2<String, String> call(Tuple2<String, PortableDataStream> tuple2) throws Exception {
String fname = tuple2._1();
PortableDataStream content = tuple2._2();
byte[] bytes = content.toArray();
String result = YUVSimpleTrans.transform(bytes);
return new Tuple2<>(fname, result);
}
});
..........
.........
}
}
Everything goes well, but, when Excecutor runs, exception throws:
Caused by: java.lang.ClassNotFoundException: test.sparker4m.Sparker4mApplication$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
test.sparker4m.Sparker4mApplication$1 is an inner class used by test.sparker4m.Sparker4mApplication and it should be "new PairFunction<Tuple2<String, PortableDataStream>, String, String>()".
I supposed that, spark's Executor process use a individual URLClassLoader to load classes, but my(sprint-boot-fat-jar's) classes are all in sparker4m-0.0.1.jar/BOOT-INF/classes, and my dependency jars are all in sparker4m-0.0.1.jar/BOOT-INF/lib, so the URLClassLoader cannot load the class.
Is there any good idea for this? needs help!

Answer 1

Finally, got solution.
The key's are:
1.It should use an uber-jar with all classes and resoures and not in "jar lib"
2.Must understand how "JarLaucher" works. JarLaucher is offered by "spring-boot-tools", it is used to "createArchive" and URLLoadClass from /BOOT-INF/*(spring-boot 2.x) and start true Main class.
so, if "1" is done, "2" is not needed, because we don't need URLLoadClass from /BOOT-INF/* any more.
The solution is : use maven-shade-plugin(with out spring-boot-maven-plugin) to build an uber-jar, and define your true main class(not JarLaucher) use org.apache.maven.plugins.shade.resource.ManifestResourceTransformer in this plugin.
After all things done, I found a question Submitting spring boot application jar to spark-submit was already fit this out, but the answer didn't point clearly that you shoud not use ${start-class}(JarLaucher) but true main class