Spring-boot+Spark in cluster, yarn-client mode - apache-spark

Question

i have tried to use spring-boot to give web service in spark,but there seems some problems.
when i debug on local,it run correctly.but as i put it on cluster and run it with spark-submit,something wrong.
source list
public List<Person> txtlist(String sqlstr) {
JavaRDD<Person> peopleRDD = sparkSession.read().textFile("hdfs://master:9000/user/root/people.txt").javaRDD().map(line -> {
String[] parts = line.split(",");
Person person = new Person();
person.setName(parts[0]);
person.setAge(Long.parseLong(parts[1].trim()));
return person;
});
Dataset<Row> peopleDF = sparkSession.createDataFrame(peopleRDD, Person.class);
peopleDF.createOrReplaceTempView("people");
Dataset<Person> sqlFrame = sparkSession.sql(sqlstr).as(Encoders.bean(Person.class));
List<Person> plist = sqlFrame.collectAsList();
return plist;
}
error list
17/11/24 00:35:30 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, slave3, executor 2):
java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.fun$1 of type org.apache.spark.api.java.function.Function in instance of org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2251)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
I tried some solutions from internet, like .setJars(), also spark-submit, but no use.
have someone used spring-boot with lamba in spark, met and solved this problem or correctly run in cluster?
Update:
I tried another kind of codes. but false
JavaRDD<String> lines = sparkSession.read().textFile("hdfs://master:9000/user/root/people.txt").javaRDD();
JavaRDD<Row> peopleRDD = lines.map(new Function<String,Row>()
{
/**
*
*/
private static final long serialVersionUID = 1L;
#Override
public Row call(String line) throws Exception {
String[] splited = line.split(",");
return RowFactory.create(splited[0],Integer.valueOf(splited[1]));
}
});
List<StructField> structFields = new ArrayList<StructField>();
structFields.add(DataTypes.createStructField("name", DataTypes.StringType, true));
structFields.add(DataTypes.createStructField("age", DataTypes.IntegerType, true));
StructType structType = DataTypes.createStructType(structFields);
the error code:
cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.
Is it maybe a Serialization problem?