Spark - Register model objects with Kyro - Caused by: java.lang.IllegalArgumentException: Class is not registered: - apache-spark

Question

I am registering the classes which has business logic and model classes with Kyro in spark . I get the below exception
> Job aborted due to stage failure: Task 14 in stage 1.0 failed 4 times,
> most recent failure: Lost task 14.3 in stage 1.0 (TID 90, **):
> java.lang.IllegalArgumentException: Class is not registered: Object[]
> Note: To register this class use: kryo.register(Object[].class); at
> com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442) at
> com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
> at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472) at
> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565) at
> org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:296)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:239)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Driver stacktrace:
Kyro Registrtor :
public class KyroSerializer implements KryoRegistrator {
#Override
public void registerClasses(Kryo kryo) {
kryo.register(People.class);
kryo.register(Lookup.class);
}
}
Model class:
class people implements Serializable{
private static final long serialVersionUID = 1L; ...... }
public class Lookup implements Serializable{
private static final long serialVersionUID = 1L;
private String code1;
private String code2;
}
Finally my Spark context :
sc.set("spark.kryo.registrator", KyroSerializer.class.getName())

Answer 1

From the exception, it seems that kryo has not registered a class object for Object[] (array whose entries are of type Object). Please try to change your code as follows:
public class KyroSerializer implements KryoRegistrator {
#Override
public void registerClasses(Kryo kryo) {
kryo.register(Object[].class); // add this line to your class
kryo.register(People.class);
kryo.register(Lookup.class);
}
}
In addition, I would register your custom registrator class with spark like this
sc.set("spark.kryo.registrator", KyroSerializer.class.getCanonicalName());
If it works for you, though, then just ignore my last remark.