apache spark: Need inputs to optimize my code - apache-spark

Question

I am building a application which reads 1.5 G data and translates. My code skeleton is as follows.
//here i pass a list of id’s to read 4000 files and form a union RDD of all the records and return it as unionbioSetId
run(){
JavaRDD<String> unionbioSetId = readDirectory(ctx, groupAID, groupBID);
JavaRDD<String> temp= unionbioSetId.coalesce(6, false);
JavaPairRDD<String, Tuple3<Double, Double, Double>> flatRDD = temp.flatMapToPair(
new PairFlatMapFunction<String, String, String>() {
return Arrays.asList(new Tuple2<String, String>(key,value));
}}).groupByKey().mapToPair(new PairFunction<Tuple2<String, Iterable<String>>, // input
String, // K
Tuple3<Double, Double, Double> // V
>() {
public Tuple2<String, Tuple3<Double, Double, Double>> call(
Tuple2<String, Iterable<String>> value) {
}).filter(new Function<Tuple2<String, Tuple3<Double, Double, Double>>, Boolean>() {
}});// group by key and map to pair,sort by key
}
String hadoopOutputPathAsString = directory;
flatRDD.saveAsTextFile(hadoopOutputPathAsString);
}
}
///////////////
num of executors:9
driver memory:2g
executor memory: 6g
execuotr cores: 12
My program is running slower than map/reduce(same code skeleton). Can any one help me in optimizing the above code skeleton to make it faster.

Answer 1

Don't call coalesce. You don't need less partitions, you need more. You have 108 worker cores, but you only use 6 of them if you go with 6 partitions. A rule of thumb is you want at least 3 * num_executors * cores_per_executor = 324 partitions.
JavaRDD<String> temp = unionbioSetId.repartition(350);
Or just don't change the number of partitions at all. When the files are read, the data is partitioned by Hadoop splits. In many cases this gives a good layout, and you would avoid the cost of repartitioning.
Read the files at once instead of reading them separately and then taking their union: sc.textFile("file1,file2,file3,...") or sc.textFile("dir/*"). This may also make a performance difference.