Text, ByteWritable, VIntWritable of Hadoop equivalent in Spark? - apache-spark

Question

Dear Members,
As we know that Text data type of Hadoop uses UTF-8 encoding. That is if the character can be stored in a byte it will store it in a byte. If a character cannot be stored in a byte then it will store it in 2 bytes. The same way, as a performance boost, for word count program in Hadoop Map phase, for mapper value data type if ByteWritable can be used then the amount of mapper output data can be reduced. IntWritable requires 4 bytes whereas ByteWritable needs 1 byte. If I use VIntWritable instead of IntWritable then if an integer can be stored in a byte it will store it in a byte otherwise it will store it in 4 bytes reduce memory footprint.
How the following Java program can be modified so that mapper's key type is Text and value is ByteWritable and reducer's key type is Text and value is VIntWritable.
// Now we have non-empty lines, lets split them into words
JavaRDD<String> words = nonEmptyLines.flatMap(new FlatMapFunction<String, String>() {
#Override
public Iterable<String> call(String s) throws Exception {
return Arrays.asList(s.split(" "));
}
});
// Convert words to Pairs, remember the TextPair class in Hadoop world
JavaPairRDD<String, Integer> wordPairs = words.mapToPair(new PairFunction<String, String, Integer>() {
public Tuple2<String, Integer> call(String s) {
return new Tuple2<String, Integer>(s, 1);
}
});
JavaPairRDD<String, Integer> wordCount = wordPairs.reduceByKey(new Function2<Integer, Integer, Integer>() {
#Override
public Integer call(Integer integer, Integer integer2) throws Exception {
return integer + integer2;
}
});
// Just for debugging, NOT FOR PRODUCTION
wordCount.foreach(new VoidFunction<Tuple2<String, Integer>>() {
#Override
public void call(Tuple2<String, Integer> stringIntegerTuple2) throws Exception {
System.out.println(String.format("%s - %d", stringIntegerTuple2._1(), stringIntegerTuple2._2()));
}
});