How to join 3 RDD tables in spark using java? - apache-spark

Question

Please bear with on this one. I have three RDDs ( coming from Hadoop ). All Three have unique keys such as ippaddress and boxnumber on which they can be matched/joined. Here are some sample data from all tables. Table A column boxnumber have to converted to number before it can be matched.
Table A:
ipaddress|boxnumber|cardnumber
94.254.57.16|59774DEa1|0D1EDF40
94.154.57.176|5F7377Ga9|0D3F796D
Table B:
cardno,boxnumber
1500914,2000096
1500413,2211469
Table C:
ipaddress|kanal|bitrate|kanaltimespent|date|country
94.254.57.16|sky|2023|003DF6A.ts|12-02-2016|chile
94.154.57.176|ITV|3425|003DF6A.ts|23-04-2014|egypt
My first attempt in java:
//TABLE A
JavaSparkContext sc = SetupSparkContext("SparkSample");
JavaRDD<ExtractTable_A> ta_RDD= ExtractTable_A.getRDD(sc);
JavaPairRDD<String, ExtractTable_A> A_PairRDD = ta_RDD.mapToPair(new PairFunction<extractTable_A, String, ExtractTable_A>()
{
#Override
public Tuple2<String, ExtractTable_A> call(ExtractTable_A extractTable_A) throws Exception
{
String [] A= extractTable_A.toString().split("|") ;
return new Tuple2<>(A[0],extractTable_A);
}
});
//TABLE B
JavaRDD<ExtractOttPdl> tb_RDD = ExtractTableB.getRDD(sc);
JavaPairRDD<String, ExtractTable_B> BPairRDD = tb_RDD.mapToPair(new PairFunction<extractTable_B, String, ExtractTable_B>()
{
#Override
public Tuple2<String, ExtractTable_B> call(ExtractTable_B extractTable_B) throws Exception
{
String [] B= extractTable_B.toString().split(",") ;
return new Tuple2<>(B[1],extractTable_B);
}
});
//TABE C
JavaRDD<ExtractTable_C> tc_RDD = ExtractTableC.getRDD(sc);
JavaPairRDD<String, ExtractTable_C> CPairRDD = tb_RDD.mapToPair(new PairFunction<extractTable_C, String, ExtractTable_C>()
{
#Override
public Tuple2<String, ExtractTableC> call(ExtractTableC extractTable_C) throws Exception
{
String [] C= extractTable_A.toString().split("|") ;
return new Tuple2<>(C[0],extractTable_A);
}
});
//At this point i need to join and create an .txt output file
The final result shoud be a file with these headers
KANAL|BITRATE|TIMESPENT|DATE|COUNTRY
===update===
I have managed to join the Table A and Table B but now i am stuck on how to join the TableC to Table A?
//Joined table A and B
JavaPairRDD<String, Tuple2<ExtractTableA, ExtractTableB>> join_1 = A_PairRDD.join(B_PairRDD);
. . .
//Joined table A and C
JavaPairRDD<String, Tuple2<ExtractTableA, ExtractTableC>> Join_2 = A_PairRDD.join(B_PairRDD);
// Output results from TableA and TableB
join_1.map(in -> {
return new ResultStringBuilder("|")
.append(Long.parseLong((in._2()._1().getCardno().trim()),16))
.append(Long.parseLong((in._2()._1().getBoxno().trim()),16))
.append(in._2()._2().getBoxno())
*** HERE I NEED TO ALSO APPEND THE COLUMN FROM TableC
.toString();
})
.saveAsTextFile("c:\outfile");

Answer 1

Remember that when you are working with the spark API, you always want to create a new RDD when you modify anything in the RDD sturcture because RDD is immutable.
In order to do three way join in this case,
you need to create a new JavaPairRDD after you join the first two tables,
because you want to have a PairRDD with new key-value pair because unique keys for Table A, B, C are different.
There could be two ways to do this (either join AB first or AC first)
The way you could join tables is like this:
Table A - Table B (PairRDD with key : boxnumber or cardnumber or maybe both)
After you join Table A and Table B, you need to create a new PairRDD with key ipaddress because you want to join with Table C.
// joinedAB is RDD resulting from join operation of Table A and C
JavaPairRDD joinedABForC = joinedAB.map(l -> new Tuple2(l[0], l));
// now joinedABForC has ipaddress as the RDD's key
// join rdd joinedABForC with Table C
After we moved the unique column to the key of the pairRdd, you can now join it with the Table C and three way join is done.
Joined Table AB - Table C (PairRdd with key : ipaddress)