Spark Java Map processing entire data set in all executors - apache-spark

Question

I have 2 executors, 2 cores each (total 4 partitions), the following function executed 22824 times for my data set containing only 5706 records. I see messages repeated 33824 (5706 x 4) in my logs:
here is my code:
Dataset<Row> dbDataSet= dbSvc.getRecords(fkId);
RDD<String> rdd= dbDataSet.toJavaRDD()
.map(row-> transformSvc.transformBlobToString(row)).rdd();
public class TransformService{
public static String transformBlobToString(Row row)
logger.info("*** transformBlobToString() Invoked on ExecutorId " + SparkEnv.get().executorId());
In logs I see
this repeated 11412 times (5706 x2 cores)
23/01/06 18:01:11 INFO TransformService: *** transformBlobToString() Invoked on ExecutorId 1
23/01/06 18:01:11 INFO TransformService: *** transformBlobToString() Invoked on ExecutorId 1
23/01/06 18:01:11 INFO TransformService: *** transformBlobToString() Invoked on ExecutorId 1
this repeated 11412 times (5706 x2 cores)
23/01/06 18:01:11 INFO TransformService: *** transformBlobToString() Invoked on ExecutorId 2
23/01/06 18:01:11 INFO TransformService: *** transformBlobToString() Invoked on ExecutorId 2
23/01/06 18:01:11 INFO TransformService: *** transformBlobToString() Invoked on ExecutorId 2
Why is this happening? What I want to do is split my total of 5706 records in 4 partitions and process 1426 in each of the partition. The function I used to fetch the DB (dbService.getRecords()) already splits the data in 4 equal parts through an incremental column. So I expected these 4 parts to be executed in parallel in 4 cores each one processing 1426 per partition. Instead it is doing entirely opposite of what I expected -- processing the entire 5706 records in each partition. Please help how I can fix this and only process 1/4rth of records in each partition.
Full code:
Mainfunction() {
Dataset < Row > topLevelDataSet = dbSvc.getRecords(id);
JavaRDD < String > stringifiedDataSet = processRecords(topLevelDataSet);
Dataset < Row > xmlDataSet = sparkSession.read().json(stringifiedDataSet);
csvDataSet = filterFormatDataFrame(xmlDataSet);
int requiredPartitions = calculatePartitions(csvDataSet);
if (requiredPartitions > dbSvc.getNumberOfDBPartitions()) {
csvDataSet.repartition((requiredPartitions).write().format("csv").option("header", "true")
.save(filepath);
}
else {
csvDataSet.write().format("csv").option("header", "true")
.save(filepath);
}
}
public Dataset < Row > getRecords(Long id) throws SQLException {
String table = "(select t1.*, ROWNUM as num_rows from (select * from myTableName where id= " + id + " ) t1) mytable";
String totalRowCount = getTotalRowCount(id, otherFIlters);
Properties properties = new Properties();
properties.setProperty("partitionColumn", "num_rows");
properties.setProperty("lowerBound", "0");
properties.setProperty("upperBound", totalRowCount);
properties.setProperty("numPartitions", 4);
properties.setProperty("fetchsize", "100");
properties.setProperty("Driver", driver);
properties.setProperty("user", user);
properties.setProperty("password", password);
Dataset < Row > records = SparkSession.getActiveSession().get().read().jdbc(jdbcUrl, table, properties);
return records;
}
}
private JavaRDD < String > processRecords(Dataset < Row > topLevelDataSet) {
JavaRDD < Row > rdd = topLevelDataSet.toJavaRDD();
// have confirmed that each Partition ONLY has 1426 records
JavaRDD < String > transformed = rdd.mapPartitions(xmlRows - > {
logger.info("*** Inside Parallellize function mapPArtition");
List < String > transformedX = new ArrayList < String > ();
int count = 0;
while (xmlRows.hasNext()) {
count++;
transformedX.add(extractBlobToString(xmlRows.next()));
}
logger.info("*** processRecords outside loop total count of records :" + count); < -Only 1405 records
return transformedX.iterator();
});
return transformed;
}
private int calculatePartitions(Dataset < Row > csvDataset) {
LongAccumulator numberOfPartitionAccumulator = sparkSession.sparkContext()
.longAccumulator("NumberOfPartitions");
csvDataset.foreachPartition(rows - > {
numberOfPartitionAccumulator.add(1);
Long dataSize = 0;
while (rows.hasNext()) {
dataSize += rows.next().json();
if (dataSize >= MAX_FILE_SIZE) {
numberOfPartitionAccumulator.add(1);
dataSize = 0;
}
}
});
return numberOfPartitionAccumulator.value().intValue();
}
An update to the original post:
another piece of critical info I found. I have stages in the DAG, stage0 executes fine as expected (each partition only processing 1/4rth of the recordset). then next stage1 "save" reexecutes the loop from the very beginning, and then the count operation stage reexecutes the loop all over again. That explains why same row was processed multiple times, but that does not explain why the stage0 steps are reexecuted in next stages. what in my code above is causing it?