ThreadPooling in pyspark seems to sum the contents of a dataframe - apache-spark

Question

I am using Threadpooling to try to parallelize multiple for loops that I have in my pyspark code.
However, it seems that when using the reduce function at the end, it is simply summing the contents of the resulting list of dataframes.
Here is the code I am using:
from multiprocessing.pool import ThreadPool
from multiprocessing import cpu_count
from pyspark.sql import functions as f
days=[180,5]
def run_threads(iterr):
for a in range(len(datepaths)-iterr, len(datepaths)-(iterr-1)):
path="s3://" + datepaths[a] + "/"
date_inpath.append(path)
uva=spark.read.option("basePath",inpath).parquet(*date_inpath)\
.select(*uvakeepcols)\
.filter((col('RESULT').isNotNull()))\
.dropDuplicates()
uva1=uva.groupBy('SEQ_ID','TOOL_ID')\
.agg(f.count('RESULT').alias('count'))\
return uva1
pool = ThreadPool(cpu_count()-1)
df_list = pool.map(run_threads, days)
pool.close()
pool.join()
df=reduce(DataFrame.unionByName, df_list)
The above code extracts the results of a particular SEQ_ID, TOOL_ID combination for 2 days and then counts the number of times the RESULT column has values.
When I use thread pooling and compare the counts for the 2 days, I get the following result:
+-------+-------+-----+
| SEQ_ID|TOOL_ID|count|
+-------+-------+-----+
|2945783| 15032| 574|
|2945783| 15032| 574|
+-------+-------+-----+
However, when I extract the data manually from each of the 2 days I get the following result:
From 180 days ago:
+-------+-------+-------------+
| SEQ_ID|TOOL_ID|count(RESULT)|
+-------+-------+-------------+
|2945783| 15032| 285|
+-------+-------+-------------+
From 5 days ago:
+-------+-------+-------------+
| SEQ_ID|TOOL_ID|count(RESULT)|
+-------+-------+-------------+
|2945783| 15032| 289|
+-------+-------+-------------+
Somehow the pooling method is adding the two counts 289+285=574
What exactly is going on here, and is there a way to correct this?
Any insight would be greatly appreciated.
Thank you