spark data frame left outer join is taking lot time - apache-spark

Question

I have two dataframes ipwithCounryName(12Mb) and ipLogs(1GB) . I would like to join two data frames based on common column ipRange. ipwithCounryName df i brodcasted Below is my code.
val ipwithCounryName_df = Init.iptoCountryBC.value
ipwithCounryName_df .createOrReplaceTempView("inputTable")
ipLogs.createOrReplaceTempView("ipTable")
val joined_table= Init.getSparkSession.sql("SELECT hostname,date,path,status,content_size,inputTable.countryName FROM ipasLong Left JOIN inputTable ON ipasLongValue >= StartingRange AND ipasLongValue <= Endingrange")
=====Physical plan===
*Project [hostname#34, date#98, path#36, status#37, content_size#105L,
countryName#5]
+- BroadcastNestedLoopJoin BuildRight, Inner, ((ipasLongValue#354L >=
StartingRange#2L) && (ipasLongValue#354L <= Endingrange#3L))
:- *Project [UDF:IpToInt(hostname#34) AS IpasLongValue#354L, hostname#34,
date#98, path#36, status#37, content_size#105L]
: +- *Filter ((isnotnull(isIp#112) && isIp#112) &&
isnotnull(UDF:IpToInt(hostname#34)))
: +- InMemoryTableScan [path#36, content_size#105L, isIp#112,
hostname#34, date#98, status#37], [isnotnull(isIp#112), isIp#112,
isnotnull(UDF:IpToInt(hostname#34))]
: +- InMemoryRelation [hostname#34, date#98, path#36, status#37,
content_size#105L, isIp#112], true, 10000, StorageLevel(disk, memory,
deserialized, 1 replicas)
: +- *Project [hostname#34, cast(unix_timestamp(date#35,
dd/MMM/yyyy:HH:mm:ss ZZZZ, Some(Asia/Calcutta)) as timestamp) AS date#98,
path#36, status#37, CASE WHEN isnull(content_size#38L) THEN 0 ELSE
content_size#38L END AS content_size#105L, UDF(hostname#34) AS isIp#112]
: +- *Filter (isnotnull(isBadData#45) && NOT isBadData#45)
: +- InMemoryTableScan [isBadData#45, hostname#34,
status#37, path#36, date#35, content_size#38L], [isnotnull(isBadData#45), NOT
isBadData#45]
: +- InMemoryRelation [hostname#34, date#35,
path#36, status#37, content_size#38L, isBadData#45], true, 10000,
StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *Project [regexp_extract(val#26,
^([^\s]+\s), 1) AS hostname#34, regexp_extract(val#26, ^.*
(\d\d/\w{3}/\d{4}:\d{2}:\d{2}:\d{2} -\d{4}), 1) AS date#35,
regexp_extract(val#26, ^.*"\w+\s+([^\s]+)\s*[(HTTP)]*.*", 1) AS path#36,
cast(regexp_extract(val#26, ^.*"\s+([^\s]+), 1) as int) AS status#37,
cast(regexp_extract(val#26, ^.*\s+(\d+)$, 1) as bigint) AS content_size#38L,
UDF(named_struct(hostname, regexp_extract(val#26, ^([^\s]+\s), 1), date,
regexp_extract(val#26, ^.*(\d\d/\w{3}/\d{4}:\d{2}:\d{2}:\d{2} -\d{4}), 1),
path, regexp_extract(val#26, ^.*"\w+\s+([^\s]+)\s*[(HTTP)]*.*", 1), status,
cast(regexp_extract(val#26, ^.*"\s+([^\s]+), 1) as int), content_size,
cast(regexp_extract(val#26, ^.*\s+(\d+)$, 1) as bigint))) AS isBadData#45]
: +- *FileScan csv [val#26] Batched:
false, Format: CSV, Location:
InMemoryFileIndex[file:/C:/Users/M1047320/Desktop/access_log_Jul95],
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<val:string>
+- BroadcastExchange IdentityBroadcastMode
+- *Project [StartingRange#2L, Endingrange#3L, CountryName#5]
+- *Filter (isnotnull(StartingRange#2L) && isnotnull(Endingrange#3L))
+- *FileScan csv [StartingRange#2L,Endingrange#3L,CountryName#5] Batched: false, Format: CSV, Location: InMemoryFileIndex[file:/C:/Users/M1047320/Documents/Spark-301/Documents/GeoIPCountryWhois.csv], PartitionFilters: [], PushedFilters: [IsNotNull(StartingRange), IsNotNull(Endingrange)], ReadSchema: struct<StartingRange:bigint,Endingrange:bigint,CountryName:string>
Join is taking more time (>30 minutes). I have one more inner join on two different dataframe of same size where join condition is "=". Its taking only 5 minutes. How should i improve my code? Please suggest

Answer 1

Please keep the filter condition in where and join the tables based on common column name.I assummed countryname is the common across both DF.
val joined_table= Init.getSparkSession.sql("SELECT hostname,date,path,status,content_size,inputTable.countryName FROM ipasLong Left JOIN inputTable ON ipasLong.countryName=inputTable.countryName
WHERE ipasLongValue >= StartingRange AND ipasLongValue <= Endingrange")
You can also directly join the dataframes.
val result=ipLogs.join(broadcast(ipwithCounryName),"joincondition","left_outer").where($"ipasLongValue" >= StartingRange && $"ipasLongValue" <= Endingrange).select("select columns")
Hope it helps you.

Answer 2

You can try increasing your JVM parameters to the capacity of your system to fully utilize it like below:
spark-submit --driver-memory 12G --conf spark.driver.maxResultSize=3g --executor-cores 6 --executor-memory 16G