Spark SQL: Empty table join with large table slow - apache-spark

Question

I have 2 table
Transaction table (partition by year)
metadata table (All key is unique no partition, 30M records)
My Spark SQL is
SELECT * FROM Transaction WHERE PARTITION_YEAR = 2022; (result: 0 record)
Very fast to get result
SELECT * FROM Metadata WHERE KEY = "A" (result: 1 record)
2-3 seconds to get result
and finally
SELECT * FROM Transaction t LEFT JOIN Metadata m ON t.key = m.key WHERE t.PARTITION_YEAR = 2022;
Pretty slow (3 mins.)
Although
SELECT * FROM (SELECT * FROM Transaction WHERE PARTITION_YEAR = 2022) t LEFT JOIN Metadata m ON t.key = m.key;
Still have to wait (3 mins)

Answer 1

From your Query behaviour , I am guessing your file format is parquet.
SELECT * FROM Metadata WHERE KEY = "A"
This is like a filter operation which supports PUSHDOWN filter and it won't scan the whole table , rather quickly look at the parquet metadata (RANGE) of your interested column (KEY) and figure out .
But when you join in Spark :
It has to bring in the Whole table of METADATA into memory, and it has to scan and shuffle the data based on your join condition. i.e. PUSHDOWN filter is not supported for joins.
Your last 2 queries are essentially same. Spark will only bring 2022 data into memory. Even though your partition might be empty, 30M records of METADATA will be loaded.
Incase you want to avoid this situation for empty partitions, you should check if the partition is empty then only fire the
The cheapest/efficient way to check :
val dfPartition = spark.sql("SELECT * FROM Transaction WHERE PARTITION_YEAR = 2022;")
if(!dfPartition.isEmpty()) // fastest.
{
//Fire your join query
}