How to efficiently select partial data from RDBMS tables in Pyspark - apache-spark

Question

Let's assume I have an employee table like this:
| employee_id | employee_name | department | created_at | updated_at |
|-------------|---------------|------------|---------------------|---------------------|
| 1 | Jessica | Finance | 2020-10-10 12:00:00 | 2020-10-10 12:00:00 |
| 2 | Michael | IT | 2020-10-10 15:00:00 | 2020-10-10 15:00:00 |
| 3 | Sheila | HR | 2020-10-11 17:00:00 | 2020-10-11 17:00:00 |
| ... | ... | ... | ... | ... |
| 1000 | Emily | IT | 2020-10-20 20:00:00 | 2020-10-20 20:00:00 |
Usually, I can batch the data in Pyspark using JDBC connection and write to GCS like this:
df = spark.read.format("jdbc") \
.option("url", "jdbc:postgresql://{ip_address}/{database}") \
.option("dbtable", table_source) \
.option("user", user_source) \
.option("password", password_source) \
.option("driver", "org.postgresql.Driver") \
.load()
df.write.parquet("gs://{bucket_name}/{target_directory}/")
When I create df like my code above and using .load(), does the data still in the database server or spark download all the data from the table and move it to spark cluster (assuming the database and spark cluster placed on different server).
And if I need to get specific data in time range let's say I need the data where created_at > 2020-10-15 00:00:00
does the code below enough? because I found it really slow when the data size reach more than 25 GB
df = spark.read.format("jdbc") \
.option("url", "jdbc:postgresql://{ip_address}/{database}") \
.option("dbtable", table_source) \
.option("user", user_source) \
.option("password", password_source) \
.option("driver", "org.postgresql.Driver") \
.load()
df.createOrReplaceTempView("get_specific_data")
get_specific_data = spark.sql('''
SELECT employee_id, employee_name, department, created_at, updated_at
FROM get_specific_data
WHERE created_at > '2020-10-15 00:00:00'
'''
get_specific_data.write.parquet("gs://{bucket_name}/{target_directory}/")
My question is more like how to efficiently get specific data in Pyspark if I know what data I need to retrieve by column created_date (or any other column, by ID, or something else). Do I need spark sql for this? or using another tools? (for the purpose of to batch the data daily)

Answer 1

Turns out if I specify only the table name in table_source it will load all the data into spark cluster.
To select specific data that I need, I can use something just like this:
last_update = '2020-10-15 00:00:00'
table_source = "(SELECT employee_id, employee_name, department, created_at, updated_at " \
"FROM get_specific_data WHERE created_at > '{0}') t1".format(last_update)
df = spark.read.format("jdbc") \
.option("url", "jdbc:postgresql://{ip_address}/{database}") \
.option("dbtable", table_source) \
.option("user", user_source) \
.option("password", password_source) \
.option("driver", "org.postgresql.Driver") \
.load()
# And then write the data to parquet file
get_specific_data.write.parquet("gs://{bucket_name}/{target_directory}/")