How to write Partitions to Postgres using foreachPartition (pySpark) - apache-spark

Question

I am new to Spark and trying to wite df partitions to Postgres
here is my code:
//csv_new is a DF with nearly 40 million rows and 6 columns
csv_new.foreachPartition(callback) // there are 19204 partitions
def callback(iterator):
print(iterator)
// the print gives me itertools.chain object
but when writing to DB with following code:
iterator.write.option("numPartitions", count).option("batchsize",
1000000).jdbc(url=url, table="table_name", mode=mode,
properties=properties)
gives an error:
*AttributeError: 'itertools.chain' object has no attribute 'write' mode is append and properties are set
Any leads on how to write the df partition to DB

Answer 1

You don't need to do that.
The documentation states it along these lines and it occurs in parallel:
df.write.format("jdbc")
.option("dbtable", "T1")
.option("url", url1)
.option("user", "User")
.option("password", "Passwd")
.option("numPartitions", "5") // to define parallelism
.save()
There are some performances aspects to consider, but that can be googled.

Answer 2

Many Thanks to #thebluephantom ,just a little add on in case the the table already exists save mode also needs to be defined.
Following was my implementation which worked :-
mode = "Append"
url = "jdbc:postgresql://DatabaseIp:port/DB Name"
properties = {"user": "username", "password": "password"}
df.write
.option("numPartitions",partitions here)
.option("batchsize",your batch size default is 1000)
.jdbc(url=url, table="tablename", mode=mode, properties=properties)