Writing a dictionary of Spark data frames to S3 bucket - python-3.x

Question

Suppose we have a dictionary of PySpark dataframes. Is there a way to write this dictionary to an S3 bucket? The purpose of this is to read these PySpark data frames and then convert them into pandas data frames. Below is some code and the errors I get:
columns = ["language","users_count"]
data = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")]
#spark = SparkSession.builder.appName('SparkByExamples.com').getOrCreate()
rdd = spark.sparkContext.parallelize(data)
df1 = rdd.toDF()
df1.printSchema()
columns = ["language","users_count"]
data = [("C", "2000"), ("Java", "10000"), ("Lisp", "300")]
#spark = SparkSession.builder.appName('SparkByExamples.com').getOrCreate()
rdd = spark.sparkContext.parallelize(data)
df2 = rdd.toDF()
df2.printSchema()
spark_dict = {df1: '1', df2: '2'}
import boto3
import pickle
s3_resource = boto3.resource('s3')
bucket='test'
key='pickle_list.pkl'
pickle_byte_obj = pickle.dumps(spark_dict)
try:
s3_resource.Object(bucket,key).put(Body=pickle_byte_obj)
except:
print("Error in writing to S3 bucket")
with this error:
An error was encountered:
can't pickle _thread.RLock objects
Traceback (most recent call last):
TypeError: can't pickle _thread.RLock objects
Also tried dumping the dictionary of PySpark data frames to a json file:
import json
flatten_dfs_json = json.dumps(spark_dict)
and got this error:
An error was encountered:
Object of type DataFrame is not JSON serializable
Traceback (most recent call last):
File "/usr/lib64/python3.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/lib64/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type DataFrame is not JSON serializable

Answer 1

Suppose we have a dictionary of PySpark dataframes. Is there a way to write this dictionary to an S3 bucket?
Yes (you might need to configure access key and secret key)
df.write.format('json').save('s3a://bucket-name/path')
The purpose of this is to read these PySpark data frames and then convert them into pandas data frames.
My 2 cents: This sounds wrong to me, you don't have to convert data to Pandas, that defeats the purpose of using Spark at the first place.