How to add user defined metadata to S3 object via Spark - apache-spark

Question

I am using spark sql dataframe to write to s3 as parquet
Dataset.write
.mode(SaveMode.Overwrite)
.parquet("s3://filepath")
in the spark configuration i have specified following options for SSE and for ACL
spark.sparkContext.hadoopConfiguration.set("fs.s3a.server-side-encryption-algorithm", "AES256")
spark.sparkContext.hadoopConfiguration.set("fs.s3a.acl.default","BucketOwnerFullControl")
how add the user define metadata to the s3 object .
Thanks
Saravanan.

Answer 1

I dont think its possible today. You can't add/update the user defined metadata for S3 objects from EMR. This is from my limited knowledge. Again, AWS Support is the best source to get this answered but I dont believe the API is exposed to allow users to add/update the user defined metadata yet from EMR

Answer 2

It's a very good question. Unfortunately, Spark does not permit it, since Spark is FileSystem agnostic (it doesnt make difference between local, HDFS, or S3).
In my mind, since all file system support kind of metadata, Spark should propose something as well...
But, as a workaround, you can always change meta-data after file have been uploaded.
Example with Java, make file public:
Gradle:
[group: 'org.apache.hadoop', name: 'hadoop-aws', version: '2.8.0'],
(provides: [group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: '1.10.60'],
[group: 'com.amazonaws', name: 'aws-java-sdk-core', version: '1.10.60'])
// Write data to S3
df
.write()
.save("s3a://BUCKET/test_json");
// (this generate test_json/_SUCCESS + test_json/part-XXX keys)
// Create S3 client
private final AmazonS3 conn;
AWSCredentials credentials = new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY);
ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setProtocol(Protocol.HTTP);
this.conn = new AmazonS3Client(credentials, clientConfig);
this.conn.setEndpoint(S3_ENDPOINT);
this.conn.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess(true));
// Get Spark generated folder files
this.conn.listObjects(BUCKET, "test_json").getObjectSummaries().forEach(obj -> {
this.conn.setObjectAcl(BUCKET, obj.getKey(), CannedAccessControlList.PublicRead);
});