configuration for `spark.hadoop.fs.s3` gets applied to `fs.s3a` not `fs.s3` - apache-spark

Question

I have read the answer posted on how to configure s3 access keys for dataproc but I have find it dissatisfactory.
Reason is because when I follow the steps and set hadoop conf for spark.hadoop.fs.s3, s3://... path still have access issue while as s3a://... path works. Test spark-shell run can be seen below.
s3 vs s3n vs s3a is topic of it's own, although I guess we don't have to worry about s3n. But I find it strange that configuration application to s3 is manifest for s3a.
here are my questions:
is this dataproc or spark? I'm assuming spark considering spark-shell is having this issue.
is there a way to configure s3 from spark-submit conf flags without code change?
Is this a bug or are we now preferring s3a over `s3?
Thanks,
***#!!!:~$ spark-shell --conf spark.hadoop.fs.s3.awsAccessKeyId=CORRECT_ACCESS_KEY \
> --conf spark.hadoop.fs.s3.awsSecretAccessKey=SECRETE_KEY
// Try to read existing path, which breaks...
scala> spark.read.parquet("s3://bucket/path/to/folder")
17/06/01 16:19:58 WARN org.apache.spark.sql.execution.datasources.DataSource: Error while looking for metadata directory.
java.io.IOException: /path/to/folder doesn't exist
at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170)
...
// notice `s3` not `s3a`
scala> spark.conf.getAll("spark.hadoop.fs.s3.awsAccessKeyId")
res3: String = CORRECT_ACCESS_KEY
scala> spark.conf.getAll("fs.s3.awsAccessKeyId")
java.util.NoSuchElementException: key not found: fs.s3.awsAccessKeyId
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:59)
... 48 elided
scala> sc
res5: org.apache.spark.SparkContext = org.apache.spark.SparkContext#426bf2f2
scala> sc.hadoopConfiguration
res6: org.apache.hadoop.conf.Configuration = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, file:/etc/hive/conf.dist/hive-site.xml
scala> sc.hadoopConfiguration.get("fs.s3.access.key")
res7: String = null <--- ugh... wtf?
scala> sc.hadoopConfiguration.get("fs.s3n.access.key")
res10: String = null <--- I understand this...
scala> sc.hadoopConfiguration.get("fs.s3a.access.key")
res8: String = CORRECT_ACCESS_KEY <--- But what is this???
// Successfull file read
scala> spark.read.parquet("s3a://bucket/path/to/folder")
ivysettings.xml file not found in HIVE_HOME or HIVE_CONF_DIR,/etc/hive/conf.dist/ivysettings.xml will be used
res9: org.apache.spark.sql.DataFrame = [whatev... ... 22 more fields]

Answer 1

There's some magic in spark-submit which picks up your AWS_ env vars and sets them for {s3, s3n, s3a} filesystens; that may be what's happening under the hood.
There is not any magic copying of the s3a settings from the s3a to s3n options in the Hadoop JARs, or anywhere else, so it may be some of the -site.xml files that is defining it.
Do use s3a:// if you have the Hadoop 2.7.x JARs on your CP anyway: better performance. V4 API auth (mandatory for some regions), ongoing maintenance.