hadoop aws versions compatibility - apache-spark

Question

Is there any reference as to what sets of versions are compatible between aws java sdk, hadoop, hadoop-aws bundle, hive, spark?
For example, I know Spark is not compatible with hive versions above Hive 2.1.1

Answer 1

You cannot drop in a later version of the AWS SDK from what which hadoop-aws was built with and expect the s3a connector to work. Ever. That is now written down quite clearly in the S3A troubleshooting docs
Whatever problem you have, changing the AWS SDK version will not fix things, only change the stack traces you see.
This may seem frustrating, given the rate at which the AWS team push out a new SDK, but you have to understand that (a) the API often changes incompatibly between versions (as you have seen), and (b) every release introduces/moves bugs which end up causing problems.
Here is the 3.x timeline of things which broke on updates of the AWS SDK.
Move 1.11.86 and some tests hang under load.
Fix: move to 1.11.134 leading to logs are full of AWS telling us off for deliberatly calling abort() on a read.
Fix: move to 1.11.199 leading to logs full of stack traces.
Fix: move to 1.11.271 and shaded JAR pulls in netty unshaded.
Every upgrade of the AWS SDK JAR causes a problem, somewhere. Sometimes an edit to the code and recompile, most commonly: logs filling up with false-alarm messages, dependency problems, threading quirks, etc. Things which can take time to surface.
what you see when you get a hadoop release is not just an aws-sdk JAR which it was compiled against, you get a hadoop-aws JAR which contains the workarounds and fixes for whatever problems that release has introduced and which were identified in the minimum of 4 weeks of testing before the hadoop release ships.
Which is why, no, you shouldn't be changing JARs unless you plan to do a complete end-to-end retest of the s3a client code, including load tests. You are encouraged to do that, the hadoop project always welcomes more testing of our pre-release code, with the Hadoop 3.1 binaries ready to play with. But trying to do it yourself by changing JARs? Sadly, an isolated exercise in pain.

Answer 2

In Hadoop documentation, it is stated that by adding hadoop-aws JAR to the build dependencies; it will pull in a compatible aws-sdk JAR.
So, I created a dummy Maven project with these dependencies to download the compatible versions
<properties>
<!-- Your exact Hadoop version here-->
<hadoop.version>3.3.1</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
Then, I checked my dependencies versions, used it in my project and it worked.