spark gcp dataproc dependency error with guava - apache-spark

Question

Our project uses gradle and scala to build spark app, but I've added gcp kms library and now when this runs on dataproc it errors with missing guava method:
java.lang.noSuchMethodError: com.google.common.base.Preconditions.checkArgument
I followed the recommendations on the following guide to shade the google libraries:
https://cloud.google.com/blog/products/data-analytics/managing-java-dependencies-apache-spark-applications-cloud-dataproc
My shadowJar definition in gradle build:
shadowJar {
zip64 true
relocate 'com.google', 'shadow.com.google'
relocate 'com.google.protobuf', 'shadow.com.google.protobuf'
relocate 'google.cloud', 'shadow.google.cloud'
exclude 'META-INF/**'
exclude "LICENSE*"
mergeServiceFiles()
archiveFileName = "myjar"
}
When running java tf on compiled fat jar this shows the guava classes including checkArgument being relocated under shadow.
But it still errors when runnin dataproc spark submit and it still seems to choose the old versions from hadoop at runtime. Below is the stack trace, starting from my KmsSymettric class which is using gcp kms decrypt:
Exception in thread "main" java.lang.NoSuchMethodError: shadow.com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
at io.grpc.Metadata$Key.validateName(Metadata.java:629)
at io.grpc.Metadata$Key.<init>(Metadata.java:637)
at io.grpc.Metadata$Key.<init>(Metadata.java:567)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:742)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:737)
at io.grpc.Metadata$Key.of(Metadata.java:593)
at io.grpc.Metadata$Key.of(Metadata.java:589)
at shadow.com.google.api.gax.grpc.GrpcHeaderInterceptor.<init>(GrpcHeaderInterceptor.java:60)
at shadow.com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:221)
at shadow.com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:194)
at shadow.com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:186)
at shadow.com.google.api.gax.rpc.ClientContext.create(ClientContext.java:155)
at shadow.com.google.cloud.kms.v1.stub.GrpcKeyManagementServiceStub.create(GrpcKeyManagementServiceStub.java:370)
at shadow.com.google.cloud.kms.v1.stub.KeyManagementServiceStubSettings.createStub(KeyManagementServiceStubSettings.java:333)
at shadow.com.google.cloud.kms.v1.KeyManagementServiceClient.<init>(KeyManagementServiceClient.java:155)
at shadow.com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:136)
at shadow.com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:127)
at mycompany.my.class.path.KmsSymmetric.decrypt(KmsSymmetric.scala:31)
My dataproc submit is:
gcloud dataproc jobs submit spark \
--cluster=${CLUSTER_NAME} \
--project ${PROJECT_ID} \
--region=${REGION} \
--jars=gs://${APP_BUCKET}/${JAR} \
--class=${CLASS} \
--app args --arg1 val1 etc
I'm using dataproc image version 1.4
What am I missing?