Access Openstack Swift from Spark - SwiftAuthenticationFailedException - apache-spark

Question

I am attempting to access Openstack Swift from Spark 2.4 but I get an error.
org.apache.hadoop.fs.swift.exceptions.SwiftAuthenticationFailedException: Authenticate as tenant '78axxxxxxxxxxxxxxxxxxxxxxxxxxxx' PasswordCredentials{username='xxxxxxxxxxxx'}
sc.hadoopConfiguration.set(s"fs.swift.service.ovh.auth.url", "https://auth.cloud.ovh.net/v3/")
sc.hadoopConfiguration.set(s"fs.swift.service.ovh.tenant", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
sc.hadoopConfiguration.set(s"fs.swift.service.ovh.username", "xxxxxxxxxxxx")
sc.hadoopConfiguration.set(s"fs.swift.service.ovh.password", "xxxxxxxxxxxxxxxxxxxx")
sc.hadoopConfiguration.set(s"fs.swift.service.ovh.http.port", "8080")
sc.hadoopConfiguration.set(s"fs.swift.service.ovh.region", "BHS3")
sc.hadoopConfiguration.set(s"fs.swift.service.ovh.public", "false")
I believe that these credentials are correct as they came directly from the openstack rc file and I can use them fine when using python-swiftclient. I have also tried using the v2.0 endpoint without success.
Unfortunately I always get this very generic error message and it won't tell me which part is failing. Is there any way to debug this better?

Answer 1

I used the example below which I received from the OVH spark submit team.
An important note is to use the tenant name instead of the tenant id from the openstack.rc file.
val hadoopConf = spark.sparkContext.hadoopConfiguration
hadoopConf.set("fs.swift.impl","org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem")
hadoopConf.set("fs.swift.service.auth.endpoint.prefix","/AUTH_")
hadoopConf.set("fs.swift.service.abc.http.port","443")
hadoopConf.set("fs.swift.service.abc.auth.url","https://auth.cloud.ovh.net/v2.0/tokens")
hadoopConf.set("fs.swift.service.abc.tenant","<TENANT NAME> or <PROJECT NAME>")
hadoopConf.set("fs.swift.service.abc.region","<REGION NAME>")
hadoopConf.set("fs.swift.service.abc.useApikey","false")
hadoopConf.set("fs.swift.service.abc.username","<USER NAME>")
hadoopConf.set("fs.swift.service.abc.password","<PASSWORD>")
https://github.com/mojtabaimani/spark-wordcount-swift-scala/blob/master/src/main/scala/com/ovh/example/SparkScalaApp.scala