spark elasticsearch read with basic auth giving 403 error - apache-spark

Question

I get this below error when I am trying to read from my ES cluster
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [DELETE] on [_search/scroll] failed; server[<HOST>:<PORT>] returned [403|Forbidden:]
at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:505)
at org.elasticsearch.hadoop.rest.RestClient.executeNotFoundAllowed(RestClient.java:476)
at org.elasticsearch.hadoop.rest.RestClient.deleteScroll(RestClient.java:541)
at org.elasticsearch.hadoop.rest.ScrollQuery.close(ScrollQuery.java:77)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.close(AbstractEsRDDIterator.scala:81)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.closeIfNeeded(AbstractEsRDDIterator.scala:74)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator$$anonfun$1.apply$mcV$sp(AbstractEsRDDIterator.scala:54)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator$$anonfun$1.apply(AbstractEsRDDIterator.scala:54)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator$$anonfun$1.apply(AbstractEsRDDIterator.scala:54)
Code used to read
// spark conf.
SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("ReadORESData").setMaster("local[*]");
// elasticsearch specific configuration.
sparkConf.set("es.nodes", "<HOST>")
.set("es.port", "<PORT>")
.set("es.net.ssl", "true")
.set("es.index.read.missing.as.empty", "true")
.set("es.net.http.auth.user", "<USERNAME>")
.set("es.net.http.auth.pass", "<PASSWORD>")
.set("es.nodes.wan.only", "true")
.set("es.nodes.discovery","false")
.set("es.input.use.sliced.partitions","false")
.set("es.resource", "<INDEX_NAME>")
.set("es.scroll.size","500");
JavaSparkContext jsc = new JavaSparkContext(sparkConf);
JavaPairRDD<String, Map<String, Object>> rdd = JavaEsSpark.esRDD(jsc);
for ( Map<String, Object> item : rdd.values().collect()) {
System.out.println(item);
}
jsc.stop();
curl is working fine from my machine.
curl -XGET 'https://<HOST>:<PORT>/<INDEX>/_search' --user <USERNAME>:<PASSWORD>
Any index I try I am seeing that same error. If invalid index it is correctly saying index not found. I am connecting to ES 5.6 cluster using these below dependencies.
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.5</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_2.11</artifactId>
<version>5.6.16</version>
</dependency>

Answer 1

Noticed that the account I was using doesn't have r/w permissions. Used a different account and it fixed that 403 error.