RangerAdminRESTClient: Error getting Roles - apache-spark

Question

I'm trying to connect my apache spark to apache Ranger using Spark AuthZ Plugin.
Kerberos is enabled for both spark and apache ranger .
i'm getting this error "WARN RangerAdminRESTClient: Error getting Roles. secureMode=true, user=hive/host_fqdn#MYCOMPANY.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=hive_policy"
apache ranger conf files :
ll /etc/ranger/admin/conf/ :
conf -> /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/conf
core-site.xml -> /etc/hadoop/conf/core-site.xml
java_home.sh
logback.xml
ranger-admin-default-site.xml
ranger-admin-env-hadoopconfdir.sh
ranger-admin-env-logback-conf-file.sh
ranger-admin-env-logdir.sh
ranger-admin-env-piddir.sh
ranger-admin-site.xml
security-applicationContext.xml
ranger-admin-site.xml content :
<configuration>
<property>
<name>ranger.jpa.jdbc.driver</name>
<value>org.postgresql.Driver</value>
<description />
</property>
<property>
<name>ranger.jpa.jdbc.url</name>
<value>jdbc:postgresql://ip#:5432/ranger</value>
<description />
</property>
<property>
<name>ranger.jpa.jdbc.user</name>
<value>rangeradmin</value>
<description />
</property>
<property>
<name>ranger.jpa.jdbc.password</name>
<value>_</value>
<description />
</property>
<property>
<name>ranger.externalurl</name>
<value>http://machine_fqdn:6080</value>
<description />
</property>
<property>
<name>ranger.scheduler.enabled</name>
<value>true</value>
<description />
</property>
<property>
<name>ranger.audit.elasticsearch.urls</name>
<value>127.0.0.1</value>
<description />
</property>
<property>
<name>ranger.audit.elasticsearch.port</name>
<value>9200</value>
<description />
</property>
<property>
<name>ranger.audit.elasticsearch.user</name>
<value />
<description />
</property>
<property>
<name>ranger.audit.elasticsearch.password</name>
<value />
<description />
</property>
<property>
<name>ranger.audit.elasticsearch.index</name>
<value />
<description />
</property>
<property>
<name>ranger.audit.elasticsearch.bootstrap.enabled</name>
<value>true</value>
</property>
<property>
<name>ranger.audit.amazon_cloudwatch.region</name>
<value>us-east-2</value>
</property>
<property>
<name>ranger.audit.amazon_cloudwatch.log_group</name>
<value>ranger_audits</value>
</property>
<property>
<name>ranger.audit.amazon_cloudwatch.log_stream_prefix</name>
<value />
</property>
<property>
<name>ranger.audit.solr.urls</name>
<value>http://##solr_host##:6083/solr/ranger_audits</value>
<description />
</property>
<property>
<name>ranger.audit.source.type</name>
<value>db</value>
<description />
</property>
<property>
<name>ranger.service.http.enabled</name>
<value>true</value>
<description />
</property>
<property>
<name>ranger.authentication.method</name>
<value>NONE</value>
<description />
</property>
<property>
<name>ranger.ldap.url</name>
<value>ldap://</value>
<description />
</property>
<property>
<name>ranger.ldap.user.dnpattern</name>
<value>uid={0},ou=users,dc=xasecure,dc=net</value>
<description />
</property>
<property>
<name>ranger.ldap.group.searchbase</name>
<value>ou=groups,dc=xasecure,dc=net</value>
<description />
</property>
<property>
<name>ranger.ldap.group.searchfilter</name>
<value>(member=uid={0},ou=users,dc=xasecure,dc=net)</value>
<description />
</property>
<property>
<name>ranger.ldap.group.roleattribute</name>
<value>cn</value>
<description />
</property>
<property>
<name>ranger.ldap.base.dn</name>
<value />
<description>LDAP base dn or search base</description>
</property>
<property>
<name>ranger.ldap.bind.dn</name>
<value />
<description>LDAP bind dn or manager dn</description>
</property>
<property>
<name>ranger.ldap.bind.password</name>
<value />
<description>LDAP bind password</description>
</property>
<property>
<name>ranger.ldap.default.role</name>
<value>ROLE_USER</value>
</property>
<property>
<name>ranger.ldap.referral</name>
<value />
<description>follow or ignore</description>
</property>
<property>
<name>ranger.ldap.ad.domain</name>
<value>example.com</value>
<description />
</property>
<property>
<name>ranger.ldap.ad.url</name>
<value />
<description>ldap://</description>
</property>
<property>
<name>ranger.ldap.ad.base.dn</name>
<value>dc=example,dc=com</value>
<description>AD base dn or search base</description>
</property>
<property>
<name>ranger.ldap.ad.bind.dn</name>
<value>cn=administrator,ou=users,dc=example,dc=com</value>
<description>AD bind dn or manager dn</description>
</property>
<property>
<name>ranger.ldap.ad.bind.password</name>
<value />
<description>AD bind password</description>
</property>
<property>
<name>ranger.ldap.ad.referral</name>
<value />
<description>follow or ignore</description>
</property>
<property>
<name>ranger.service.https.attrib.ssl.enabled</name>
<value>false</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.keyalias</name>
<value>myKey</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.pass</name>
<value>_</value>
</property>
<property>
<name>ranger.service.host</name>
<value>machine_fqdn</value>
</property>
<property>
<name>ranger.service.http.port</name>
<value>6080</value>
</property>
<property>
<name>ranger.service.https.port</name>
<value>6182</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.file</name>
<value>/etc/ranger/admin/keys/server.jks</value>
</property>
<property>
<name>ranger.solr.audit.user</name>
<value />
<description />
</property>
<property>
<name>ranger.solr.audit.user.password</name>
<value />
<description />
</property>
<property>
<name>ranger.audit.solr.zookeepers</name>
<value />
<description />
</property>
<property>
<name>ranger.ldap.user.searchfilter</name>
<value>(uid={0})</value>
<description />
</property>
<property>
<name>ranger.ldap.ad.user.searchfilter</name>
<value>(sAMAccountName={0})</value>
<description />
</property>
<property>
<name>ranger.sso.providerurl</name>
<value>https://127.0.0.1:8443/gateway/knoxsso/api/v1/websso</value>
</property>
<property>
<name>ranger.sso.publicKey</name>
<value />
</property>
<property>
<name>ranger.sso.enabled</name>
<value>false</value>
</property>
<property>
<name>ranger.sso.browser.useragent</name>
<value>Mozilla,chrome</value>
</property>
<property>
<name>ranger.admin.kerberos.token.valid.seconds</name>
<value>30</value>
</property>
<property>
<name>ranger.admin.kerberos.cookie.domain</name>
<value>machine_fqdn</value>
</property>
<property>
<name>ranger.admin.kerberos.cookie.path</name>
<value>/</value>
</property>
<property>
<name>ranger.admin.kerberos.principal</name>
<value>hive/_HOST#MYCOMPANY.COM</value>
</property>
<property>
<name>ranger.admin.kerberos.keytab</name>
<value>/etc/security/hdfs.keytab</value>
</property>
<property>
<name>ranger.spnego.kerberos.principal</name>
<value>HTTP/_HOST#MYCOMPANY.COM</value>
</property>
<property>
<name>ranger.spnego.kerberos.keytab</name>
<value>/etc/security/hdfs.keytab</value>
</property>
<property>
<name>ranger.lookup.kerberos.principal</name>
<value>hive/_HOST#MYCOMPANY.COM</value>
</property>
<property>
<name>ranger.lookup.kerberos.keytab</name>
<value>/etc/security/hdfs.keytab</value>
</property>
<property>
<name>ranger.kerberos.principal</name>
<value>hive/_HOST#MYCOMPANY.COM</value>
</property>
<property>
<name>ranger.kerberos.keytab</name>
<value>/etc/security/hdfs.keytab</value>
</property>
<property>
<name>ranger.supportedcomponents</name>
<value />
</property>
<property>
<name>ranger.downloadpolicy.session.log.enabled</name>
<value>false</value>
</property>
<property>
<name>ranger.kms.service.user.hdfs</name>
<value>hdfs</value>
</property>
<property>
<name>ranger.kms.service.user.hive</name>
<value>hive</value>
</property>
<property>
<name>ranger.kms.service.user.hbase</name>
<value>hbase</value>
</property>
<property>
<name>ranger.kms.service.user.om</name>
<value>om</value>
</property>
<property>
<name>ranger.audit.hive.query.visibility</name>
<value>true</value>
<description />
</property>
<property>
<name>ranger.service.https.attrib.keystore.credential.alias</name>
<value>keyStoreCredentialAlias</value>
</property>
<property>
<name>ranger.tomcat.ciphers</name>
<value />
</property>
<property>
<name>ranger.audit.solr.collection.name</name>
<value>ranger_audits</value>
</property>
<property>
<name>ranger.audit.solr.config.name</name>
<value>ranger_audits</value>
</property>
<property>
<name>ranger.audit.solr.configset.location</name>
<value />
</property>
<property>
<name>ranger.audit.solr.no.shards</name>
<value>1</value>
</property>
<property>
<name>ranger.audit.solr.max.shards.per.node</name>
<value>1</value>
</property>
<property>
<name>ranger.audit.solr.no.replica</name>
<value>1</value>
</property>
<property>
<name>ranger.audit.solr.acl.user.list.sasl</name>
<value>solr,infra-solr</value>
</property>
<property>
<name>ranger.audit.solr.bootstrap.enabled</name>
<value />
</property>
<property>
<name>ranger.audit.solr.max.retry</name>
<value />
<description>Maximum no. of retry to setup solr</description>
</property>
<property>
<name>ranger.admin.cookie.name</name>
<value>RANGERADMINSESSIONID</value>
</property>
</configuration>
core-site.xml content :
<configuration>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
</configuration>
spark conf files : ll /opt/spark/conf
core-site.xml : to enable kerberos (same as above file)
hive-site.xml
ranger-admin-default-site.xml (same as above file)
ranger-admin-site.xml (same as above file)
ranger-spark-audit.xml
ranger-spark-security.xml
spark-defaults.conf
hive-site.xml content:
<configuration>
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://ip#:8020/hive/warehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.postgresql.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:postgresql://ip#:5432/hivemetastoredb</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>kube</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>kubeadmin</value>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.execution.engine</name>
<value>mr</value>
</property>
<property>
<name>hive.metastore.port</name>
<value>9083</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://hive-metastore-service.my-hdfs.svc.cluster.local:9083</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.input.dir.recursive</name>
<value>true</value>
</property>
<property>
<name>hive.server2.authentication</name>
<value>KERBEROS</value>
<description>authenticationtype</description>
</property>
<property>
<name>hive.server2.authentication.kerberos.principal</name>
<value>hive/_HOST#MYCOMPANY.COM</value>
<description>HiveServer2 principal. If _HOST is used as the FQDN portion, it will be replaced with the actual hostname of the running instance.</description>
</property>
<property>
<name>hive.server2.authentication.kerberos.keytab</name>
<value>/etc/security/hdfs.keytab</value>
<description>Keytab file for HiveServer2 principal</description>
</property>
</configuration>
ranger-spark-security.xml content :
<configuration>
<property>
<name>ranger.plugin.spark.policy.rest.url</name>
<value>http://ranger_machine_ip#:6080</value>
</property>
<property>
<name>ranger.plugin.spark.service.name</name>
<value>hive_policy</value>
</property>
<property>
<name>ranger.plugin.spark.policy.cache.dir</name>
<value>/</value>
</property>
<property>
<name>ranger.plugin.spark.policy.pollIntervalMs</name>
<value>5000</value>
</property>
<property>
<name>ranger.plugin.spark.policy.source.impl</name>
<value>org.apache.ranger.admin.client.RangerAdminRESTClient</value>
</property>
<property>
<name>ranger.plugin.spark.enable.implicit.userstore.enricher</name>
<value>true</value>
<description>Enable UserStoreEnricher for fetching user and group attributes if using macros or scripts in row-filters since Ranger 2.3</description>
</property>
<property>
<name>ranger.plugin.hive.policy.cache.dir</name>
<value>/</value>
<description>As Authz plugin reuses hive service def, a policy cache path is required for caching UserStore and Tags for "hive" service def, while "ranger.plugin.spark.policy.cache.dir config" is the path for caching policies in service. </description>
</property>
</configuration>
spark-defaults.conf content :
spark.kubernetes.driver.master k8s://master_ip#:6443
spark.kubernetes.authenticate.serviceAccountName spark
spark.kubernetes.namespace my-hdfs
spark.executor.memory 1g
spark.driver.memory 2g
spark.kubernetes.container.image spark-3.2.2
spark.storage.memoryFraction 0
spark.executor.cores 1
spark.executor.instances 3
spark.sql.extensions org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension
spark.kerberos.keytab=/etc/security/hdfs.keytab
spark.kerberos.principal=hive/host_fqdn#MYCOMPANY.COM
spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf
Any help will be appreciated; THX