how to secure / disable the spark web ui - .htaccess

Question

I'm trying to password protect the spark web ui of my spark cluster. I've looked at the security doc. Usually the spark doc has many examples on how to do things, but for some reason, none is provided in this case. I don't feel comfortable enough for creating my own javax servlet filter, nor properly connecting it to whatever it is supposed to be connected to.
So I've tried protecting it with an nginx htaccess setup - this would be way enough for my purpose. unfortunately, when I run the cluster it avoids the 8080 port and switches to 8081 - saying that 8080 is not accessible.
Has anyone tried to password protect a spark web ui?

Answer 1

Disclaimer: It is an extremely naive approach and you shouldn't depend on it in a production environment. Moreover I assume you don't use this instance of Nginx and you have access to standard ports (80|443).
Configure Spark to use a port of your choice. You can use SPARK_MASTER_WEBUI_PORT variable. Below I assume it is 8080.
Generate self-signed certificates for your server. You can find multiple good resources how to do it so just to make this answer complete lets use example from Linode guide:
openssl req -new -x509 -sha256 -days 365 -nodes -out /path/to/nginx.pem -keyout /path/to/nginx.key
Make sure that key has limited access rights
chmod 400 /path/to/nginx.key
Generate htpasswd file
htpasswd -b -c /path/to/passwdfile username password
Remove default configuration from nginx/sites-enabled
Create a simple reverse proxy configuration and add it to ``nginx/sites-enabled`
server {
# Adjust port number if cannot use ports below 1024
listen 443 ssl;
ssl_certificate /path/to/nginx.pem;
ssl_certificate_key /path/to/nginx.key;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://127.0.0.1:8080;
auth_basic "closed site";
auth_basic_user_file /path/to/passwdfile;
}
}
Configure your system to reject remote connections to web UI port.
To make it work web UI has to be accessible from localhost so everyone who has access to your master can reach web UI directly.