How to use local Docker images when submitting Spark jobs (2.3) natively to Kubernetes? - apache-spark

Question

I am trying to submit a Spark Job on Kubernetes natively using Apache Spark 2.3.
When I use a Docker image on Docker Hub (for Spark 2.2), it works:
bin/spark-submit \
--master k8s://http://localhost:8080 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar
However, when I try to build a local Docker image,
sudo docker build -t spark:2.3 -f kubernetes/dockerfiles/spark/Dockerfile .
and submit the job as:
bin/spark-submit \
--master k8s://http://localhost:8080 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=spark:2.3 \
local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar
I get the following error; that is "repository docker.io/spark not found: does not exist or no pull access, reason=ErrImagePull, additionalProperties={})"
status: [ContainerStatus(containerID=null, image=spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=rpc error: code = 2 desc = repository docker.io/spark not found: does not exist or no pull access, reason=ErrImagePull, additionalProperties={}), additionalProperties={}), additionalProperties={})]
2018-03-15 11:09:54 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state:
pod name: spark-pi-3a1a6e8ce615395fa7df81eac06d58ed-driver
namespace: default
labels: spark-app-selector -> spark-8d9fdaba274a4eb69e28e2a242fe86ca, spark-role -> driver
pod uid: 5271602b-2841-11e8-a78e-fa163ed09d5f
creation time: 2018-03-15T11:09:25Z
service account name: default
volumes: default-token-v4vhk
node name: mlaas-p4k3djw4nsca-minion-1
start time: 2018-03-15T11:09:25Z
container images: spark:2.3
phase: Pending
status: [ContainerStatus(containerID=null, image=spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=Back-off pulling image "spark:2.3", reason=ImagePullBackOff, additionalProperties={}), additionalProperties={}), additionalProperties={})]
Also, I tried to run a local Docker registry as described in:
https://docs.docker.com/registry/deploying/#run-a-local-registry
docker run -d -p 5000:5000 --restart=always --name registry registry:2
sudo docker tag spark:2.3 localhost:5000/spark:2.3
sudo docker push localhost:5000/spark:2.3
I can do this successfully:
docker pull localhost:5000/spark:2.3
However, when I submit the Spark job:
bin/spark-submit \
--master k8s://http://localhost:8080 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=localhost:5000/spark:2.3 \
local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar
I again got ErrImagePull:
status: [ContainerStatus(containerID=null, image=localhost:5000/spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=rpc error: code = 2 desc = Error while pulling image: Get http://localhost:5000/v1/repositories/spark/images: dial tcp [::1]:5000: getsockopt: connection refused, reason=ErrImagePull, additionalProperties={}), additionalProperties={}), additionalProperties={})]
Is there a way in Spark 2.3 to use local Docker images when submitting jobs natively to Kubernetes?
Thank you in advance.

Answer 1

I guess you using something like a minikube for set-up a local Kubernetes cluster and in most of cases it using a virtual machines to spawn a cluster.
So, when Kubernetes trying to pull image from localhost address, it connecting to virtual machine local address, not to your computer address. Moreover, your local registry bind only on localhost and not accessible from virtual machines.
The idea of a fix is to make your local docker registry accessible for your Kubernetes and to allow pull images from local insecure registry.
So, first of all, bind your docker registry on your PC to all interfaces:
docker run -d -p 0.0.0.0:5000:5000 --restart=always --name registry registry:2
Then, check your local IP address of the PC. It will be something like 172.X.X.X or 10.X.X.X. The way of the check is depends of your OS, so just google it if you don't know how to get it.
After, start your minikube with an additional option:
minikube start --insecure-registry="<your-local-ip-address>:5000", where a 'your-local-ip-address' is your local IP address.
Now you can try to run a spark job with a new address of a registry and K8s has be able to download your image:
spark.kubernetes.container.image=<your-local-ip-address>:5000/spark:2.3