the tensorflow docker gpu image doesn't detect my GPU - linux

Question

Running the latest docker with:
docker run -it -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyter jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'
code:
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
gives me:
2020-07-27 19:44:03.826149: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-07-27 19:44:03.826179: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (-1)
2020-07-27 19:44:03.826201: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
I'm on Pop_OS 20.04, have tried installing the CUDA drivers from the Pop repository as well as from NVidia. No dice. Any help appreciated.
Running
docker run --gpus all nvidia/cuda:10.0-base nvidia-smi
gives me:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.05 Driver Version: 450.51.05 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2080 On | 00000000:09:00.0 On | N/A |
| 0% 52C P5 15W / 225W | 513MiB / 7959MiB | 17% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+

Answer 1

As per the docs here and here, you have to add a "gpus" argument when creating a the docker container to have gpu support.
So you should start your container something like this. The "--gpus all" makes all the gpus available on the host to be visible to the container.
docker run -it --gpus all -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyter jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'
Also you can try running nvidia-smi on the tensorflow image to quickly check if gpu is accessible on the container.
docker run -it --rm --gpus all tensorflow/tensorflow:latest-gpu-jupyter nvidia-smi
Would return this in my case.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:07:00.0 On | N/A |
| 0% 45C P8 8W / 166W | 387MiB / 8116MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
As you can see, I'm running an older nvidia driver(440.100), so I cannot confirm that this would solve your problem. I'm also on Pop_OS 20.04 and didn't install anything other than docker along with dependencies and nvidia-container-toolkit.
Also I would highly suggest avoiding the latest tag when creating containers as it might cause you to unknowingly upgrade to a newer image. Go with version numbered images.
For example tensorflow/tensorflow:2.3.0-gpu-jupyter.