Caffe2 doesn't seem to see cuDNN? - linux

Question

I am trying to install SpConv (spatially sparse convolution library), however when running python3 setup.py bdist_wheel, I get an error which seems to be related to Caffe2 not being able to see cuDNN, as I infer from this message:
CMake Error at /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
libraries. Please set the proper cuDNN prefixes and / or install cuDNN.
I have tried reinstalling cuDNN but no luck. Can I manually point Caffe2 to cuDNN? This is because usually cuda is located in /usr/local/cuda/, however I have it in /usr/lib/cuda.
Full output below:
running bdist_wheel
running build
running build_py
running build_ext
Release
|||||CMAKE ARGS||||| ['-DCMAKE_PREFIX_PATH=/home/ivan/.local/lib/python3.6/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.6', '-DSPCONV_BuildTests=OFF', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/mnt/home/ivan/second.pytorch/second/spconv/build/lib.linux-x86_64-3.6/spconv', '-DCMAKE_BUILD_TYPE=Release']
-- Caffe2: CUDA detected: 9.1
-- Caffe2: CUDA nvcc is: /usr/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr
-- Caffe2: Header version is: 9.1
-- Could NOT find CUDNN (missing: CUDNN_INCLUDE_DIR)
CMake Warning at /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:131 (message):
Caffe2: Cannot find cuDNN library. Turning the option off
Call Stack (most recent call first):
/home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
/home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
CMakeLists.txt:20 (find_package)
-- Autodetected CUDA architecture(s): 6.1
-- Added CUDA NVCC flags for: -gencode;arch=compute_61,code=sm_61
CMake Error at /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
libraries. Please set the proper cuDNN prefixes and / or install cuDNN.
Call Stack (most recent call first):
/home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
CMakeLists.txt:20 (find_package)
-- Configuring incomplete, errors occurred!
See also "/mnt/home/ivan/second.pytorch/second/spconv/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log".
See also "/mnt/home/ivan/second.pytorch/second/spconv/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
File "setup.py", line 99, in <module>
zip_safe=False,
File "/home/ivan/.local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 204, in run
self.run_command('build')
File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "setup.py", line 40, in run
self.build_extension(ext)
File "setup.py", line 82, in build_extension
subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/mnt/home/ivan/second.pytorch/second/spconv', '-DCMAKE_PREFIX_PATH=/home/ivan/.local/lib/python3.6/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.6', '-DSPCONV_BuildTests=OFF', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/mnt/home/ivan/second.pytorch/second/spconv/build/lib.linux-x86_64-3.6/spconv', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
System: Ubuntu 18.04, CUDA 9.1, cuDNN 7.6.3

Answer 1

Found the problem. I needed to set CUDNN_INCLUDE_DIR to /usr/lib/cuda/include (i.e. where the cudnn.h file is located).
Lesson learnt: take time to understand the error, missing: CUDNN_INCLUDE_DIR was key.

Answer 2

Another way to fix this:
Acquire cuDNN from here. You may have to register to do this, but the registration is free.
Unpack the cuDNN archive file.
Move the cuDNN files as follows:
sudo cp cuda/include/cudnn*.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*
Alternately, place a symoblic link at /usr/local/ pointing to the cuda directory produced by unpacking the cuDNN archive.