Pytorch not detecting multiple GPUs - pytorch

Question

I have 10 GPUs available and 1 GPU (e.g. GPU#9) is in use by another torch process. I would like to run another process on any of the remaining GPUs (e.g. GPU#2, GPU#3, GPU#4) but I always get the error message:
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable
I tried several options and none of them worked:
OPTION 1: Selecting GPUs on python script
import os
os.environ[“CUDA_DEVICE_ORDER”] = ‘PCI_BUS_ID’
os.environ[“CUDA_VISIBLE_DEVICES”] = ‘2,3,4’
print(f’[INFO] Using GPU: {torch.cuda.current_device()}‘)
print(f’[INFO] Available GPUs: {torch.cuda.device_count()}')
for d in range(torch.cuda.device_count()):
print(torch.cuda.get_device_name(d))
It recognises the 3 GPUs and the process will now assign them as GPU#0, GPU#1 and GPU#3 within that process.
[INFO] Using GPU: 2,3,4
[INFO] Available GPUs: 10
GeForce GTX 1080 Ti
OPTION 2: Selecting GPU on command line:
CUDA_VISIBLE_DEVICES=2,3,4 python gdxray_cganTrainer.py
and I check on linux using
env | grep CUDA_VISIBLE_DEVICES
CUDA_VISIBLE_DEVICES=2,3,4
None of the above 2 options work as still get the CUDA-capable devices are busy or unavailable. It looks like it knows that the already running process on GPU#9 was assigned as GPU#0 and when I select GPU#2,3,4 the first GPU will be assigned as GPU#0 in the new process. But it shouldn't be as they are different processes
I am using torch 1.8.0, python 3.8.8, cudatoolkit 11.1.1
Any help?