Even for single-instance training, PyTorch DistributedDataParallel (DDP) is generally recommended over PyTorch DataParallel (DP) because DP's strategy is less performant and it uses more memory on the default device. (Per this PyTorch forums thread)
Hugging Face recommend to run distributed training via the python -m torch.distributed.launch launcher, because their Trainer API supports DDP but will fall back to DP if you don't. (Per this HF forums thread)
I recently ran in to this problem: scaling a HF training job from p3.8xlarge to p3.16xlarge increased memory consumption on (I think) one of the GPUs to the point where I had to significantly reduce batch size to avoid CUDA Out of Memory errors - basically losing all scaling advantage.
So the good news is for p3.16xl+ I can just enable SageMaker Distributed Data Parallel and the PyToch DLC will automatically launch via torch.distributed for me.
The bad news for use cases with smaller workloads or wanting to test before they scale up, is that SMDistributed doesn't support all multi-GPU instance types. No p3.8xl or g series, for example. I did try manually setting the sagemaker_distributed_dataparallel_enabled environment variable, but no joy.
So how else can we launch HF Trainer scripts with PyTorch DDP on SageMaker?
Great question, thanks for asking! PyTorch DDP runs data parallel workers in multiple processes, that must be launched and managed by developers. DDP should be seen as a managed allreduce, more than a managed data-parallelism library, since it requires you to launch and manage the workers and even assigning resources to workers. In order to launch the DDP processes in a SageMaker Training job you have many options:
If you do multi-GPU, single-machine, you can use torch.multiprocessing.spawn, as shown in this official PyTorch demo (that is broken by the way)
If you do multi-GPU, single-machine, you can also use the Ray Train library to launch those processes. I was able to use it in a Notebook, but not in the DLC yet (recent library that is a bit rough to learn and make work, see all my issues here). Ray Train should work on multi-node too.
If you do multi-GPU, any-machine, you can use torch.distributed.launch, wrapped in a launcher script in shell or Python. Example here https://gitlab.aws.dev/cruchant/a2d2-segmentation/-/blob/main/3_2D-Seg-Audi-A2D2-Distributed-Training-DDP.ipynb
You can also launch those processes with the SageMaker MPI integration instead of torch.distributed. Unfortunately, we didn't create documentation for this, so no one uses it nor pitches it. But it looks cool, because it allows to run copies of your script directly in the EC2 machines without the need to invoke an intermediary PyTorch launcher. Example here
So for now, my recommendation would be to go the route (3), which is the closest to what the PyTorch community does, so provides easier development and debugging path.
Notes:
PyTorch DDP evolves fast. In PT 1.10 torch.distributed is replaced by torchrun, and a torchX tool is being created to...simplify things!).
Not having to manage that mess is a reason why SageMaker Distributed Data Parallel is a great value prop: you only need to edit your script, and the SM service handles process creation. Unfortunately, as you point out, SMDP being limited to P3 and P4 training jobs seriously limits its use.
Below are important PT DDP concepts to understand to alter single-GPU code into multi-machine code
Unlike Apache Spark, which takes care of workload partitioning on your behalf, Pytorch distributed training requires the user to assign specific pieces of work to specific GPUs. In the following section, we assume that we train on GPU.
In PyTorch DDP, each GPU runs a customized copy of you training code. A copy of the training code running on one GPU is generally called a rank, a data parallel replica, a process, a worker, but other names may exist.
For PyTorch DDP to launch a training cluster on the MxN GPUs spread over your M machines, you must specify to PyTorch DDP the number of machines you have and the number of processes to launch per machine. This is respectively done by the parameters -nnodes and -nproc_per_node of the torch.distributed.launch utility. You must run the torch.distributed.lauch once on each node of the training cluster. You can achieve this parallel command with multiple tools, for example with MPI or SageMaker Training as mentioned above. In order to establish the necessary handshakes and form a cluster, you must also specify in the torch.distributed.launch command -node_rank, which must take a unique machine ID between 0 and N-1 on each of the machines, and -master_addr and -master_port, optional if you run a single-machine cluster, which must be the same across all machines.
In the init_process_group DDP initialization method running from within each data parallel replica script, you must specify the world size and replica ID, respectively with the world_size and rank parameters. Hence you must have a way to communicate to each script a unique ID, generally called the global rank. The global rank can help you personalize the work done by each GPU, for example saving a model just from one card, or running validation only in one card. In a cluster composed of 3 machines having 4 GPUs each, global ranks would range from 0 to 11. Within a machine, in order to assign DDP data parallel replicas to available GPUs, the script running in each replica must be assigned a GPU ID, unique within the machine it's running on. This is called the local rank and can be set as an argument by the PyTorch DDP torch.distributed.launch. In a cluster composed of 3 machines having 4 GPUs each, on each machine the DDP processes would have local ranks ranging from 0 to 3
I met this issue when I running my python file in linux.
I searched some answers in google like use the code below:
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
The system can be running but without ant out information. Actually I found the 2 measn ignore all information while 1 means give all information like error or normal output.
because I need to use GPU so my original code is:
os.environ["CUDA_VISIBLE_DEVICES"]="0"
But if I keep the code my output would be an error information like the title.
How can I do? and of course I need use the GPU and the codes can be running in colab which indicated that my code has no problem.
some guys even said uninstall tensorboard...that must a wrong way.
Or should I download tensorflow gpu not tensorflow in m,y virtual enviroment? when I USE THE tensorflow gpu version, the error is core dumped.
If when forcing the os.environ["CUDA_VISIBLE_DEVICES"]="0" doesn't work, then this means that your tensorflow gpu installation did not succeed. You must ensure you have the right combination of TensorFlow + CUDA + CUDNN. That is why you get the error, because due to improper versions/installation TF falls back on CPU.
From the docs I can see that developer can utilize all cores available in the computer to speed up the recognition process using --cpus n flag (https://pypi.org/project/face-recognition/)
which is to use when running face_recognition --cpus 4 ./pictures_of_people_i_know/ ./unknown_pictures/ in a command line.
Is there an equivalent of this flag to be used within a script or do I need to use Python's multiprocessing module ? Docs don't mention this.
I’ve got a simple model consisting only of convolutions (even no activation between) and I wanted to benchmark it in Caffe2 on ARM Android device using multiple cores.
When I run
./speed_benchmark --init_net=model_for_inference-simplified-init-net.pb --net=model_for_inference-simplified-predict-net.pb --iter=1
it runs on single core.
Speed benchmark was built using:
scripts/build_android.sh -DANDROID_ABI=arm64-v8a -DANDROID_TOOLCHAIN=clang -DBUILD_BINARY=ON
On X86 it has been build via
mkdir build
cd build
cmake .. -DBUILD_BINARY=ON
and setting OMP_NUM_THREADS=8 helps but not on ARM
Do I need to change the building command for arm, set some environmental variables, some binary arguments or something else?
I didn't know that I need to set the engine information in the model like described in https://caffe2.ai/docs/mobile-integration.html
After updating prediction net by:
for op in predict_net.op:
if op.type == 'Conv':
op.engine = 'NNPACK'
more cores started to be used
I have a machine with 2 GPUs.
Quite often, one is used in production (i.e doing predictions with the already trained model), while the other is used for training and experimenting new models.
While I was using theano, I had no problem running my scripts on only one GPU by specifying a flag as follow
THEANO_FLAGS="device=cuda0" training_script.py
THEANO_FLAGS="device=cuda1" prediction_script.py
Is there a simple way to do the same in Keras with a Tensorflow backend ? Default behavior seem to map all the memory of all the GPUs for one session
(Please note that I don't really care if each script maps a whole GPU separately, even if they could work using less memory)
You can easily choose one gpu. Just fill 0 or 1 on CUDA_VISIBLE_DEVICES
import os
os.environ["CUDA_VISIBLE_DEVICES"]="1"
Furthermore if you want to spesify a portion of gpu for the selected gpu above, add:
from keras import backend as K
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4 #what portion of gpu to use
session = tf.Session(config=config)
K.set_session(session)