How to monitor GPU memory usage when training a DNN? - pytorch

Question

I give a result example. I want to ask how to get the data like this graph.

Answer 1

You can use pytorch commands such as torch.cuda.memory_stats to get information about current GPU memory usage and then create a temporal graph based on these reports.

Answer 2

I think this is the best
torch.cuda.mem_get_info
it returns the global free and total GPU memory occupied for a given device using cudaMemGetInfo.

Answer 3

Another method is using nvidia-smi which looks like this
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66 Driver Version: 450.66 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 00000000:01:00.0 On | N/A |
| 0% 50C P8 12W / 215W | 1088MiB / 8113MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1091 G /usr/lib/xorg/Xorg 24MiB |
| 0 N/A N/A 1158 G /usr/bin/gnome-shell 48MiB |
+-----------------------------------------------------------------------------+
And use subprocess to get the string for example
import subprocess
import re
command = 'nvidia-smi'
while True:
p = subprocess.check_output(command)
ram_using = re.findall(r'\b\d+MiB+ /', str(p))[0][:-5]
ram_total = re.findall(r'/ \b\d+MiB', str(p))[0][3:-3]
ram_percent = int(ram_using) / int(ram_total)
Or just split the line with str(p).split('\n') and count the string length