Is training in tensorflow multithreaded? - multithreading

Suppose I'm trying to train the following network for the cifar-10
https://www.tensorflow.org/tutorials/images/deep_cnn
I would like to know if the basic operations involved in stochastic gradient descent ( or some optimization technique) like computation of gradient etc multithreaded?
More precisely, if I run the above code on a single core machine and on many core machine like Intel Xeon Phi, will it run faster on the many-core machine? ( One can assume one core in both the machines are similar) If yes, what is the exact cause of the speed-up or which computations run faster on the many-core machine?

There are 3 kinds of parallelism in tensorflow.
input pipeline: input, preprocessing and network are run in different thread. This can be achieved in tf.QueueRunner or tf.data.Dataset in newer version of tensorflow.
inter op parallelism: if one node is independent from another, then they can be executed at the same time. So this depends on the structure of your network.
intra op parallelism: one node can use multi-thread, for example, ops implemented with Eigen can use multi-thread in one node.
BTW, inter and intra op parallelism threads appear in Session config, most cpu ops can benefit from it. However, GPU for training is highly recommended because of the high speed.

Related

Running two different independent PyTorch programs on a single GPU

I have a single NVIDIA GPU which has a memory of 16GB. I have to run two different (and independent; meaning, two different problems: one is a vision type task, another is NLP task) Python programs. The codes are written using PyTorch and both the codes can use GPU.
I have tested that program 1 takes roughly 5GB of GPU memory, and the rest is free. If I run the two programs, will it hamper the model performance or will it cause any process conflicts?
Linked question; but it does not necessarily mean PyTorch codes
I do not know the details of how this works, but I can tell from experience that both programs will run well (as long as they do not need more than 16GB of RAM when combined), and execution times should stay roughly the same.
However, computer vision usually requires a lot of IO (mostly reading images), if the other task needs to read files too, this part may become slower than when running both programs individually.
It should work fine.
In one of my projects, I faced the problem of lack of GPU memory while working with multiple models. After loading them, my models used to take up most of the GPU memory. And during model inference, very less memory used to remain for the data. As we know, if your models are loaded on GPU then you also need to load your data on your GPU. So when you do batch inference (for eg giving 16 images at a time to the model) the complete batch is loaded on the GPU. This again takes more GPU memory. Your program crashes if it does not get enough GPU memory.
If you think GPU memory is not the issue in your case then everything should work fine. You also do not need to worry about conflicts because both processes will allocate their own GPU memory and will work independently. There would be no performance issues.

AWS, Cuda, Tensorflow

When I'm running my Python code on the most powerfull AWS GPU instances (with 1 or 8 x Tesla v100 16mb aka. P3.x2large or P3.16xlarge) they are both only 2-3 times faster than my DELL XPS Geforce 1050-Ti laptop?
I'm using Windows, Keras, Cuda 9, Tensorflow 1.12 and the newest Nvidia drivers.
When I check the GPU load via GZU the GPU max. run at 43% load for a very short period - each time. The controller runs at max. 100%...
The dataset I use is matrices in JSON format and the files are located on a Nitro drive at 10TB with MAX 64.000 IOPS. No matter if the folder contains 10TB, 1TB or 100mb...the training is still very very slow per iteration?
All advises are more than welcome!
UPDATE 1:
From the Tensorflow docs:
"To start an input pipeline, you must define a source. For example, to construct a Dataset from some tensors in memory, you can use tf.data.Dataset.from_tensors() or tf.data.Dataset.from_tensor_slices(). Alternatively, if your input data are on disk in the recommended TFRecord format, you can construct a tf.data.TFRecordDataset."
Before I had matrices stored in JSON format (Made by Node). My TF runs in Python.
I will now only save the coordinates in Node and save it in JSON format.
The question is now: In Python what is the best solution to load data? Can TF use the coordinates only or do I have to make the coordinates back to matrices again or what?
The performance of any machine learning model depends on many things. Including but not limited to: How much pre-processing you do, how much data you copy from CPU to GPU, Op bottlenecks, and many more. Check out the tensorflow performance guide as a first step. There are also a few videos from the tensorflow dev summit 2018 that talk about performance. How to properly use tf.data, and how to debug performance are two that I recommend.
The only thing I can say for sure is that JSON is a bad format for this purpose. You should switch to tfrecord format, which uses protobuf (better than JSON).
Unfortunately performance and optimisation of any system takes a lot of effort and time, and can be a rabbit hole that just keeps going down.
First off, you should be having a really good reason to go for an increased computational overhead with Windows-based AMI.
If your CPU is at ~100%, while GPU is <100%, then your CPU is likely the bottleneck. If you are on cloud, consider moving to instances with larger CPU-count (CPU is cheap, GPU is scarce). If you can't increase CPU count, moving some parts of your graph to GPU is an option. However, tf.data-based input pipeline is run entirely on CPU (but highly scalable due to C++ implementation). Prefetching to GPUs might also help here, but the cost of spawning another background thread to populate the buffer for downstream might damp this effect. Another option is to do some or all pre-processing steps offline (i.e. prior to training).
A word of caution on using Keras as the input pipeline. Keras relies on Python´s multithreading (and optionally multiprocessing) libraries, which may both lack performance (when doing heavy I/O or augmentations on-the-fly) and scalability (when running on multiple CPUs) compared to GIL-free implementations. Consider performing preprocessing offline, pre-loading input data, or using alternative input pipelines (as the aforementioned TF native tf.data, or 3rd party ones, like Tensorpack).

Tensorflow. How to distribute ops between GPUs

I am running a very large Tensorflow model on google cloud ml-engine.
When using the scale tier basic_gpu (with batch_size=1) I get errors like:
Resource exhausted: OOM when allocating tensor with shape[1,155,240,240,16]
because the model is too large to fit in one GPU.
Using the tier comple_model_m_gpu which provides 4 GPUs I can spread the operations between the 4 GPUs.
However, I remember reading that communication between GPUs is slow and can create a bottleneck in training. Is this true?
If so, is there a recommended way of spreading operations between the GPUs that prevents this problem?
I recommend the following guide:
Optimizing for GPU
From the guide:
The best approach to handling variable updates depends on the model,
hardware, and even how the hardware has been configured.
A few suggestions based on the guide:
Try using P100s which have 16 GB of RAM (compared to 12 on the K80s). They are also significantly faster, although they also cost more
Place the variables on CPU: tf.train.replica_device_setter(worker_device=worker, ps_device='/cpu:0', ps_tasks=1)
Using Tesla P100 GPUs instead of Tesla K80 GPUs fixes this issue because P100s have something called Page Migration Engine.
Page Migration Engine frees developers to focus more on tuning for
computing performance and less on managing data movement. Applications
can now scale beyond the GPU's physical memory size to virtually
limitless amount of memory.

Pre-processing on multiple CPUs in parallel and feeding output to tensorflow which trains on Multi-GPUs

I am trying to use Tensorflow for my work on a classification problem.
Before feeding the input in the form of images, I want to do some pre-processing on the images. I would like to carry out this pre-processing on multiple CPU cores in parallel and feed them to the TensorFlow graph which I want to run in a Multi-GPU setting (I have 2 TitanX GPUs).
The reasoning I want this setup is, so that while the GPU is performing the training, the CPUs keep on doing their job of pre-processing and hence after each iteration, the GPU does not remain idle. I have been looking into the TensorFlow API for this, but could not locate something which specifically addresses such a scenario.
So, multiple CPU cores should keep on pre-processing a list of files and fill in a queue from which TensorFlow extracts its batch of data. Whenever this queue is full, CPU cores should wait and again start processing when the queue (or part of it) is vacated due to feeding of examples to TensorFlow graph.
I have two questions specifically :
How may I achieve this setup ?
Is it a good idea to have this setup ?
A clear example would be a great help.

sklearn and Tensorflow with dual CPU machine

I am thinking about building a dual-CPU machine for machine learning. I already have a fast GPU in my current rig but I am limited to 32GB of DDR3, I have an i7-4790k and I am planning to upgrade to dual E5 2683 v3's.
I need CPU computing power for sklearn and grid search. Does Sklearn work on 2 cpu's the same it does on 1? Will it use all the cores on both CPU's when n_jobs=-1?
Will tensorflow only work on the one CPU when training on my GPU? If I just copy and pasted the MNIST for experts tutorial on the TF website would it use both CPU's and my GPU without specifying the devices?
I choose not to put this on the superuser forum because it is more about the software than the hardware.
From what I read even if you add an instruction like
with tf.Session() as sess:
with tf.device("/cpu:0"):
...
It treats it as a recommendation and might use the GPU when it sees fit.
I guess it might use the other CPU

Resources