I have a for loop in my code and in each iteration I augment some processed data and train my TF model again. After a while it takes longer than expected to process my code. I suspect about CPU usage since I running on multiple cores. How can I fix that?
I'm trying to train a gensim sgns model and in the process I measure the loss during which I'm calculating as
loss = model.running_training_loss / model.corpus_count,
however, I noticed that if I change my worker thread I get different losses keeping all other parameters same. Especially if I keep my worker thread a 1 I get a really high loss and If I increase threads I get less loss. An instance
thread loss
worker=1 20.40519721
worker=10 2.714875407
worker=16 1.239528453
Up through gensim 3.5.0, the loss value reported may not be very sensible, only resetting the tally each call to train(), rather than each internal epoch. There are some fixes forthcoming in this issue:
https://github.com/RaRe-Technologies/gensim/pull/2135
What version of gensim are you using, and what is your code doing to collect the loss data?
I read all sorts of texts about it, and none seem to answer this very basic question. It's always ambiguous:
In a stateful = False LSTM layer, does keras reset states after:
Each sequence; or
Each batch?
Suppose I have X_train shaped as (1000,20,1), meaning 1000 sequences of 20 steps of a single value. If I make:
model.fit(X_train, y_train, batch_size=200, nb_epoch=15)
Will it reset states for every single sequence (resets states 1000 times)?
Or will it reset states for every batch (resets states 5 times)?
Cheking with some tests, I got to the following conclusion, which is according to the documentation and to Nassim's answer:
First, there isn't a single state in a layer, but one state per sample in the batch. There are batch_size parallel states in such a layer.
Stateful=False
In a stateful=False case, all the states are resetted together after each batch.
A batch with 10 sequences would create 10 states, and all 10 states are resetted automatically after it's processed.
The next batch with 10 sequences will create 10 new states, which will also be resetted after this batch is processed
If all those sequences have length (timesteps) = 7, the practical result of these two batches is:
20 individual sequences, each with length 7
None of the sequences are related. But of course: the weights (not the states) will be unique for the layer, and will represent what the layer has learned from all the sequences.
A state is: Where am I now inside a sequence? Which time step is it? How is this particular sequence behaving since its beginning up to now?
A weight is: What do I know about the general behavior of all sequences I've seen so far?
Stateful=True
In this case, there is also the same number of parallel states, but they will simply not be resetted at all.
A batch with 10 sequences will create 10 states that will remain as they are at the end of the batch.
The next batch with 10 sequences (it's required to be 10, since the first was 10) will reuse the same 10 states that were created before.
The practical result is: the 10 sequences in the second batch are just continuing the 10 sequences of the first batch, as if there had been no interruption at all.
If each sequence has length (timesteps) = 7, then the actual meaning is:
10 individual sequences, each with length 14
When you see that you reached the total length of the sequences, then you call model.reset_states(), meaning you will not continue the previous sequences anymore, now you will start feeding new sequences.
In Keras there are two modes for maintaining states:
The default mode (stateful = False) where the state is reset after each sample. This is because, samples/sequences in a batch are always independent and each sample/sequence has a separate state which is the representation of that sequence. Another implication is that all the signal required to process a sequence in a batch is contained in that batch and state from previous batch is not required.
The stateful mode where the state is never reset. It is up to the user to reset state before a new epoch, but Keras itself wont reset the state. In this mode the state is propagated from sample "i" of one batch to sample"i" of the next batch. Generally it is recommended to reset state after each epoch, as the state may grow for too long and become unstable. However in my experience with small size datasets (20,000- 40,000 samples) resetting or not resetting the state after an epoch does not make much of a difference to the end result. For bigger datasets it may make a difference.
Stateful model will be useful if you have patterns that span over 100s of time steps. Otherwise the default mode is sufficient. In my experience setting the batch size roughly equivalent to the size (time steps) of the patterns in the data also helps.
The stateful setup could be quite difficult to grasp at first. The key to understand is if there are very long sequences which need to be divided into small sub-sequences and processed, the sub-sequences are lined up across batches and not within a batch. Within a batch the sequences are always independent. However if state needs to be carried over from one sub-sequence to another, the sub-sequences should be lined up across batches i.e. Sequence i of batch n+1 is a continuation of sequence i from batch n and so on. So in this case the final state of sequence i from batch n is passed over as the initial state of sequence i in batch n+1.
References
https://github.com/fchollet/keras/issues/98
https://keras.io/getting-started/faq/#how-can-i-use-stateful-rnns
In the doc of the RNN code you can read this :
Note on using statefulness in RNNs :
You can set RNN layers to be 'stateful', which means that the states
computed for the samples in one batch will be reused as initial states
for the samples in the next batch. This assumes a one-to-one mapping
between samples in different successive batches.
I know that this doesn't answer directly your question, but to me it confirms what I was thinking : when a LSTM is not stateful, the state is reset after every sample. They don't work by batches, the idea in a batch is that every sample is independant from each other.
So you have 1000 reset of the state for your example.
Everyone seems to be making it too confusing. Keras LSTM resets state after every batch.
Here is a good blog: https://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/
Read LSTM State Within A Batch and Stateful LSTM for a One-Char to One-Char Mapping topics in this blog. It shows why it must reset it after batch only.
Expanding on #Nassim_Ben's answer, it is true that each sequence is considered independent for each instance of the batch. However, you need to keep in mind that the RNNs hidden state and cell memory get's passed along to the next cell for 20 steps. The hidden state and cell memory is typically set to zero for the very first cell in the 20 cells.
After the 20th cell, and after the hidden state (only, not cell memory) gets passed onto the layers above the RNN, the state gets reset. I'm going to assume that they mean cell memory and hidden state here.
So yes, it does get reset for all 1000 instances, however, considering that your batch_size=200, it gets reset 5 times, with each batch getting reset after they are done passing information through those 20 steps. Hopefully you got your head around this.
Here's a project I did where I had the same question. Pay special attention to cell 15 and it's explanation in the blob after cell 11. I kept appending letters because the state was getting reset otherwise.
I am using Keras/TensorFlow (GPU) to create a time series forecasting model. I have 100x of time series and want to train a network for each of them.
Running a few time series in a row is fine, but once I run 100x or 1000x then it appears that the training time of each model increase slowly (but surely). Is there a simple reason for this ?
Below is code to reproduce the issue (note that it could take a while to run).
https://gist.github.com/mannsi/c5666c4b786c35c3443beea6d13a32fe
On my machine the first iteration takes 10s, iteration #250 takes 16s and iteration #500 takes 25s.
I am new to Neural Networks and Keras/TF so maybe this is totally normal but I did not factor this in when doing my back-of-the-envelope time calculations.
System info:
python 3.5
keras (1.2.2)
tensorflow-gpu(1.0.0)
EDIT: I tested the same code on a TensorFlow CPU backend and I see the exact same behavior there.
It's possible that there is some overhead building up in the computation graph over each iteration. Use the Keras backend function K.clear_session() to reset the underlying Tensorflow session between each run.
Could it be that your gpu warms up and therefore the power is lowered to reduce temperature?
How long does the first iteration take if you relaunch it after having done many iterations?
As your model parameters didn't change, you have to compile only once the model. Then you can build a loop for fitting it.
You instantiate a model and compile it in every loop, that's why your memory consumption grows continuously.
I have a Large-Scale Gradient Descent optimization problem that I am running using Matlab. The code has got two parts:
A Sequential update part that fires every iteration that updates the parameter vector.
A validation error computation part that fires every 10 iterations or so using the parameter value at the end of the corresponding iteration in which its fired.
The way that I am running this now is to do (1) and (2) sequentially. But (2) takes a lot of time and its not the core part of my routine - I made it just to check the progress and plot the error of my model. Is it possible in Matlab to run (2) in a parallel manner to (1) ? Please note that (1) cannot be run in parallel since it performs sequential update. So a simple 'parfor' usage is not a solution, unless there is a really smart way of doing that.
I don't think Matlab has any way of multi-threading outside of the (rather restricted) parallel computing toolbox. There is a work over which may help you though:
Open 2 sessions of Matlab, sessions A and B (or instances, or workspaces, however you call it)
Matlab session A:
Calculate the 10 iterations of your sequential process (1)
Saves the result in a file (adequately and uniquely named)
Goes on to calculate the next 10 iterations (back to the top of this loop basically)
In parralel:
Matlab session B:
Check periodically for the existence of the file written by process A (define a timer that will do that at the time interval which make sense for your process, a few seconds or a few minutes ...)
If the file exist => load it then do the validation computation (your process (2)) and display/report the results.
note: This only works if process (1) doesn't need the result of process (2) to run its iterations, but if it is the case I don't know how you could parallelise anyway.
If you have multiple cores on your machine that should run smoothly, if you have a single core then the 2 sessions will have to share and you will see a performance impact.