Is there a possibility to calculate the variance of target values for each batch using keras custom callback?
Maybe in a similar manner to the Create keras callback to save model predictions and targets for each batch during training ?
Im using keras from tensorflow backend and python3.
Cheers,
Maks
Related
I am trying to implement a CRF layer in a TensorFlow sequential model for a NER problem. I am not sure how to do it. Previously when I implemented CRF, I used CRF from keras with tensorflow as backend i.e. I created the entire model in keras instead of tensorflow and then passed the entire model through CRF. It worked.
But now I want to develop the model in Tensorflow as tensorflow2.0.0 beta already has keras inbuilt in it and I am trying to build a sequential layer and add CRF layer after a bidirectional lstm layer. Although I am not sure how to do that. I have gone through the CRF documentation in tensorflow-addons and it contains different functions such as forward CRF etc etc but not sure how to implement them as a layer ? I am wondering is it possible at all to implement a CRF layer inside a sequential tensorflow model or do I need to build the model graph from scratch and then use CRF functions ? Can anyone please help me with it. Thanks in advance
In the training process:
You can refer to this API:
tfa.text.crf_log_likelihood(
inputs,
tag_indices,
sequence_lengths,
transition_params=None
)
The inputs are the unary potentials(just like that in the logistic regression, and you can refer to this answer) and here in your case, they are the logits(it is usually not the distributions after the softmax activation function) or states of the BiLSTM for each character in the encoder(P1, P2, P3, P4 in the diagram above; ).
The tag_indices are the target tag indices, and the sequence_lengths represent the sequence lengths in a batch.
The transition_params are the binary potentials(also how the tag transits from one time step to the next), you can create the matrix yourself or you just let the API do it for you.
In the inference process:
You just utilize this API:
tfa.text.viterbi_decode(
score,
transition_params
)
The score stands for the same input like that in the training(the P1, P2, P3, P4 states) and the transition_params are also that trained in the training process.
I have tried to write my own custom loss function in Keras. But writing complex loss functions are normally requiring deep knowledge of TensorFlow and Keras Backend. Do I need to study them to write my own loss function or is there an alternative method for writing custom loss functions for neural networks in Keras?
My loss function requires the probability of prediction of all classes and some way to point out the probability corresponding to the label class.
I would like to visualize gradients of a seq2seq model using Keras Tensorboard callback. If I'm using a regular LSTM cell in my encoder and decoder, I get nice non-zero gradients:
However if I change the rnn cell to CuDNNLSTM some gradients turn to zero, which seem to be incorrect:
The both models seem to train correctly.
So, what's wrong with visualisations of CuDNNLSTM gradients? Is there a bug in Keras Tensorboard callback?
Code that I am running is a slightly modified Keras lstm_seq2seq example: https://gist.github.com/nicolas-ivanov/1818d6502d5f1496e5fbe14889eddca1
There is ImageDataGenerator.flow() function, in which we pass our training images and it returns augmented images. But what does ImageDataGenerator.fit() function does?
For some transformations such as centering or scaling, the ImageDataGenerator needs statistics on the data such the feature-wise mean, standard deviation etc. The fit() method collects these statistics. This API is similar to how the preprocessing functions in scikit-learn work.
I want to get the weights of each node of every layer in the DNNClassifier, trained using the estimator API of tensorflow. I found that it is possible to get weights of each node in keras. Is it possible for estimator API? Thanks for your help.
input_func = tf.estimator.inputs.pandas_input_fn(x=X_train,y=y_train,batch_size=10,num_epochs=1000,shuffle=True)
dnn_model = tf.estimator.DNNClassifier(hidden_units=[10,10,10],feature_columns=feat_cols,n_classes=2
model.train(input_fn,steps=6000)
I have used the above code to train the model. I want to further extract the weights of each node of hidden layer.
Yes, it should be possible to do so.
You can extract the trainable variables names with:
train_var_names = [var.name for var in tf.trainable_variables()]
These are usually named 'layer-0/kernel' and 'layer-0/bias'. You can then access their values (after training your network) through your estimator (which I'll assume is named dnn_model from your question). As an example:
weights_0 = dnn_model.get_variable_value(train_var_names[0])