As the title states, does Keras (w. Tensorflow backend) normalize the kernel weights compared to e.g. Tensorflow? For example, if two identical networks are implemented with Keras respectively Tensorflow, will the kernel-weights differ?
If you use Tensorflow in backend of Keras, there are no reasons for the implementation to be different.
You can check by yourself here : https://github.com/keras-team/keras/tree/master/keras/layers
Related
I have h5 weights from a Keras model.
I want to rewrite the Keras model into a tf.keras model (using TF2.x).
I know that only the high level API changed, but do you know if I still can use the h5 weights?
Most likely they can be loaded, but is the structure different between Keras and tf.keras weights?
Thanks
It seems that they are the same
cudos to Mohsin hasan answer
In the past, when I had to convert tf.keras model to keras model, I
did following:
Train model in tf.keras
Save only the weights tf_model.save_weights("tf_model.hdf5")
Make Keras model architecture using all layers in keras (same as the tf keras one)
load weights by layer names in keras: keras_model.load_weights(by_name=True)
This seemed to work for me. Since, I was using out of box architecture
(DenseNet169), I had to very less work to replicate tf.keras network
to keras.
And the answer from Alex Cohn
tf.keras HDF5 model and Keras HDF5 models are not different things,
except for inevitable software version update synchronicity. This is
what the official docs say:
tf.keras is TensorFlow's implementation of the Keras API specification. This is a high-level API to build and train models that
includes first-class support for TensorFlow-specific functionality
If the convertor can convert a keras model to tf.lite, it will deliver
same results. But tf.lite functionality is more limited than tf.keras.
If this feature set is not enough for you, you can still work with
tensorflow, and enjoy its other advantages.
I know that tensorflow probability supports mixture density networks now with the following layer :
https://www.tensorflow.org/probability/api_docs/python/tfp/layers/MixtureNormal
I was wondering if there is any convolutional version of this layer or maybe any existing library or implementation of this layer.
something along the lines mentioned here :
https://www.h-its.org/software/deep-convolutional-mixture-density-networks/
Any help is appreciated.
I am trying to implement a CRF layer in a TensorFlow sequential model for a NER problem. I am not sure how to do it. Previously when I implemented CRF, I used CRF from keras with tensorflow as backend i.e. I created the entire model in keras instead of tensorflow and then passed the entire model through CRF. It worked.
But now I want to develop the model in Tensorflow as tensorflow2.0.0 beta already has keras inbuilt in it and I am trying to build a sequential layer and add CRF layer after a bidirectional lstm layer. Although I am not sure how to do that. I have gone through the CRF documentation in tensorflow-addons and it contains different functions such as forward CRF etc etc but not sure how to implement them as a layer ? I am wondering is it possible at all to implement a CRF layer inside a sequential tensorflow model or do I need to build the model graph from scratch and then use CRF functions ? Can anyone please help me with it. Thanks in advance
In the training process:
You can refer to this API:
tfa.text.crf_log_likelihood(
inputs,
tag_indices,
sequence_lengths,
transition_params=None
)
The inputs are the unary potentials(just like that in the logistic regression, and you can refer to this answer) and here in your case, they are the logits(it is usually not the distributions after the softmax activation function) or states of the BiLSTM for each character in the encoder(P1, P2, P3, P4 in the diagram above; ).
The tag_indices are the target tag indices, and the sequence_lengths represent the sequence lengths in a batch.
The transition_params are the binary potentials(also how the tag transits from one time step to the next), you can create the matrix yourself or you just let the API do it for you.
In the inference process:
You just utilize this API:
tfa.text.viterbi_decode(
score,
transition_params
)
The score stands for the same input like that in the training(the P1, P2, P3, P4 states) and the transition_params are also that trained in the training process.
To create RNN cells, there are classes like GRUCell and LSTMCell which can be used later to create RNN layers.
And also there are 2 other classes as CudnnGRU and CudnnLSTM which can be directly used to create RNN layers.
In the documentation they say that the latter classes have cuDNN implementation. Why should I use or not use this cuDNN implemented classes over classical RNN implementations when I'm creating a RNN model..?
In short: cudnnGRU and cudnnLSTM can/ must be used on GPU, normal rnn implementations not. So if you have tensorflow-gpu, cudnn implementation of RNN cells would run faster.
CuDNNLSTM and CuDNNGRU are the fast implementation backed by CuDNN. Both can only be run on the GPU, with the TensorFlow backend. The cuDNN is a GPU-accelerated library of primitives for deep neural networks.
The cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN is part of the NVIDIA Deep Learning SDK.
The cuDNN highlights include:
Up to 3x faster training of ResNet-50 and GNMT on Tesla V100 vs.
Tesla P100
Improved NHWC support for pooling and strided convolution
Get Improved performance for common workloads such as ResNet50 and SSD as batchnorm now supports NHWC data layout with an added option
to fuse batchnorm with Add and ReLu operations
I am trying to implement Gaussian Mixture Model using keras with tensorflow backend. Is there any guide or example on how to implement it?
Are you sure that it is what you want? you want to integrate a GMM into a neural network?
Tensorflow and Keras are libraries to create, train and use neural networks models. The Gaussian Mixture Model is not a neural network.