Shoutout to all Pytorch core developers!
I would like to implement a custom version of the typical LSTM cell as it is implemented in Pytorch, say, change one of the activation functions at a gate. For this, I would like to see how the LSTM is implemented in Pytorch at the moment. I can find some code here, but unfortunately, I cannot find the exact LSTM computations there, including the implementation of the gate formulas as they are also specified in the docstrings.
Related posts can for example be found here, but all they delivered me is that nobody has found the LSTM cell code in github.
Where is the LSTM cell implemented in the Pytorch Github exactly?
Related
I know that pytorch hasn't implemented LSTM with peephole connections yet, but I wanted to implement one. So, by looking at the source code of LSTM, I cannot see how the gates are used exactly. Nevertheless, as I understand, I can implement my own LSTM using LSTMCell. So, I found this code on the pytorch github for custom LSTMs Here, from lines 93 to 118.
So, to implement the peephole LSTM, I just need to multiply the cellgate with each of the ingate, forgetgate, and outgate, like this:
ingate = torch.sigmoid(ingate * cellgate)
forgetgate = torch.sigmoid(forgetgate * cellgate)
outgate = torch.sigmoid(outgate * cellgate)
cellgate = torch.tanh(cellgate)
PS: the above code should be added to the whole class in the above link; in other words, will this piece of code work if I added to the code in the above link?
Do I understand how to do it correctly? And, in general, is there another way or existing way to implement peephole LSTM?
I have a quantized model in pytorch and now I want to extract the parameter of the quantized linear layer and implement the forward manually.
I search the source code but only find this function.
def forward(self, x: torch.Tensor) -> torch.Tensor:
return torch.ops.quantized.linear(
x, self._packed_params._packed_params, self.scale, self.zero_point)
But no where I can find how torch.ops.quantized.linear is defined.
Can someone give me a hind how the forward of quantized linear are defined?
In answer to the question of where torch.ops.quantized.linear is, I was looking for the same thing but was never able to find it. I believe it's probably somewhere in the aten (C++ namespace). I did, however, find some useful PyTorch-based implementations in the NVIDIA TensorRT repo below. It's quite possible these are the ones actually called by PyTorch via some DLLs. If you're trying to add quantization to a custom layer, these implementations walk you through it.
You can find the docs here and the GitHub page here.
For the linear layer specifically, see the QuantLinear layer here
Under the hood, this calls TensorQuantFunction.apply() for post-training quantization or FakeTensorQuantFunction.apply() for quantization-aware training.
I am trying to implement the AntisymmetricRNN described in this paper: https://arxiv.org/abs/1902.09689.
Working in Keras, I guess I have to implement my own layer so I have read https://keras.io/layers/writing-your-own-keras-layers/. Instead of starting from a plain layer as explained there, I reckon the best would probably be to extend one of the existing RNN, but Keras has
RNN
SimpleRNNCell
SimpleRNN
The documentation isn't verbose enough for someone my level about what these classes do/are, and consequently I am having a hard time figuring out what should be my starting point.
Any help, both in terms of where to start and what to actually look out for, and all sorts of suggestions are greatly appreciated. Thank you.
In Keras, all recurrent layers are RNN layers with a certain Cell.
The definition is RNN(cell=someCell)
So, the LSTM layer follows the same principle, an LSTM(units=...) layer is equal to an RNN(cell=LSTMCell(units=...), ...) layer.
That said, to implement your recurrent layer (if it doesn't break the recurrent flow step by step or jump steps), you need to implement your own cell. You can study what is happening in the LSTMCell code, compare it with the papers and adjust the weights and formulas to your need.
So you will have your own RNN(cell=yourCell).
I'm having trouble understanding an implementation in Keras of conditional variational autoencoders. The associated jupyter notebook is here.
My issue is, I don't see how you would pass the test set through the model. I know you need to use the recognition network for training and the prior network for testing. I think that means you're supposed to concatenate the result of the prior network and some function of the input, but I only see an encoder (which takes in both x and y) and a decoder. Where in this implementation is the prior network (just x)?
I'm going through the Keras autoencoder tutorial on this link and I noticed that, while all previous examples used all available cores on my machine, the example for the Convolutional autoencoder used a single core, therefore taking quite a bit longer than the previous examples.
I've trained it used the theano backend (as opposed to using tensorflow as suggested in the tutorial) though.
Is this expected behaviour or is something wrong here?