Keras Sequential model with cRelu activation - keras

I have a problem with creating a Dense model with 3 Layers in which the activation function is cRelu.
cRelu concatenates two relu (a negative and a positive) and creates a tensor twice the size in it's output.
When trying to add another layer after it, I always get a size mismatch error
model = Sequential()
model.add(Dense(N, input_dim=K, activation=crelu))
model.add(Dense(N//2, activation=crelu))
How do I tell the next layer to expect a 2N input and to N?

Keras doesn't expect the activation function to change the output shape. If you want to change it, you should wrap the crelu functionality in a layer and specify the corresponding output shape:
import tensorflow as tf
from keras.layers import Layer
class cRelu(Layer):
def __init__(self, **kwargs):
super(cRelu, self).__init__(**kwargs)
def build(self, input_shape):
super(cRelu, self).build(input_shape)
def call(self, x):
return tf.nn.crelu(x)
def compute_output_shape(self, input_shape):
"""
All axis of output_shape, except the last one,
coincide with the input shape.
The last one is twice the size of the corresponding input
as it's the axis along which the two relu get concatenated.
"""
return (*input_shape[:-1], input_shape[-1]*2)
Then you can use it as follows
model = Sequential()
model.add(Dense(N, input_dim=K))
model.add(cRelu())
model.add(Dense(N//2))
model.add(cRelu())

Related

how to handle different size of input data using Pytorch built in neural network

I build a simple pytorch model as below. However, I receive error message that mat1 and mat2 size are not aligned. How do I tweek the code to allow the flexibility of different dimension of data?
class simpleNet(nn.Module):
def __init__(self, **input_dim, hidden_size, num_classes**):
"""
:param input_dim: input feature dimension
:param hidden_size: hidden dimension
:param num_classes: total number of classes
"""
super(TwoLayerNet, self).__init__()
# hidden layer
self.hidden = nn.Linear(input_dim, hidden_size)
# Second fully connected layer that outputs our 10 labels
self.output = nn.Linear(hidden_size, num_classes)
def forward(self, x):
out = None
x = self.hidden(x)
x = torch.sigmoid(x)
x = self.output(x)
out = x
trying to build a toy neural network using Pytorch.
For your neural network to work, your output from your previous layer should be equal to your input for next layer, since its a code snippet for just your architecture without the initializations code, I cannot tell what you can simplify, not having equals in transition is not a good practice though. However, you can use reshape function from torch to make your output of previous layer equal to your next layer to make it work as a brute force method. Refer to: https://pytorch.org/docs/stable/generated/torch.reshape.html

How to add BatchNormalization loss to gradient calculation in tensorflow 2.0 using keras subclass API

Using the keras subclass API it is easy enough to add a a batch normalization layer however the layer.losses list always appears empty. What is the correct method of including in the train loss when doing tape.gradient(loss, lossmodel.trainable_variables) where lossmodel is some separate keras subclass model defining a more complicated loss function that must include the gradient losses?
For example, this is minimal model with ONLY the batch norm layer. It has no loss AFAIK
class M(tf.keras.Model):
def __init__(self, axis):
super().__init__()
self.layer = tf.keras.layers.BatchNormalization(axis=axis, scale=False, center=True, virtual_batch_size=1, input_shape=(6,))
def call(self, x):
out = self.layer(x)
return out
m = M(1)
In [77]: m.layer.losses
Out[77]: []

Custom layer in keras not doing anything

I am trying to make a layer which flips an image in the horizontal axis and then adds this image to the batch dimension. The code is as follows:
class FlipLayer(keras.layers.Layer):
def __init__(self, input_layer):
super(FlipLayer, self).__init__()
def get_output_shape(self, input_shape):
return (2 * input_shape[0],) + input_shape[1:]
def get_output(self, input):
return keras.layers.Concatenate([
input,
flipim(input)
], axis=0)
Where 'flipim' is just a function flipping the numpy array in the desired axis. Keras does not give any error when compiling the model using this function, however it isn't doing anything. When I use this layer as my last layer and check the output, it is still the same size in the batch dimension compared to the previous layer.

How do I add LSTM, GRU or other recurrent layers to a Sequential in PyTorch

I like using torch.nn.Sequential as in
self.conv_layer = torch.nn.Sequential(
torch.nn.Conv1d(196, 196, kernel_size=15, stride=4),
torch.nn.Dropout()
)
But when I want to add a recurrent layer such as torch.nn.GRU it won't work because the output of recurrent layers in PyTorch is a tuple and you need to choose which part of the output you want to further process.
So is there any way to get
self.rec_layer = nn.Sequential(
torch.nn.GRU(input_size=2, hidden_size=256),
torch.nn.Linear(in_features=256, out_features=1)
)
to work? For this example, let's say I want to feed torch.nn.GRU(input_size=2, hidden_size=20)(x)[1][-1] (the last hidden state of the last layer) into the following Linear layer.
I made a module called SelectItem to pick out an element from a tuple or list
class SelectItem(nn.Module):
def __init__(self, item_index):
super(SelectItem, self).__init__()
self._name = 'selectitem'
self.item_index = item_index
def forward(self, inputs):
return inputs[self.item_index]
SelectItem can be used in Sequential to pick out the hidden state:
net = nn.Sequential(
nn.GRU(dim_in, dim_out, batch_first=True),
SelectItem(1)
)

Keras layer input type

I have designed a layer in Keras. This is the first layer of the network. The input to this layer must be an RGB image ie of shape (height , width , 3). However when i run the code , i get the following error.
ValueError: Layer sequential_1 was called with an input that isn't a symbolic tensor. Received type: . Full input: [<main.CountPix object at 0x7fa9a5e81518>]. All inputs to the layer should be tensors.
How should I input my image or what should I modify in my layer?
class CountPix(Layer):
def __init__(self, **kwargs):
super(CountPix, self).__init__(**kwargs)
def build(self, input_shape):
# Create a trainable weight variable for this layer.
self.kernel = self.add_weight(name='kernel', shape=((200,200,3)),initializer='uniform',trainable=True)
super(MyLayer, self).build(input_shape) # Be sure to call this somewhere!
You need to define an input.
from keras.layers import Input
input_X = Input(shape=(height, width, 3), dtype='float32', name='input_image')
Also, if your self.kernel line you need to explicitly tell keras it is has an input shape similar to this example:
from keras.layers import Dense
from keras.models import Sequential
model = Sequential()
model.add(Dense(32, input_shape=(height, width, 3)))

Resources