How can I implement this locally-connected architecture in Keras? - keras

Question

My input data looks like this:
> x <- rnorm(10*9, sd = 10) %>% matrix(10) %>% round
> colnames(x) <- c(paste0(2014, c("a","b", "c")), paste0(2015, c("a","b", "c")), paste0(2016, c("a","b", "c")))
> x
2014a 2014b 2014c 2015a 2015b 2015c 2016a 2016b 2016c
[1,] 1 -11 3 3 6 5 17 5 15
[2,] 9 8 0 -1 10 8 -3 -11 6
[3,] -6 22 -3 1 -1 -4 -3 11 -9
[4,] 10 -15 0 -2 4 14 11 -11 3
[5,] 5 4 5 5 15 -9 2 5 1
[6,] -24 16 9 -7 2 -12 1 18 -2
[7,] 1 13 5 -14 1 -10 15 -1 14
[8,] -8 4 4 -15 -1 -20 -6 14 5
[9,] 10 19 -15 15 -4 3 -1 -11 8
[10,] 10 -11 -9 -1 16 3 24 -8 4
My outcome variable is continuous (i.e.: this is a regression problem).
I want to fit a model with an architecture that looks like this:
Basically, I've got granular data from separate years that aggregate to form a set of annual phenomena, which may themselves interact. If I had enough data, I could just fit a bunch of fully-connected layers. But those would be inefficient with my modest sample size.
This isn't exactly a conv net, because I don't want the "tiles" to overlap.
I also want apply both dropout and a global L2 penalty.
I'm new to Keras, but not to neural nets. How can I implement this, and how is it referred-to in Keras terminology?

Answer 1

You can use the functional API to have multiple inputs and create that computation graph. Something along the lines of:
inputs = [Input(shape=(3,)) for _ in range(3)]
latents = list()
for i in range(3):
latent = Dense(3, activation='relu')(inputs[i])
latent = Dense(3, activation='relu')(latent)
latents.append(latent)
merged = concatenate(latents)
out = Dense(4, activation='relu')(merged)
out = Dense(4, activation='relu')(out)
out = Dense(1)(out)
Your architecture diagram assumes you have fixed number year inputs, in this case 3 years. If you have variable number of years you have to use shared Dense layers and use TimeDistributed wrapper to apply the Dense layers to every year before merging:
in = Inputs(shape=(3,3)) # this time we have 2d array of 3 years
latent = TimeDistributed(Dense(3, activation='relu'))(in) # apply same dense to every year
latent = TimeDistributed(Dense(3, activation='relu'))(latent)
merged = Flatten()(latent)
out = ...
This time the Dense layers are shared across years, they have the same weights essentially.