How Kera LSTM inputs corresponds with output - python-3.x

Question

I am trying to predict the stock price with help of investor sentiments and previous stock price.
head of data frame is as under:
time_p close sent_sum output
2007-01-03 10:00:00 10.837820 0.4 10.6838
2007-01-03 11:00:00 10.849175 0.6 10.8062
2007-01-03 12:00:00 10.823942 -0.3 10.7898
2007-01-03 13:00:00 10.810063 -0.2 10.7747
2007-01-03 14:00:00 10.680111 0.1 10.7078
How I preprocess Data?
Above df contains stock data where,time_p is hourly datetime(not included in model) that coresponds to houly closing price close, sent_sum is invostor sentiment and output is labels for model. output is shifted upword with df.output.shitf(-8) in other words I want to predict +1 hour into future based upon -7 hours close(price) plus -7hours sent_sum (investor sentimnets).
I am trying to fit a model like this:
import tensorflow as tf
from pandas_datareader import data
import urllib.request, json
from sklearn.preprocessing import MinMaxScaler
from sklearn import metrics
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.layers.recurrent import LSTM
from keras import optimizers
import math
import keras as k
import numpy as np
import matplotlib.pyplot as plt
data = pd.read_csv('AAPL_final.csv')
raw= data.iloc[:,[2,3]].values
raw2= data.iloc[:,[4]].values
#############scalling fo data######
scaler = MinMaxScaler(feature_range=(-1, 1))
scaler_y = MinMaxScaler(feature_range=(-1, 1))
scaled_x = scaler.fit_transform(raw)
scaled_y = scaler_y.fit_transform(raw2)
########tran test set##############
train= scaled_x[:14000].reshape(2000,7,2) # Train_X data
train_= scaled_y[:14000].reshape(2000,7,1) #train_Y
test_xdata= scaled_x[14000:17542].reshape(506,7,2)# Test_x
test_ydata= scaled_y[14000:17542].reshape(506,7,1)#Test_y
train_x,train_y= train, train_
test_x, test_y = test_xdata, test_ydata
print('shapes of tranx,teainy,testx and testy',train_x.shape, train_y.shape, test_x.shape, test_y.shape)
model = Sequential()
model.add(LSTM(100,input_shape=(7,2),return_sequences=True))
model.add(Dropout(0.1))
model.add(LSTM(100,return_sequences=True))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='sgd',metrics=['accuracy', 'mae', 'mape', 'cosine'])#sgd#rmsprop
My Questiton: I suspect that once I have alreay shifted the label data with -7 points round the way matched current inputs with +7 hours in future time period is it ok to write train_= scaled_y[:14000].reshape(2000,7,1) #train_Y in (2000,7,1) shape or I am doing something worge.
Secondly, I am confused with how keras_lstm matches input with labels, I mean how input_shape really works?
Is there any good way to fit this model?, please suggest.
I shall be grateful for the help.

Answer 1

You can do something like following on the scaled_x and scaled_y
I used toy dataset to show an example, here data and labels are of shape ((150, 4), (150,)) initially, using the following script:
seq_length = 10
dataX = []
dataY = []
for i in range(0, 150 - seq_length, 1):
dataX.append(data[i:i+seq_length])
dataY.append(labels[i+seq_length-1])
import numpy as np
dataX = np.reshape(dataX, (-1, seq_length, 4))
dataY = np.reshape(dataY, (-1, 1))
# dataX.shape, dataY.shape
Output: ((140, 10, 4), (140, 1))
Like this example you can create the sequence with the 7 days data, with target for next day.
Keras LSTM layer expects the input to be 3 dims as (batch_size, seq_length, input_dims) as this
input_dims = # an integer
seq_length = #an integer
model = Sequential()
model.add(LSTM(128, activation='relu', input_shape=(seq_length, input_dims), return_sequences=True))
Note: batch_size is not used on defining the layer, model will fill itself while fit.