how to text classification using LSTM - keras

Question

I am doing text classification using LSTM model, I got 98% accuracy in validation data but when I am submitting It gets 0 scores, please help me how to do, I am a beginner to NLP.
I have data like this
train.head()
id category text
0 959 0 5573 1189 4017 1207 4768 8542 17 1189 5085 5773
1 994 0 6315 7507 6700 4742 1944 2692 3647 4413 6700
2 995 0 5015 8067 5335 1615 7957 5773
3 996 0 2925 7199 1994 4647 7455 5773 4518 2734 2807 8...
4 997 0 7136 1207 6781 237 4971 3669 6193
I am applying tokenizer here :
from keras.preprocessing.text import Tokenizer
max_features = 1000
tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(list(X_train))
X_train = tokenizer.texts_to_sequences(X_train)
X_test = tokenizer.texts_to_sequences(X_test)
I am applying sequence padding here:
from keras.preprocessing import sequence
max_words = 30
X_train = sequence.pad_sequences(X_train, maxlen=max_words)
X_test = sequence.pad_sequences(X_test, maxlen=max_words)
print(X_train.shape,X_test.shape)
Here my model:
batch_size = 64
epochs = 5
max_features = 1000
embed_dim = 100
num_classes = train['category'].nunique()
model = Sequential()
model.add(Embedding(max_features, embed_dim, input_length=X_train.shape[1]))
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(LSTM(100, dropout=0.2))
model.add(Dense(num_classes, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
Layer (type) Output Shape Param #
=================================================================
embedding_2 (Embedding) (None, 30, 100) 100000
_________________________________________________________________
conv1d_3 (Conv1D) (None, 30, 32) 9632
_________________________________________________________________
max_pooling1d_3 (MaxPooling1 (None, 15, 32) 0
_________________________________________________________________
conv1d_4 (Conv1D) (None, 15, 32) 3104
_________________________________________________________________
max_pooling1d_4 (MaxPooling1 (None, 7, 32) 0
_________________________________________________________________
lstm_2 (LSTM) (None, 100) 53200
_________________________________________________________________
dense_2 (Dense) (None, 2) 202
=================================================================
Total params: 166,138
Trainable params: 166,138
Non-trainable params: 0
_________________________________________________________________
None
Here my epochs:
model_history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=epochs, batch_size=batch_size, verbose=1)
Train on 2771 samples, validate on 693 samples
Epoch 1/5
2771/2771 [==============================] - 2s 619us/step - loss: 0.2816 - acc: 0.9590 - val_loss: 0.1340 - val_acc: 0.9668
Epoch 2/5
2771/2771 [==============================] - 1s 238us/step - loss: 0.1194 - acc: 0.9664 - val_loss: 0.0809 - val_acc: 0.9668
Epoch 3/5
2771/2771 [==============================] - 1s 244us/step - loss: 0.0434 - acc: 0.9843 - val_loss: 0.0258 - val_acc: 0.9899
Epoch 4/5
2771/2771 [==============================] - 1s 236us/step - loss: 0.0150 - acc: 0.9958 - val_loss: 0.0423 - val_acc: 0.9899
Epoch 5/5
2771/2771 [==============================] - 1s 250us/step - loss: 0.0064 - acc: 0.9984 - val_loss: 0.0532 - val_acc: 0.9899
after I will applied predict function to test data:
my submission file like this :
submission.head()
id category
0 3729 0.999434
1 3732 0.999128
2 3761 0.999358
3 5 0.996779
4 7 0.998702
my actual submission file like this :
submission.head()
id category
0 3729 1
1 3732 1
2 3761 1
3 5 1
4 7 1

Answer 1

Looks like you need to transform your results back into words! When you tokenized and padded, that turned words into numbers. You just need to change them back! For example:
transformed_category = []
for cat in submission['category']:
transformed_category.append(tokenizer.word_index(cat))
For education sake... It does this because math can't really be performed on strings---at least, not as readily as it can be done with numbers. So any time you've got text in your neural networks, they need to be turned into a numerical representation prior to getting fed into the network. Vectorizers (which your tokenizer did) and 'one-hot' or 'categorical' are the most common methods. In either case, once you get your results back out of the network, you can turn them back into words for humans. :)
edit after comments
Hi! So yes, I was looking at the columns askew. You're getting values of 1 (or really close) because sigmoid can only choose between 0 and 1, but, then, it looks like you wanted that, since your loss is binary_crossentropy. With sigmoid activation, large values will asymtotically approach 1. So I'd say you need to re-think your output layer. It looks like you're sending in arrays of numbers, and it looks like you want to get out a category spanning broader than 0 to 1, so consider turning your Y data into categoricals, using softmax as your final output activation, and changing your loss to categorical_crossentropy