Why is keras tokenizer applying lowercase() to it's own tokens? - keras

Question

I am running my first cnn text-classifier using the IMDB dataset with the in-built
tf.keras.datasets.imdb.load_data()
I understand the AttributeError: 'int' object has no attribute 'lower' error indicates that a lowercase function is being applied to int objects (seems to be from the tokenizer). However, I don't know why it is throwing this in this case as I am loading it directly though the in-built tf.keras.datasets.imdb.load_data().
I am not experienced with using embedding in text-classification.
The code excluding the CNN model is:
import tensorflow as tf
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.layers import Embedding, LSTM
from keras.layers import Conv1D, Flatten, MaxPooling1D
from keras.datasets import imdb
import wandb
from wandb.keras import WandbCallback
import numpy as np
from keras.preprocessing import text
import imdb
wandb.init(mode="disabled") # disabled for debugging
config = wandb.config
# set parameters:
config.vocab_size = 1000
config.maxlen = 1000
config.batch_size = 32
config.embedding_dims = 10
config.filters = 16
config.kernel_size = 3
config.hidden_dims = 250
config.epochs = 10
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.imdb.load_data()
tokenizer = text.Tokenizer(num_words=config.vocab_size)
tokenizer.fit_on_texts(X_train)
X_train = tokenizer.texts_to_matrix(X_train)
X_test = tokenizer.texts_to_matrix(X_test)
X_train = sequence.pad_sequences(X_train, maxlen=config.maxlen)
X_test = sequence.pad_sequences(X_test, maxlen=config.maxlen)
Line 34 referred to in the error is tokenizer = text.Tokenizer(num_words=config.vocab_size)
The exact error thrown (includes Deprecation warnings) is:
C:\Users\Keegan\anaconda3\envs\oldK\lib\site-
packages\tensorflow_core\python\keras\datasets\imdb.py:129:
VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-
or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If
you meant to do this, you must specify 'dtype=object' when creating the ndarray.
x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
C:\Users\Keegan\anaconda3\envs\oldK\lib\site-
packages\tensorflow_core\python\keras\datasets\imdb.py:130: VisibleDeprecationWarning: Creating
an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or
ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must
specify 'dtype=object' when creating the ndarray.
x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
Traceback (most recent call last):
File "imdb-cnn.py", line 34, in <module>
tokenizer.fit_on_texts(X_train)
File "C:\Users\Keegan\anaconda3\envs\oldK\lib\site-packages\keras_preprocessing\text.py",
line 217, in fit_on_texts
text = [text_elem.lower() for text_elem in text]
File "C:\Users\Keegan\anaconda3\envs\oldK\lib\site-packages\keras_preprocessing\text.py", line 217, in <listcomp>
text = [text_elem.lower() for text_elem in text]
AttributeError: 'int' object has no attribute 'lower'
The Anaconda venv has Python 3.7.1, Tensorflow 2.1.0 and Keras 2.3.1

Answer 1

The Keras tokenizer has an attribute lower which can be set either to True or False.
I guess the reason why the pre-packaged IMDB data is by default lower-cased is that the dataset is pretty small. If you did not lower-case it the capitalized and lower-cased words would get different embeddings, but the capitalized forms probably are not frequently enough in the training data to train the embeddings appropriately. This of course changes, once you use pre-trained embeddings or pre-trained contextualized models such as BERT which were pre-trained on large data.