TensorFlow - Simple Feed Forward NN not training - python-3.x

Question

I'm new to TensorFlow and just constructed my first very small network!! My code runs, but it has the same accuracy all along; it doesn't change with training. My data has 15 features and 6 classes. Maybe I'll add more features if that makes it easier and better. In short, my question is:
What's a general procedure for debugging TensorFlow code?
My network architecture was determined arbitrarily, so maybe I should change the number of neurons per layer, not entirely sure.
sess1 = tf.Session()
num_predictors = len(training_predictors_tf.columns)
num_classes = len(training_classes_tf.columns)
feature_data = tf.placeholder(tf.float32, [None, num_predictors])
actual_classes = tf.placeholder(tf.float32, [None, num_classes])
weights1 = tf.Variable(tf.truncated_normal([num_predictors, 50], stddev=0.0001))
biases1 = tf.Variable(tf.ones([50]))
weights2 = tf.Variable(tf.truncated_normal([50, 45], stddev=0.0001))
biases2 = tf.Variable(tf.ones([45]))
weights3 = tf.Variable(tf.truncated_normal([45, 25], stddev=0.0001))
biases3 = tf.Variable(tf.ones([25]))
weights4 = tf.Variable(tf.truncated_normal([25, num_classes], stddev=0.0001))
biases4 = tf.Variable(tf.ones([num_classes]))
hidden_layer_1 = tf.nn.relu(tf.matmul(feature_data, weights1) + biases1)
hidden_layer_2 = tf.nn.relu(tf.matmul(hidden_layer_1, weights2) + biases2)
hidden_layer_3 = tf.nn.relu(tf.matmul(hidden_layer_2, weights3) + biases3)
out = tf.matmul(hidden_layer_3, weights4) + biases4
model = tf.nn.softmax_cross_entropy_with_logits(labels=actual_classes, logits=out)
# cost = -tf.reduce_sum(actual_classes*tf.log(model))
cross_entropy = tf.reduce_mean( model)
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(cross_entropy)
# train_step = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(cross_entropy)
sess1.run(tf.global_variables_initializer())
correct_prediction = tf.equal(tf.argmax(out, 1), tf.argmax(actual_classes, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
for i in range(1, 30001):
sess1.run(
train_step,
feed_dict={
feature_data: training_predictors_tf.values,
actual_classes: training_classes_tf.values.reshape(len(training_classes_tf.values), num_classes)
}
)
if i%5000 == 0:
print(i, sess1.run(
accuracy,
feed_dict={
feature_data: training_predictors_tf.values,
actual_classes: training_classes_tf.values.reshape(len(training_classes_tf.values), num_classes)
}
))
And this is my output:
5000 0.3627
10000 0.3627
15000 0.3627
20000 0.3627
25000 0.3627
30000 0.3627
EDIT: I scaled my data as explained here, in the range of [-5;0] but it still does not train the network any better :(
Snippet of unscaled data (one-hot encoding first 6 columns):
2017-06-27 0 0 0 1 0 0 20120.0 20080.0 20070.0 20090.0 ... 20050.0 20160.0 20130.0 20160.0 20040.0 20040.0 20040.0 31753.0 36927.0 41516.0
2017-06-28 0 0 1 0 0 0 20150.0 20120.0 20080.0 20150.0 ... 20060.0 20220.0 20160.0 20130.0 20130.0 20040.0 20040.0 39635.0 31753.0 36927.0
2017-06-29 0 0 0 1 0 0 20140.0 20150.0 20120.0 20140.0 ... 20090.0 20220.0 20220.0 20160.0 20100.0 20130.0 20040.0 50438.0 39635.0 31753.0
2017-06-30 0 1 0 0 0 0 20210.0 20140.0 20150.0 20130.0 ... 20150.0 20270.0 20220.0 20220.0 20050.0 20100.0 20130.0 58983.0 50438.0 39635.0
2017-07-03 0 0 0 1 0 0 20020.0 20210.0 20140.0 20210.0 ... 20140.0 20250.0 20270.0 20220.0 19850.0 20050.0 20100.0 88140.0 58983.0 50438.0

Answer 1

Debugging your network and improving it are two different things. To improve it, once you've chosen a type of classifier (for instance a neural network), you should use the training and validation accuracies, and adjust your hyperparameters in functions of both of these. See the chapter Practical methodology of Goodfellow and alii's book for instance to know how to tune the hyperparameters (it's a bit long, but pure gold !).
As to debugging, this is the harder part. You usually do that by printing some "key tensor" values every once in a while. You clearly have a bug somewhere, or your accuracy would change at least a bit during training. A common problem causing that is exploding gradient, that causes Nans to appear very early in training (sometimes infinity or even 0s at strange places), which then basically stops any update in your network. I'd suggest printing your loss, and maybe the norm of your gradients, that should tell you if this is the problem. If so, the quick-and-dirty solution is to use a smaller LR at start and then augment it. The real solution is to use gradient clipping.
Example of how to print multiple tensor values:
if i%5000 == 0:
acc_val, loss_val, predictions = sess1.run(
[accuracy, cross_entropy, tf.argmax(out, 1)],
feed_dict={
feature_data: training_predictors_tf.values,
actual_classes: training_classes_tf.values.reshape(len(training_classes_tf.values), num_classes)
})
print(i, acc_val, loss_val) # You can also print predictions, but it will be very big so it'll be harder to see the others. Doing it would allow you to check if, for instance, the model always predicts class 0...