I am trying to do a multivariate linear regression and I am having some issues. Namely, I am getting the following error:
ValueError: Cannot feed value of shape (3,) for Tensor 'X:0', which has shape '(1, 3)'
I have 3 feature variables, which I call trainX and 1 label, which I call trainY. Their shapes are the following (they are numpy arrays):
trainX.shape:
(2500, 3)
trainY.shape:
(2500,)
The following piece of code defines the tensors that I use to compute the model:
X = tf.compat.v1.placeholder("float", [1, 3], name="X")
Y = tf.compat.v1.placeholder("float", [1], name="Y")
W = tf.Variable(tf.zeros([3, 1]), name="W")
b = tf.Variable(tf.zeros([1]), name="b")
I calculate the predicted label and the cost function and the optimizer by doing:
predicted_y = tf.matmul(X, W) + b
cost = tf.reduce_sum(tf.pow(predicted_y-Y, 2)) / (2 * n)
optimizer = tf.compat.v1.train.GradientDescentOptimizer(learning_rate).minimize(cost)
I am getting the error in the tensor-flow session, namely in the following piece of code:
with tf.Session() as sess:
sess.run(init)
for epoch in range(training_epochs):
for (_x, _y) in zip(trainX, trainY):
sess.run(optimizer, feed_dict={X: _x, Y: _y})
if (epoch + 1) % 100 == 0:
c = sess.run(cost, feed_dict={X: trainX, Y: trainY})
print("Epoch", (epoch + 1), ": cost =", c, "W =", sess.run(W), "b =", sess.run(b))
# Storing necessary values to be used outside the Session
training_cost = sess.run(cost, feed_dict={X: trainX, Y: trainY})
weight = sess.run(W)
bias = sess.run(b)
Any help would be greatly appreciated.
The problem is that _x is a vector with 3 elements, while X expects a matrix with 1 line and 3 columns. One possible solution would be to reshape _x:
_x = np.reshape(_x, [1, 3])
Another possibility would be to change the placeholder to the input shape:
X = tf.compat.v1.placeholder("float", [3], name="X")
Often one wants to train on more than one example. In this case, you may want to define the placeholders to allow any number of inputs:
X = tf.compat.v1.placeholder("float", [None, 3], name="X")
Y = tf.compat.v1.placeholder("float", [None], name="Y")
Then we could for example we use batches of 100:
with tf.Session() as sess:
sess.run(init)
for epoch in range(training_epochs):
for i in range(trainX.shape[0] % 100):
sess.run(optimizer, feed_dict={X: trainX[i*100:(i+1)*100, ...], Y: trainY[i*100:(i+1)*100]})
Related
I want to calculate my-self loss. I want to predict N point by deep learning. Thus, the output of network is N point (N*3).
The numpy calculation should be:
import numpy as np
point1 = np.random.random(size=[10, 30, 3])
point2 = np.random.random(size=[10, 30, 3])
losses = []
for s in range(10):
loss = 0
for p in range(30):
p1 = point1[s, p, :]
dis = p1 - point2[s, :, :]
dis = np.linalg.norm(dis, axis=1)
loss += dis.min()
losses.append(loss)
print(loss)
In pytorch, the point should be:
point1 = np.random.random(size=[10, 30, 3])
point2 = np.random.random(size=[10, 30, 3])
point1 = torch.from_numpy(point1)
point2 = torch.from_numpy(point2)
How can I calculate the loss in pytorch?
Any suggestion is appreciated！
I have a dataset of 25000 colored pictures 100*100(*3) and I am trying to build a simple neural network with one convolutional layer. Its pictures of cells that are infected or not by Malaria, so my output is 2.
But it seems like I have a dimension mismatch, and I don't know where my error comes from.
My neural network :
def simple_nn(X_training, Y_training, X_test, Y_test):
input = 100*100*3
batch_size = 25
X = tf.placeholder(tf.float32, [batch_size, 100, 100, 3])
#Was:
# W = tf.Variable(tf.zeros([input, 2]))
# b = tf.Variable(tf.zeros([2]))
#Now:
W = tf.Variable(tf.truncated_normal([4, 4, 3, 3], stddev=0.1))
B = tf.Variable(tf.ones([3])/10) # What should I put here ??
init = tf.global_variables_initializer()
# model
#Was:
# Y = tf.nn.softmax(tf.matmul(tf.reshape(X, [-1, input]), W) + b)
#Now:
stride = 1 # output is still 28x28
Ycnv = tf.nn.conv2d(X, W, strides=[1, stride, stride, 1], padding='SAME')
Y = tf.nn.relu(Ycnv + B)
# placeholder for correct labels
Y_ = tf.placeholder(tf.float32, [None, 2])
# loss function
cross_entropy = -tf.reduce_sum(Y_ * tf.log(Y))
# % of correct answers found in batch
is_correct = tf.equal(tf.argmax(Y,1), tf.argmax(Y_,1))
accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))
learning_rate = 0.00001
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
train_step = optimizer.minimize(cross_entropy)
sess = tf.Session()
sess.run(init)
#Training here...
My error :
Traceback (most recent call last):
File "neural_net.py", line 135, in <module>
simple_nn(X_training, Y_training, X_test, Y_test)
File "neural_net.py", line 69, in simple_nn
cross_entropy = -tf.reduce_sum(Y_ * tf.log(Y))
...
ValueError: Dimensions must be equal, but are 2 and 3 for 'mul' (op: 'Mul') with input shapes: [?,2], [25,100,100,3].
I used a simple layer before, and it was working. I changed my weight and bias, and honestly, I don't know why my bias are setup like this, I followed a tutorial (https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/#11) but it is not explained.
I also replaced my Y to a conv2D.
And I don't know what my ouput should be if I want to get a vector of size 2*1 as a result.
You have correctly defined your labels as
Y_ = tf.placeholder(tf.float32, [None, 2])
So the last dimension is 2. However, the output from the convolution step is not directly suitable for comparing it to the labels. What I mean is the following: if you do
Ycnv = tf.nn.conv2d(X, W, strides=[1, stride, stride, 1], padding='SAME')
Y = tf.nn.relu(Ycnv + B)
The dimensions of this are going to be four as the error says:
ValueError: Dimensions must be equal, but are 2 and 3 for 'mul' (op: 'Mul') with input shapes: [?,2], [25,100,100,3].
So it is impossible to multiply directly (or operate) the output from convolution with the labels. What I recommend is to flatten (reshape it to only one dimension) the output of convolution and pass it to a fully connected layer of 2 units (as much as classes you have). Like this:
Y = tf.reshape(Y, [1,-1])
logits = tf.layers.dense(Y, units= 2)
and you can pass this to the loss.
Also I recommend you to change the loss to a more approprite version. For example, tf.losses.sigmoid_cross_entropy.
Also, the way you use convolutions is strange. Why do you put handmade filters in the convolution? besides you should have to initialize and before it putting them in a collection. In conclusion I recommend you to delete all the following code:
W = tf.Variable(tf.truncated_normal([4, 4, 3, 3], stddev=0.1))
B = tf.Variable(tf.ones([3])/10) # What should I put here ??
init = tf.global_variables_initializer()
# model
#Was:
# Y = tf.nn.softmax(tf.matmul(tf.reshape(X, [-1, input]), W) + b)
#Now:
stride = 1 # output is still 28x28
Ycnv = tf.nn.conv2d(X, W, strides=[1, stride, stride, 1], padding='SAME')
Y = tf.nn.relu(Ycnv + B)
and substitute it by:
conv1 = tf.layers.conv2d(X, filters=64, kernel_size=3,
strides=1, padding='SAME',
activation=tf.nn.relu, name="conv1")
Also the init = tf.global_variable_initializer() should be at the end of the graph construction becuase, if not, there will be variables it won't catch.
My final working code is:
def simple_nn():
inp = 100*100*3
batch_size = 2
X = tf.placeholder(tf.float32, [batch_size, 100, 100, 3])
Y_ = tf.placeholder(tf.float32, [None, 2])
#Was:
# W = tf.Variable(tf.zeros([input, 2]))
# b = tf.Variable(tf.zeros([2]))
#Now:
# model
#Was:
# Y = tf.nn.softmax(tf.matmul(tf.reshape(X, [-1, input]), W) + b)
#Now:
stride = 1 # output is still 28x28
conv1 = tf.layers.conv2d(X, filters=64, kernel_size=3,
strides=1, padding='SAME',
activation=tf.nn.relu, name="conv1")
Y = tf.reshape(conv1, [1,-1])
logits = tf.layers.dense(Y, units=2, activation=tf.nn.relu)
# placeholder for correct labels
# loss function
cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(labels=Y_, logits=logits)
loss = tf.reduce_mean(cross_entropy)
# % of correct answers found in batch
is_correct = tf.equal(tf.argmax(Y,1), tf.argmax(Y_,1))
accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))
learning_rate = 0.00001
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
train_step = optimizer.minimize(cross_entropy)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
...
My network transposes an image, with size 62*71, to a vector of 124 outputs. In the test, I got the same output for each input. I checked 4000 cases.
I cannot seem to signify the problem because the learning seems to be fine, there is an improvement of the error and get a relatively low error.
Someone maybe knows what is the problem?
#load data
data_in= np.transpose(np.loadtxt("images_in_10000.csv", delimiter=',',dtype=np.float32))
data_out= np.transpose(np.loadtxt("out_to_image_10000.csv", delimiter=',',dtype=np.float32))
x_train = data_in[0:6000, :]
x_test = data_in[6000:10001,:]
y_train = data_out[0:6000, :]
y_test = data_out[6000:10001, :]
#parametersa
batch=100
epochs=7
learning_rate=0.01
n = x_test.shape[1] #4392
m = x_train.shape[0] #6000
d = y_test.shape[1] #124
l = y_test.shape[0] #4000
trainX = tf.placeholder(tf.float32, [batch, n])
trainY = tf.placeholder(tf.float32, [batch, d])
testX = tf.placeholder(tf.float32, [l, n])
testY = tf.placeholder(tf.float32, [l, d])
W_c1= tf.Variable(tf.random_normal([5, 5, 1, 32]))
W_c2= tf.Variable(tf.random_normal([5, 5, 32, 64]))
W_fc= tf.Variable(tf.random_normal([18 * 16 * 64, 128]))
W_out= tf.Variable(tf.random_normal([128, d]))
b_c1= tf.Variable(tf.random_normal([32]))
b_c2=tf.Variable(tf.random_normal([64]))
b_fc=tf.Variable(tf.random_normal([128]))
b_out=tf.Variable(tf.random_normal([d]))
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def maxpool2d(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
def convolutional_neural_network(x):
x = tf.reshape(x, shape=[-1,61,72, 1])
conv1 = tf.nn.relu(conv2d(x, W_c1) + b_c1)
conv1 = maxpool2d(conv1)
conv2 = tf.nn.relu(conv2d(conv1, W_c2) + b_c2)
conv2 = maxpool2d(conv2)
fc = tf.reshape(conv2, [-1, 18 * 16 * 64])
fc = tf.nn.relu(tf.matmul(fc, W_fc) + b_fc)
output = tf.matmul(fc, W_out) + b_out
return output
prediction = convolutional_neural_network(trainX)
cost =tf.reduce_mean(tf.pow(prediction-trainY,2))
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)
prediction_t = convolutional_neural_network(testX)
losstest = tf.reduce_mean(tf.pow(prediction_t - testY, 2))
k=0
a = np.linspace(0, m - batch, m / batch, dtype=np.int32)
costshow = [0] * (len(a) * epochs)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for epoch in range(epochs):
epoch_loss = 0
for i in (np.linspace(0,m - batch, m / batch, dtype=np.int32)):
x = x_train[i:i + batch, :]
y = y_train[i:i + batch, :]
sess.run(optimizer, feed_dict={trainX: x, trainY: y})
cost_val = sess.run(cost, feed_dict={trainX: x, trainY: y})
costshow[k]=cost_val
print("Epoch=", '%04d' % (epoch + 1), "loss=", " {:.9f}".format(cost_val))
k = k + 1
print("finsh train-small ")
result = sess.run(prediction_t, feed_dict={testX: x_test})
test_loss = sess.run(losstest, feed_dict={testX: np.asarray(x_test), testY: np.asarray(y_test)})
print("Testing loss=", test_loss)
The metric behind a picture is clearly defined. The values of an image often ranges from 0-1 or 0-255. For CNN's you should normalize your input values (0-1).
Thus you have to be careful with your weight initialization. For example, if your have a bias of 0.6 and a value of 0.6, you get a 1.2 as image value and your plotting program thinks you are in the 0-255 range and everything is black.
So try to use the glorot-initializer for the weights and zero-initializer for the bias initializer:
Weights:
tf.get_variable("weight", shape=[5, 5, 1, 32], initializer=tf.glorot_uniform_initializer())
Bias:
tf.get_variable("bias", shape=[32], initializer=tf.zeros_initializer())
Furthermore, tf.Variabel is deprecated. It is better to use tf.get_variable.
This question may have been asked but I failed to find it.
What is the simplest way to constantly get batches of data from a dataset? Is there a built-in tensorflow function to do so?
for instance:
for i in num_trains:
x_batch, y_batch = get_batch(x_train, y_train, batch_size)
sess.run(train_step, feed_dict={x:x_batch,y:y_batch})
If there is not such a built in function, how would you implement it? I tried myself but I could not figure out how I can get a new batch different from the previous ones each time I call the function.
Thanks!
You can try:
# Feed batch data
def get_batch(inputX, inputY, batch_size):
duration = len(inputX)
for i in range(0,duration//batch_size):
idx = i*batch_size
yield inputX[idx:idx+batch_size], inputY[idx:idx+batch_size]
You can also use tensorflow's dataset API as well:
dataset = tf.data.Dataset.from_tensor_slices((train_x, train_y))
dataset = dataset.batch(batch_size)
Getting the batch:
X = np.arange(100)
Y = X
batch = get_batch(X, Y, 5)
batch_x, batch_y = next(batch)
print(batch_x, batch_y)
#[0 1 2 3 4] [0 1 2 3 4]
batch_x, batch_y = next(batch)
print(batch_x, batch_y)
#[5 6 7 8 9] [5 6 7 8 9]
Typically for running over the dataset for multiple epochs, you would do:
for epoch in range(number of epoch):
for step in range(size_of_dataset//batch_size):
for x_batch, y_batch in get_batch(x_train, y_train, batch_size):
sess.run(train_step, feed_dict={x:x_batch,y:y_batch})
Using the dataset API:
dataset = tf.data.Dataset.from_tensor_slices((X, Y))
dataset = dataset.batch(5)
iterator = dataset.make_initializable_iterator()
train_x, train_y = iterator.get_next()
with tf.Session() as sess:
sess.run(iterator.initializer)
for i in range(2):
print(sess.run([train_x, train_y]))
#[array([0, 1, 2, 3, 4]), array([0, 1, 2, 3, 4])]
#[array([5, 6, 7, 8, 9]), array([5, 6, 7, 8, 9])]
My network starts to learn and looks okay on the first batch and then suddenly stops with TypeError on the second batch only! Why was it okay on the first batch then? Or why did it break after the first? Stupefying error... Here are the details:
I have built a CNN that is trying to predict 124 features for each image. The images are of size 61 x 72 pixels and the output vector of numbers are of size 124 x 1.
The images are floating point matrices with numbers between 1 and -1.
The information I'm trying to predict is in a CSV file, with each line describing an image. When I load the data for the training process I process each line and reshape them, also get the pictures the network is learning.
When I run my program, I get the following error on the second batch, however:
"TypeError: Fetch argument 2.7674865e+09 has invalid type , must be a string or Tensor. (Can not convert a float32 into a Tensor or Operation.)"
Can you please help pinpoint what the problem is? Here's my code:
import tensorflow as tf
import numpy as np
data_in=np.loadtxt(open("images.csv"), delimiter=',',dtype=np.float32);
data_out=np.loadtxt(open("outputmix-124.csv"),
delimiter=',',dtype=np.float32);
x_train = data_in[0:6000, :]
x_test = data_in[6000:10000,:]
y_train = data_out[0:6000, :]
y_test = data_out[6000:10000, :]
batch=600
epochs=10
n = x_test.shape[1] #4392
m = x_train.shape[0] #6000
d = y_test.shape[1] #124
l = y_test.shape[0] #4000
trainX= tf.placeholder(tf.float32, [batch, n], name="X")
trainY = tf.placeholder(tf.float32, [batch, d])
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def maxpool2d(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],
padding='SAME')
def convolutional_neural_network(x):
weights = {'W_c1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
'W_c2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
'W_fc': tf.Variable(tf.random_normal([18 * 16 * 64, 1024])),
'out': tf.Variable(tf.random_normal([1024, d]))}
biases = {'b_c1': tf.Variable(tf.random_normal([32])),
'b_c2': tf.Variable(tf.random_normal([64])),
'b_fc': tf.Variable(tf.random_normal([1024])),
'out': tf.Variable(tf.random_normal([d]))}
x = tf.reshape(x, shape=[-1,61,72, 1])
conv1 = tf.nn.relu(conv2d(x, weights['W_c1']) + biases['b_c1'])
conv1 = maxpool2d(conv1)
conv2 = tf.nn.relu(conv2d(conv1, weights['W_c2']) + biases['b_c2'])
conv2 = maxpool2d(conv2)
fc = tf.reshape(conv2, [-1, 18 * 16 * 64])
fc = tf.nn.relu(tf.matmul(fc, weights['W_fc']) + biases['b_fc'])
fc = tf.nn.dropout(fc, keep_rate)
output = tf.matmul(fc, weights['out']) + biases['out']
return output
def train_neural_network(x):
prediction = convolutional_neural_network(x)
cost =tf.reduce_mean(tf.pow(prediction-trainY,2))
optimizer = tf.train.AdamOptimizer().minimize(cost)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for epoch in range(epochs):
epoch_loss = 0
for i in (np.linspace(0,m - batch, m / batch, dtype=np.int32)):
x = x_train[i:i + batch, :]
y = y_train[i:i + batch, :]
sess.run(optimizer, feed_dict={trainX: x, trainY: y})
cost = sess.run(cost, feed_dict={trainX: x, trainY: y})
print("Epoch=", '%04d' % (epoch + 1), "loss=", "
{:.9f}".format(cost))
epoch_loss += cost
print('Epoch', epoch, 'completed out of', epochs, 'loss:',
epoch_loss)
train_neural_network(trainX)
This is a fairly typical mistake. The problem is with the variable cost. First you assign the loss calculation tensor to it in the second line of the function train_neural_network():
cost =tf.reduce_mean(tf.pow(prediction-trainY,2))
Then when you run the training and the cost calculation, you do this, and this is where it gets messed up:
cost = sess.run(cost, feed_dict={trainX: x, trainY: y})
because you assign the value of the loss to cost, which is now a simple floating point number, instead of a Tensor. The next time around sess.run() gets a floating point number instead of a tensor as a first argument, and the error above is printed.
Use something like cost_val for storing the value of the loss and leave cost to store the reference to the tensor. You of course need to update the lines that print the value as well, so these three lines I've changed:
cost_val = sess.run(cost, feed_dict={trainX: x, trainY: y})
print("Epoch=", '%04d' % (epoch + 1), "loss=", " {:.9f}".format(cost_val))
epoch_loss += cost_val
I'm posting the full revised version here (tested code; note I've generated test data instead of loading; this is a loadable and testable example for anyone but you need to change it back to load your actual data):
import tensorflow as tf
import numpy as np
keep_rate = 0.8
#data_in=np.loadtxt(open("images.csv"), delimiter=',',dtype=np.float32);
#data_out=np.loadtxt(open("outputmix-124.csv"),
# delimiter=',',dtype=np.float32);
data_in = np.random.normal( size = ( 10000, 4392 ) )
data_out = np.random.normal( size = ( 10000, 124 ) )
x_train = data_in[0:6000, :]
x_test = data_in[6000:10000,:]
y_train = data_out[0:6000, :]
y_test = data_out[6000:10000, :]
batch=600
epochs=10
n = x_test.shape[1] #4392
m = x_train.shape[0] #6000
d = y_test.shape[1] #124
l = y_test.shape[0] #4000
trainX = tf.placeholder(tf.float32, [batch, n], name="X")
trainY = tf.placeholder(tf.float32, [batch, d])
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def maxpool2d(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],
padding='SAME')
def convolutional_neural_network(x):
weights = {'W_c1': tf.Variable(tf.random_normal([5, 5, 1, 32])),
'W_c2': tf.Variable(tf.random_normal([5, 5, 32, 64])),
'W_fc': tf.Variable(tf.random_normal([18 * 16 * 64, 1024])),
'out': tf.Variable(tf.random_normal([1024, d]))}
biases = {'b_c1': tf.Variable(tf.random_normal([32])),
'b_c2': tf.Variable(tf.random_normal([64])),
'b_fc': tf.Variable(tf.random_normal([1024])),
'out': tf.Variable(tf.random_normal([d]))}
x = tf.reshape(x, shape=[-1,61,72, 1])
conv1 = tf.nn.relu(conv2d(x, weights['W_c1']) + biases['b_c1'])
conv1 = maxpool2d(conv1)
conv2 = tf.nn.relu(conv2d(conv1, weights['W_c2']) + biases['b_c2'])
conv2 = maxpool2d(conv2)
fc = tf.reshape(conv2, [-1, 18 * 16 * 64])
fc = tf.nn.relu(tf.matmul(fc, weights['W_fc']) + biases['b_fc'])
fc = tf.nn.dropout(fc, keep_rate)
output = tf.matmul(fc, weights['out']) + biases['out']
return output
def train_neural_network(x):
prediction = convolutional_neural_network(x)
cost =tf.reduce_mean(tf.pow(prediction-trainY,2))
optimizer = tf.train.AdamOptimizer().minimize(cost)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for epoch in range(epochs):
epoch_loss = 0
for i in (np.linspace(0,m - batch, m / batch, dtype=np.int32)):
x = x_train[i:i + batch, :]
y = y_train[i:i + batch, :]
sess.run(optimizer, feed_dict={trainX: x, trainY: y})
cost_val = sess.run(cost, feed_dict={trainX: x, trainY: y})
print("Epoch=", '%04d' % (epoch + 1), "loss=", " {:.9f}".format(cost_val))
epoch_loss += cost_val
print('Epoch', epoch, 'completed out of', epochs, 'loss:',
epoch_loss)
train_neural_network(trainX)