Implementing Wide and Deep Neural Network using Functional API - python-3.x

Question

I am trying to build a Wide and Deep Neural Network using Keras Functional API. I am getting a value to shape mismatch error. I don't understand where I am wrong. I am implementing this on the Fashion MNIST dataset.The X_train shape is (60000,28,28) and Y_train is (60000,). I am guessing that the error is because of the line : input_ = keras.layers.... but I don't understand how to resolve it.
Code :
# Building a Non Sequnetial Model using Functional API One Use of it is in Wide and Deep Neural Networks
input_ = keras.layers.Input(shape=X_train.shape[1:]) # This will return shape of the input [28,28],remeber we dont have to set it to the number of neurons in the layer
hidden1 = keras.layers.Dense(100,activation = "relu")(input_) # We have to call it as a function
hidden2 = keras.layers.Dense(100,activation = "relu")(hidden1)
concat_layer = keras.layers.concatenate([input_,hidden2])
output = keras.layers.Dense(10,activation="softmax")(concat_layer)
model = keras.models.Model(inputs=[input_], outputs=[output])
model.compile(loss = keras.losses.sparse_categorical_crossentropy,optimizer = keras.optimizers.SGD(lr = 0.8),metrics= ["accuracy"])
Tensorboard_cb = keras.callbacks.TensorBoard(Path_Tensor)
model.fit(X_train,Y_train,validation_split=0.2,epochs=100,callbacks=[Tensorboard_cb])
Error :
Epoch 1/100
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-97-675a2b302d27> in <module>
----> 1 model.fit(X_train,Y_train,validation_split=0.2,epochs=100,callbacks=[Tensorboard_cb])
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\training.py in _method_wrapper(self, *args, **kwargs)
106 def _method_wrapper(self, *args, **kwargs):
107 if not self._in_multi_worker_mode(): # pylint: disable=protected-access
--> 108 return method(self, *args, **kwargs)
109
110 # Running inside `run_distribute_coordinator` already.
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1096 batch_size=batch_size):
1097 callbacks.on_train_batch_begin(step)
-> 1098 tmp_logs = train_function(iterator)
1099 if data_handler.should_sync:
1100 context.async_wait()
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\def_function.py in __call__(self, *args, **kwds)
778 else:
779 compiler = "nonXla"
--> 780 result = self._call(*args, **kwds)
781
782 new_tracing_count = self._get_tracing_count()
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
821 # This is the first call of __call__, so we have to initialize.
822 initializers = []
--> 823 self._initialize(args, kwds, add_initializers_to=initializers)
824 finally:
825 # At this point we know that the initialization is complete (or less
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\def_function.py in _initialize(self, args, kwds, add_initializers_to)
695 self._concrete_stateful_fn = (
696 self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access
--> 697 *args, **kwds))
698
699 def invalid_creator_scope(*unused_args, **unused_kwds):
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
2853 args, kwargs = None, None
2854 with self._lock:
-> 2855 graph_function, _, _ = self._maybe_define_function(args, kwargs)
2856 return graph_function
2857
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\function.py in _maybe_define_function(self, args, kwargs)
3211
3212 self._function_cache.missed.add(call_context_key)
-> 3213 graph_function = self._create_graph_function(args, kwargs)
3214 self._function_cache.primary[cache_key] = graph_function
3215 return graph_function, args, kwargs
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
3073 arg_names=arg_names,
3074 override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075 capture_by_value=self._capture_by_value),
3076 self._function_attributes,
3077 function_spec=self.function_spec,
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
984 _, original_func = tf_decorator.unwrap(python_func)
985
--> 986 func_outputs = python_func(*func_args, **func_kwargs)
987
988 # invariant: `func_outputs` contains only Tensors, CompositeTensors,
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\eager\def_function.py in wrapped_fn(*args, **kwds)
598 # __wrapped__ allows AutoGraph to swap in a converted function. We give
599 # the function a weak reference to itself to avoid a reference cycle.
--> 600 return weak_wrapped_fn().__wrapped__(*args, **kwds)
601 weak_wrapped_fn = weakref.ref(wrapped_fn)
602
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\func_graph.py in wrapper(*args, **kwargs)
971 except Exception as e: # pylint:disable=broad-except
972 if hasattr(e, "ag_error_metadata"):
--> 973 raise e.ag_error_metadata.to_exception(e)
974 else:
975 raise
ValueError: in user code:
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\training.py:806 train_function *
return step_function(self, iterator)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\training.py:796 step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1211 run
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2585 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2945 _call_for_each_replica
return fn(*args, **kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\training.py:789 run_step **
outputs = model.train_step(data)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\training.py:749 train_step
y, y_pred, sample_weight, regularization_losses=self.losses)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\compile_utils.py:204 __call__
loss_value = loss_obj(y_t, y_p, sample_weight=sw)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\losses.py:149 __call__
losses = ag_call(y_true, y_pred)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\losses.py:253 call **
return ag_fn(y_true, y_pred, **self._fn_kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\util\dispatch.py:201 wrapper
return target(*args, **kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\losses.py:1567 sparse_categorical_crossentropy
y_true, y_pred, from_logits=from_logits, axis=axis)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\util\dispatch.py:201 wrapper
return target(*args, **kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\backend.py:4783 sparse_categorical_crossentropy
labels=target, logits=output)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\util\dispatch.py:201 wrapper
return target(*args, **kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\nn_ops.py:4176 sparse_softmax_cross_entropy_with_logits_v2
labels=labels, logits=logits, name=name)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\util\dispatch.py:201 wrapper
return target(*args, **kwargs)
c:\users\na462\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\nn_ops.py:4091 sparse_softmax_cross_entropy_with_logits
logits.get_shape()))
ValueError: Shape mismatch: The shape of labels (received (32, 1)) should equal the shape of logits except for the last dimension (received (32, 28, 10)).

Answer 1

you forgot to add Flatten Layer, you input is 3D (None,28,28) but your output is only 2D (None,10), since you did not Flatten so you are getting (None, 28, 10) as out which is wrong.
also i think you did not use one hot encoding, because i can see (32,1) as your output shape rather then (32, 10). you need to change your last Dense layer neuron to 1, or use one hot encoding.
if you want to use Dense(1) then you have to make sure that your mnist dataset output is numeric, such as 0,1,2,3,4..9. but if you want to use Dense(10) then you have to use onehot encoding, your output will be [1,0,0,0,0,0,0,0,0,0] this is equal to 0 in decimal number system. [0,1,0,0,0,0,0,0,0,0] this is equal to 1 in decimal number system. it will have ten columns, each represent single decimal number, if a number in a column is 1 which mean it is that decimal number.
1st column mean 0, [1,0,0,0,0,0,0,0,0,0]
2nd column mean 1 [0,1,0,0,0,0,0,0,0,0]
3rd column mean 2 [0,0,1,0,0,0,0,0,0,0]
so on
last column mean 9 [0,0,0,0,0,0,0,0,0,1]
as for flatten layer, if you input is image then image has width, height and color, which are 3 dimensions. but as you can see above your output is either a single number or 10 columns of 0s and 1s. in order to make 3 dimension in your 1 dimension output you have to use flatten layer. also there is another hidden dimension which represent your data samples, or batch size. in your case it is 32. so your input is (32, 28, 28). and your output is (32, 10).
another thing is you can also use flatten layer first after you feed input. you can give input in the form of (None, 784) here 784 is (28 multiple 28)