Restoring saved model in Tensorflow v1.14 - python-3.x

Question

I am using tensorflow v1.14. I have a saved model and I'm trying to restore the model using the following code:
loader = tf.train.import_meta_graph("models/fcnn0/model.ckpt.meta")
graph = tf.get_default_graph()
sess = tf.Session()
loader.restore(sess, "models/fcnn0/model.ckpt")
I used to use the same piece of code in Tensorflow v1.13 and it used to work without errors. But now I'm getting the error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.DataLossError: file is too short to be an sstable
[[{{node save/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sandesh/PycharmProjects/fading/finding_code/src/load_32_64.py", line 8, in <module>
loader.restore(sess, "models/fcnn_32_64_aenc_1331_747_3870000/model.ckpt")
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: file is too short to be an sstable
[[node save/RestoreV2 (defined at home/sandesh/PycharmProjects/fading/finding_code/src/load_32_64.py:5) ]]
Original stack trace for 'save/RestoreV2':
File "home/sandesh/PycharmProjects/fading/finding_code/src/load_32_64.py", line 5, in <module>
loader = tf.train.import_meta_graph("models/fcnn_32_64_aenc0/model.ckpt.meta")
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1449, in import_meta_graph
**kwargs)[0]
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1473, in _import_meta_graph_with_return_elements
**kwargs))
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/meta_graph.py", line 857, in import_scoped_meta_graph_with_return_elements
return_elements=return_elements)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def
_ProcessNewOps(graph)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3751, in <listcomp>
for c_op in c_api_util.new_tf_operations(self)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
Can someone point me as to what I'm doing wrong? Thanks in advance.

Answer 1

I was looking into the folder where the model files were saved and found that the model.ckpt.meta file had not been written to disk properly. I reran the training and saved the model and then it worked perfectly.