PaddingFIFOQueue.enqueue_many throwing Value Error: Shapes must be equal rank - python-3.x

Question

I have an RNN model, to which I'm trying to feed in varying length input sequences through input pipelines randomly sampling from multiple TF record files containing serialized SequenceExamples, batch padding and shuffling across multiple batches
each sequence example has 3 elements, as length: constant, input:1-D array, labels:1-D array
Procedure is as follows
def read_file_queue(self,filename_queue):
reader = tf.TFRecordReader()
key, ex = reader.read(filename_queue)
context_features = {
"seq-len": tf.FixedLenFeature([],dtype=tf.int64)
}
sequence_features = {
"tokens": tf.FixedLenSequenceFeature([],dtype=tf.int64),
"labels": tf.FixedLenSequenceFeature([],dtype=tf.int64)
}
context_parsed, sequence_parsed = tf.parse_single_sequence_example(serialized=ex,
context_features=context_features,
sequence_features=sequence_features)
return context_parsed["seq-len"], sequence_parsed["tokens"],sequence_parsed["labels"]
def get_batch_data(self):
fqueue = tf.train.string_input_producer(self.data_filelist,
shuffle=True,
num_epochs=self.num_epochs)
# read from multiple tf records as defined by read_threads
ex = [self.read_file_fmt(fqueue) for _ in range(self.read_threads)]
print(ex)
# ex = self.read_file_fmt(fqueue)
pad_output = self.padding_pipeline(ex)
shuffle_output = self.shuffle_pipeline(pad_output)
return shuffle_output
def padding_pipeline(self, input):
padding_queue = tf.PaddingFIFOQueue(
capacity=self.pad_capacity,
dtypes=[tf.int64, tf.int64, tf.int64],
shapes=[[], [None], [None]])
# use enqueue_many instead enqueue because
# the input is list of tuples from each tf record reader thread
padding_enqueue_op = padding_queue.enqueue_many(input) # <<< !!!!! error here !!!!!
padding_queue_runner = tf.train.QueueRunner(padding_queue, [padding_enqueue_op] * self.pad_threads)
tf.train.add_queue_runner(padding_queue_runner)
padding_dequeue_op = padding_queue.dequeue_up_to(self.batch_size)
return padding_dequeue_op
def shuffle_pipeline(self,input):
shuffle_queue = tf.RandomShuffleQueue(
capacity=self.shuffle_capacity,
min_after_dequeue=self.shuffle_min_after_dequeue,
dtypes=[tf.int64, tf.int64, tf.int64],
shapes=None)
shuffle_enqueue_op = shuffle_queue.enqueue(input)
shuffle_queue_runner = tf.train.QueueRunner(
shuffle_queue, [shuffle_enqueue_op] * self.shuffle_threads)
tf.train.add_queue_runner(shuffle_queue_runner)
shuffle_dequeue_op = shuffle_queue.dequeue()
return shuffle_dequeue_op
For which I'm getting the following error:
ValueError: Shapes must be equal rank, but are 0 and 1 From merging
shape 0 with other shapes. for
'padding_fifo_queue_EnqueueMany/component_0' (op: 'Pack') with input
shapes: [], [?], [?].
I'm sure I'm doing something silly here, however, I could not find what is that im doing wrong..

Answer 1

Taking a hint from here, maybe you should have the following?
padding_queue = tf.PaddingFIFOQueue(
capacity=self.pad_capacity,
dtypes=[tf.int64, tf.int64, tf.int64],
shapes=[None, [None], [None]])
By the way, if you could add some basic script for generating random data in the format you are using, it would be easier to replicate. Thanks.