queue reader with tf.py_func yields memory leak - memory-leaks

Question

I am trying to write a queue-reader that walks through a large file and runs a python function on each line before passing it to the actual operation.
I use a string_input_producer to read for a single .tsv file. Then I create a queue with tf.TextLineReader and enhance each line with tf.py_func. Doing this, I notice some memory leakage that only comes into effect if tf.py_func is called (yes, even as a noop).
Running the below code yields the following result:
$ python test_memory.py 2> /dev/null
run WITHOUT tf.py_func
00001/50000, 1.4260% mem
05001/50000, 1.4512% mem
10001/50000, 1.4512% mem
15001/50000, 1.4512% mem
20001/50000, 1.4512% mem
25001/50000, 1.4516% mem
30001/50000, 1.4516% mem
35001/50000, 1.4516% mem
40001/50000, 1.4516% mem
45001/50000, 1.4516% mem
50000/50000, 1.4516% mem
===========================
run WITH tf.py_func
00001/50000, 1.4975% mem
05001/50000, 1.5051% mem
10001/50000, 1.5066% mem
15001/50000, 1.5081% mem
20001/50000, 1.5110% mem
25001/50000, 1.5137% mem
30001/50000, 1.5148% mem
35001/50000, 1.5165% mem
40001/50000, 1.5195% mem
45001/50000, 1.5210% mem
50000/50000, 1.5235% mem
===========================
As you can see, running the code without tf.py_func keeps the used memory stable, whereas running it WITH the python function makes it constantly increase. this effect is way more pronounced on files with larger rows.
test_memory.py:
import os
import sys
import psutil
import tensorflow as tf
def py_funner(x, do_py=True):
'''
this function returns the exact input.
if do_py==True, it passes the data through a python noop using tf.py_func
'''
if do_py:
def py_func(y):
# this is just another noop.
return y
# py_func wraps a python function as a tensorflow op.
return tf.py_func(py_func, [x], [tf.string], stateful=False)[0]
else:
return x
def get_data(do_py=True):
# take the code as input. the effect is way more pronounced on larger files,
# e.g., a tsv that encode image data in base64, as for ms-celeb-1m
in_str = os.__file__
# produce a queue that reads the one file row by row.
input_queue = tf.train.string_input_producer([in_str])
reader = tf.TextLineReader()
ind, row = reader.read(input_queue)
# call the wrapper to either include tf.py_func or not.
return py_funner(row, do_py=do_py)
def main():
# get the current proccess to monitor memory usage
process = psutil.Process(os.getpid())
# execute the same code both with a tf.py_func noop and without it
for tt in [False, True]:
print 'run WITH%s tf.py_func'%('' if tt else 'OUT')
# generate the data queue
data = get_data(do_py=tt)
# start the session and the queue coordinator
sess = tf.Session()
coord = tf.train.Coordinator()
queue_threads = tf.train.start_queue_runners(sess, coord=coord)
# read a lot of the file
max_iter = 50000
for i in range(max_iter):
run_ops = [data]
d = sess.run(run_ops)
mem = process.memory_percent()
print '\r%05d/%d, %.4f%% mem'%(i+1, max_iter, mem),
sys.stdout.flush()
if i%5000==0:
print
print '\n==========================='
if __name__=='__main__':
main()
I'm grateful for any kind of pointers or ideas how to further debug this?! Maybe is there a way to see whether the python function keeps a storage of some kind?
thanks!!