SLURM, using srun to print outputs - slurm

Question

I am using srun to run my program, however, it cannot print the output.
me#home:~$ srun -p K80q --gres=gpu:1 -N 1 python3 main.py
2019-05-15 19:56:43.305156: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-15 19:56:43.543516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-05-15 19:56:43.543567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-15 19:56:43.900189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-15 19:56:43.900248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-15 19:56:43.900257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-15 19:56:43.900619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10761 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)
I only got the above output and it cannot print the information I expected. How can I fix it?
By the way, simply define a test code
import tensorflow
if __name__ == '__main__':
for i in range(10):
print('Hello')
It can print Hello 10 times.
Update:
After 20 minutes, it outputs some information I expected. How can I make it output immediately?

Answer 1

Try the -u option of srun:
-u, --unbuffered
By default the connection between slurmstepd and the user launched application is over a pipe. The stdio output written by
the application is
buffered by the glibc until it is flushed or the output is set as unbuffered. See setbuf(3). If this option is specified the
tasks are executed
with a pseudo terminal so that the application output is unbuffered.