I am trying to write a shell that needs to run socket connections on a seperate thread. On my testings, when print() is used while cmd.Cmd.cmdloop() waiting for input, the print is displaying wrong.
from core.shell import Shell
import time
import threading
def test(shell):
time.sleep(2)
shell.write('Doing test')
if __name__ == '__main__':
shell = Shell(None, None)
testThrd = threading.Thread(target=test, args=(shell,))
testThrd.start()
shell.cmdloop()
When the above command runs, here is what happens:
python test.py
Welcome to Test shell. Type help or ? to list commands.
>>asd
*** Unknown syntax: asd
>>[17:59:25] Doing test
As you can see, printing from another threads add output after prompt >> not in a new line. How can I do it so that it appears in a new line and prompt appears?
What you can do, is redirect stdout from your core.shell.Shell to a file like object such as StringIO. You would also redirect the output from your thread into a different file like object.
Now, you can have some third thread read both of these objects and print them out in whatever fashion you want.
You said core.shell.Shell inherits from cmd.Cmd, which allows redirection as a parameter to the constructor:
import io
import time
import threading
from core.shell import Shell
def test(output_obj):
time.sleep(2)
print('Doing test', file=output_obj)
cmd_output = io.StringIO()
thr_output = io.StringIO()
shell = Shell(stdout=cmd_output)
testThrd = threading.Thread(target=test, args=(thr_output,))
testThrd.start()
# in some other process/thread
cmd_line = cmd_output.readline()
thr_line = thr_output.readline()
That's quite difficult. Both your threads are sharing the same stdout. So the output from each of those threads are concurrently sent to your stdout buffer where they are printed in some arbitrary order.
What you need to do is coordinate the output from both threads, and that's a tough nut to crack. Even bash doesn't do that!
That said, maybe you can try using a lock to make sure your threads access stdout in a controlled manner. Check out: http://effbot.org/zone/thread-synchronization.htm
Related
My case is a little bit specific. I'm trying to run a Python program using Python for testing purposes. The case is as follows:
# file1.py
print("Hello world")
# file1.test.py
import io
import sys
import os
import unittest
EXPECTED_OUTPUT = "Hello world"
class TestHello(unittest.TestCase):
def test_hello(self):
sio = io.StringIO()
sys.stdout = sio
os.system("python3 path/to/file1.py")
sys.stdout = sys.__stdout__
print("captured value:", sio.getvalue())
self.assertEqual(sio.getvalue(), EXPECTED_STDOUT)
if __name__ == "__main__":
unittest.main()
But nothing ends up in the sio variable. This way and similar ways are introduced online but they don't seem to work for me. My Python version is 3.8.10 but it doesn't really matter if this works better in some other version, I can switch to that.
Note: I know that if I was using an importable object this might be easier, but right now I need to know how to catch the output of another file.
Thanks!
stdout redirection does not work like this - this will change the stdout variable inside your Python process. But by using os.system, you are running another process, that will re-use the same terminal pseudo-files your parent process is using.
If you want to log a subprocess, the way to do it is to use the subprocess modules calls, which allow you to redirect the subprocess output. https://docs.python.org/3/library/subprocess.html
Also, the subprocess won't be able to use a StringIO object from the parent process (it is not an O.S. level object, just an in-process Python object with a write method). The docs above include instructions about using the special object subprocess.PIPE which allows for in-memory communication, or, you can just pass an ordinary filesystem file, which you can read afterwards.
I am starting two processes via multiprocessing and this is working fine. The only problems which I have are the print and debug statements from these two processes.
The hope is, to use the REPL and start the processes, like in the background. However, I do not get this to run. I always get the debug statements and therefore can't use the REPL anymore. This is how I call the processes:
processes = [
Process(target=start_viewer, args=()),
Process(target=start_server, args=(live, amount, fg))
]
for p in processes:
p.start()
Any idea on how to "mute" the process, or get them in the background?
If I correct understand you, you want to not show printing from one of processes.
You can achieve this by redirect output of the Python Interpreter.
Add sys.stdout = open("/dev/null", 'w') to the process which you want to "mute".
Full working example below.
from multiprocessing import Process
from time import sleep
import sys
def start_viewer():
sys.stdout = open("/dev/null", 'w')
while True:
print("start_viewer")
sleep(1)
def start_server():
while True:
print("start_server")
sleep(1)
if __name__ == '__main__':
processes = [
Process(target=start_viewer, args=()),
Process(target=start_server, args=())
]
for p in processes:
p.start()
Be aware that /dev/null is like passing prints to nowhere, if you want to save it you can use text file. Also to achieve multi os support you should use os.devnull.
Consider the following function
import time
def foo():
for i in range(5):
print(f"{i}. Hello world!")
time.sleep(1)
I would like to save all these print calls in a variable without preventing them from reaching the terminal in real time. Essentially, print would output to stdout and a variable.
I have tried:
from contextlib import redirect_stdout
import io
stdout = io.StringIO()
with redirect_stdout(stdout):
foo()
stdout_content = stdout.getvalue()
print(stdout_content)
However, this blocks printing to the terminal until foo returns.
I would like foo to keep printing to the terminal in real time while an object stores the calls.
How can this be achieved?
One approach is to provide your own file like object to redirect_stdout. Your object class will implement the write by writing both to a file and the original sys.stdout.
You can read about sys.stdout here.
You can see the various file like classes here for solid examples.
I'm using Python 3.7.4 and I have created two functions, the first one executes a callable using multiprocessing.Process and the second one just prints "Hello World". Everything seems to work fine until I try redirecting the stdout, doing so prevents me from getting any printed values during the process execution. I have simplified the example to the maximum and this is the current code I have of the problem.
These are my functions:
import io
import multiprocessing
from contextlib import redirect_stdout
def call_function(func: callable):
queue = multiprocessing.Queue()
process = multiprocessing.Process(target=lambda:queue.put(func()))
process.start()
while True:
if not queue.empty():
return queue.get()
def print_hello_world():
print("Hello World")
This works:
call_function(print_hello_world)
The previous code works and successfully prints "Hello World"
This does not work:
with redirect_stdout(io.StringIO()) as out:
call_function(print_hello_world)
print(out.getvalue())
With the previous code I do not get anything printed in the console.
Any suggestion would be very much appreciated. I have been able to narrow the problem to this point and I think is related to the process ending after the io.StringIO() is already closed but I have no idea how to test my hypothesis and even less how to implement a solution.
This is the workaround I found. It seems that if I use a file instead of a StringIO object I can get the things to work.
with open("./tmp_stdout.txt", "w") as tmp_stdout_file:
with redirect_stdout(tmp_stdout_file):
call_function(print_hello_world)
stdout_str = ""
for line in tmp_stdout_file.readlines():
stdout_str += line
stdout_str = stdout_str.strip()
print(stdout_str) # This variable will have the captured stdout of the process
Another thing that might be important to know is that the multiprocessing library buffers the stdout, meaning that the prints only get displayed after the function has executed/failed, to solve this you can force the stdout to flush when needed within the function that is being called, in this case, would be inside print_hello_world (I actually had to do this for a daemon process that needed to be terminated if it ran for more than a specified time)
sys.stdout.flush() # This will force the stdout to be printed
import multiprocessing as mp
import time
"""
1. send item via pipe
2. Receive on the other end by a generator
3. if the pipe is closed on the sending side, retrieve
all item left and then quit.
"""
def foo(conn):
for i in range(7):
time.sleep(.3)
conn.send(i)
conn.close()
def bar(conn):
while True:
try:
yield conn.recv()
except EOFError:
break
if __name__ == '__main__':
"""Choose which start method is used"""
recv_conn, send_conn = mp.Pipe(False)
p = mp.Process(target = foo, args = (send_conn,)) # f can only send msg.
p.start()
# send_conn.close()
for i in bar(recv_conn):
print(i)
I'm using Python 3.4.1 on Ubuntu 14.04 and the code is not working. At the end of the program, there is no EOFError, which should terminates the code, although the Pipe has been closed. Closing the Pipe inside a function does not close the Pipe. Why is this the case?
Uncomment your send_conn.close() line. You should be closing pipe ends in processes that don't need them. The issue is that once you launch the subprocess, the kernel is tracking two open references to the send connection of the pipe: one in the parent process, one in your subprocess.
The send connection object is only being closed in your subprocess, leaving it open in the parent process, so your conn.recv() call won't raise EOFError. The pipe is still open.
This answer may be useful to you as well.
I verified that this code works in Python 2.7.6 if you uncomment the send_conn.close() call.