My case is a little bit specific. I'm trying to run a Python program using Python for testing purposes. The case is as follows:
# file1.py
print("Hello world")
# file1.test.py
import io
import sys
import os
import unittest
EXPECTED_OUTPUT = "Hello world"
class TestHello(unittest.TestCase):
def test_hello(self):
sio = io.StringIO()
sys.stdout = sio
os.system("python3 path/to/file1.py")
sys.stdout = sys.__stdout__
print("captured value:", sio.getvalue())
self.assertEqual(sio.getvalue(), EXPECTED_STDOUT)
if __name__ == "__main__":
unittest.main()
But nothing ends up in the sio variable. This way and similar ways are introduced online but they don't seem to work for me. My Python version is 3.8.10 but it doesn't really matter if this works better in some other version, I can switch to that.
Note: I know that if I was using an importable object this might be easier, but right now I need to know how to catch the output of another file.
Thanks!
stdout redirection does not work like this - this will change the stdout variable inside your Python process. But by using os.system, you are running another process, that will re-use the same terminal pseudo-files your parent process is using.
If you want to log a subprocess, the way to do it is to use the subprocess modules calls, which allow you to redirect the subprocess output. https://docs.python.org/3/library/subprocess.html
Also, the subprocess won't be able to use a StringIO object from the parent process (it is not an O.S. level object, just an in-process Python object with a write method). The docs above include instructions about using the special object subprocess.PIPE which allows for in-memory communication, or, you can just pass an ordinary filesystem file, which you can read afterwards.
Related
Suppose I have a python script program.py
that looks as follows:
# program.py
import random
class Program
def __init__(self):
self.data = []
def run():
while True:
self.data.append(random.random()]
if __name__ == "__main__":
program = Program()
program.run()
Now suppose that I have another script script.py that calls program.py as a separate process.
# script.py
import subprocess
import time
process = subprocess.Popen(["py", "program.py"])
while True:
data = get_program_data(process)# how?
time.sleep(10)
The purpose of script.py is to illustrate the fact that I don't have access to the class Program. In my case this is due to the fact that I will be triggering program.py from a .NET application. I thought I'll try to understand how to deal with this problem from a python script first then apply it to the .NET application. So here is my question (keep in mind that I can alter the code in program.py and script.py, but script.py can not access program.py):
How should I go about accessing self.data from the process? I have been searching all over and I'm not quite sure in what direction I should be going. In my case, I will need to trigger difference commands for different ind of data generated in Program, i.e. get_program_data1(), get_program_data2(),....
The way I have been "solving" this issue is to have a file controller.json that script.py modifies and program.py reads and acts accordingly. I does not feel quite right doing and so I want your opinion about this. Remember that ultimately, script.py is a .NET application. Thanks
Will Multithreading and Multiprocessing work with or without this,
if __name__=="__main__" ??
__name__ is a global variable that is automatically initialized to refer to a string—either the name of the module in which __name__ appears or, the special value, "__main__", if it appears in a script or top-level program.
This pattern is used in many Python source files:
if __name__ == "__main__":
# test or demo the module
...
The test/demo code is run if you run the source file as a script, or it doesn't get run if the source file is imported as a module.
Importing multiprocessing or threading doesn't change what if __name__ == "__main__" does, and if __name__ == "__main__" doesn't change what multiprocessing or threading do. Those modules are no different in that respect from any other Python module (e.g., math, or time, or sys)
Consider the following function
import time
def foo():
for i in range(5):
print(f"{i}. Hello world!")
time.sleep(1)
I would like to save all these print calls in a variable without preventing them from reaching the terminal in real time. Essentially, print would output to stdout and a variable.
I have tried:
from contextlib import redirect_stdout
import io
stdout = io.StringIO()
with redirect_stdout(stdout):
foo()
stdout_content = stdout.getvalue()
print(stdout_content)
However, this blocks printing to the terminal until foo returns.
I would like foo to keep printing to the terminal in real time while an object stores the calls.
How can this be achieved?
One approach is to provide your own file like object to redirect_stdout. Your object class will implement the write by writing both to a file and the original sys.stdout.
You can read about sys.stdout here.
You can see the various file like classes here for solid examples.
I'm using Python 3.7.4 and I have created two functions, the first one executes a callable using multiprocessing.Process and the second one just prints "Hello World". Everything seems to work fine until I try redirecting the stdout, doing so prevents me from getting any printed values during the process execution. I have simplified the example to the maximum and this is the current code I have of the problem.
These are my functions:
import io
import multiprocessing
from contextlib import redirect_stdout
def call_function(func: callable):
queue = multiprocessing.Queue()
process = multiprocessing.Process(target=lambda:queue.put(func()))
process.start()
while True:
if not queue.empty():
return queue.get()
def print_hello_world():
print("Hello World")
This works:
call_function(print_hello_world)
The previous code works and successfully prints "Hello World"
This does not work:
with redirect_stdout(io.StringIO()) as out:
call_function(print_hello_world)
print(out.getvalue())
With the previous code I do not get anything printed in the console.
Any suggestion would be very much appreciated. I have been able to narrow the problem to this point and I think is related to the process ending after the io.StringIO() is already closed but I have no idea how to test my hypothesis and even less how to implement a solution.
This is the workaround I found. It seems that if I use a file instead of a StringIO object I can get the things to work.
with open("./tmp_stdout.txt", "w") as tmp_stdout_file:
with redirect_stdout(tmp_stdout_file):
call_function(print_hello_world)
stdout_str = ""
for line in tmp_stdout_file.readlines():
stdout_str += line
stdout_str = stdout_str.strip()
print(stdout_str) # This variable will have the captured stdout of the process
Another thing that might be important to know is that the multiprocessing library buffers the stdout, meaning that the prints only get displayed after the function has executed/failed, to solve this you can force the stdout to flush when needed within the function that is being called, in this case, would be inside print_hello_world (I actually had to do this for a daemon process that needed to be terminated if it ran for more than a specified time)
sys.stdout.flush() # This will force the stdout to be printed
I am trying to write a shell that needs to run socket connections on a seperate thread. On my testings, when print() is used while cmd.Cmd.cmdloop() waiting for input, the print is displaying wrong.
from core.shell import Shell
import time
import threading
def test(shell):
time.sleep(2)
shell.write('Doing test')
if __name__ == '__main__':
shell = Shell(None, None)
testThrd = threading.Thread(target=test, args=(shell,))
testThrd.start()
shell.cmdloop()
When the above command runs, here is what happens:
python test.py
Welcome to Test shell. Type help or ? to list commands.
>>asd
*** Unknown syntax: asd
>>[17:59:25] Doing test
As you can see, printing from another threads add output after prompt >> not in a new line. How can I do it so that it appears in a new line and prompt appears?
What you can do, is redirect stdout from your core.shell.Shell to a file like object such as StringIO. You would also redirect the output from your thread into a different file like object.
Now, you can have some third thread read both of these objects and print them out in whatever fashion you want.
You said core.shell.Shell inherits from cmd.Cmd, which allows redirection as a parameter to the constructor:
import io
import time
import threading
from core.shell import Shell
def test(output_obj):
time.sleep(2)
print('Doing test', file=output_obj)
cmd_output = io.StringIO()
thr_output = io.StringIO()
shell = Shell(stdout=cmd_output)
testThrd = threading.Thread(target=test, args=(thr_output,))
testThrd.start()
# in some other process/thread
cmd_line = cmd_output.readline()
thr_line = thr_output.readline()
That's quite difficult. Both your threads are sharing the same stdout. So the output from each of those threads are concurrently sent to your stdout buffer where they are printed in some arbitrary order.
What you need to do is coordinate the output from both threads, and that's a tough nut to crack. Even bash doesn't do that!
That said, maybe you can try using a lock to make sure your threads access stdout in a controlled manner. Check out: http://effbot.org/zone/thread-synchronization.htm