I'm not sure but I imagine that there may be issues similar to mine, but I have not found any that has been satisfactory.
When I open my Jupyter Lab and execute a cell as below (code01), it remains with the status of * (see figure01 below) meaning that it is still running the code, but the output of the out1.txt file is printed correctly.
I would like to know if it is normal for the cell to remain running in this circumstances described from code01.
code01:
import sys
file = open('out1.txt', 'a')
sys.stdout = file
print("house")
file.close()
figure01:
Because you redirect the stdout to a file and then close it you are breaking the IPython kernel underneath: there is no way for any stdout to be correctly processed by the kernel afterwards (cannot write to a closed file). You can reproduce it by executing your code in the IPython console rather than in a notebook. To fix this you could rebind the original stdout back:
import sys
file = open('out1.txt', 'a')
stdout = sys.stdout
sys.stdout = file
print("house")
# before close, not after!
sys.stdout = stdout
file.close()
But this is still not 100% safe; you should ideally use context managers instead:
from contextlib import redirect_stdout
with open('out1.txt', 'a') as f:
with redirect_stdout(f):
print('house')
But for this particular case why not to make use the file argument of the print() function?
with open('out1.txt', 'a') as f:
print('house', file=f)
Related
I have a very simple (test) code which I'm running either from a Linux shell, or in interactive mode, and I have two different behaviours I cannot figure out the reason of.
I have a file generated by a Popen call, previously, where each line is a file path. This is the code used to generate the file:
with open('find.txt','w') as f:
find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)
(Incidentally, I was trying to build a PIPE originally, namely inputting the output of this command to a grep command, and since I wasn't successful in any way, I decided to break the problem down and just read the file paths from a file, and process them one by one. So maybe there is a common issue that is blocking me somewhere in this procedure).
Since in this second step I wasn't even able to open and process the files by opening the addresses contained in each line of the find.txt file, I just tried to print the file lines out, because for sure they're available in there:
with open('find.txt','r') as g:
for l in g.readlines():
print(l)
Now, the interesting part:
if I paste the lines above into a python shell, everything works fine and I get my outputs as expected
if, on the other hand, I try to run python test.py, where test.py is the name of the file containing the lines above, no output appears in the shell's stdout.
I've tried sys.stdout.flush() to no avail. I've also inserted some dummy print() statements along the way: everything gets printed but what's after the g.readlines() statement.
Here's the full script I'm trying to make work (a pre-precursor of what I'm actually after, tbh).
#!/usr/bin/env python3
import subprocess
import sys
with open('find.txt','w') as f:
find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)
print('hello')
with open('find.txt','r') as g:
print('hello?')
for l in g.readlines():
print('help me!')
print(l)
sys.stdout.flush()
output being:
{ancis:>106> python test.py
hello
hello?
{ancis:>106>
EDIT
I've quickly tried the very same lines (but without the call to find, which isn't available) on my python installation in Windows: it works as expected)
Based on that, I've tried to run the simpler code below:
print('hello')
with open('find.txt','r') as g:
print('hello?')
for l in g.readlines():
print('help me!')
print(l)
sys.stdout.flush()
as a script, in Linux - This also works w/o problems.
This should mean that somehow I'm messing things up with the call to Popen... But what?
This is a race condition.
Your call to
find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)
is opening another process and running your find command which takes a bit of time to fully execute.
Python then continues on and reaches the reading of the file portion before the command is fully executed and the file is generated.
Want to test it out?
Add a time.sleep(1) just before the opening of the file.
Full test script:
#!/usr/bin/env python3
import subprocess
import time
with open('find.txt','w') as f:
find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)
time.sleep(1)
with open('find.txt','r') as g:
for l in g:
print(l)
To block until the process is complete you can use find.communicate().
With this you can also optionally set a timeout if that's something that you want.
#!/usr/bin/env python3
import subprocess
with open('find.txt','w') as f:
find = subprocess.Popen(["find",".","-name","myfile.out"],stdout=f)
find.communicate()
with open('find.txt','r') as g:
for l in g:
print(l)
Source:
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate
When I run vim in a subprocess, I have the script that created that subprocess continue printing miscellaneous status updates. However the updates get printed on top of my vim instance. How do I create a behavior where I only see the output when I quit vim (:q).
I'm running the subprocess like this:
def callVim():
# Open the file with the text editor
EDITOR = os.environ.get('EDITOR','vim')
call([EDITOR, temp_loc])
try:
print('Starting vim process...')
vim = Process(target = callVim)
vim.start()
except KeyboardInterrupt:
print("Terminating vim server.")
vim.terminate()
vim.join()
And this is what the overlaid output looks like ('Check if target updated...' being the output):
Thank you for any and all help.
Any kind of suggestion is welcome.
I'm running tmux if that matters?
EDIT:
So i tried rerouting the sys.stdout to a variable before calling on vim, like so:
so = sys.stdout = StringIO()
EDITOR = os.environ.get('EDITOR','vim')
call([EDITOR, self.temp_loc])
But it still printed over vim.
EDIT2:
Also I need to keep the output somewhere because i use it elsewhere in the program.
I'm using python3, and I want my program to utilize utf-8 for stdout and stderr without my having to set the PYTHONIOENCODING environment variable to utf-8 outside of the python program.
In other words, I want the choice of forcing utf-8 output to be contained within the python program itself, no matter what setting of PYTHONIOENCODING or lack thereof might have been specified in the invoker's environment.
The only python3-specific way to accomplish this that I've figured out so far is to start the program as follows:
#!/usr/bin/python3.6
import os
import sys
if os.environ.get('PYTHONIOENCODING', '').lower() != 'utf-8':
os.environ['PYTHONIOENCODING'] = 'utf-8'
sys.exit(os.execl(
sys.executable,
os.path.basename(sys.executable),
*sys.argv
))
# Remainder of my program ...
Ideally, I'd like to implement this without having to re-exec the python3 interpreter, but I fear that I'm probably out of luck.
Any suggestions? Thank you very much.
ADDENDUM:
Per the comments in the answer, below, I tried the following, but it doesn't print anything ...
#!/usr/bin/python3.6
open(sys.stdout.buffer.fileno(), 'w', encoding='utf8')
open(sys.stderr.buffer.fileno(), 'w', encoding='utf8')
print('Seems to be working')
sys.exit(0)
I also tried utf-8 (with the hyphen), and it doesn't print anything, either.
Some people suggest
sys.stdout = codecs.getwriter("UTF-8")(sys.stdout)
Others say that may break some libraries.
As of Python 3.7, TextIOWrapper has reconfigure method for you:
sys.stdin.reconfigure(encoding="utf-8")
sys.stdout.reconfigure(encoding="utf-8")
sys.stderr.reconfigure(encoding="utf-8")
And now I understand better what lenz suggested, and I figured out the answer:
#!/usr/bin/python3.6
import os
import sys
sys.stdout = os.fdopen(sys.stdout.buffer.fileno(), 'w', encoding='utf8')
sys.stderr = os.fdopen(sys.stderr.buffer.fileno(), 'w', encoding='utf8')
I am going off of this thread.
First, I added import sys to the top of my file. The print statement that I want to log is inside of an if/else statement, specifically in the else portion. It looks like this:
else:
sys.stdout = open('logfile', 'w')
print("hi")
This does create a file called logfile in the same directory as the .py file, but it is emtpy and does not include "hi".
I'm writing a small script that runs through a directory and attempts to obtain version numbers by running "filename --version". Now for the most part it works or fails in a manageable way. However I've come across one file "iptables-restore" in /sbin/ that fails when --version is passed to it. However the failure leaves the prompt in a state that requires a ctrl+z or ctrl+c to return to the prompt, and thus halts my script.
Here's the code:
try:
subOut = subprocess.check_output([fname, "--version"])
except subprocess.CalledProcessError as cpE:
fObj.write('{0:25}RETURN CODE:{1:15}\t\t{2:30}\n'.format(fnamecolon, cpE.returncode, cpE.output))
fnamecolon = ''
pass
except Exception as e:
pass
I just wondered if there's an elegant way to handle this - either via a return code or perhaps an Exception.
I'm fairly new to Python and am doing this for practice really - so any guidance is greatly appreciated.
Thanks all!
So this works better - probably still some things I'm not understanding though...
try:
# devnull=open(os.devnull, 'w')
try:
from subprocess import DEVNULL # Python 3
except ImportError:
DEVNULL = open(os.devnull, 'r+b', 0)
subOut = Popen([fname, "--version"], stdin=DEVNULL, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = subOut.stdout.read()
except subprocess.CalledProcessError as cpE:
fObj.write('{0:25}RETURN CODE:{1:15}\t\t{2:30}\n'.format(fnamecolon, cpE.returncode, cpE.output))
fname = ''
pass
except Exception as e:
# fObj.write('{0:30}{1:30}{2:30}\n'.format(fname, e.returncode, e.output))
pass
I can reproduce it: iptables-restore --version commands waits for input without stdin=subprocess.DEVNULL suggested by #mata:
#!/usr/bin/env python3
import subprocess
cp = subprocess.run(['iptables-restore', '--version'],
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
print(cp.returncode, cp.stdout, cp.stderr)