I am calling several bash scripts in a loop:
import subprocess
for script in scripts:
cmd = './{}'.format(script)
subprocess.call(cmd, shell=True)
My issue is that a few scripts will hang and I don't understand why. It seems as though subprocess is just waiting after script has finished executing. Note that I am not using stdout=subprocess.PIPE, etc, that the documentation specifically suggests can overflow a buffer and cause a related issue.
Any suggestions? I can set timeout on the process, but would prefer not to resort to this measure.
When sigterm, the python output reads:
File "/home/ubuntu/anaconda3/lib/python3.6/subprocess.py", line 269, in call
return p.wait(timeout=timeout)
File "/home/ubuntu/anaconda3/lib/python3.6/subprocess.py", line 1457, in wait
(pid, sts) = self._try_wait(0)
File "/home/ubuntu/anaconda3/lib/python3.6/subprocess.py", line 1404, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
Related
I have a simple cmd_str containing a set of lines. Using exec, I can run those lines juts fine. However, running those lines in a separate process when shell=True is failing. Is this dues to missing quotes? what is happening under the hood?
import subprocess
cmd_str = """
import sys
for r in range(10):
print('rob')
"""
exec(cmd_str) # works ok
full_cmd = f'python3 -c "exec( "{cmd_str}" )"'
process = subprocess.Popen([full_cmd],
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(output, error) = process.communicate()
exit_code = process.wait()
output_msg = output.decode("utf-8", 'ignore')
error_msg = error.decode("utf-8", 'ignore').strip()
Your approach is slightly inaccurate. I believe the problem you're having has to do with the subprocess usage. The first thing you must realise is that exec
is a way to send and execute python code to and from the interpreter directly. This is why it works inside python programs(and it is generally not a good approach). Subprocesses on the other hand handle command like they are being called directly from the terminal or shell. This means that you no longer need to include exec cause you are already interacting with the python interpreter when you call python -c.
To get this to run in a subprocess environment all you need to do is
full_cmd = f'python3 -c "{cmd_str}"'
process = subprocess.Popen(full_cmd,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
Also, notice the absence of square brackets when calling subprocess.Popen, this is because that particular approach works slightly different and if you want to use the square brackets your command will have to be
full_cmd = ['python3', '-c', f'{cmd_str}']
process = subprocess.Popen(full_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
And with these few changes, everything should work OK
I have simple script, that logs to a logfile. This is the core of my script:
with open('/var/log/mail/mail.log', 'a', buffering=1) as f:
for line in sys.stdin:
f.write(line)
The file being written /var/log/mail/mail.log has to be rotated regularly by logrotate. At the moment, when logrotate rotates the file, my script does not realize it, and continues writing to the old (now renamed) file.
logrotate has the possibility to execute command after the file has been rotated. Normally, when rsyslog is logging, the command would be:
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
But in my case, I need to send some signal to my script, and handle the signal in my script.
Also, I don't know in advance the PID my script is running as.
How can I best implement this ?
As a solution for this you can watch if the inode of the open log file is the same as the path. If not reopen the file. This only works on Unix.
import os, sys, stat
logfile = '/var/log/mail/mail.log'
while True:
with open(logfile, 'a', buffering=1) as f:
f_ino = os.stat(logfile)[stat.ST_INO]
f_dev = os.stat(logfile)[stat.ST_DEV]
for line in sys.stdin:
f.write(line)
try:
if os.stat(logfile)[stat.ST_INO] != f_ino or
os.stat(logfile)[stat.ST_DEV] != f_dev:
break
except OSError: # Was IOError with python < 3.4
pass
Closing the file is not required as it’s handled by the with context manager.
The try..except OSError block is used to catch any error by the system function os.stat. It can be that during the change of the file the function returns an OSError (For example a FileNotFoundError). In this case it will pass the exception and retry the check if the inode is changed. If we omit this try..except block you might end up with a program that terminates.
I have a program that is supposed to create a text file. When called from subprocess.run() in python3, the program runs but it does not create the text file. The program works as expected when called from the terminal.
import subprocess as subp
...
comm=[os.getcwd()+'/test/myprogram.bin','arg1','arg2']
compl_proc = subp.run(comm,
capture_output=True,
text=True,
check=True)
The file was in the python script directory, because I never told subprocess.run() what is the current working directory of the subprocess. So cwd='...' is added.
import subprocess as subp
import os
...
comm=[os.getcwd()+'/test/myprogram.bin','arg1','arg2']
compl_proc = subp.run(comm,
cwd=os.getcwd()+'/test/',
capture_output=True,
text=True,
check=True)
I am trying to do a script that listen to a directory waiting to new files, then send to a Nextcloud. The files may be big ones,so I want to check if they are complete before sending. I thought about using lsof +D path/to/directory and check if the files are in the output of the command and send them when the file is not. The code would be something like:
command=list()
command.append("lsof")
command.append("+D")
command.append("/path/to/dir")
lsof = subprocess.check_output(command, stderr = subprocess.STDOUT)
But I get subprocess.CalledProcessError returned non-zero exit status 1
Can someone help to ecexute the command and get the output into the variable?
EDIT:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['lsof', '+D', '/home/CLI2Cache/sync']' returned non-zero exit status 1
There are couple of workarounds for this. You can use shell=True in the check_output -
lsof = subprocess.check_output(command, shell=True, stderr = subprocess.STDOUT)
Please note that shell=True is not safe as it also gives access to a lot of shell commands which might lead to some vulnerabilities if the command is user-specified or not sanitized properly. Please go through this to understand the risks.
A better way would be to use subprocess.Popen -
lsof = subprocess.Popen(command, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
try:
output, errs = lsof.communicate(timeout=20)
except TimeoutExpired:
lsof.kill()
output, errs = proc.communicate()
communicate is also useful, if you want to send the input to the spawned process and get corresponding output at each step.
I've been playing with using the subprocess module to run python scripts as sub-processes and have come accross a problem with reading output line by line.
The documentation I have read indicates that you should be able to use subprocess and call readline() on stdout, and this does indeed work if the script I am calling is a bash script. However when I run a python script readline() blocks until the whole script has completed.
I have written a couple of test scripts that repeat the problem. In the test scripts I attmept to run a python script (tst1.py) as a sub-process from within a python script (tst.py) and then read the output of tst1.py line by line.
tst.py starts tst1.py and tries to read the output line by line:
#!/usr/bin/env python
import sys, subprocess, multiprocessing, time
cmdStr = 'python ./tst1.py'
print(cmdStr)
cmdList = cmdStr.split()
subProc = subprocess.Popen(cmdList, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
while(1):
# this call blocks until tst1.py has completed, then reads all the output
# it then reads empty lines (seemingly for ever)
ln = subProc.stdout.readline()
if ln:
print(ln)
tst1.py simply loops printing out a message:
#!/usr/bin/env python
import time
if __name__ == "__main__":
x = 0
while(x<20):
print("%d: sleeping ..." % x)
# flushing stdout here fixes the problem
#sys.stdout.flush()
time.sleep(1)
x += 1
If tst1.py is written as a shell script tst1.sh:
#!/bin/bash
x=0
while [ $x -lt 20 ]
do
echo $x: sleeping ...
sleep 1
let x++
done
readline() works as expected.
After some playing about I discovered the situation can be resolved by flushing stdout in tst1.py, but I do not understand why this is required. I was wondering if anyone had an explanation for this behaviour ?
I am running redhat 4 linux:
Linux lb-cbga-05 2.6.9-89.ELsmp #1 SMP Mon Apr 20 10:33:05 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Because if the output is buffered somewhere the parent process won't see it until the child process exists at that point the output is flushed and all fd's are closed. As for why it works with bash without explicitly flushing the output, because when you type echo in a most shells it actually forks a process that executes echo (which prints something) and exists so the output is flushed too.