I made 2 classes:
Input which implements read method, and
Output which implements write method
I am trying to call a shell command and capture the input and output. This is my code:
import subprocess
command = "date"
output = Output()
input = Input()
process = subprocess.Popen(command,
stdout=output, stdin=input, shell=False)
but when I check the output instance it says that it contains no data.
Technically, those class instances are probably invalid parameters, unless they inherit from a file object.
According to the documentation (emphasis mine):
stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. DEVNULL indicates that the special file os.devnull will be used. With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the applications should be captured into the same file handle as for stdout.
Related
I'm developing a Python code that can run two applications and exchange information between them during their run time.
The basic scheme is something like:
start a subprocess with the 1st application
start a subprocess with the 2nd application
1st application performs some calculation, writes a file A, and waits for input
2nd application reads file A, performs some calculation, writes a file B, and waits for input
1st application reads file B, performs some calculation, writes a file C, and waits for input
...and so on until some condition is met
I know how to start one Python subprocess, and now I'm learning how to pass/receive information during run time.
I'm testing my Python code using a super-simple application that just reads a file, makes a plot, closes the plot, and returns 0.
I was able to pass an input to a subprocess using subprocess.communicate() and I could tell that the subprocess used that information (plot opens and closes), but here the problems started.
I can only send an input string once. After the first subprocess.communicate() in my code below, the subprocess hangs there. I suspect I might have to use subprocess.stdin.write() instead, since I read subprocess.communicate() will wait for the end of the file and I wish to send multiple times different inputs during the application run instead. But I also read that the use of stdin.write() and stdout.read() is discouraged. I tried this second alteranative (see #alternative in the code below), but in this case the application doesn't seem to receive the inputs, i.e. it doesn't do anything and the code ends.
Debugging is complicated because I haven't found a neat way to output what the subprocess is receiving as input and giving as output. (I tried to implement the solutions described here, but I must have done something wrong: Python: How to read stdout of subprocess in a nonblocking way, A non-blocking read on a subprocess.PIPE in Python)
Here is my working example. Any help is appreciated!
import os
import subprocess
from subprocess import PIPE
# Set application name
app_folder = 'my_folder_path'
full_name_app = os.path.join(app_folder, 'test_subprocess.exe')
# Start process
out_app = subprocess.Popen([full_name_app], stdin=PIPE, stdout=PIPE)
# Pass argument to process
N = 5
for n in range(N):
str_to_communicate = f'{{\'test_{n+1}.mat\', {{\'t\', \'y\'}}}}' # funny looking string - but this how it needs to be passed
bytes_to_communicate = str_to_communicate.encode()
output_communication = out_app.communicate(bytes_to_communicate)
# output_communication = out_app.stdin.write(bytes_to_communicate) # alternative
print(f'Communication command #{n+1} sent')
# Terminate process
out_app.terminate()
import sys
data = []
for line in sys.stdin:
data.append(line)
I did not use .read() or .readline()
but this code works reading the data. It read the data line by line. Input data is separated by '\n'. data is typed by user. like 'input()' .
My Question:
Is .read() or .readline() not necessary?
I wonder why for loop works on sys.stdin and how it reads data line by line?
If you look at the documentation for sys, you'll see that sys.stdin (and sys.stdout / sys.stderr) are just file objects.
These streams are regular text files like those returned by the open() function.
The documentation about open() says:
The type of file object returned by the open() function depends on the mode. When open() is used to open a file in a text mode ('w', 'r', 'wt', 'rt', etc.), it returns a subclass of io.TextIOBase (specifically io.TextIOWrapper).
TextIOWrapper inherits from IOBase, which has an __iter__ method. This __iter__ method allows looping over lines in the file. I can't find where this is noted in the Python documentation, but it is given in the source code for IOBase.
IOBase (and its subclasses) support the iterator protocol, meaning that an IOBase object can be iterated over yielding the lines in a stream.
I have a command-line program which when triggered provides a list of options and prompts the user to select an option to continue.
Now I have launched the command line program using subprocess call, now the command line program prompts for the value, how do I pass the value from user to the command line program via subprocess call?
subprocess.call just waits for the process to finish and gives return code, no way to interact with it. if you instead use subprocess.Popen that gives you the ability to communicate with the subprocess while it is running via stdin and stdout
import subprocess, sys
program = subprocess.Popen("python3",
# give us a pipes to coommunicate
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
data = input("input to subprocess")
[out,err] = program.communicate((data+"\n").encode())
print(out.decode())
print(err.decode(), file=sys.stderr)
Doing a bit of input then some output then more input can get messy though since there reading from stdout is blocking so determining when the output has stopped for more input is tricky.
Working on python script that uses stdin and stdout. The default
behaviour is the script will write to a file, so I have this function
that is called to handle writing to a file:
def run_loop(data,name):
with open(name, 'w') as fo:
fo.write("webstat output "+t) # t is a date str created in global scope
for x in data:
fo.write(processer(x))
fo.write('\n')
I have a few places in the script where I use this function, so it
would be less convenient to have to rewrite it or write a substituent
of it to work with stdout. When I try using stdout with open,
sys.stdout says its a type _io.TextIOWrapper. I get an error when
tying to call run_loop(same_data,sys.stdout) I get this error:
TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper
I'm wondering if there is away to use open() with stdout? I would
guess that since stdout is file io, I should be able to write to it as
a file. Ideally there would be a way to change _io.TextIOWrapper into
an object that open() can use, but I've been looking at types of the
methods inside of sys.stdout and they don't seem to provide this
facility. I'm not sure why the fo object from open() couldn't be
set to stdout?
The behaviour I'm trying to achieve is
cat input-data.txt | myscript.py > out.txt
I've been able to get the stdin portion to work, but I'm still at a
loss for how to write to stdout with open().
Supposedly, I could forego open(), if I could keep compatibility with
the other places I call this function: data being a list and name
usually being a string of the file name where the output should be saved.
Python 3.7.5rc1
I am new to python and programming and I am trying to understand this code. I have spent the past few hours reading documentation and watching videos on subprocessing but I am still confused (I added snidbits of information of what I found online to comment the code as best I could).
Here are some questions I have pertaining to the code below:
when is subprocess used?
when should I use Popen verses the more convenient handles with subprocess?
what does PIPE do?
what does close_fds do?
basically I need this line of code explained
my_process=Popen(['player',my_video_File_path], stdin=PIPE, close_fds=True)
full code here:
#run UNIX commands we need to create a subprocess
from subprocess import Popen, PIPE
import os
import time
import RPi.GPIO as GPIO
my_video_file_path='/home/pi/green1.mp4'
#stdin listens for information
# PIPE connnects the stdin with stdout
#pipe, (like a pipe sending info through a tunnel from one place to another )
#STDIN (channel 0):
#Where your command draws the input from. If you don’t specify anything special this will be your keyboard input.
#STDOUT (channel 1):
#Where your command’s output is sent to. If you don’t specify anything special the output is displayed in your shell.
#to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE.
#Popen interface can be used directly with subprocess
# with pipe The return value is an open file object connected to the pipe, which can be read or written depending on whether mode is 'r' (default) or 'w'.
#If we pass everything as a string, then our command is passed to the shell;
#"On Unix, if args is a string, the string is interpreted as the name or path of the program to execute. "
my_process=Popen(['player',my_video_File_path], stdin=PIPE, close_fds=True)
GPIO.setmode(GPIO.BCM)
GPIO.setup(17,GPIO.IN,pull_up_down=GPIO.PUD_UP)
GPIO.setup(22,GPIO.IN,pull_up_down=GPIO.PUD_UP)
while True:
button_state=GPIO.input(17)
button_state1=GPIO.input(22)
if button_state==False:
print("quite video")
my_process.stdin.write("q")
time.sleep(.09)
if button_state1==False:
print("full video")
my_process.stdin.write("fs")
time.sleep(5)
Regarding the difference between subprocess and Popen, here is a line from the python 3.5 docs:
For more advanced use cases, the underlying Popen interface can be used directly. (compared to using subprocess)
So, this means that subprocess is a wrapper on Popen.
PIPE is used so that your python script communicate with the subprocess via standard IO operations (you can think of print in python as a type of stdout).
So, when you do my_process.stdin.write("fs") you are sending this text (or piping it) to the standard input of your subprocess. Then your subprocess reads this text and does whatever processing it needs to do.
To further understand subprocessing, try to read standard input into a python program to see how it works. You can follow this How do you read from stdin in Python? question to work on this exercise.
Also, it might be worthwhile to learn about piping in the more general linux style. Try to read through this article.