So how do I pass binary data using stdin to a executable command that I want to run using subprocess.run()?
The documentation is pretty vague about using the stdin for passing the data to external executable. I'm working on linux machine using python3 and I want to invoke dd of=/somefile.data bs=32 (which takes the input from stdin if I understand the man page correctly) and I have the binary data in bytearray that I want to pass to the command through stdin so that I do not have to write it to a temporary file and invoke the dd using that file as a input.
My requirement is simply to pass data that I have in bytearray to the dd command to be written to disk. What is the correct way to achieve this using the subprocess.run() and stdin?
Edit: I meant something like the following:
ba = bytearray(b"some bytes here")
#Run the dd command and pass the data from ba variable to its stdin
You can pass the output of one command to another by calling Popen directly.
file_cmd1 = <your dd command>
file_cmd2 = <command you want to pass dd output to>
proc1 = Popen(sh_split(file_cmd1), stdout=subprocess.PIPE)
proc2 = Popen(file_cmd2, [shell=True], stdin=proc1.stdout, stdout=subprocess.PIPE)
proc1.stdout.close()
This, as far as I know, will work just fine on a byte output from command 1.
In your case what you most like want to do is the following when you just want to pass data to the stdin of the process:
out = bytearray(b"Some data here")
p = subprocess.Popen(sh_split("dd of=/../../somefile.data bs=32"), stdin=subprocess.PIPE)
out = p.communicate(input=b''.join(out))[0]
print(out.decode())#Prints the output from the dd
Specifically for stdin to subprocess.run() as asked by the OP, use input as follows:
#!/usr/bin/python3
import subprocess
data = bytes("Hello, world!", "ascii")
p = subprocess.run(
"cat -", # The - means 'cat from stdin'
input=data,
# stdin=... <-- don't use this
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
print(p.stdout.decode("ascii"))
print(p.returncode)
# Hello, world!
# 0
Related
I'm trying to execute:
actual = subprocess.run(['echo 123 | ./ft_ssl md5 -s ' + data + ' -p'], stdout=subprocess.PIPE)
actual = actual.stdout.decode('utf-8')
and after that variable actual equals "123 | ./ft_ssl md5 -s fuck -p\n".
Python run only echo for all the input and ignore | operation.
Help, please, what i have to do to run two commands with this logical operation?
You can avoid echo altogether. Piping can be simulated here by passing the echo argument to the stdin of the call to ./ft_ssl
actual = subprocess.Popen(['./ft_ssl', 'md5', '-s', data, '-p'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
output = actual.communicate(b'123')
See the docs for more details about communicate
I am trying to get grep to work using pipes and subprocess. I've double-checked the cat and I know it's working, but for some reason the grep isn't returning anything, even though when I run it through the terminal, it works just fine. I'm wondering if I have the command constructed correctly, as it doesn't give the desired output, and I can't figure out why.
I'm trying to retrieve a few specific lines of data from a file I've already retrieved from a server. I've been having a lot of issues with getting grep to work and perhaps I do not simply understand how it works.
p1 = subprocess.Popen(["cat", "result.txt"], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "tshaper"], stdin=p1.stdout,
stdout=subprocess.PIPE)
o = p1.communicate()
print(o)
p1.stdout.close()
out, err = p2.communicate()
print(out)
The output for the file I have when I run this command (cat result.txt | grep "tshaper") on the terminal:
tshaper.1.devname=eth0
tshaper.1.input.burst=0
tshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.rate=25000
tshaper.1.output.status=enabled
tshaper.1.status=enabled
tshaper.status=disabled
My results running the command in the script:
(b'', b'')
where the tuple is the stdout, stderr respectively of the p2 process.
EDIT:
I changed the command based on the Popen documentation to
p1 = subprocess.Popen(['result.txt', 'cat'], shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, cwd=os.getcwd())
to the p1 subprocess statement. While I was able to get the output in stderr, it didn't really change anything, saying
(b'', b'cat: 1: cat: result.txt: not found\n')
FYI: You got this error: (b'', b'cat: 1: cat: result.txt: not found\n') because you have changed the seq. of commands in your Popen method: ['result.txt', 'cat'] (Based on your question).
I have written a working solution which provides the expected output.
Python3.6.6 has been used for it.
result.txt file:
I have changed some lines to test the grep command.
tshaper.1.devname=eth0
ashaper.1.input.burst=0
bshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
cshaper.1.output.rate=25000
tshaper.1.output.status=enabled
dshaper.1.status=enabled
tshaper.status=disabled
Code:
I have made an understandable printing but it is not necessary if you only need the output of grep. (It is a bytesl-like object in Python3)
import subprocess
p1 = subprocess.Popen(['cat', 'result.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2 = subprocess.check_output(["grep", "tshaper"], stdin=p1.stdout)
print("\n".join(p2.decode("utf-8").split(" ")))
Otuput:
You can see the grep filters the lines from cat command as it is expected.
>>> python3 test.py
tshaper.1.devname=eth0
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.status=enabled
tshaper.status=disabled
I am trying to get a file name and pass it to a command using popen. Then I want to print the output. This is my code:
filePath = tkinter.filedialog.askopenfilename(filetypes=[("All files", "*.*")])
fileNameStringForm = (basename(filePath ))
fileNameByteForm = fileNameStringForm.encode(encoding='utf-8')
process = subprocess.Popen(['gagner','-arg1'], shell = True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process .communicate(fileNameByteForm )
stdout, stderr = process .communicate() <<------ERROR POINTS TO THIS LINE
stringOutput = stdout.decode('urf-8')
print(stringOutput)
I am getting the following error:
ValueError: Invalid file object: <_io.BufferedReader name=9>
I have looked at other similar questions but nothing seems to have solved my problem. Can some show me where I am going wrong in the code?
Edit:
If I were to run the command in a command line it would be:
gagner -arg1 < file1
What you are doing is not what you are describing in the supposed command line argument. You are actually executing this:
echo "file1" | gagner -arg1
You will need to make sure that you pass in the file contents yourself. Popen will not open and read the file for you.
According to the documentation, what communicate() does is
interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
So, once you have run
process.communicate(fileNameByteForm)
your sub process has finished and the pipes have been closed. The second call will then fail as a result.
What you want to do instead is
stdout, stderr = process.communicate(input_data)
which will pipe your input data into the sub process and read stdout and stderr.
I have the following R script (myscript.r)
#!/usr/bin/env Rscript
dat <- read.table(file('stdin'), sep=" ",header=FALSE)
# do something with dat
# later with user given "param_1"
With that script we can run it the following way;
$ cat data_no_*.txt | ./myscript.r
What I want to do is to make the script takes additional parameter from user:
$ cat data_no_*.txt | ./myscript.r param_1
What should I do to modify the myscript.r to accommodate that?
For very basic usage, have a look at ?commandArgs.
For more complex usage, two popular packages for command-line arguments and options parsing are getopt and optparse. I use them all the time, they get the job done. I also see argparse, argparser, and GetoptLong but have never used them before. One I missed: Dirk recommended that you look at docopt which does seem very nice and easy to use.
Finally, since you seem to be passing arguments via pipes you might find this OpenRead() function useful for generalizing your code and allowing your arguments to be pipes or files.
I wanted to test docopt so putting it all together, your script could look like this:
#!/usr/bin/env Rscript
## Command-line parsing ##
'usage: my_prog.R [-v -m <msg>] <param> <file_arg>
options:
-v verbose
-m <msg> Message' -> doc
library(docopt)
opts <- docopt(doc)
if (opts$v) print(str(opts))
if (!is.null(opts$message)) cat("MESSAGE: ", opts$m)
## File Read ##
OpenRead <- function(arg) {
if (arg %in% c("-", "/dev/stdin")) {
file("stdin", open = "r")
} else if (grepl("^/dev/fd/", arg)) {
fifo(arg, open = "r")
} else {
file(arg, open = "r")
}
}
dat.con <- OpenRead(opts$file_arg)
dat <- read.table(dat.con, sep = " ", header = FALSE)
# do something with dat and opts$param
And you can test running:
echo "1 2 3" | ./test.R -v -m HI param_1 -
or
./test.R -v -m HI param_1 some_file.txt
We built littler to support just that via its r executable.
Have a look at its examples, it may fit your bill.
I'm seeking for ways to write data to the existing process's STDIN from external processes, and found similar question How do you stream data into the STDIN of a program from different local/remote processes in Python? in stackoverlow.
In that thread, #Michael says that we can get file descriptors of existing process in path like below, and permitted to write data into them on Linux.
/proc/$PID/fd/
So, I've created a simple script listed below to test writing data to the script's STDIN (and TTY) from external process.
#!/usr/bin/env python
import os, sys
def get_ttyname():
for f in sys.stdin, sys.stdout, sys.stderr:
if f.isatty():
return os.ttyname(f.fileno())
return None
if __name__ == "__main__":
print("Try commands below")
print("$ echo 'foobar' > {0}".format(get_ttyname()))
print("$ echo 'foobar' > /proc/{0}/fd/0".format(os.getpid()))
print("read :: [" + sys.stdin.readline() + "]")
This test script shows paths of STDIN and TTY and then, wait for one to write it's STDIN.
I launched this script and got messages below.
Try commands below
$ echo 'foobar' > /dev/pts/6
$ echo 'foobar' > /proc/3308/fd/0
So, I executed the command echo 'foobar' > /dev/pts/6 and echo 'foobar' > /proc/3308/fd/0 from other terminal. After execution of both commands, message foobar is displayed twice on the terminal the test script is running on, but that's all. The line print("read :: [" + sys.stdin.readline() + "]") was not executed.
Are there any ways to write data from external processes to the existing process's STDIN (or other file descriptors), i.e. invoke execution of the lineprint("read :: [" + sys.stdin.readline() + "]") from other processes?
Your code will not work.
/proc/pid/fd/0 is a link to the /dev/pts/6 file.
$ echo 'foobar' > /dev/pts/6
$ echo 'foobar' > /proc/pid/fd/0
Since both the commands write to the terminal. This input goes to terminal and not to the process.
It will work if stdin intially is a pipe.
For example, test.py is :
#!/usr/bin/python
import os, sys
if __name__ == "__main__":
print("Try commands below")
print("$ echo 'foobar' > /proc/{0}/fd/0".format(os.getpid()))
while True:
print("read :: [" + sys.stdin.readline() + "]")
pass
Run this as:
$ (while [ 1 ]; do sleep 1; done) | python test.py
Now from another terminal write something to /proc/pid/fd/0 and it will come to test.py
I want to leave here an example I found useful. It's a slight modification of the while true trick above that failed intermittently on my machine.
# pipe cat to your long running process
( cat ) | ./your_server &
server_pid=$!
# send an echo to your cat process that will close cat and in my hypothetical case the server too
echo "quit\n" > "/proc/$server_pid/fd/0"
It was helpful to me because for particular reasons I couldn't use mkfifo, which is perfect for this scenario.