I am trying to get grep to work using pipes and subprocess. I've double-checked the cat and I know it's working, but for some reason the grep isn't returning anything, even though when I run it through the terminal, it works just fine. I'm wondering if I have the command constructed correctly, as it doesn't give the desired output, and I can't figure out why.
I'm trying to retrieve a few specific lines of data from a file I've already retrieved from a server. I've been having a lot of issues with getting grep to work and perhaps I do not simply understand how it works.
p1 = subprocess.Popen(["cat", "result.txt"], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "tshaper"], stdin=p1.stdout,
stdout=subprocess.PIPE)
o = p1.communicate()
print(o)
p1.stdout.close()
out, err = p2.communicate()
print(out)
The output for the file I have when I run this command (cat result.txt | grep "tshaper") on the terminal:
tshaper.1.devname=eth0
tshaper.1.input.burst=0
tshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.rate=25000
tshaper.1.output.status=enabled
tshaper.1.status=enabled
tshaper.status=disabled
My results running the command in the script:
(b'', b'')
where the tuple is the stdout, stderr respectively of the p2 process.
EDIT:
I changed the command based on the Popen documentation to
p1 = subprocess.Popen(['result.txt', 'cat'], shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, cwd=os.getcwd())
to the p1 subprocess statement. While I was able to get the output in stderr, it didn't really change anything, saying
(b'', b'cat: 1: cat: result.txt: not found\n')
FYI: You got this error: (b'', b'cat: 1: cat: result.txt: not found\n') because you have changed the seq. of commands in your Popen method: ['result.txt', 'cat'] (Based on your question).
I have written a working solution which provides the expected output.
Python3.6.6 has been used for it.
result.txt file:
I have changed some lines to test the grep command.
tshaper.1.devname=eth0
ashaper.1.input.burst=0
bshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
cshaper.1.output.rate=25000
tshaper.1.output.status=enabled
dshaper.1.status=enabled
tshaper.status=disabled
Code:
I have made an understandable printing but it is not necessary if you only need the output of grep. (It is a bytesl-like object in Python3)
import subprocess
p1 = subprocess.Popen(['cat', 'result.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2 = subprocess.check_output(["grep", "tshaper"], stdin=p1.stdout)
print("\n".join(p2.decode("utf-8").split(" ")))
Otuput:
You can see the grep filters the lines from cat command as it is expected.
>>> python3 test.py
tshaper.1.devname=eth0
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.status=enabled
tshaper.status=disabled
Related
So how do I pass binary data using stdin to a executable command that I want to run using subprocess.run()?
The documentation is pretty vague about using the stdin for passing the data to external executable. I'm working on linux machine using python3 and I want to invoke dd of=/somefile.data bs=32 (which takes the input from stdin if I understand the man page correctly) and I have the binary data in bytearray that I want to pass to the command through stdin so that I do not have to write it to a temporary file and invoke the dd using that file as a input.
My requirement is simply to pass data that I have in bytearray to the dd command to be written to disk. What is the correct way to achieve this using the subprocess.run() and stdin?
Edit: I meant something like the following:
ba = bytearray(b"some bytes here")
#Run the dd command and pass the data from ba variable to its stdin
You can pass the output of one command to another by calling Popen directly.
file_cmd1 = <your dd command>
file_cmd2 = <command you want to pass dd output to>
proc1 = Popen(sh_split(file_cmd1), stdout=subprocess.PIPE)
proc2 = Popen(file_cmd2, [shell=True], stdin=proc1.stdout, stdout=subprocess.PIPE)
proc1.stdout.close()
This, as far as I know, will work just fine on a byte output from command 1.
In your case what you most like want to do is the following when you just want to pass data to the stdin of the process:
out = bytearray(b"Some data here")
p = subprocess.Popen(sh_split("dd of=/../../somefile.data bs=32"), stdin=subprocess.PIPE)
out = p.communicate(input=b''.join(out))[0]
print(out.decode())#Prints the output from the dd
Specifically for stdin to subprocess.run() as asked by the OP, use input as follows:
#!/usr/bin/python3
import subprocess
data = bytes("Hello, world!", "ascii")
p = subprocess.run(
"cat -", # The - means 'cat from stdin'
input=data,
# stdin=... <-- don't use this
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
print(p.stdout.decode("ascii"))
print(p.returncode)
# Hello, world!
# 0
I'm trying to execute:
actual = subprocess.run(['echo 123 | ./ft_ssl md5 -s ' + data + ' -p'], stdout=subprocess.PIPE)
actual = actual.stdout.decode('utf-8')
and after that variable actual equals "123 | ./ft_ssl md5 -s fuck -p\n".
Python run only echo for all the input and ignore | operation.
Help, please, what i have to do to run two commands with this logical operation?
You can avoid echo altogether. Piping can be simulated here by passing the echo argument to the stdin of the call to ./ft_ssl
actual = subprocess.Popen(['./ft_ssl', 'md5', '-s', data, '-p'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
output = actual.communicate(b'123')
See the docs for more details about communicate
I'm building a large project with SCONS, for reasons out of this topic (large story) I need to pass the object files options in the final linkage command inside a file.
Eg:
gcc -o program.elf #objects_file.txt -T linker_file.ld
This command works since I've tested it manually. But now I need to run it embedded in the Project build files. My first approach/idea has been to collect all the options into a file in the following way:
dbg_exe = own_env.Program('../' + target_path, components)
own_env.AddPreAction(dbg_exe, 'echo \'$SOURCES\' > objects_file.txt')
note: the $sources contains all the object files I need.
As I expected the command seems to be executed , I see the command printed in the terminal but for some reason it has not been executed since I don't find the objects_file.txt anywhere.
It's curious that if I copy & paste the printed lines in the same terminal the command execution is successful so I suppose the syntax constructed is correct.
I tried also a shorter test code:
own_env.AddPreAction(dbg_exe, 'ls -l > salida_ls.txt')
... and another surprise , this time I get syntax error in the console:
scons: done reading SConscript files.
scons: Building targets ...
ls -l > salida_ls.txt
ls: cannot access '>': No such file or directory
ls: cannot access 'salida_ls.txt': No such file or directory
a simple 'ls -l' works fine.
Any idea why this kind of bash commands don't work as expected? Is the > redirection symbol affecting the SCONS?
Some maybe useful information:
OS Windows10
Terminal mingw32
SCons v2.3.1
After searching I've found out that this is something related with the redefinition of the SPAWN construction variable:
def w32api_spawn(sh, escape, cmd, args, e_env):
print "CMD value"
print sh
print escape
print cmd
print args
print e_env
print " ********************************** "
if cmd == "SHELL":
return SCons.Platform.win32.spawn(sh,escape,args[1], args[1:],e_env)
cmdline = cmd + ' ' + string.join(args[1:], ' ')
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= _subprocess.STARTF_USESHOWWINDOW
proc = subprocess.Popen(
cmdline,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
startupinfo=startupinfo,
shell = False,
env = None
)
data, err = proc.communicate()
print data
rv = proc.wait()
if rv:
print "====="
print err
print "====="
return rv
Looks like you'll need to swap back to the default SPAWN for that Program().
Add this to the top of that SConscript
from SCons.Platform.win32 import spawn
Then replace the logic you pasted above with:
dbg_exe = own_env.Program('../' + target_path, components, SPAWN=spawn)
own_env.AddPreAction(dbg_exe, 'echo \'$SOURCES\' > objects_file.txt')
This assumes that you're only building on win32. If that's not true you'll need to conditionally add the SPAWN to your Program() above only when you're on win32.
Finally I found a workaround running a python native function to build th efile I needed. Unfortunately I cannot afford more time with this issue, I didn't find the real reason and solution but it is clear is not something related with the normal SCONS performing but with the trick performed in the SPAWN.
scons_common.GenerateObjectsFile('../' + objects_file, components)
I am trying to get a file name and pass it to a command using popen. Then I want to print the output. This is my code:
filePath = tkinter.filedialog.askopenfilename(filetypes=[("All files", "*.*")])
fileNameStringForm = (basename(filePath ))
fileNameByteForm = fileNameStringForm.encode(encoding='utf-8')
process = subprocess.Popen(['gagner','-arg1'], shell = True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process .communicate(fileNameByteForm )
stdout, stderr = process .communicate() <<------ERROR POINTS TO THIS LINE
stringOutput = stdout.decode('urf-8')
print(stringOutput)
I am getting the following error:
ValueError: Invalid file object: <_io.BufferedReader name=9>
I have looked at other similar questions but nothing seems to have solved my problem. Can some show me where I am going wrong in the code?
Edit:
If I were to run the command in a command line it would be:
gagner -arg1 < file1
What you are doing is not what you are describing in the supposed command line argument. You are actually executing this:
echo "file1" | gagner -arg1
You will need to make sure that you pass in the file contents yourself. Popen will not open and read the file for you.
According to the documentation, what communicate() does is
interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
So, once you have run
process.communicate(fileNameByteForm)
your sub process has finished and the pipes have been closed. The second call will then fail as a result.
What you want to do instead is
stdout, stderr = process.communicate(input_data)
which will pipe your input data into the sub process and read stdout and stderr.
I'm seeking for ways to write data to the existing process's STDIN from external processes, and found similar question How do you stream data into the STDIN of a program from different local/remote processes in Python? in stackoverlow.
In that thread, #Michael says that we can get file descriptors of existing process in path like below, and permitted to write data into them on Linux.
/proc/$PID/fd/
So, I've created a simple script listed below to test writing data to the script's STDIN (and TTY) from external process.
#!/usr/bin/env python
import os, sys
def get_ttyname():
for f in sys.stdin, sys.stdout, sys.stderr:
if f.isatty():
return os.ttyname(f.fileno())
return None
if __name__ == "__main__":
print("Try commands below")
print("$ echo 'foobar' > {0}".format(get_ttyname()))
print("$ echo 'foobar' > /proc/{0}/fd/0".format(os.getpid()))
print("read :: [" + sys.stdin.readline() + "]")
This test script shows paths of STDIN and TTY and then, wait for one to write it's STDIN.
I launched this script and got messages below.
Try commands below
$ echo 'foobar' > /dev/pts/6
$ echo 'foobar' > /proc/3308/fd/0
So, I executed the command echo 'foobar' > /dev/pts/6 and echo 'foobar' > /proc/3308/fd/0 from other terminal. After execution of both commands, message foobar is displayed twice on the terminal the test script is running on, but that's all. The line print("read :: [" + sys.stdin.readline() + "]") was not executed.
Are there any ways to write data from external processes to the existing process's STDIN (or other file descriptors), i.e. invoke execution of the lineprint("read :: [" + sys.stdin.readline() + "]") from other processes?
Your code will not work.
/proc/pid/fd/0 is a link to the /dev/pts/6 file.
$ echo 'foobar' > /dev/pts/6
$ echo 'foobar' > /proc/pid/fd/0
Since both the commands write to the terminal. This input goes to terminal and not to the process.
It will work if stdin intially is a pipe.
For example, test.py is :
#!/usr/bin/python
import os, sys
if __name__ == "__main__":
print("Try commands below")
print("$ echo 'foobar' > /proc/{0}/fd/0".format(os.getpid()))
while True:
print("read :: [" + sys.stdin.readline() + "]")
pass
Run this as:
$ (while [ 1 ]; do sleep 1; done) | python test.py
Now from another terminal write something to /proc/pid/fd/0 and it will come to test.py
I want to leave here an example I found useful. It's a slight modification of the while true trick above that failed intermittently on my machine.
# pipe cat to your long running process
( cat ) | ./your_server &
server_pid=$!
# send an echo to your cat process that will close cat and in my hypothetical case the server too
echo "quit\n" > "/proc/$server_pid/fd/0"
It was helpful to me because for particular reasons I couldn't use mkfifo, which is perfect for this scenario.