I am trying to get a file name and pass it to a command using popen. Then I want to print the output. This is my code:
filePath = tkinter.filedialog.askopenfilename(filetypes=[("All files", "*.*")])
fileNameStringForm = (basename(filePath ))
fileNameByteForm = fileNameStringForm.encode(encoding='utf-8')
process = subprocess.Popen(['gagner','-arg1'], shell = True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process .communicate(fileNameByteForm )
stdout, stderr = process .communicate() <<------ERROR POINTS TO THIS LINE
stringOutput = stdout.decode('urf-8')
print(stringOutput)
I am getting the following error:
ValueError: Invalid file object: <_io.BufferedReader name=9>
I have looked at other similar questions but nothing seems to have solved my problem. Can some show me where I am going wrong in the code?
Edit:
If I were to run the command in a command line it would be:
gagner -arg1 < file1
What you are doing is not what you are describing in the supposed command line argument. You are actually executing this:
echo "file1" | gagner -arg1
You will need to make sure that you pass in the file contents yourself. Popen will not open and read the file for you.
According to the documentation, what communicate() does is
interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
So, once you have run
process.communicate(fileNameByteForm)
your sub process has finished and the pipes have been closed. The second call will then fail as a result.
What you want to do instead is
stdout, stderr = process.communicate(input_data)
which will pipe your input data into the sub process and read stdout and stderr.
Related
So how do I pass binary data using stdin to a executable command that I want to run using subprocess.run()?
The documentation is pretty vague about using the stdin for passing the data to external executable. I'm working on linux machine using python3 and I want to invoke dd of=/somefile.data bs=32 (which takes the input from stdin if I understand the man page correctly) and I have the binary data in bytearray that I want to pass to the command through stdin so that I do not have to write it to a temporary file and invoke the dd using that file as a input.
My requirement is simply to pass data that I have in bytearray to the dd command to be written to disk. What is the correct way to achieve this using the subprocess.run() and stdin?
Edit: I meant something like the following:
ba = bytearray(b"some bytes here")
#Run the dd command and pass the data from ba variable to its stdin
You can pass the output of one command to another by calling Popen directly.
file_cmd1 = <your dd command>
file_cmd2 = <command you want to pass dd output to>
proc1 = Popen(sh_split(file_cmd1), stdout=subprocess.PIPE)
proc2 = Popen(file_cmd2, [shell=True], stdin=proc1.stdout, stdout=subprocess.PIPE)
proc1.stdout.close()
This, as far as I know, will work just fine on a byte output from command 1.
In your case what you most like want to do is the following when you just want to pass data to the stdin of the process:
out = bytearray(b"Some data here")
p = subprocess.Popen(sh_split("dd of=/../../somefile.data bs=32"), stdin=subprocess.PIPE)
out = p.communicate(input=b''.join(out))[0]
print(out.decode())#Prints the output from the dd
Specifically for stdin to subprocess.run() as asked by the OP, use input as follows:
#!/usr/bin/python3
import subprocess
data = bytes("Hello, world!", "ascii")
p = subprocess.run(
"cat -", # The - means 'cat from stdin'
input=data,
# stdin=... <-- don't use this
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
print(p.stdout.decode("ascii"))
print(p.returncode)
# Hello, world!
# 0
I'm building a large project with SCONS, for reasons out of this topic (large story) I need to pass the object files options in the final linkage command inside a file.
Eg:
gcc -o program.elf #objects_file.txt -T linker_file.ld
This command works since I've tested it manually. But now I need to run it embedded in the Project build files. My first approach/idea has been to collect all the options into a file in the following way:
dbg_exe = own_env.Program('../' + target_path, components)
own_env.AddPreAction(dbg_exe, 'echo \'$SOURCES\' > objects_file.txt')
note: the $sources contains all the object files I need.
As I expected the command seems to be executed , I see the command printed in the terminal but for some reason it has not been executed since I don't find the objects_file.txt anywhere.
It's curious that if I copy & paste the printed lines in the same terminal the command execution is successful so I suppose the syntax constructed is correct.
I tried also a shorter test code:
own_env.AddPreAction(dbg_exe, 'ls -l > salida_ls.txt')
... and another surprise , this time I get syntax error in the console:
scons: done reading SConscript files.
scons: Building targets ...
ls -l > salida_ls.txt
ls: cannot access '>': No such file or directory
ls: cannot access 'salida_ls.txt': No such file or directory
a simple 'ls -l' works fine.
Any idea why this kind of bash commands don't work as expected? Is the > redirection symbol affecting the SCONS?
Some maybe useful information:
OS Windows10
Terminal mingw32
SCons v2.3.1
After searching I've found out that this is something related with the redefinition of the SPAWN construction variable:
def w32api_spawn(sh, escape, cmd, args, e_env):
print "CMD value"
print sh
print escape
print cmd
print args
print e_env
print " ********************************** "
if cmd == "SHELL":
return SCons.Platform.win32.spawn(sh,escape,args[1], args[1:],e_env)
cmdline = cmd + ' ' + string.join(args[1:], ' ')
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= _subprocess.STARTF_USESHOWWINDOW
proc = subprocess.Popen(
cmdline,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
startupinfo=startupinfo,
shell = False,
env = None
)
data, err = proc.communicate()
print data
rv = proc.wait()
if rv:
print "====="
print err
print "====="
return rv
Looks like you'll need to swap back to the default SPAWN for that Program().
Add this to the top of that SConscript
from SCons.Platform.win32 import spawn
Then replace the logic you pasted above with:
dbg_exe = own_env.Program('../' + target_path, components, SPAWN=spawn)
own_env.AddPreAction(dbg_exe, 'echo \'$SOURCES\' > objects_file.txt')
This assumes that you're only building on win32. If that's not true you'll need to conditionally add the SPAWN to your Program() above only when you're on win32.
Finally I found a workaround running a python native function to build th efile I needed. Unfortunately I cannot afford more time with this issue, I didn't find the real reason and solution but it is clear is not something related with the normal SCONS performing but with the trick performed in the SPAWN.
scons_common.GenerateObjectsFile('../' + objects_file, components)
I am trying to get grep to work using pipes and subprocess. I've double-checked the cat and I know it's working, but for some reason the grep isn't returning anything, even though when I run it through the terminal, it works just fine. I'm wondering if I have the command constructed correctly, as it doesn't give the desired output, and I can't figure out why.
I'm trying to retrieve a few specific lines of data from a file I've already retrieved from a server. I've been having a lot of issues with getting grep to work and perhaps I do not simply understand how it works.
p1 = subprocess.Popen(["cat", "result.txt"], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "tshaper"], stdin=p1.stdout,
stdout=subprocess.PIPE)
o = p1.communicate()
print(o)
p1.stdout.close()
out, err = p2.communicate()
print(out)
The output for the file I have when I run this command (cat result.txt | grep "tshaper") on the terminal:
tshaper.1.devname=eth0
tshaper.1.input.burst=0
tshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.rate=25000
tshaper.1.output.status=enabled
tshaper.1.status=enabled
tshaper.status=disabled
My results running the command in the script:
(b'', b'')
where the tuple is the stdout, stderr respectively of the p2 process.
EDIT:
I changed the command based on the Popen documentation to
p1 = subprocess.Popen(['result.txt', 'cat'], shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, cwd=os.getcwd())
to the p1 subprocess statement. While I was able to get the output in stderr, it didn't really change anything, saying
(b'', b'cat: 1: cat: result.txt: not found\n')
FYI: You got this error: (b'', b'cat: 1: cat: result.txt: not found\n') because you have changed the seq. of commands in your Popen method: ['result.txt', 'cat'] (Based on your question).
I have written a working solution which provides the expected output.
Python3.6.6 has been used for it.
result.txt file:
I have changed some lines to test the grep command.
tshaper.1.devname=eth0
ashaper.1.input.burst=0
bshaper.1.input.rate=25000
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
cshaper.1.output.rate=25000
tshaper.1.output.status=enabled
dshaper.1.status=enabled
tshaper.status=disabled
Code:
I have made an understandable printing but it is not necessary if you only need the output of grep. (It is a bytesl-like object in Python3)
import subprocess
p1 = subprocess.Popen(['cat', 'result.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2 = subprocess.check_output(["grep", "tshaper"], stdin=p1.stdout)
print("\n".join(p2.decode("utf-8").split(" ")))
Otuput:
You can see the grep filters the lines from cat command as it is expected.
>>> python3 test.py
tshaper.1.devname=eth0
tshaper.1.input.status=enabled
tshaper.1.output.burst=0
tshaper.1.output.status=enabled
tshaper.status=disabled
I'm using the the following function to fork commands in my Turtle script:
forkCommand shellCommand = do
pid <- inshell (shellCommand <> "& echo $!") empty
return $ PID (lineToText pid)
The reason for doing this is because I want to get the PID of the forked process that I'm running.
The issue is that the command I'm ruining isn't streaming any stdout. For example you could set shellCommand to:
"python -c \"print('Hello, World')\""
and you won't see the print occur.
I'm having some trouble using FIFOs for stdin.
I have a script like this:
#!/usr/bin/env ruby
while true do
data = gets
puts "Got: #{data}"
end
Then I run it like:
$ ./script < input.fifo &
$ echo testdata > input.fifo
It will print something like:
Got: testdata
Got:
Got:
Got:
Got:
Got:
etc.
My suspicion is that something is wrong with the FIFO. That something is not getting cleared out after it is sent to the script.
I tried the same thing with a C program with a similar input loop using a scanf("%d" ...) and it acted like this:
$ echo 1 > input.fifo
Got: 1
Got: 1
Got: 1
Got: 1
etc.
So it would seem that the last thing in the FIFO gets stuck there. In the ruby example, it is a null line, because gets captures the \n. In the second it is the 1 itself.
Can anyone offer any insight?
Thanks!
The situation is simple, after:
'echo 1 > input.fifo', the file 'input.fifo' was opened, "1" was written to it,
and it is closed.
The problem is that it is closed. When fifo closed from writing side, this is equal "end of file" for reading side. So if you check return code from "scanf" in your C example it will be equal to EOF constant.
And after "end of file" when you read all data from fifo, and try read something,
the "reading" code will alaways return immediately and report "end of file".