Is there any way for a writer to know that a reader has closed its end of a named pipe (or exited), without writing to it?
I need to know this because the initial data I write to the pipe is different; the reader is expecting an initial header before the rest of the data comes.
Currently, I detect this when my write() fails with EPIPE. I then set a flag that says "next time, send the header". However, it is possible for the reader to close and re-open the pipe before I've written anything. In this case, I never realize what he's done, and don't send the header he is expecting.
Is there any sort of async event type thing that might help here? I'm not seeing any signals being sent.
Note that I haven't included any language tags, because this question should be considered language-agnostic. My code is Python, but the answers should apply to C, or any other language with system call-level bindings.
If you are using an event loop that is based on the poll system call you can register the pipe with an event mask that contains EPOLLERR. In Python, with select.poll,
import select
fd = open("pipe", "w")
poller = select.poll()
poller.register(fd, select.POLLERR)
poller.poll()
will wait until the pipe is closed.
To test this, run mkfifo pipe, start the script, and in another terminal run, for example, cat pipe. As soon as you quit the cat process, the script will terminate.
Oddly enough, it appears that when the last reader closes the pipe, select indicates that the pipe is readable:
writer.py
#!/usr/bin/env python
import os
import select
import time
NAME = 'fifo2'
os.mkfifo(NAME)
def select_test(fd, r=True, w=True, x=True):
rset = [fd] if r else []
wset = [fd] if w else []
xset = [fd] if x else []
t0 = time.time()
r,w,x = select.select(rset, wset, xset)
print 'After {0} sec:'.format(time.time() - t0)
if fd in r: print ' {0} is readable'.format(fd)
if fd in w: print ' {0} is writable'.format(fd)
if fd in x: print ' {0} is exceptional'.format(fd)
try:
fd = os.open(NAME, os.O_WRONLY)
print '{0} opened for writing'.format(NAME)
print 'select 1'
select_test(fd)
os.write(fd, 'test')
print 'wrote data'
print 'select 2'
select_test(fd)
print 'select 3 (no write)'
select_test(fd, w=False)
finally:
os.unlink(NAME)
Demo:
Terminal 1:
$ ./pipe_example_simple.py
fifo2 opened for writing
select 1
After 1.59740447998e-05 sec:
3 is writable
wrote data
select 2
After 2.86102294922e-06 sec:
3 is writable
select 3 (no write)
After 2.15910816193 sec:
3 is readable
Terminal 2:
$ cat fifo2
test
# (wait a sec, then Ctrl+C)
There is no such mechanism. Generally, according to the UNIX-way, there are no signals for streams opening or closing, on either end. This can only be detected by reading or writing to them (accordingly).
I would say this is wrong design. Currently you are trying to have the receiver signal their availability to receive by opening a pipe. So either you implement this signaling in an appropriate way, or incorporate the "closing logic" in the sending part of the pipe.
Related
I have a script that opens a subprocess, this process awaits multiple inputs.
I tried with subprocess.Popen() but when combining the 3 standards streams it gets stuck in deadlocks...
So I tired to use wexpect (Apparently Windows version of pexecpt) it also didn't worked.
Now I'm using Sarge and my code right now looks like this:
import os
import subprocess
import sarge
import sys
def get_env():
env = os.environ.copy()
return env
def interactive(list_command: list, instruction_dict: dict):
env = get_env()
std_out = ""
for a_number in instruction_dict:
capture_stdout = sarge.Capture(buffer_size=-1)
capture_stderr = sarge.Capture(buffer_size=-1)
child_proc = sarge.Command(list_command, stdout=capture_stdout, stderr=capture_stderr,
shell=True, env=env)
child_proc.run(input=subprocess.PIPE,
async_=True)
print(capture_stdout.readlines(timeout=1.0), capture_stderr.readlines(timeout=1.0))
if instruction_dict[a_number][0] is not None:
#if capture_stdout.expect(instruction_dict[a_number][0]) is not None or capture_stderr.expect(instruction_dict[a_number][0]) is not None:
# self.test_manager.log_event(msg="found line : " + instruction_dict[a_number][0] + "in PIPE", current_event_type=LogLevel.DEBUG, screen_only=False)
print("in exec line : ", child_proc.stdout.readline(timeout=1.0), child_proc.stderr.readlines(timeout=1.0))
else:
print("in exec line : ", child_proc.stdout.readlines(timeout=1.0), child_proc.stderr.readlines(timeout=1.0))
if instruction_dict[a_number][1] is not None:
print(f"sending \"{instruction_dict[a_number][1]}\" to process in interactive")
temp_bytes = str.encode((instruction_dict[a_number][1]+"\n"))
child_proc.stdin.write(temp_bytes)
try:
child_proc.stdin.flush() # blind flush
except:
pass
print("Last line : ", child_proc.stdout.readlines(timeout=1.0), child_proc.stderr.readlines(timeout=1.0))
child_proc.wait()
std_out += child_proc.stdout.read().decode('utf-8')
child_proc.kill() # kill the subprocess
print("output: ", std_out)
interactive(list_command=["process.exe", "-delete_datamodel", "MODELNAME"],instruction_dict={1 : (None, "y")})
Output looks like this:
[] []
in exec line : [] []
send "y" to process in interactive
Last line : [] [b'User root connected to DB MODELNAME\r\n']
output: Are you sure to delete DB MODELNAME? (y/n)\n\nDelete Data Model Ok\n
In console it looks like this:
C:\>process.exe -delete_data_model -db_name MODELNAME
Are you sure to delete DB MODELNAME ? (y/n)
y
User root connected to DB MODELNAME
Delete Data Model Ok
There are multiple problems:
I can't catch the first stdout "Are you sure to delete DB MODELNAME ? (y/n)" during the loop I only get [].
capture_stdout.expect crash but in fact it might be the same problem as the first one, if it can't read the first stdout.
lines are going in stderr instead of stdout (it might be because of the process, I don't know how to test this)
for another command I ll need to answer more than one interactive by looking of the question before answering them that way my instruction_dict will look like this
instruction_dict = {1 : ("Are you sure to recover the DB", "y"),
2 : ("Do you want to stop the service", "y"),
3 : ("Do you want to stop the DB", "y"),
4 : ("Do you want to drop and create the DB", "y")}
Is this even possible? I have search for a working exemple and didn't found one, I know I might be bad at searching but...
Sorry for the terrible syntax(and bad english), I want to know if there is a pure python solution (I have one solution working in tcl but I'm trying to get rid of it/ I didn't wrote the tcl code).
I am beginning to think my search skills are lacking.
I trying to find any articles on how with Groovy, to open an interactive process, read its output and then write to the process depending on the output text. All I can find is how printing, reading and writing with files. Nothing about how to Write to a interactive process.
The process is asking for a password
Write the password to process
Something like this if possible:
def process = "some-command.sh".execute()
process.in.eachLine { line ->
if (line.contains("enter password")) {
process.out.write("myPassword")
}
}
This here works reading from the process output:
def process = "some-command.sh".execute()
process.in.eachLine { line ->
println line
}
Though it stops when the process is asking for input. It does not print out the line with the question.
Edit: Found out why it did not print the line with the ask password. It was not a new line. The question was a simple print (not println). How do I read when there is not yet a new line?
I have been told expect can be used, but I am looking for a solution which does not require a dependency.
1.bat
#echo off
echo gogogo
set /P V=input me:
echo V=%V%
this script waits for input just after :
gogogo
input me:
this means that eachLine not triggered for input me because no new line after it
however the previous line gogogo could be caught
and following script works for gogogo but does not work for input me
groovy
def process = "1.bat".execute()
process.in.eachLine { line ->
if (line.contains("gogogo")) {
process.out.write("myPassword\n".getBytes("UTF-8"))
process.out.flush()
}
}
groovy2
probably this could be optimized.. following script works without new line:
def process = "1.bat".execute()
def pout = new ByteArrayOutputStream()
def perr = new ByteArrayOutputStream()
process.consumeProcessOutput(pout, perr) //starts listening threads and returns immediately
while(process.isAlive()){
Thread.sleep(1234)
if(pout.toString("UTF-8").endsWith("input me:")){
process.out.write("myPassword\n".getBytes("UTF-8"))
process.out.flush()
}
}
I need to read output from a child process as it's produced -- perhaps not on every write, but well before the process completes. I've tried solutions from the Python3 docs and SO questions here and here, but I still get nothing until the child terminates.
The application is for monitoring training of a deep learning model. I need to grab the test output (about 250 bytes for each iteration, at roughly 1-minute intervals) and watch for statistical failures.
I cannot change the training engine; for instance, I cannot insert stdout.flush() in the child process code.
I can reasonably wait for a dozen lines of output to accumulate; I was hopeful of a buffer-fill solving my problem.
Code: variations are commented out.
Parent
cmd = ["/usr/bin/python3", "zzz.py"]
# test_proc = subprocess.Popen(
test_proc = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
out_data = ""
print(time.time(), "START")
while not "QUIT" in str(out_data):
out_data = test_proc.stdout
# out_data, err_data = test_proc.communicate()
print(time.time(), "MAIN received", out_data)
Child (zzz.py)
from time import sleep
import sys
for _ in range(5):
print(_, "sleeping", "."*1000)
# sys.stdout.flush()
sleep(1)
print("QUIT this exercise")
Despite sending lines of 1000+ bytes, the buffer (tested elsewhere as 2kb; here, I've gone as high as 50kb) filling doesn't cause the parent to "see" the new text.
What am I missing to get this to work?
Update with regard to links, comments, and iBug's posted answer:
Popen instead of run fixed the blocking issue. Somehow I missed this in the documentation and my experiments with both.
universal_newline=True neatly changed the bytes return to string: easier to handle on the receiving end, although with interleaved empty lines (easy to detect and discard).
Setting bufsize to something tiny (e.g. 1) didn't affect anything; the parent still has to wait for the child to fill the stdout buffer, 8k in my case.
export PYTHONUNBUFFERED=1 before execution did fix the buffering problem. Thanks to wim for the link.
Unless someone comes up with a canonical, nifty solution that makes these obsolete, I'll accept iBug's answer tomorrow.
subprocess.run always spawns the child process, and blocks the thread until it exits.
The only option for you is to use p = subprocess.Popen(...) and read lines with s = p.stdout.readline() or p.stdout.__iter__() (see below).
This code works for me, if the child process flushes stdout after printing a line (see below for extended note).
cmd = ["/usr/bin/python3", "zzz.py"]
test_proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
out_data = ""
print(time.time(), "START")
while not "QUIT" in str(out_data):
out_data = test_proc.stdout.readline()
print(time.time(), "MAIN received", out_data)
test_proc.communicate() # shut it down
See my terminal log (dots removed from zzz.py):
ibug#ubuntu:~/t $ python3 p.py
1546450821.9174328 START
1546450821.9793346 MAIN received b'0 sleeping \n'
1546450822.987753 MAIN received b'1 sleeping \n'
1546450823.993136 MAIN received b'2 sleeping \n'
1546450824.997726 MAIN received b'3 sleeping \n'
1546450825.9975247 MAIN received b'4 sleeping \n'
1546450827.0094354 MAIN received b'QUIT this exercise\n'
You can also do it with a for loop:
for out_data in test_proc.stdout:
if "QUIT" in str(out_data):
break
print(time.time(), "MAIN received", out_data)
If you cannot modify the child process, unbuffer (from package expect - install with APT or YUM) may help. This is my working parent code without changing the child code.
test_proc = subprocess.Popen(
["unbuffer"] + cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)
So I'm connected to an IRC chat using socket.connec
I log in by passing my variables via socket.send
The log in is successful and then I sit in a while true loop using
Socket.recv(1024)
If I just continually print the response everything looks fine, but let's say I want to add to the end of the string... I noticed that socket.recv doesn't always get the full message (only grabs up to 1024 as expected) and the remainder of the message is in the next iteration of the loop.
This makes its it impossible to process the feedback line by line.
Is there a better way to constantly read the data without it getting trunked? Is it possible to figure out the size of the response before receiving it so the buffer can be set dynamically?
TCP is a stream-based protocol. Buffer the bytes received and only extract complete messages from the stream.
For complete lines, look for newline characters in the buffer.
Example server:
import socket
class Client:
def __init__(self,socket):
self.socket = socket
self.buffer = b''
def getline(self):
# if there is no complete line in buffer,
# add to buffer until there is one.
while b'\n' not in self.buffer:
data = self.socket.recv(1024)
if not data:
# socket was closed
return ''
self.buffer += data
# break the buffer on the first newline.
# note: partition(n) return "left of n","n","right of n"
line,newline,self.buffer = self.buffer.partition(b'\n')
return line + newline
srv = socket.socket()
srv.bind(('',5000))
srv.listen(1)
conn,where = srv.accept()
client = Client(conn)
print(f'Client connected on {where}')
while True:
line = client.getline()
if not line:
break
print(line)
Example client:
s=socket()
s.connect(('127.0.0.1',5000))
s.sendall(b'line one\nline two\nline three\nincomplete')
s.close()
Output on server:
Client connected on ('127.0.0.1', 2667)
b'line one\n'
b'line two\n'
b'line three\n'
When controlling a process using a PTY master/slave pair, I would like to indicate to the process in question that stdin has closed and I have no more content to send, but I would still like to receive output from the process.
The catch is that I only have one file descriptor (the PTY "master") which handles both input from the child process and output to the child process. So closing the descriptor would close both.
Example in python:
import subprocess, pty, os
master,slave = pty.openpty()
proc = subprocess.Popen(["/bin/cat"], stdin=slave, stdout=slave)
os.close(slave) # now belongs to child process
os.write(master,"foo")
magic_close_fn(master) # <--- THIS is what I want
while True:
out = os.read(master,4096)
if out:
print out
else:
break
proc.wait()
You need to get separate read and write file descriptors. The simple way to do that is with a pipe and a PTY. So now your code would look like this:
import subprocess, pty, os
master, slave = pty.openpty()
child_stdin, parent_stdin = os.pipe()
proc = subprocess.Popen(["/bin/cat"], stdin=child_stdin, stdout=slave)
os.close(child_stdin) # now belongs to child process
os.close(slave)
os.write(parent_stdin,"foo") #Write to the write end (our end) of the child's stdin
#Here's the "magic" close function
os.close(parent_stdin)
while True:
out = os.read(master,4096)
if out:
print out
else:
break
proc.wait()
I had to do this today, ended up here and was sad to see no answer. I achieved this using a pair of ptys rather than a single pty.
stdin_master, stdin_slave = os.openpty()
stdout_master, stdout_slave = os.openpty()
def child_setup():
os.close(stdin_master) # only the parent needs this
os.close(stdout_master) # only the parent needs this
with subprocess.Popen(cmd,
start_new_session=True,
stderr=subprocess.PIPE,
stdin=stdin_slave,
stdout=stdout_slave,
preexec_fn=child_setup) as proc:
os.close(stdin_slave) # only the child needs this
os.close(stdout_slave) # only the child needs this
stdin_pty = io.FileIO(stdin_master, "w")
stdout_pty = io.FileIO(stdout_master, "r")
stdin_pty.write(b"here is your input\r")
stdin_pty.close() # no more input (EOF)
output = b""
while True:
try:
output += stdout_pty.read(1)
except OSError:
# EOF
break
stdout_pty.close()
I think that what you want is to send the CTRL-D (EOT - End Of Transmission) caracter, isn't you? This will close the input in some applications, but others will quit.
perl -e 'print qq,\cD,'
or purely shell:
echo -e '\x04' | nc localhost 8080
Both are just examples. BTW the CTRL-D caracter is \x04 in hexa.