I'm trying to read non-blocking from subprocess pipe on windows python3.
I made an implementation that pipes a FFMPEG video stream. It works on Linux but since FNCTL doesn't work on windows I'm looking at other ways of avoiding read/write blocking.
Since readline() isn't blocking the read stream this solution should work, however I need to recreate the read(bytes) function with readline(). I need to be able to read entire frames of size X (1920*1080*3) from the stream with readline. I'm guessing '\n' occurences needs to be handled aswell.
The code so far:
def enqueue_output(self, out, queue):
# This function should read(SIZE) from stream and add it to queue
# IT DOES NOT CURRENTLY WORK
temp = bytes()
SIZE = 1920*1080*3
while True:
for line in iter(out.readline, b''):
if len(temp) >= SIZE:
queue.put(line)
temp += line
ffmpeg = sp.Popen(ffmpegCmd, stdin=sp.PIPE,
stdout=sp.PIPE,stderr=sp.PIPE, close_fds=ON_POSIX, shell=True)
q = Queue()
t = Thread(target=self.enqueue_output, args=(ffmpeg.stdout, q))
t.daemon = True # thread dies with the program
t.start()
The Read function. Follow question, would it be better to have the byte assemble code in the read function?
img_thread = threading.Thread(target=self.reader,
args=(q,))
img_thread.start()
def reader(self, q): # TODO : catch exceptions
s_buf = ''
while True:
# read line without blocking
try: line = q.get_nowait() # or q.get(timeout=.1)
except Empty:
print('no output yet')
else: # got line
print(line)
try:
img = np.frombuffer(line, dtype=np.uint8)
img = img.reshape((1080, 1920, 3))
except Exception as e:
print(e)
pass
else:
cv2.imshow("test", img)
cv2.waitKey(1)
Some Example data of how it looks when i just send readline() :
b'\x05\x07\t\x06\x08\n'
b'\x05\x07\t\x06\x08\n'
b'\x07\t\x0b\x05\x07\t\t\x0b\r\x06\x08\n'
b'\x00\x02\x04\x01\x03\x05\x05\x07\t\x05\x07\t\x05\x07\t\x05\x07\t\x01\x03\x05\x00\x02\x04\t\x0b\r\x06\x08\n'
b'\x05\x08\x08\x05\x08\x08\x03\x06\x06\x01\x04\x04\x02\x05\x05\x06\t\t\x05\x08\x08\x05\x08\x08\x05\x08\x08\x05\x08\x08\x05\x08\x08\x05\x08\x08\x05\x08\x08\x05\x08\x08\x02\x04\x06\x05\x07\t\x05\x07\t\x02\x04\x06\x05\x07\t\x05\x07\t\x05\x07\t\x05\x07\t\x05\x07\t\x06\x08\n'
b'\x05\x07\t\x00\x00\x02\x07\t\x0b\x03\x05\x07\x00\x00\x00\x00\x00\x00\x03\x02\x04\x02\x01\x03\t\x08\n'
b'\x0c\x0b\r\x08\x07\t\x06\x05\x07\x04\x03\x05\x00\x00\x00\x05\x04\x06\x05\x04\x06\x05\x04\x06\x05\x04\x06\x06\x05\x07\x06\x05\x07\x06\x05\x07\x06\x05\x07\x06\x05\x07\x05\x04\x06\x06\x05\x07\x06\x05\x07\x05\x04\x06\x05\x04\x06\x05\x04\x06\x06\x05\x07\x05\x04\x06\x08\x07\t\t\x08\n'
b'\t\x08\n'
b'\x08\x07\t\x06\x05\x07\x05\x04\x06\x06\x05\x07\x08\x07\t\x02\x01\x03\x02\x01\x03\x08\x07\t\x06\x05\x07\x06\x05\x07\x06\x05\x07\x08\x07\t\x06\x05\x07\x06\x05\x07\x06\x05\x07\x06\x05\x07\x05\x04\x06\x05\x04\x06\x05\x04\x06\x05\x04\x06\x04\x04\x04\x05\x05\x05\x04\x04\x04\x04\x04\x04\x05\x05\x05\x05\x05\x05\x04\x04\x04\x05\x05\x05\x03\x03\x03\x03\x03\x03\x05\x05\x05\x06\x06\x06\x08\x08\x08\x06\x06\x06\x05\x05\x05\x02\x02\x02\x00\x00\x00\x04\x00\x03\x08\x04\x07\n'
b'\x06\t\x08\x04\x07\x07\x03\x06\x07\x03\x06\x07\x03\x06\x07\x03\x06\x07\x03\x06\x06\x02\x05\x05\x01\x04\x06\x02\x05\x06\x02\x05\x07\x03\x06\x08\x04\x07\x03\x03\x03\n'
b'\n'
b'\n'
b'\t\t\t\x06\x06\x06\t\t\t\x05\x05\x05\x04\x04\x04\t\t\t\x06\x06\x06\x05\x05\x05\x05\x05\x05\x06\x06\x06\x06\x06\x06\x08\x08\x08\x08\x08\x08\x08\x08\x08\x06\x06\x06\x06\x06\x06\x05\x05\x05\x04\x04\x04\x04\x04\x04\x05\x05\x05\x05\x05\x05\x05\x05\x05\x01\x01\x01\x05\x05\x05\x05\x05\x05\x02\x02\x02\x03\x03\x03\x08\x08\x08\x08\x08\x08\x04\x04\x04\x06\x03\x04\x06\x03\x04\x05\x02\x03\x05\x02\x03\x06\x03\x04\x06\x03\x04\x05\x02\x03\x05\x02\x03\x05\x02\x03\x06\x03\x04\x06\x03\x04\x05\x02\x03\x05\x02\x03\x05\x02\x03\x05\x02\x03\x04\x01\x02\x01\x01\x01\x00\x00\x00\x03\x03\x03\x05\x05\x05\x06\x06\x06\x08\x08\x08\x06\x06\x06\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x04\x04\x04\x05\x05\x05\x05\x05\x05\x05\x05\x05\x06\x06\x06\x06\x06\x06\x05\x05\x05\x05\x05\x05\x04\x04\x04\x04\x04\x04\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x06\x06\x06\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x04\x04\x04\x04\x04\x04\x03\x03\x03\x03\x03\x03\x06\x06\x06\x08\x08\x08\x02\x02\x02\x05\x05\x05\r\r\r\x11\x11\x11\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x07\x04\x05\x06\x03\x04\x06\x03\x04\x04\x01\x02\x06\x03\x04\x0b\x08\t\x08\x05\x06\x04\x01\x02\x07\x04\x05\x06\x03\x04\x06\x03\x04\x06\x03\x04\x06\x03\x04\x06\x03\x04\x06\x03\x04\x06\x03\x04\x04\x04\x04\x04\x04\x04\x00\x03\x03\x00\x03\x03\x00\x03\x03\x00\x03\x03\x03\x06\x06\x03\x06\x06\x02\x05\x05\x02\x05\x05\x01\x04\x04\x01\x04\x04\x01\x04\x04\x01\x04\x04\x04\x04\x04\x04\x04\x04\x05\x02\x03\x05\x02\x03\x05\x02\x03\x05\x02\x03\x06\x03\x04\x06\x03\x04\x06\x03\x04\x08\x05\x06\x08\x05\x06\x08\x05\x06\x04\x01\x02\x04\x01\x02\x05\x02\x03\x06\x03\x04\x07\x04\x05\x07\x04\x05\x05\x05\x05\x05\x05\x05\x03\x03\x03\x03\x03\x03\x03\x03\x03\x03\x03\x03\x04\x04\x04\x06\x06\x06\x05\x05\x05\x04\x04\x04\x03\x03\x03\x04\x04\x04\x05\x05\x05\x05\x05\x05\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x03\x03\x03\x03\x03\x03\x03\x03\x03\x03\x03\x03\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x01\x04\x04\x01\x04\x04\x01\x04\x04\x00\x03\x03\x02\x05\x05\x00\x01\x01\x00\x01\x01\x02\x05\x05\x03\x06\x06\x02\x05\x05\x00\x02\x02\x00\x03\x03\x00\x03\x03\x01\x04\x04\x02\x05\x05\x02\x05\x05\x01\x03\x05\x01\x03\x05\x00\x01\x06\x00\x01\x06\x01\x02\x07\x01\x02\x07\x01\x02\x07\x02\x03\x08\x02\x03\x08\x02\x03\x08\x00\x01\x06\x00\x01\x06\x01\x04\x04\x01\x04\x04\x01\x04\x04\x02\x05\x05\x02\x05\x05\x03\x06\x06\x02\x05\x05\x02\x05\x05\x00\x03\x03\x00\x03\x03\x00\x03\x03\x00\x03\x03\x00\x03\x03\x00\x03\x03\x00\x02\x02\x00\x02\x02\x00\x01\x01\x00\x02\x02\x02\x05\x05\x01\x04\x04\x04\x04\x04\x03\x03\x03\x04\x04\x04\x03\x03\x03\x04\x04\x04\x04\x04\x04\x02\x05\x05\x02\x05\x05\x01\x04\x04\x05\x08\x08\x07\x0c\x0c\x06\x0b\x0b\x04\t\t\x06\x0b\x0b\x05\n'
b'\n'
b'\x00\x02\x02\x03\x08\x08\x03\x08\x08\x03\x08\x08\x03\x08\x08\x03\x08\x08\x03\x08\x08\x01\x06\x06\x01\x06\x06\x03\x08\x08\x03\x08\x08\x04\t\t\x04\t\t\x03\x08\x08\x03\x08\x08\x03\x08\x08\x03\x08\x08\x04\t\t\x04\t\t\x04\t\t\x04\t\t\x04\t\t\x04\t\t\x04\t\t\x04\t\t\x03\x08\x08\x03\x08\x08\x03\x08\x08\x03\x08\x08\x04\t\t\x04\t\t\x04\t\t\x05\n'
b'\n'
b'\t\x0c\x0c\t\x0c\x0c\x03\x06\x06\x02\x05\x05\x03\x06\x06\x06\t\t\t\x0c\x0c\x08\x0b\x0b\x08\x0b\x0b\x06\t\t\x05\x08\x08\x02\x05\x05\x00\x03\x03\x03\x06\x06\x06\t\t\x03\x06\x06\x06\t\t\x01\x04\x04\x01\x04\x04\x05\x08\x08\x06\t\t\x02\x05\x05\x05\x08\x08\n'
b'\r\r\x02\x05\x05\x06\t\t\x08\x0b\x0b\x06\t\t\x05\x08\x08\x07\n'
Any help would be appreciated!
If anyone has a better advice on how to avoid the read/write block on subprocess for windows that would be great too.
Related
I am trying to write some non-blocking FIFO code with kqueue on my BSD machine. Here's the small server code: server.py
import os
import selectors
sel = selectors.KqueueSelector()
TMP_PATH="/tmp/myfifo"
def fifo_read(fd, mask):
data = os.read(fd, 8)
print("fd:{} gives:{} \n", fd, data)
sel.unregister(fd)
print("unregistered")
def fifo_accept(listen_fd, mask):
print("accepted {}".format(listen_fd))
fd = os.dup(listen_fd)
print("duped to {}".format(fd))
sel.register(fd, selectors.EVENT_READ, fifo_read)
if __name__ == "__main__":
try:
os.unlink(TMP_PATH)
except:
pass
os.mkfifo(TMP_PATH)
listen_fd = os.open(TMP_PATH, os.O_RDONLY, mode=0o600)
sel.register(listen_fd, selectors.EVENT_READ, fifo_accept)
while True:
events = sel.select()
for key, mask in events:
cb = key.data
cb(key.fileobj, mask)
sel.close()
Now, when I run a client.py as:
import os
TMP_PATH="/tmp/myfifo"
fd = os.open(TMP_PATH, os.O_WRONLY, mode=0o600)
res = os.write(fd, b"1234567")
print("sent {}".format(res))
When I run the client, I get:
sent 7
But on server, it runs to inifinite loop. Now I understand why the infinite loop is happening. I actually tried mimicking the socket way of using selectors in this Python Docs example.
Here's what I have tried:
I did try the code without duplicating the fd, but it's still in infinite loop.
I tried calling sel.unregister on the original listen_fd, but in this case, running the client the second time doesn't work (which is expected).
Can anyone please let me know if I'm missing something?
So I found one solution to this problem. With sockets, we get a new socket object on accept. So we need to emulate that behaviour by calling unregister on the original fileobj, open again and call register on that.
Fixed code:
import os
import selectors
sel = selectors.KqueueSelector()
try:
os.unlink("./myfifo")
except:
pass
os.mkfifo("./myfifo", 0o600)
def cb(fp):
sel.unregister(fp)
print(f"got {fp.read()}")
fp.close()
fp2 = open("./myfifo", "rb")
sel.register(fp2, selectors.EVENT_READ, cb)
if __name__ == "__main__":
orig_fp = open("./myfifo", "rb")
print("open done")
ev = sel.register(orig_fp, selectors.EVENT_READ, cb)
print(f"registration done for {ev}")
while True:
events = sel.select()
print(events)
for key, mask in events:
key.data(key.fileobj)
I've been programming for a while in python but this is my first in multiprocessing.
I made a program that scrapes a local weather station for the ambient temperature using beautifulsoup4 every minute. The program also reads temperatures from several sensors and uploads everything to a Mysql database. This all works fine but on occasion (once every day) getting the data from the local weather station fails in retrieving the webpage. This causes beautifulsoup to start an infinite loop which effectively stops all functionality of the program. To combat this I tried to try my hand on multiprocessing.
I've coded a check that kills the extra thread if that is still running after 10 seconds. Here is where things go wrong, normally the beautifulsoup thread closes after 2-4 seconds when its finished. However in the case where the beautifulsoup gets stuck in its loop not only the thread is terminated but the entire program stops doing stuff altogether.
I've copied the relevant snippets of code. Please note that some vars are declared outside of the snippets, the code works with exception of the problem described above. Btw I am very much aware that there is a plethora of ways to make my code more efficient. Refining the code is something that I'll do when its working stable :) Thanks in advance for your help!
Imports:
...
from multiprocessing import Process, Queue
import multiprocessing
from bs4 import BeautifulSoup #sudo apt-get install python3-bs4
Beutifulsoup section:
def get_ZWS_temp_out(temp):
try:
if 1==1:
response = requests.get(url)
responsestr = str(response)
if "200" in responsestr:
soup = BeautifulSoup(response.content, 'html.parser')
tb = soup.findAll("div", {"class": "elementor-element elementor-element-8245410 elementor-widget__width-inherit elementor-widget elementor-widget-wp-widget-live_weather_station_widget_outdoor"})
tb2 = tb[0].findAll("div", {"class": "lws-widget-big-value"})
string = str(tb2[0])[-10:][:4]
stringt = string[:1]
if stringt.isdigit() == True:
#print("getal ok")
string = string
elif stringt == '-':
#print("minteken")
string = string
elif stringt == '>':
#print("temp < 10")
string = string[-3:]
temp = float(string)
except Exception as error:
print(error)
Q.put(temp)
return(temp)
Main program:
Q = Queue()
while 1 == 1:
strings = time.strftime("%Y,%m,%d,%H,%M,%S")
t = strings.split(',')
time_numbers = [ int(x) for x in t ]
if last_min != time_numbers[4]:
targettemp = get_temp_target(targettemp)
p = Process(target=get_ZWS_temp_out, name="get_ZWS_temp_out", args=(ZWS_temp_out,))
p.start()
i = 0
join = True
while i < 10:
i = i + 1
time.sleep(1)
if p.is_alive() and i == 10: #checks to quit early otherwise another iteration
print(datetime.datetime.fromtimestamp(time.time()).strftime("%Y-%m-%d %H:%M:%S"),": ZWS getter is running for too long... let's kill it...")
# Terminate ZWS query
p.terminate()
i = 10
join = False
if join == True:
p.join()
Thanks in advance for your time :)
I have to manually stop the program which gives the following output:
pi#Jacuzzi-pi:~ $ python3 /home/pi/Jacuzzi/thermometer.py
temperature sensors observer and saving program, updates every 3,5 seconds
2019-10-28 03:50:11 : ZWS getter is running for too long... let's kill it...
^CTraceback (most recent call last):
File "/home/pi/Jacuzzi/thermometer.py", line 283, in <module>
ZWS_temp_out = Q.get()
File "/usr/lib/python3.5/multiprocessing/queues.py", line 94, in get
res = self._recv_bytes()
File "/usr/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
I believe your program is waiting infinitely to pull items from the queue you've created. I can't see the line in the code you've posted, but it appears in the error message:
ZWS_temp_out = Q.get()
Since the get_ZWS_temp_out process is the one that adds items to the queue, you need to make sure that the process is running before you call Q.get(). I suspect this line of code gets executed between the act of terminating the timed-out process and restarting a new process, where instead it should be called after the new process is created.
Based on what Rob found out this is the updated (working) code for the main program, the others are unchanged:
Q = Queue()
while 1 == 1:
strings = time.strftime("%Y,%m,%d,%H,%M,%S")
t = strings.split(',')
time_numbers = [ int(x) for x in t ]
if last_min != time_numbers[4]:
targettemp = get_temp_target(targettemp)
p = Process(target=get_ZWS_temp_out, name="get_ZWS_temp_out", args=(ZWS_temp_out,))
p.start()
i = 0
completion = True
while i < 10:
i = i + 1
time.sleep(1)
if p.is_alive() and i == 10: #checks to quit early otherwise another iteration
print(datetime.datetime.fromtimestamp(time.time()).strftime("%Y-%m-%d %H:%M:%S"),": ZWS getter is running for too long... let's kill it...")
# Terminate ZWS query
p.terminate()
i = 10
completion = False
if completion == True:
p.join()
ZWS_temp_out = Q.get()
The code below used to work in spyder to communicate over serial with an arduino. In the console window of spyder, I would see lines of data being printed out:
78.7,77.9,100,80
78.7,77.9,100,80
78.7,77.9,100,80
78.7,77.9,100,80 ...
Data is from two temperature probes, a flowmeter, and the thermostat set temp.
I upgraded my Kubuntu 18.04 system to all things python3. Now, the code runs, but the spyder3 console window shows no visible characters, but scrolls blank lines. The rest of my python code for parsing and plotting this data doesn't work.
I've spent all day trying to fix this with no luck. I'm guessing it's a simple fix for someone with more experience than me.
The only difference between the old working code and the code below is that the print statements have parentheses added to remove the syntax error.
python
""" This code was originally copied from:
Listen to serial, return most recent numeric values
Lots of help from here:
http://stackoverflow.com/questions/1093598/pyserial-how-to-read-last-line-sent-from-serial-device
"""
from threading import Thread
import time
import serial
last_received = ''
def receiving(ser):
global last_received
buffer = ''
while True:
buffer = buffer + ser.read(ser.inWaiting())
if '\n' in buffer:
lines = buffer.split('\n') # Guaranteed to have at least 2 entries
last_received = lines[-2]
#If the Arduino sends lots of empty lines, you'll lose the
#last filled line, so you could make the above statement conditional
#like so: if lines[-2]: last_received = lines[-2]
buffer = lines[-1]
class SerialData(object):
def __init__(self, init=50):
try:
self.ser = serial.Serial(
port='/dev/ttyACM0',
baudrate=9600,
bytesize=serial.EIGHTBITS,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
timeout=0.1,
xonxoff=0,
rtscts=0,
interCharTimeout=None
)
except serial.serialutil.SerialException:
#no serial connection
self.ser = None
else:
Thread(target=receiving, args=(self.ser,)).start()
def next(self):
if not self.ser:
return '81.3,78.1,10.0,60.0,0' #100 #return anything so we can test when Arduino isn't connected
#return a float value or try a few times until we get one
for i in range(40):
raw_line = last_received
try:
# return float(raw_line.strip())
return str(raw_line.strip())
except ValueError:
print('bogus data',raw_line)
time.sleep(.005)
return 0.
def __del__(self):
if self.ser:
self.ser.close()
def write(self,val):
self.ser.write(val)
if __name__=='__main__':
s = SerialData()
for i in range(500):
time.sleep(.015)
print( s.next())
One of the most significant differences between Python 2.x and 3.x is the way text strings are encoded. For Python 3.x everything is Unicode, as compared to ASCII for 2.x, so you just need to decode the raw bytes you read from the serial port:
buffer = buffer + ser.read(ser.inWaiting()).decode('utf-8')
EDIT: now you seem to have a different problem involving an exception. It looks like your port is open, to be sure you can change the way you handle the exception when you instantiate the port:
except serial.serialutil.SerialException as e:
print(e)
self.ser = None
Once you know the error you should be able to handle it. Quite likely your port was not properly closed in an earlier session.
Bingo! That last suggestion fixed the program and the python GUI interface I wrote for it is working with these corrections.
python
from threading import Thread
import time
import serial
last_received = ''
def receiving(ser):
global last_received
buffer = ''
while True:
buffer = buffer + ser.read(ser.inWaiting()).decode('utf-8')
if '\n' in buffer:
lines = buffer.split('\n') # Guaranteed to have at least 2 entries
last_received = lines[-2]
#If the Arduino sends lots of empty lines, you'll lose the
#last filled line, so you could make the above statement conditional
#like so: if lines[-2]: last_received = lines[-2]
buffer = lines[-1]
class SerialData(object):
def __init__(self, init=50):
try:
self.ser = serial.Serial(
port='/dev/ttyACM1',
baudrate=9600,
bytesize=serial.EIGHTBITS,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
timeout=0.1,
xonxoff=0,
rtscts=0,
interCharTimeout=None
)
except serial.serialutil.SerialException as e:
print(e)
#no serial connection
self.ser = None
else:
Thread(target=receiving, args=(self.ser,)).start()
def next(self):
# if not self.ser:
# return '81.3,78.1,10.0,60.0,0' #100 #return anything so we can test when Arduino isn't connected
#return a float value or try a few times until we get one
for i in range(40):
raw_line = last_received
try:
# return float(raw_line.strip())
return str(raw_line.strip())
except ValueError:
print('bogus data',raw_line)
time.sleep(.005)
return 0.
def __del__(self):
if self.ser:
self.ser.close()
def write(self,val):
self.ser.write(val)
if __name__=='__main__':
s = SerialData()
for i in range(500):
time.sleep(.015)
print (s.next())
I'm parsing the last line of a continuously updating log file. If it matches, I want to return the match to a list and start another function using that data. I need to keep watching for new entries and parse them even while the new function continues.
I've been working this from a few different angles for about a week with varying success. I tried threading, but ran into issues getting the return value, I tried using a global var but couldn't get it working. I'm now trying asyncio, but having even more issues getting that to work.
def tail():
global match_list
f.seek(0, os.SEEK_END)
while True:
line = f.readline()
if not line:
time.sleep(0.1)
continue
yield line
def thread():
while True:
tail()
def somefun(list):
global match_list
#do things here
pass
def main():
match_list = []
f = open(r'file.txt')
thread=threading.Thread(target=thread, args=(f,))
thread.start()
while True:
if len(match_list) >= 1:
somefun(match_list)
if __name__ == '__main__':
main()
Wrote the above from memory..
I want tail() to return the line to a list that somefun() can use.
I'm having issues getting it to work, I will use threading or asyncio.. anything to get it running at this point.
In asyncio you might use two coroutines, one that reads from file, and the other that processes the file. Since they communicate using queue, they don't need the global variable. For example:
import os, asyncio
async def tail(f, queue):
f.seek(0, os.SEEK_END)
while True:
line = f.readline()
if not line:
await asyncio.sleep(0.1)
continue
await queue.put(line)
async def consume(queue):
lines = []
while True:
next_line = await queue.get()
lines.append(next_line)
# it is not clear if you want somefun to receive the next
# line or *all* lines, but it's easy to do either
somefun(next_line)
def somefun(line):
# do something with line
print(f'line: {line!r}')
async def main():
queue = asyncio.Queue()
with open('file.txt') as f:
await asyncio.gather(tail(f, queue), consume(queue))
if __name__ == '__main__':
asyncio.run(main())
# or, on Python older than 3.7:
#asyncio.get_event_loop().run_until_complete(main())
The beauty of an asyncio-based solution is that you can easily start an arbitrary number of such coroutines in parallel (e.g. you could start gather(main1(), main2()) in an outer coroutine, and run that), and have them all share the same thread.
with a few small fixes you almost run this :) (comments inside)
match_list # should be at the module scope
def tail():
# f = open(...) ???
f.seek(0, os.SEEK_END)
while True:
line = f.readline()
if not line:
time.sleep(0.1)
continue
yield line
def thread():
for line in tail():
match_list.append(line) # append line
print("thread DONE!")
def somefun(list):
#do things here
while match_list:
line = match_list.pop(0)
print(line)
def main():
match_list = []
f = open(r'file.txt')
thread=threading.Thread(target=thread, args=(f,))
thread.start()
while True:
if match_list:
somefun(match_list)
time.sleep(0.1) # <-- don't burn the CPU :)
I'm working on a bot for a competition that receives its input through sys.stdin and uses Python's print() for output. I have the following:
import sys
def main():
while True:
line = sys.stdin.readline()
parts = line.split()
if len(parts) > 0:
# do stuff
The problem is that the input comes in through a stream and using the above, blocks me from printing anything back until the stream is closed. What can I do to make this work?
By turning blocking off you can only read a character at a time. So, there is no way to get readline() to work in a non-blocking context. I assume you just want to read key presses to control the robot.
I have had no luck using select.select() on Linux and created a way with tweaking termios settings. So, this is Linux specific but works for me:
import atexit, termios
import sys, os
import time
old_settings=None
def init_any_key():
global old_settings
old_settings = termios.tcgetattr(sys.stdin)
new_settings = termios.tcgetattr(sys.stdin)
new_settings[3] = new_settings[3] & ~(termios.ECHO | termios.ICANON) # lflags
new_settings[6][termios.VMIN] = 0 # cc
new_settings[6][termios.VTIME] = 0 # cc
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, new_settings)
#atexit.register
def term_any_key():
global old_settings
if old_settings:
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
def any_key():
ch_set = []
ch = os.read(sys.stdin.fileno(), 1)
while ch is not None and len(ch) > 0:
ch_set.append( ord(ch[0]) )
ch = os.read(sys.stdin.fileno(), 1)
return ch_set
init_any_key()
while True:
key = any_key()
if key is not None:
print(key)
else:
time.sleep(0.1)
A better Windows or cross-platform answer is here: Non-blocking console input?
You can use selectors for handle I/O multiplexing:
https://docs.python.org/3/library/selectors.html
Try this out:
#! /usr/bin/python3
import sys
import fcntl
import os
import selectors
# set sys.stdin non-blocking
orig_fl = fcntl.fcntl(sys.stdin, fcntl.F_GETFL)
fcntl.fcntl(sys.stdin, fcntl.F_SETFL, orig_fl | os.O_NONBLOCK)
# function to be called when enter is pressed
def got_keyboard_data(stdin):
print('Keyboard input: {}'.format(stdin.read()))
# register event
m_selector = selectors.DefaultSelector()
m_selector.register(sys.stdin, selectors.EVENT_READ, got_keyboard_data)
while True:
sys.stdout.write('Type something and hit enter: ')
sys.stdout.flush()
for k, mask in m_selector.select():
callback = k.data
callback(k.fileobj)
The above code will hold on the line
for k, mask in m_selector.select():
until a registered event occurs, returning a selector_key instance (k) and a mask of monitored events.
In the above example we registered only one event (Enter key press):
m_selector.register(sys.stdin, selectors.EVENT_READ, got_keyboard_data)
The selector key instance is defined as follows:
abstractmethod register(fileobj, events, data=None)
Therefore, the register method sets k.data as our callback function got_keyboard_data, and calls it when the Enter key is pressed:
callback = k.data
callback(k.fileobj)
A more complete example (and hopefully more useful) would be to multiplex stdin data from user with incoming connections from network:
import selectors
import socket
import sys
import os
import fcntl
m_selector = selectors.DefaultSelector()
# set sys.stdin non-blocking
def set_input_nonblocking():
orig_fl = fcntl.fcntl(sys.stdin, fcntl.F_GETFL)
fcntl.fcntl(sys.stdin, fcntl.F_SETFL, orig_fl | os.O_NONBLOCK)
def create_socket(port, max_conn):
server_addr = ('localhost', port)
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.setblocking(False)
server.bind(server_addr)
server.listen(max_conn)
return server
def read(conn, mask):
global GO_ON
client_address = conn.getpeername()
data = conn.recv(1024)
print('Got {} from {}'.format(data, client_address))
if not data:
GO_ON = False
def accept(sock, mask):
new_conn, addr = sock.accept()
new_conn.setblocking(False)
print('Accepting connection from {}'.format(addr))
m_selector.register(new_conn, selectors.EVENT_READ, read)
def quit():
global GO_ON
print('Exiting...')
GO_ON = False
def from_keyboard(arg1, arg2):
line = arg1.read()
if line == 'quit\n':
quit()
else:
print('User input: {}'.format(line))
GO_ON = True
set_input_nonblocking()
# listen to port 10000, at most 10 connections
server = create_socket(10000, 10)
m_selector.register(server, selectors.EVENT_READ, accept)
m_selector.register(sys.stdin, selectors.EVENT_READ, from_keyboard)
while GO_ON:
sys.stdout.write('>>> ')
sys.stdout.flush()
for k, mask in m_selector.select():
callback = k.data
callback(k.fileobj, mask)
# unregister events
m_selector.unregister(sys.stdin)
# close connection
server.shutdown()
server.close()
# close select
m_selector.close()
You can test using two terminals.
first terminal:
$ python3 test.py
>>> bla
open another terminal and run:
$ nc localhost 10000
hey!
back to the first
>>> qwerqwer
Result (seen on the main terminal):
$ python3 test.py
>>> bla
User input: bla
>>> Accepting connection from ('127.0.0.1', 39598)
>>> Got b'hey!\n' from ('127.0.0.1', 39598)
>>> qwerqwer
User input: qwerqwer
>>>
#-----------------------------------------------------------------------
# Get a character from the keyboard. If Block is True wait for input,
# else return any available character or throw an exception if none is
# available. Ctrl+C isn't handled and continues to generate the usual
# SIGINT signal, but special keys like the arrows return the expected
# escape sequences.
#
# This requires:
#
# import sys, select
#
# This was tested using python 2.7 on Mac OS X. It will work on any
# Linux system, but will likely fail on Windows due to select/stdin
# limitations.
#-----------------------------------------------------------------------
def get_char(block = True):
if block or select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], []):
return sys.stdin.read(1)
raise error('NoChar')
This is a posix solution, similar to the answer by swdev.
As they stated, you have to play with termios.VMIN and termios.VTIME to catch more than one char without requiring user to press Enter. Trying to only use raw mode will be a problem as special keys like arrows can mess next keypress.
Here we use tty.setcbreak() or tty.setraw() as a shortcut, but they have short internals.
import termios
import tty
import sys
import select
def get_enter_key():
fd = sys.stdin.fileno()
orig_fl = termios.tcgetattr(fd)
try:
tty.setcbreak(fd) # use tty.setraw() instead to catch ^C also
mode = termios.tcgetattr(fd)
CC = 6
mode[CC][termios.VMIN] = 0
mode[CC][termios.VTIME] = 0
termios.tcsetattr(fd, termios.TCSAFLUSH, mode)
keypress, _, _ = select.select([fd], [], [])
if keypress:
return sys.stdin.read(4095)
finally:
termios.tcsetattr(fd, termios.TCSANOW, orig_fl)
try:
while True:
print(get_enter_key())
except KeyboardInterrupt:
print('exiting')
sys.exit()
note that there are two potential timeouts you could add here:
one is adding last parameter to select.select()
another is playing with VMIN and VTIME
Might I suggest nobreak? If'n you are willing to use curses.
https://docs.python.org/3/library/curses.html#curses.window.nodelay
You should be able to get read of a stream with either
sys.stdin.read(1)
to read utf-8 decoded chars or:
sys.stdin.buffer.read(1)
to read raw chars.
I would do this if I wanted to get raw data from the stdin and do something with it in a timely manner, without reading a newline or filling up the internal buffer first. This is suitable for running programs remotely via ssh where tty is not available, see:
ssh me#host '/usr/bin/python -c "import sys; print(sys.stdin.isatty())"'
There are some other things to think about to make programs work as expected in this scenario. You need to flush the output when you're done to avoid buffering delays, and it could be easy to assume a program hasn't read the input, when you've simply not flushed the output.
stdout.write("my data")
stdout.flush()
But usually it's not the input reading that's the problem but that the terminal (or program) supplying the input stream is not handing it over when you expect, or perhaps it's not reading your output when you expect. If you have a tty to start with (see ssh check above) you can put it into raw mode with the tty module.
import sys
import termios
import tty
old = termios.tcgetattr(sys.stdin)
tty.setraw(sys.stdin)
c = None
try:
c = sys.stdin.read(1)[0]
finally:
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old)
print(c)
... if using Mac/Linux. If using Windows you could use msvcrt.getch().
Use a generator - thankfully sys.stdin is already a generator!
A generator enables you to work on an infinite stream. Always when you call it, it returns the next element. In order to build a generator you need the yield keyword.
for line in sys.stdin:
print line
if a_certain_situation_happens:
break
Do not forget to place a break statement into the loop if a certain, wished situation happens.
You can find more information about generators on:
http://www.dabeaz.com/generators/index.html
http://linuxgazette.net/100/pramode.html