I have a serial device (Arduino) regularly outputting log data, which shold be written in a Log file. Also the device takes spontaneous commands over serial. I send the commands to a Raspberry over Telegram, which are handled and sent to the arduino by Telepot, which runs in a separate thread.
How can I make sure that the two processes get along with each other?
I am a complete Beginner in Multithreading.
Here is a shortened version of my Code:
import time
import datetime
import telepot
import os
import serial
from time import sleep
ser = None
bot = None
def log(data):
with open('logfile', 'w') as f:
file.write("Timestamp" + data)
#The handle Function is called by the telepot thread,
#whenever a message is received from Telegram
def handle(msg):
chat_id = msg['chat']['id']
command = msg['text']
print( 'Command Received: %s' % command)
if command = '/start':
bot.sendMessage(chat_id, 'welcome')
elif command == 'close_door':
#This serial write could possibly happen while a
#ser.readline() is executed, which would crash my program.
ser.write("Close Door")
elif command == 'LOG':
#Here i should make sure that nothing
#is waiting from the Arduino
#so that the next two Serial lines are the Arduinos
#respoonce to the "LOG" command.
#and that hanlde is the only
#function talking to the Serial port now.
ser.write("LOG")
response = ser.readline()
response += "\0000000A" + ser.readline()
#The Arduinos response is now saved as one string
#and sent to the User.
bot.sendMessage(chat_id, response)
print("Command Processed.")
bot = telepot.Bot('BOT TOKEN')
bot.message_loop(handle)
ser = serial.Serial("Arduino Serial Port", 9600)
print( 'I am listening ...')
while True:
#anything to make it not run at full speed (Recommendations welcome)
#The log updates are only once an hour.
sleep(10)
#here i need to make sure it does not collide with the other thread.
while ser.in_waiting > 0:
data = ser.readline()
log(data)
This code is not my actual code, but it should represent exactly what I'm trying to do.
My last resort would be to put the serial code in the threads loop function, But this would require me to change the libary which would be ugly.
I looked up some stuff about Queues in Asincio, and locking functions. However i don't really understand how to apply that. Also I don't use the async telepot.
After reading more on locking and threads, I found an answer with help of the links provided in this Question: Locking a method in Python?
It was often recommended to use Queues, however I don't know how.
My solution (code may have errors, but the principle works)
import time
import random
import datetime
import telepot
import os
import serial
from time import sleep
#we need to import the Lock from threading
from threading import Lock
ser = None
bot = None
def log(data):
with open('logfile', 'w') as f:
file.write("Timestamp" + data)
#create a lock:
ser_lock = Lock()
#The handle Function is called by the telepot thread,
#whenever a message is received from Telegram
def handle(msg):
#let the handle function use the same lock:
global ser_lock
chat_id = msg['chat']['id']
command = msg['text']
print( 'Command Received: %s' % command)
if command == '/start':
bot.sendMessage(chat_id, 'welcome')
elif command == 'close_door':
#This serial write could possibly happen while a
#ser.readline() is executed, which would crash my program.
with ser_lock:
ser.write("Close Door")
elif command == 'LOG':
#Here i should make sure that nothing
#is waiting from the Arduino
#so that the next two Serial lines are the Arduinos
#respoonce to the "LOG" command.
#and that hanlde is the only
#function talking to the Serial port now.
#the lock will only be open when no other thread is using the port.
#This thread will wait untill it's open.
with ser_lock:
while ser.in_waiting > 0:
data = ser.readline()
log(data)
#Should there be any old data, just write it to a file
#now i can safely execute serial writes and reads.
ser.write("LOG")
response = ser.readline()
response += "\0000000A" + ser.readline()
#The Arduinos response is now saved as one string
#and sent to the User.
bot.sendMessage(chat_id, response)
print("Command Processed.")
bot = telepot.Bot('BOT TOKEN')
bot.message_loop(handle)
ser = serial.Serial("Arduino Serial Port", 9600)
print( 'I am listening ...')
while True:
#anything to make it not run at full speed (Recommendations welcome)
#The log updates are only once a
sleep(10)
#here i need to make sure it does not collide with the other thread.
with ser_lock:
while ser.in_waiting > 0:
data = ser.readline()
log(data)
Related
Here is my problem : I have 2 programs communicating thanks to zmq on an arbitrary tcp port.
When the #1 receives message from #2 he has to call some function.
If #1 receives a message before the current function ends, I'd like #1 to interrupt the current function and call the new one.
I tried to use threading.Event to interrupt function.
I don't know if zmq is the right option for my needs or if the socket types fine.
To simplify I show the simplest version possible,here is what I tried :
p1.py
import zmq
from threading import Event
port_p2 = "6655"
context = zmq.Context()
socket = context.socket(zmq.PAIR)
socket.connect("tcp://localhost:%s" % port_p2)
print("port 6655")
__exit1 = Event()
__exit2 = Event()
def action1():
__exit1.clear()
__exit2.set()
while not __exit1.is_set():
for i in range(1, 20):
print(i)
time.sleep(1)
__exit1.set()
def action2():
__exit2.clear()
__exit1.set()
while not __exit2.is_set():
for i in range(1, 20):
print(i * 100)
time.sleep(1)
__exit2.set()
if __name__ == "__main__":
try:
while True:
try:
string = socket.recv(flags=zmq.NOBLOCK)
# message received, process it
string = str(string, 'utf-8')
if "Action1" in string:
action1()
if "Action2" in string:
action2()
except zmq.Again as e:
# No messages waiting to be processed
pass
time.sleep(0.1)
except(KeyboardInterrupt, SystemExit):
print("exit")
and p2.py
import time
import random
port_p1 = "6655"
context = zmq.Context()
socket_p1 = context.socket(zmq.PAIR)
socket_p1.bind("tcp://*:%s" % port_p1)
print("port 6655")
if __name__ == "__main__":
while True:
i = random.choice(range(1, 10))
print(i)
try:
if random.choice([True, False]):
print("Action 1")
socket_p1.send(b'Action1')
else:
socket_p1.send(b'Action2')
print("Action 2")
except zmq.Again as e:
pass
time.sleep(i)
For my purpose I didn't want / can't use system signals
I'd appreciate any input and don't hesitate to ask for precision, I have to confess that I had trouble writing this down.
Thank you
Q : …like #1 to interrupt the current function…
Given you have forbidden to use signals, #1 can but passively signal (be it over the present ZeroMQ infrastructure or not) the function, not to continue further and return in a pre-mature fashion ( so the fun() has to get suitably modified for doing that active re-checking, best in some reasonably granular progressive fashion, regularly checking actively the #1 if did passively signal ( "tell" the fun() ) to RET early, due to whatever reason and way the #1 had and used to do that ).
The other chance is to extend the already present ZeroMQ infrastructure ( the Context()-instance(s) ) with a socket-monitor artillery and make the fun() .connect()-directly to the socket-monitor resources to actively learn about any new message arriving to #1 ( i.e. autonomously, without #1's initiative ) and deciding to return in a pre-mature fashion, in those cases, where feasible according to your application logic.
For the socket-monitor case, the API documentation has all details needed for implementation, which would otherwise go way beyond the scope of the Stack Overflow post.
What I’m Trying to Do:
I’m trying to create a client script that can listen for potential messages from the server and receive input from a user at any point that tells the script to send a message to the server all on a single socket. The server I’m working with will only connect to one client at a time, so everything is on one socket.
The script will facilitate interaction between a maya ui and a file with prewritten functions that determine what to send to the server.
How it works:
I have the script with two threads. The parent thread is the adding messages thread and there is a child thread that runs the listening in the background. The child thread has a constant background loop, listening for any messages from the server (for example an error message) and reads a message queue to see if anything has been added. If something is added to the queue, the listening loop stops, sends a message, then starts the listening loop again. The parent thread allows the user to add a command into the message queue using the add_message() attribute. The maya ui will have buttons that call functions to add commands to the message queue. I made this with python 2.7
Here is an simplified example of the client. I added a single message before starting the listening loop so you can see what it is supposed to look like.
import socket
import struct
import threading
import Queue
import time
class ThreadedClient(threading.Thread):
def __init__(self, host, port):
threading.Thread.__init__(self)
#set up queues
self.send_q = Queue.Queue(maxsize = 10)
#declare instance variables
self.host = host
self.port = port
#connect to socket
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.connect((self.host, self.port))
#LISTEN
def listen(self):
while True: #loop forever
try:
print 'checking for message...'
# stops listening if there's a message to send
if self.send_q.empty() == False:
self.send_message()
else:
print 'no message'
print 'listening...'
message = self.s.recv(4096)
print 'RECEIVED: ' + message
except socket.timeout:
pass
def start_listen(self):
t = threading.Thread(target = self.listen())
t.start()
#t.join()
print 'started listen'
#ADD MESSAGE
def add_message(self, msg):
#put message into the send queue
self.send_q.put(msg)
print 'ADDED MSG: ' + msg
#self.send_message()
#SEND MESSAGE
def send_message(self):
#send message
msg = self.get_send_q()
if msg == "empty!":
print "nothing to send"
else:
self.s.sendall(msg)
print 'SENDING: ' + msg
#restart the listening
#self.start_listen()
#SAFE QUEUE READING
#if nothing in q, prints "empty" instead of stalling program
def get_send_q(self):
if self.send_q.empty():
print "empty!"
return "empty!"
else:
msg = self.send_q.get()
return msg
if __name__ == '__main__':
port = 8001
address = 'localhost'
s = ThreadedClient(address, port)
s.start()
print('Server started, port: ', port)
s.add_message('hello world')
s.start_listen()
s.add_message('hello world')
And here is an example server for the client:
import socket
import sys
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Bind the socket to the port
server_address = ('localhost', 8001)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
# Listen for incoming connections
sock.listen(1)
while True:
# Wait for a connection
print >>sys.stderr, 'waiting for a connection'
connection, client_address = sock.accept()
try:
print >>sys.stderr, 'connection from', client_address
# Receive the data in small chunks and retransmit it
while True:
data = connection.recv(16)
print >>sys.stderr, 'received "%s"' % data
if data:
print >>sys.stderr, 'sending data back to the client'
connection.sendall(data)
else:
print >>sys.stderr, 'no more data from', client_address
break
finally:
# Clean up the connection
connection.close()
Problem:
Once I start the listening loop thread, the script will no longer take any more input. Just as a test I added a line after starting the listening thread that would add a message to the queue and nothing happens.
Previous Approaches:
This question is essentially exactly what I’m trying to do as well, although it was never solved: How to handle chat client using threading and queues?
I have looked at Grab user input asynchronously and pass to an Event loop in python about creating a main loop that accepts user input and have tried implementing the queue system but am getting stuck.
I’ve also tried out How to use threading to get user input realtime while main still running in python approach but I can’t use raw_input() for my final usage of the code. I have also tried socket.timeout, but also loops without taking input.
One approach I was considering was making the script asynchronous, but from what I’ve read I don’t think it will resolve the issue.
TL/DR:
Is there a way to create a script that has a client listening to the server loop running in the background while at the same time being able to accept real time commands from the user?
I would greatly appreciate any help or nudge in the right direction, I've been stuck on this a while now.
The script works. There were just some errors throwing it off. In the start_listen() function
t = threading.Thread(target = self.listen())
should be
t = threading.Thread(target = self.listen)
and then in the __init__() after
self.s.connect((self.host, self.port))
add
self.s.settimeout(.1)
so that the script will cycle through the while loop.
I'm trying to create a code that is non-blocking and that allows me to create multiple clients to do some request on a server. However, I can't create more than 1 client simultaneously!
CLIENT.PY
import asyncio
PYTHONASYNCIODEBUG = 1
#ECHO CLIENT PROTOCOL
async def tcp_echo_client(message, loop):
# Send request to server
reader, writer = await asyncio.open_connection('127.0.0.1', 8888, loop=loop)
print('Send: %r' % message)
writer.write(message.encode())
# Receive the information
if message == '1':
await asyncio.Task(read_position(reader))
else:
await asyncio.ensure_future(read_server(reader))
# Close the connection
print('Close the socket')
writer.close()
#ASYNCIO COROUTINES TO REQUEST INFORMATION
async def read_server(reader):
server_message = await reader.read()
print(type(server_message))
print('Received: %r' % server_message.decode())
async def read_position(reader):
while True:
print("I'm Here")
server_message = await reader.read(50)
position = server_message.split()
print(position)
print(type(position))
print('Received: %r' % server_message.decode())
#FUNCTION THAT CREATES THE CLIENT
def main(message):
'''This function creates the client'''
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(tcp_echo_client(message, loop))
finally:
pass
# This is how I create a new client
if __name__ == '__main__':
message = '2'
main(message)
message = '3'
main(message)
I want to create multiples clients, however, the code is blocking in the first main when I send the message('1'). I don't know why the code is blocking if I'm using asyncio. My server accepts multiples connections, because if I run this code seperatly I can do everything. The propose of this is to create clients every time I click a button at my Kivy app to send a request to the server.
This problems exists because I want to control a Robot and do a lot of things simultaneously, however with a blocking code I can't do it because I'm get stuck
Maybe it's a stupid question but I've only started coded 2 months ago and I haven't any help.
Your main function doesn't "create the client", as its docstring claims. It creates the client and runs it to completion. This is why multiple invocations of main() result in serial execution. main() being a regular function, that's exactly what you'd expect, asyncio doesn't change that. It's useful to remember that asyncio is single-threaded, so it can't do some "run in the background" magic, unless you cooperate.
To cooperate, you need to tell aysncio to start both clients, and then await them in parallel:
async def main(messages):
loop = asyncio.get_event_loop()
# launch the coroutines in parallel
tasks = [loop.create_task(tcp_echo_client(msg, loop)) for msg in messages]
# don't exit until all of them are done
await asyncio.gather(*tasks)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main(['2', '3']))
Note that when awaiting your coroutines, you don't need to wrap them in asyncio.ensure_future() or asyncio.Task() - asyncio will handle that automatically. await read_position(reader) and await read_server(reader) would work just fine and have the same meaning as the longer versions.
i'm trying to build a program to send a string to process Tangki and Tangki2 then send a bit of array data each to process Outdata, but it seems not working correctly. but when i disable gate to the Outdata everything works flawlessly.
this is the example code:
import os
from multiprocessing import Process, Pipe
from time import sleep
import cv2
def outdata(input1,input2):
while(1):
room=input1.recv()
room2=input2.recv()
def tangki(keran1,selang1): ##============tangki1
a=None
x,y,degree,tinggi=0,0,0,0
dout=[]
while(1):
frame=keran1.recv()
dout.append([x,y,degree,tinggi])
selang1.send(dout)
print ("received from: {}".format(frame))
def tangki2(keran3,selang2): ##=================tangki2
x,y,degree,tinggi=0,0,0,0
dout2=[]
while(1):
frame=keran3.recv()
dout2.append([x,y,degree,tinggi])
selang2.send(dout2)
print("received from: {}".format(frame))
def pompa(gate1,gate2):
count=0
while(1):
count+=1
gate1.send("gate 1, val{}".format(count))
gate2.send("gate 2, val{}".format(count))
if __name__ == '__main__':
pipa1, pipa2 = Pipe()
pipa3, pipa4 = Pipe()
tx1,rx1 = Pipe()
tx2,rx2 = Pipe()
ptangki = Process(target=tangki, args=(pipa2, tx1))
ptangki2 = Process (target=tangki2, args=(pipa4, tx2))
ppompa = Process(target=pompa, args=(pipa1,pipa3))
keran = Process(target=outdata, args=(rx1,rx2))
ptangki.start()
ptangki2.start()
ppompa.start()
keran.start()
ptangki.join()
ptangki2.join()
ppompa.join()
keran.join()
at exact count reach 108 the process hang, not responding whatsoever. when i TOP it, the python3 process has gone, it seems that selang1 and selang2 causing the problem. i've search in google and it might be a Pipe Deadlock. so the question is how to prevent this from happening since i've already dump all data in pipe via repeated reading both in input1 and input2.
Edit: it seems that the only problem was the communication between tangki and tangki2 to outdata
it's actually because buffer size limit? but adding dout=[x,y,degree,tinggi] and dout=[x,y,degree,tinggi] reset the size of data to minimal, or by assigning dout=[0,0,0,0] and dout2=[0,0,0,0] right after selang1.send(dout) and selang2.send(dout2)
I try to try to write some kind of renderer for the command line that should be able to print data from stdin and from another data source using asyncio and blessed, which is an improved version of python-blessings.
Here is what I have so far:
import asyncio
from blessed import Terminal
#asyncio.coroutine
def render(term):
while True:
received = yield
if received:
print(term.bold + received + term.normal)
async def ping(renderer):
while True:
renderer.send('ping')
await asyncio.sleep(1)
async def input_reader(term, renderer):
while True:
with term.cbreak():
val = term.inkey()
if val.is_sequence:
renderer.send("got sequence: {0}.".format((str(val), val.name, val.code)))
elif val:
renderer.send("got {0}.".format(val))
async def client():
term = Terminal()
renderer = render(term)
render_task = asyncio.ensure_future(renderer)
pinger = asyncio.ensure_future(ping(renderer))
inputter = asyncio.ensure_future(input_reader(term, renderer))
done, pending = await asyncio.wait(
[pinger, inputter, renderer],
return_when=asyncio.FIRST_COMPLETED,
)
for task in pending:
task.cancel()
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(client())
asyncio.get_event_loop().run_forever()
For learning and testing purposes there is just a dump ping that sends 'ping' each second and another routine, that should grab key inputs and also sends them to my renderer.
But ping only appears once in the command line using this code and the input_reader works as expected. When I replace input_reader with a pong similar to ping everything is fine.
This is how it looks when typing 'pong', although if it takes ten seconds to write 'pong':
$ python async_term.py
ping
got p.
got o.
got n.
got g.
It seems like blessed is not built to work correctly with asyncio: inkey() is a blocking method. This will block any other couroutine.
You can use a loop with kbhit() and await asyncio.sleep() to yield control to other coroutines - but this is not a clean asyncio solution.