Here is my problem : I have 2 programs communicating thanks to zmq on an arbitrary tcp port.
When the #1 receives message from #2 he has to call some function.
If #1 receives a message before the current function ends, I'd like #1 to interrupt the current function and call the new one.
I tried to use threading.Event to interrupt function.
I don't know if zmq is the right option for my needs or if the socket types fine.
To simplify I show the simplest version possible,here is what I tried :
p1.py
import zmq
from threading import Event
port_p2 = "6655"
context = zmq.Context()
socket = context.socket(zmq.PAIR)
socket.connect("tcp://localhost:%s" % port_p2)
print("port 6655")
__exit1 = Event()
__exit2 = Event()
def action1():
__exit1.clear()
__exit2.set()
while not __exit1.is_set():
for i in range(1, 20):
print(i)
time.sleep(1)
__exit1.set()
def action2():
__exit2.clear()
__exit1.set()
while not __exit2.is_set():
for i in range(1, 20):
print(i * 100)
time.sleep(1)
__exit2.set()
if __name__ == "__main__":
try:
while True:
try:
string = socket.recv(flags=zmq.NOBLOCK)
# message received, process it
string = str(string, 'utf-8')
if "Action1" in string:
action1()
if "Action2" in string:
action2()
except zmq.Again as e:
# No messages waiting to be processed
pass
time.sleep(0.1)
except(KeyboardInterrupt, SystemExit):
print("exit")
and p2.py
import time
import random
port_p1 = "6655"
context = zmq.Context()
socket_p1 = context.socket(zmq.PAIR)
socket_p1.bind("tcp://*:%s" % port_p1)
print("port 6655")
if __name__ == "__main__":
while True:
i = random.choice(range(1, 10))
print(i)
try:
if random.choice([True, False]):
print("Action 1")
socket_p1.send(b'Action1')
else:
socket_p1.send(b'Action2')
print("Action 2")
except zmq.Again as e:
pass
time.sleep(i)
For my purpose I didn't want / can't use system signals
I'd appreciate any input and don't hesitate to ask for precision, I have to confess that I had trouble writing this down.
Thank you
Q : …like #1 to interrupt the current function…
Given you have forbidden to use signals, #1 can but passively signal (be it over the present ZeroMQ infrastructure or not) the function, not to continue further and return in a pre-mature fashion ( so the fun() has to get suitably modified for doing that active re-checking, best in some reasonably granular progressive fashion, regularly checking actively the #1 if did passively signal ( "tell" the fun() ) to RET early, due to whatever reason and way the #1 had and used to do that ).
The other chance is to extend the already present ZeroMQ infrastructure ( the Context()-instance(s) ) with a socket-monitor artillery and make the fun() .connect()-directly to the socket-monitor resources to actively learn about any new message arriving to #1 ( i.e. autonomously, without #1's initiative ) and deciding to return in a pre-mature fashion, in those cases, where feasible according to your application logic.
For the socket-monitor case, the API documentation has all details needed for implementation, which would otherwise go way beyond the scope of the Stack Overflow post.
Related
I have a simple twisted TCP server running absolutely fine, it basically deals with database requests and displays the right things its just an echo client with a bunch of functions, the database that is being read also updates I have this refresh function to open the database and refresh it however if I add this to the message functions it'll take too long to respond as the refresh function takes around 6/7 seconds to complete, my initial idea was to have this function in a while loop and running constantly refreshing every 5/10 mins but after reading about the global interpreter lock its made me think that that isn't possible, any suggestions on how to run this function in the background of my code would be greatly appreciated
I've tried having it in a thread but it doesn't seem to run at all when I start the thread, I put it under the if name == 'main': function and no luck!
Here is my refresh function
def refreshit()
Application = win32com.client.Dispatch("Excel.Application")
Workbook = Application.Workbooks.open(database)
Workbook.RefreshAll()
Workbook.Save()
Application.Quit()
xlsx = pd.ExcelFile(database)
global datess
global refss
df = pd.read_excel(xlsx, sheet_name='Sheet1')
datess = df.groupby('documentDate')
refss = df.groupby('reference')
class Echo(Protocol):
global Picked_DFS
Picked_DFS = None
label = None
global errors
global picked
errors = []
picked = []
def dataReceived(self, data):
"""
As soon as any data is received, write it back.
"""
response = self.handle_message(data)
print('responding with this')
print(response)
self.transport.write(response)
def main():
f = Factory()
f.protocol = Echo
reactor.listenTCP(8000, f)
reactor.run()
if __name__ == '__main__':
main()
I had tried this to no avail
if __name__ == '__main__':
main()
thread = Thread(target = refreshit())
thread.start()
thread.join()
You have an important error on this line:
thread = Thread(target = refreshit())
Though you have not included the definition of refreshit (perhaps a function to consider renaming), I assume refreshit is a function that performs your refresh.
In this case, what you are doing here is calling refreshit and waiting for it to return a value. Then, the value it returns is used as the target of the Thread you create here. This is probably not what you meant. Instead:
thread = Thread(target = refreshit)
That is, refreshit itself is what you want the target of the thread to be.
You also need to be sure to sequence your operations so that everything gets to run concurrently:
if __name__ == '__main__':
# Start your worker/background thread.
thread = Thread(target = refreshit)
thread.start()
# Run Twisted
main()
# Cleanup/wait on your worker/background thread.
thread.join()
You may also just want to use Twisted's thread support instead of using the threading module directly (but this is not mandatory).
if __name__ == '__main__':
# Start your worker/background thread.
thread = Thread(target = refreshit)
thread.start()
# Run Twisted
main()
# Cleanup/wait on your worker/background thread.
thread.join()
I have a serial device (Arduino) regularly outputting log data, which shold be written in a Log file. Also the device takes spontaneous commands over serial. I send the commands to a Raspberry over Telegram, which are handled and sent to the arduino by Telepot, which runs in a separate thread.
How can I make sure that the two processes get along with each other?
I am a complete Beginner in Multithreading.
Here is a shortened version of my Code:
import time
import datetime
import telepot
import os
import serial
from time import sleep
ser = None
bot = None
def log(data):
with open('logfile', 'w') as f:
file.write("Timestamp" + data)
#The handle Function is called by the telepot thread,
#whenever a message is received from Telegram
def handle(msg):
chat_id = msg['chat']['id']
command = msg['text']
print( 'Command Received: %s' % command)
if command = '/start':
bot.sendMessage(chat_id, 'welcome')
elif command == 'close_door':
#This serial write could possibly happen while a
#ser.readline() is executed, which would crash my program.
ser.write("Close Door")
elif command == 'LOG':
#Here i should make sure that nothing
#is waiting from the Arduino
#so that the next two Serial lines are the Arduinos
#respoonce to the "LOG" command.
#and that hanlde is the only
#function talking to the Serial port now.
ser.write("LOG")
response = ser.readline()
response += "\0000000A" + ser.readline()
#The Arduinos response is now saved as one string
#and sent to the User.
bot.sendMessage(chat_id, response)
print("Command Processed.")
bot = telepot.Bot('BOT TOKEN')
bot.message_loop(handle)
ser = serial.Serial("Arduino Serial Port", 9600)
print( 'I am listening ...')
while True:
#anything to make it not run at full speed (Recommendations welcome)
#The log updates are only once an hour.
sleep(10)
#here i need to make sure it does not collide with the other thread.
while ser.in_waiting > 0:
data = ser.readline()
log(data)
This code is not my actual code, but it should represent exactly what I'm trying to do.
My last resort would be to put the serial code in the threads loop function, But this would require me to change the libary which would be ugly.
I looked up some stuff about Queues in Asincio, and locking functions. However i don't really understand how to apply that. Also I don't use the async telepot.
After reading more on locking and threads, I found an answer with help of the links provided in this Question: Locking a method in Python?
It was often recommended to use Queues, however I don't know how.
My solution (code may have errors, but the principle works)
import time
import random
import datetime
import telepot
import os
import serial
from time import sleep
#we need to import the Lock from threading
from threading import Lock
ser = None
bot = None
def log(data):
with open('logfile', 'w') as f:
file.write("Timestamp" + data)
#create a lock:
ser_lock = Lock()
#The handle Function is called by the telepot thread,
#whenever a message is received from Telegram
def handle(msg):
#let the handle function use the same lock:
global ser_lock
chat_id = msg['chat']['id']
command = msg['text']
print( 'Command Received: %s' % command)
if command == '/start':
bot.sendMessage(chat_id, 'welcome')
elif command == 'close_door':
#This serial write could possibly happen while a
#ser.readline() is executed, which would crash my program.
with ser_lock:
ser.write("Close Door")
elif command == 'LOG':
#Here i should make sure that nothing
#is waiting from the Arduino
#so that the next two Serial lines are the Arduinos
#respoonce to the "LOG" command.
#and that hanlde is the only
#function talking to the Serial port now.
#the lock will only be open when no other thread is using the port.
#This thread will wait untill it's open.
with ser_lock:
while ser.in_waiting > 0:
data = ser.readline()
log(data)
#Should there be any old data, just write it to a file
#now i can safely execute serial writes and reads.
ser.write("LOG")
response = ser.readline()
response += "\0000000A" + ser.readline()
#The Arduinos response is now saved as one string
#and sent to the User.
bot.sendMessage(chat_id, response)
print("Command Processed.")
bot = telepot.Bot('BOT TOKEN')
bot.message_loop(handle)
ser = serial.Serial("Arduino Serial Port", 9600)
print( 'I am listening ...')
while True:
#anything to make it not run at full speed (Recommendations welcome)
#The log updates are only once a
sleep(10)
#here i need to make sure it does not collide with the other thread.
with ser_lock:
while ser.in_waiting > 0:
data = ser.readline()
log(data)
I'm trying to create a code that is non-blocking and that allows me to create multiple clients to do some request on a server. However, I can't create more than 1 client simultaneously!
CLIENT.PY
import asyncio
PYTHONASYNCIODEBUG = 1
#ECHO CLIENT PROTOCOL
async def tcp_echo_client(message, loop):
# Send request to server
reader, writer = await asyncio.open_connection('127.0.0.1', 8888, loop=loop)
print('Send: %r' % message)
writer.write(message.encode())
# Receive the information
if message == '1':
await asyncio.Task(read_position(reader))
else:
await asyncio.ensure_future(read_server(reader))
# Close the connection
print('Close the socket')
writer.close()
#ASYNCIO COROUTINES TO REQUEST INFORMATION
async def read_server(reader):
server_message = await reader.read()
print(type(server_message))
print('Received: %r' % server_message.decode())
async def read_position(reader):
while True:
print("I'm Here")
server_message = await reader.read(50)
position = server_message.split()
print(position)
print(type(position))
print('Received: %r' % server_message.decode())
#FUNCTION THAT CREATES THE CLIENT
def main(message):
'''This function creates the client'''
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(tcp_echo_client(message, loop))
finally:
pass
# This is how I create a new client
if __name__ == '__main__':
message = '2'
main(message)
message = '3'
main(message)
I want to create multiples clients, however, the code is blocking in the first main when I send the message('1'). I don't know why the code is blocking if I'm using asyncio. My server accepts multiples connections, because if I run this code seperatly I can do everything. The propose of this is to create clients every time I click a button at my Kivy app to send a request to the server.
This problems exists because I want to control a Robot and do a lot of things simultaneously, however with a blocking code I can't do it because I'm get stuck
Maybe it's a stupid question but I've only started coded 2 months ago and I haven't any help.
Your main function doesn't "create the client", as its docstring claims. It creates the client and runs it to completion. This is why multiple invocations of main() result in serial execution. main() being a regular function, that's exactly what you'd expect, asyncio doesn't change that. It's useful to remember that asyncio is single-threaded, so it can't do some "run in the background" magic, unless you cooperate.
To cooperate, you need to tell aysncio to start both clients, and then await them in parallel:
async def main(messages):
loop = asyncio.get_event_loop()
# launch the coroutines in parallel
tasks = [loop.create_task(tcp_echo_client(msg, loop)) for msg in messages]
# don't exit until all of them are done
await asyncio.gather(*tasks)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main(['2', '3']))
Note that when awaiting your coroutines, you don't need to wrap them in asyncio.ensure_future() or asyncio.Task() - asyncio will handle that automatically. await read_position(reader) and await read_server(reader) would work just fine and have the same meaning as the longer versions.
I try to try to write some kind of renderer for the command line that should be able to print data from stdin and from another data source using asyncio and blessed, which is an improved version of python-blessings.
Here is what I have so far:
import asyncio
from blessed import Terminal
#asyncio.coroutine
def render(term):
while True:
received = yield
if received:
print(term.bold + received + term.normal)
async def ping(renderer):
while True:
renderer.send('ping')
await asyncio.sleep(1)
async def input_reader(term, renderer):
while True:
with term.cbreak():
val = term.inkey()
if val.is_sequence:
renderer.send("got sequence: {0}.".format((str(val), val.name, val.code)))
elif val:
renderer.send("got {0}.".format(val))
async def client():
term = Terminal()
renderer = render(term)
render_task = asyncio.ensure_future(renderer)
pinger = asyncio.ensure_future(ping(renderer))
inputter = asyncio.ensure_future(input_reader(term, renderer))
done, pending = await asyncio.wait(
[pinger, inputter, renderer],
return_when=asyncio.FIRST_COMPLETED,
)
for task in pending:
task.cancel()
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(client())
asyncio.get_event_loop().run_forever()
For learning and testing purposes there is just a dump ping that sends 'ping' each second and another routine, that should grab key inputs and also sends them to my renderer.
But ping only appears once in the command line using this code and the input_reader works as expected. When I replace input_reader with a pong similar to ping everything is fine.
This is how it looks when typing 'pong', although if it takes ten seconds to write 'pong':
$ python async_term.py
ping
got p.
got o.
got n.
got g.
It seems like blessed is not built to work correctly with asyncio: inkey() is a blocking method. This will block any other couroutine.
You can use a loop with kbhit() and await asyncio.sleep() to yield control to other coroutines - but this is not a clean asyncio solution.
I am attempting to make a few thousand dns queries. I have written my script to use python-adns. I have attempted to add threading and queue's to ensure the script runs optimally and efficiently.
However, I can only achieve mediocre results. The responses are choppy/intermittent. They start and stop, and most times pause for 10 to 20 seconds.
tlock = threading.Lock()#printing to screen
def async_dns(i):
s = adns.init()
for i in names:
tlock.acquire()
q.put(s.synchronous(i, adns.rr.NS)[0])
response = q.get()
q.task_done()
if response == 0:
dot_net.append("Y")
print(i + ", is Y")
elif response == 300:
dot_net.append("N")
print(i + ", is N")
tlock.release()
q = queue.Queue()
threads = []
for i in range(100):
t = threading.Thread(target=async_dns, args=(i,))
threads.append(t)
t.start()
print(threads)
I have spent countless hours on this. I would appreciate some input from expedienced pythonista's . Is this a networking issue ? Can this bottleneck/intermittent responses be solved by switching servers ?
Thanks.
Without answers to the questions, I asked in comments above, I'm not sure how well I can diagnose the issue you're seeing, but here are some thoughts:
It looks like each thread is processing all names instead of just a portion of them.
Your Queue seems to be doing nothing at all.
Your lock seems to guarantee that you actually only do one query at a time (defeating the purpose of having multiple threads).
Rather than trying to fix up this code, might I suggest using multiprocessing.pool.ThreadPool instead? Below is a full working example. (You could use adns instead of socket if you want... I just couldn't easily get it installed and so stuck with the built-in socket.)
In my testing, I also sometimes see pauses; my assumption is that I'm getting throttled somewhere.
import itertools
from multiprocessing.pool import ThreadPool
import socket
import string
def is_available(host):
print('Testing {}'.format(host))
try:
socket.gethostbyname(host)
return False
except socket.gaierror:
return True
# Test the first 1000 three-letter .com hosts
hosts = [''.join(tla) + '.com' for tla in itertools.permutations(string.ascii_lowercase, 3)][:1000]
with ThreadPool(100) as p:
results = p.map(is_available, hosts)
for host, available in zip(hosts, results):
print('{} is {}'.format(host, 'available' if available else 'not available'))