I am using aiozmq for a simple RPC program.
I have created a client and server.
When the server is running, the client runs just fine.
I have a timeout set in the client to raise an exception in the event of no server being reachable.
The client code is below. When I run it without the server running, I get an expected exception but the script doesn't actually return to the terminal. It still seems to be executing.
Could someone firstly explain how this is happening and secondly how to fix it?
import asyncio
from asyncio import TimeoutError
from aiozmq import rpc
import sys
import os
import signal
import threading
import sys
import traceback
#signal.signal(signal.SIGINT, signal.SIG_DFL)
async def client():
print("waiting for connection..")
client = await rpc.connect_rpc(
connect='tcp://127.0.0.1:5555',
timeout=1
)
print("got client")
for i in range(100):
print("{}: calling simple_add".format(i))
ret = await client.call.simple_add(1, 2)
assert 3 == ret
print("calling slow_add")
ret = await client.call.slow_add(3, 5)
assert 8 == ret
client.close()
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.set_debug(True)
future = asyncio.ensure_future(client())
try:
loop.run_until_complete(future)
except TimeoutError:
print("Timeout occurred...")
future.cancel()
loop.stop()
#loop.run_forever()
main_thread = threading.currentThread()
for t in threading.enumerate():
if t is main_thread:
print("skipping main_thread...")
continue
print("Thread is alive? {}".format({True:'yes',
False:'no'}[t.is_alive()]))
print("Waiting for thread...{}".format(t.getName()))
t.join()
print(sys._current_frames())
traceback.print_stack()
for thread_id, frame in sys._current_frames().items():
name = thread_id
for thread in threading.enumerate():
if thread.ident == thread_id:
name = thread.name
traceback.print_stack(frame)
print("exiting..")
sys.exit(1)
#os._exit(1)
print("eh?")
The result of running the above is below. Note again that the program was still running, I had to to exit.
> python client.py
waiting for connection..
got client
0: calling simple_add
Timeout occurred...
skipping main_thread...
{24804: <frame object at 0x00000000027C3848>}
File "client.py", line 54, in <module>
traceback.print_stack()
File "client.py", line 60, in <module>
traceback.print_stack(frame)
exiting..
^C
I also tried sys.exit() which also didn't work:
try:
loop.run_until_complete(future)
except:
print("exiting..")
sys.exit(1)
I can get the program to die, but only if I use os._exit(1). sys.exit() doesn't seem to cut it. I doesn't appear that there are any other threads preventing the interpreter from dying. (Unless I'm mistaken?) What else could be stopping the program from exiting?
Related
I have below code, where I am using OpenCV to start webcam video. Along with that I also have a thread running that pings www.google.com to check network connectivity.
import time
import cv2
import os
from threading import Thread
stopThread = False
def CheckNetwork():
global stopThread
while True:
time.sleep(60)
host = "www.google.com"
response = os.system("ping " + host)
if response == 0:
print("Internet host reachable")
else:
print("Internet host not reachable")
if stopThread:
break
def main():
global stopThread
Thread(target=CheckNetwork).start()
cam = cv2.VideoCapture(0)
while True:
ret_val, img = cam.read()
cv2.imshow('Camera', img)
key = cv2.waitKey(1)
if key == ord('q'):
stopThread = True
break
cv2.destroyAllWindows()
main()
This code is running fine. If I have to close the application by pressing q, OpenCV window closes but application keeps running for 60sec because of the thread and only after 60sec whole application terminates safely.
I wanted to know if this is a good way to close the threads. Is there any better way available which can immediately terminate threads in Python?
There's no native way of stopping a thread in Python. Instead of using a stop flag, you can also use ctypes that calls the Python API to raise an exception in the thread.
import ctypes
# Other imports...
class ThreadWithException(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
# code here...
def get_id(self):
# returns id of the respective thread
if hasattr(self, '_thread_id'):
return self._thread_id
for id, thread in threading._active.items():
if thread is self:
return id
def raise_exception(self):
thread_id = self.get_id()
res = ctypes.pythonapi.PyThreadState_SetAsyncExc(thread_id,
ctypes.py_object(SystemExit))
if res > 1:
ctypes.pythonapi.PyThreadState_SetAsyncExc(thread_id, 0)
print('Exception raise failure')
I've developed a pattern to use for a commanding a python daemon through cmd module shell using an eventloop. However, its not ready yet because I can't figure out how to gracefully exit the two applications (I'm still learning asyncio and can't figure out the following problem). When the cmd module is commanded to exit I get:
(Cmd) exit
Shutting down client...
Traceback (most recent call last):
File "client_loop.py", line 10, in <module>
loop.run_until_complete(test.cmdloop())
File "...\asyncio\base_events.py", line 467, in run_until_complete
future = tasks.ensure_future(future, loop=self)
File "...\lib\asyncio\tasks.py", line 526, in ensure_future
raise TypeError('An asyncio.Future, a coroutine or an awaitable is '
TypeError: An asyncio.Future, a coroutine or an awaitable is required
I'm not good with asyncio yet, what am I doing wrong? Sorry for the long question but the files/errors make it long and hopefully makes it easier to debug.
Here are the supporting files:
shell.py
# implementation of the (Cmd) prompt with history functionality
# standard imports
import cmd as cmd
class Shell(cmd.Cmd):
def __init__(self, **kwargs):
cmd.Cmd.__init__(self, **kwargs)
self.eventloop = None
self.shutdown_client = None
self.tcp_echo_client = None
def set_eventloop(self, loop):
self.eventloop = loop
def set_funcs(self, tcp_echo_client, shutdown_client):
self.tcp_echo_client = tcp_echo_client
self.shutdown_client = shutdown_client
def do_exit(self,*args):
"""
Exits the shell gracefully
:param args:
:return:
"""
print('Shutting down client...')
self.shutdown_client(self.eventloop)
return True
def default(self, line):
try:
self.eventloop.run_until_complete(self.tcp_echo_client(line, self.eventloop))
except SystemExit:
pass
server.py
# server logic to parse arguments coming over the TCP socket and echo it back
# standard imports
import asyncio
async def handle_echo(reader, writer):
data = await reader.read(100)
message = data.decode()
addr = writer.get_extra_info('peername')
print("Received %r from %r" % (message, addr))
print("Send: %r" % message)
writer.write(data)
await writer.drain()
print("Close the client socket")
writer.close()
loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888, loop=loop)
server = loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
# Close the server
for task in asyncio.Task.all_tasks():
loop.run_until_complete(task)
server.close()
loop.run_until_complete(server.wait_closed())
loop.stop()
loop.close()
exit(0)
client.py
# client functions to send message over TCP and process response
# standard imports
import asyncio
# user imports
import shell
async def tcp_echo_client(message, loop):
reader, writer = await asyncio.open_connection('127.0.0.1', 8888,
loop=loop)
print('Send: %r' % message)
writer.write(message.encode())
data = await reader.read(100)
print('Received: %r' % data.decode())
print('Close the socket')
writer.close()
def shutdown_client(loop):
loop.stop()
# Find all running tasks:
pending = asyncio.Task.all_tasks()
# Run loop until tasks done:
loop.run_until_complete(asyncio.gather(*pending))
loop = asyncio.get_event_loop()
test = shell.Shell()
test.set_eventloop(loop)
test.set_funcs(tcp_echo_client, shutdown_client)
loop.run_until_complete(test.cmdloop())
loop.close()
The problem lies in the fact that cmd doesn't require asyncio to process input from the user and send a message to through the TCP/IP socket. Simply removing asyncio from the client side solved the problem. It still offers the shell implementation and the client-server pattern. Here is the new code:
server.py
# server logic to parse arguments coming over the TCP socket and echo it back
# standard imports
import asyncio
async def handle_echo(reader, writer):
data = await reader.read(100)
message = data.decode()
addr = writer.get_extra_info('peername')
print("Received %r from %r" % (message, addr))
print("Send: %r" % message)
writer.write(data)
await writer.drain()
print("Close the client socket")
writer.close()
loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888, loop=loop)
server = loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
# Close the server
for task in asyncio.Task.all_tasks():
loop.run_until_complete(task)
server.close()
loop.run_until_complete(server.wait_closed())
loop.stop()
loop.close()
exit(0)
client_shell.py
# implementation of a shell prompt (using Cmd module) to send message over TCP and process response
# standard imports
import socket
import cmd as cmd
class Shell(cmd.Cmd):
def __init__(self, **kwargs):
cmd.Cmd.__init__(self, **kwargs)
def do_exit(self,*args):
"""
Exits the shell gracefully
:param args:
:return:
"""
print('Shutting down client...')
return True
def default(self, line):
try:
self._tcp_echo_client(line.encode())
except SystemExit:
pass
def _tcp_echo_client(self, message):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
print('Send: %r' % message)
s.connect(('127.0.0.1', 8888))
s.sendall(message)
data = s.recv(1000)
print('Received: %r' % data.decode())
print('Close the socket')
if __name__ == '__main__':
Shell().cmdloop()
I want to use timeout to stop the blocking function of mqtt, I use a the timeout_decorator module, it can stop command function but cannot stop blocking function, subscribe.simple.
The following code runs successfully
import time
import timeout_decorator
#timeout_decorator.timeout(5, timeout_exception=StopIteration)
def mytest():
print("Start")
for i in range(1,10):
time.sleep(1)
print("{} seconds have passed".format(i))
if __name__ == '__main__':
mytest()
the result as follow:
Start
1 seconds have passed
2 seconds have passed
3 seconds have passed
4 seconds have passed
Traceback (most recent call last):
File "timeutTest.py", line 12, in <module>
mytest()
File "/home/gyf/.local/lib/python3.5/site-packages/timeout_decorator/timeout_decorator.py", line 81, in new_function
return function(*args, **kwargs)
File "timeutTest.py", line 8, in mytest
time.sleep(1)
File "/home/gyf/.local/lib/python3.5/site-packages/timeout_decorator/timeout_decorator.py", line 72, in handler
_raise_exception(timeout_exception, exception_message)
File "/home/gyf/.local/lib/python3.5/site-packages/timeout_decorator/timeout_decorator.py", line 45, in _raise_exception
raise exception()
timeout_decorator.timeout_decorator.TimeoutError: 'Timed Out'
but I failed with the subscribe.simple API
import timeout_decorator
#timeout_decorator.timeout(5)
def sub():
# print(type(msg))
print("----before simple")
# threading.Timer(5,operateFail,args=)
msg = subscribe.simple("paho/test/simple", hostname=MQTT_IP,port=MQTT_PORT,)
print("----after simple")
return msg
publish.single("paho/test/single", "cloud to device", qos=2, hostname=MQTT_IP,port=MQTT_PORT)
try:
print("pub")
msg = sub()
print(msg)
except StopIteration as identifier:
print("error")
The result infinitely wait
pub
----before simple
I want the function which include subscribe.simple API can stop after 5 seconds.
Asyncio won't be able to handle blocking function in the same thread. therefore using asyncio.wait_for failed. However, inspired by this blog post I used loop.run_in_executor to keep control on the blocking thread.
from paho.mqtt import subscribe
import asyncio
MQTT_IP = "localhost"
MQTT_PORT = 1883
msg = None
def possibly_blocking_function():
global msg
print("listenning for message")
msg = subscribe.simple(
"paho/test/simple",
hostname=MQTT_IP,
port=MQTT_PORT,
)
print("message received!")
async def main():
print("----before simple")
try:
await asyncio.wait_for(
loop.run_in_executor(None, possibly_blocking_function), timeout=5
)
except asyncio.TimeoutError:
pass
print("----after simple")
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Output :
----before simple
listenning for message
----after simple
Please note this is not perfect, the program won't end since there are running tasks. You can exit it using various solution but this is out of scope since I am still looking for a clean way to close that stuck thread.
This is the relevant code of my python program:
import discord
import asyncio
class Bot(discord.Client):
def __init__(self):
super().__init__()
#asyncio.coroutine
def my_background_task(self):
yield from self.wait_until_ready()
while not self.is_closed:
yield from asyncio.sleep(3600*24) # <- This is line 76 where it fails
doSomething()
bot = Bot()
loop = asyncio.get_event_loop()
try:
loop.create_task(bot.my_background_task())
loop.run_until_complete(bot.login('username', 'password'))
loop.run_until_complete(bot.connect())
except Exception:
loop.run_until_complete(bot.close())
finally:
loop.close()
The program occasionally quits (on its own, while it should not) with no other errors or warning other than
Task was destroyed but it is pending!
task: <Task pending coro=<my_background_task() running at bin/discordBot.py:76> wait_for=<Future pending cb=[Task._wakeup()]>>
How to ensure the program won't randomly quit? I have Python 3.4.3+ on Xubuntu 15.10.
This is because the discord client module needs control once every minute or so.
This means that any function that steals control for more than a certain time causes discord's client to enter an invalid state (which will manifest itself as an exception some point later, perhaps upon next method call of client).
To ensure that the discord module client can ping the discord server, you should use a true multi-threading solution.
One solution is to offload all heavy processing onto a separate process (a separate thread will not do, because Python has a global interpreter lock) and use the discord bot as a thin layer whose responsibility is to populate work queues.
Related reading:
https://discordpy.readthedocs.io/en/latest/faq.html#what-does-blocking-mean
Example solution... this is WAY beyond the scope of the problem, but I already had the code mostly written. If I had more time, I would write a shorter solution :)
2 parts, discord interaction and processing server:
This is the discord listener.
import discord
import re
import asyncio
import traceback
import websockets
import json
# Call a function on other server
async def call(methodName, *args, **kwargs):
async with websockets.connect('ws://localhost:9001/meow') as websocket:
payload = json.dumps( {"method":methodName, "args":args, "kwargs": kwargs})
await websocket.send(payload)
#...
resp = await websocket.recv()
#...
return resp
client = discord.Client()
tok = open("token.dat").read()
#client.event
async def on_ready():
print('Logged in as')
print(client.user.name)
print(client.user.id)
print('------')
#client.event
async def on_error(event, *args, **kwargs):
print("Error?")
#client.event
async def on_message(message):
try:
if message.author.id == client.user.id:
return
m = re.match("(\w+) for (\d+).*?", message.content)
if m:
g = m.groups(1)
methodName = g[0]
someNumber = int(g[1])
response = await call(methodName, someNumber)
if response:
await client.send_message(message.channel, response[0:2000])
except Exception as e:
print (e)
print (traceback.format_exc())
client.run(tok)
This is the worker server for processing heavy requests. You can make this part sync or async.
I chose to use some magic called a websocket to send data from one python process to another one. But you can use anything you want. You could make one script write files into a dir, and the other script could read the files out and process them, for example.
import tornado
import tornado.websocket
import tornado.httpserver
import json
import asyncio
import inspect
import time
class Handler:
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def consume(self, text):
return "You said {0} and I say hiya".format(text)
async def sweeps(self, len):
await asyncio.sleep(len)
return "Slept for {0} seconds asynchronously!".format(len)
def sleeps(self, len):
time.sleep(len)
return "Slept for {0} seconds synchronously!".format(len)
class MyService(Handler, tornado.websocket.WebSocketHandler):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def stop(self):
Handler.server.stop()
def open(self):
print("WebSocket opened")
def on_message(self, message):
print (message)
j = json.loads(message)
methodName = j["method"]
args = j.get("args", ())
method = getattr(self, methodName)
if inspect.iscoroutinefunction(method):
loop = asyncio.get_event_loop()
task = loop.create_task(method(*args))
task.add_done_callback( lambda res: self.write_message(res.result()))
future = asyncio.ensure_future(task)
elif method:
resp = method(*args)
self.write_message(resp)
def on_close(self):
print("WebSocket closed")
application = tornado.web.Application([
(r'/meow', MyService),
])
if __name__ == "__main__":
from tornado.platform.asyncio import AsyncIOMainLoop
AsyncIOMainLoop().install()
http_server = tornado.httpserver.HTTPServer(application)
Handler.server = http_server
http_server.listen(9001)
asyncio.get_event_loop().run_forever()
Now, if you run both processes in separate python scripts, and tell your bot "sleep for 100", it will sleep for 100 seconds happily!
The asyncio stuff functions as a make-shift work queue, and you can properly separate the listener from the backend processing by running them as separate python scripts.
Now, no matter how long your functions run in the 'server' part, the client part will never be prevented from pinging the discord server.
Image failed to upload, but... anyway, this is how to tell the bot to sleep and reply... note that the sleep is synchronous.
http://i.imgur.com/N4ZPPbB.png
I don't think problem happens while asyncio.sleep. Anyway you shouldn't suppress exception you got:
bot = Bot()
loop = asyncio.get_event_loop()
try:
# ...
except Exception as e:
loop.run_until_complete(bot.close())
raise e # <--- reraise exception you got while execution to see it (or log it here)
finally:
# ...
You have to manually stop your task on exit:
import discord
import asyncio
class Bot(discord.Client):
def __init__(self):
super().__init__()
#asyncio.coroutine
def my_background_task(self):
yield from self.wait_until_ready()
while not self.is_closed:
yield from asyncio.sleep(3600*24) # <- This is line 76 where it fails
doSomething()
bot = Bot()
loop = asyncio.get_event_loop()
try:
task = loop.create_task(bot.my_background_task())
loop.run_until_complete(bot.login('username', 'password'))
loop.run_until_complete(bot.connect())
except Exception:
loop.run_until_complete(bot.close())
finally:
task.cancel()
try:
loop.run_until_complete(task)
except Exception:
pass
loop.close()
import sys
import socket
import threading
import time
QUIT = False
class ClientThread(threading.Thread): # Class that implements the client threads in this server
def __init__(self, client_sock): # Initialize the object, save the socket that this thread will use.
threading.Thread.__init__(self)
self.client = client_sock
def run(self): # Thread's main loop. Once this function returns, the thread is finished and dies.
global QUIT # Need to declare QUIT as global, since the method can change it/
done = False
cmd = self.readline() #Read data from the socket and process it
while not done:
if 'quit' == cmd:
self.writeline('Ok, bye')
QUIT = True
done = True
elif 'bye' == cmd:
self.writeline('Ok, bye')
done = True
else:
self.writeline(self.name)
cmd = self.readline()
self.client.close() # Make sure socket is closed when we're done with it
return
def readline(self): # Helper function, read up to 1024 chars from the socket, and returns them as a string
result = self.client.recv(1024)
if None != result: # All letters in lower case and without and end of line markers
result = result.strip().lower()
return result
def writeline(self, text): # Helper function, writes the given string to the socket with and end of line marker at end
self.client.send(text.strip() + '\n')
class Server: # Server class. Opens up a socket and listens for incoming connections.
def __init__(self): # Every time a new connection arrives, new thread object is created and
self.sock = None # defers the processing of the connection to it
self.thread_list = []
def run(self): # Server main loop: Creates the server (incoming) socket, listens > creates thread to handle it
all_good = False
try_count = 0 # Attempt to open the socket
while not all_good:
if 3 < try_count: # Tried more than 3 times without success, maybe post is in use by another program
sys.exit(1)
try:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Create the socket
port = 80
self.sock.bind(('127.0.0.1', port)) # Bind to the interface and port we want to listen on
self.sock.listen(5)
all_good = True
break
except socket.error, err:
print('Socket connection error... Waiting 10 seconds to retry.')
del self.sock
time.sleep(10)
try_count += 1
print( 'Server is listening for incoming connections.')
print('Try to connect through the command line with:')
print('telnet localhost 80')
print('and then type whatever you want.')
print()
print("typing 'bye' finishes the thread. but not the server",)
print("eg. you can quit telnet, run it again and get a different ",)
print("thread name")
print("typing 'quit' finishes the server")
try:
while not QUIT:
try:
self.sock.settimeout(0.500)
client = self.sock.accept()[0]
except socket.timeout:
time.sleep(1)
if QUIT:
print('Received quit command. Shutting down...')
break
continue
new_thread = ClientThread(client)
print('Incoming Connection. Started thread ',)
print(new_thread.getName())
self.thread_list.append(new_thread)
new_thread.start()
for thread in self.thread_list:
if not thread.isAlive():
self.thread_list.remove(thread)
thread.join()
except KeyboardInterrupt:
print('Ctrl+C pressed... Shutting Down')
except Exception as err:
print('Exception caught: %s\nClosing...' % err)
for thread in self.thread_list:
thread.join(1.0)
self.sock.close()
if "__main__" == __name__:
server = Server()
server.run()
print('Terminated')
Resolved many issues, these are the ones left, thank you guys!
1st error: socket.error, err.This specifically tells me that this no longer works in Python 3.4, but does not offer an alternative.
2nd Error: except socket.error, err: Python 3.4 does not support this syntax
3rd Error: self.readline(), I also have to assume writeline does not work also.
In this case, self.readline() is totally not working, I get an error that says AttributeError: 'ClientThread' object has no attribute 'readline'
This only happens after the thread is created. Console shows:
Incoming connection. Started thread.
Thread-1
Then flags that error.
4th Error: Cannot get 2to3 to run? Terminal says not recognised as internal command, and python console gives a big FU.
Can I get any rectification suggestions for the following errors please?
There are multiple issues that prevent your code from woring on python3
you're using python2 print statements, so your code can't possibly run on python3 where print() is a function.
the recv and send methods require/return bytes in python3, not str
the syntax for catching errors is except ExceptionClass as name
The first step in porting Python 2 code to Python 3 is to run it though the 2to3 program that comes with Python.
If you run your code through 2to3 with the -w option, it will fix a lot of your problems automagically.
> 2to3 -w --no-diffs socktest1.py
If you want to see what would be changed, but not change anything;
> 2to3 socktest1.py |less