I'm trying to implement asynchronous client and server using pyzmq and asyncio in python3.5. I've used the asyncio libraries provided by zmq. Below is my code for client(requester.py) and server(responder.py). My requirement is to use only REQ and REP zmq sockets to achieve async client-server.
requester.py:
import asyncio
import zmq
import zmq.asyncio
async def receive():
message = await socket.recv()
print("Received reply ", "[", message, "]")
return message
async def send(i):
print("Sending request ", i,"...")
request = "Hello:" + str(i)
await socket.send(request.encode('utf-8'))
print("sent:",i)
async def main_loop_num(i):
await send(i)
# Get the reply.
message = await receive()
print("Message :", message)
async def main():
await asyncio.gather(*(main_loop_num(i) for i in range(1,10)))
port = 5556
context = zmq.asyncio.Context.instance()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:%d" % port)
asyncio.get_event_loop().run_until_complete(asyncio.wait([main()]))
responder.py:
import asyncio
import zmq
import zmq.asyncio
async def receive():
message = await socket.recv()
print("Received message:", message)
await asyncio.sleep(10)
print("Sleep complete")
return message
async def main_loop():
while True:
message = await receive()
print("back to main loop")
await socket.send(("World from %d" % port).encode('utf-8'))
print("sent back")
port = 5556
context = zmq.asyncio.Context.instance()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:%d" % port)
asyncio.get_event_loop().run_until_complete(asyncio.wait([main_loop()]))
The output that I'm getting is:
requester.py:
Sending request 5 ...
sent: 5
Sending request 6 ...
Sending request 1 ...
Sending request 7 ...
Sending request 2 ...
Sending request 8 ...
Sending request 3 ...
Sending request 9 ...
Sending request 4 ...
responder.py:
Received message: b'Hello:5'
Sleep complete
back to main loop
sent back
From the output, I assume that the requester has sent multiple requests, but only the first one has reached the responder. Also, the response sent by responder for the first request has not even reached back to the requester. Why does this happen? I have used async methods everywhere possible, still the send() and recv() methods are not behaving asynchronously. Is it possible to make async req-rep without using any other sockets like router, dealer, etc?
ZMQs REQ-REP sockets expect a strict order of one request - one reply - one request - one reply - ...
your requester.py starts all 10 requests in parallel:
await asyncio.gather(*(main_loop_num(i) for i in range(1,10)))
when sending the second request ZMQ complains about this:
zmq.error.ZMQError: Operation cannot be accomplished in current state
Try to change your main function to send one request at a time:
async def main():
for i in range(1, 10):
await main_loop_num(i)
If you need to send several requests in parallel then you can't use a REQ-REP socket pair but for example a DEALER-REP socket pair.
Related
I need only 1 client to connect to my websocket server, and when it sends a message with "pattern 1" my server will send all messages to it for that pattern. If he now needs a "pattern 2" he sends the message to the server, and the server needs to keep sending the "pattern 1" and start sending the "pattern 2" messages. Without disconnecting from my server. It's possible?
When I use the while loop, the server always gets stuck on "pattern 1" and when I take the while True, it always needs messages to enter the "pattern 1" function again.
What I need is, if you received the "pattern 1" message, stay in the function and always receive the updates, if you receive the "pattern 2" message from the client, enter the "pattern 2" function and continue to receive "pattern 1" and "pattern 2" at the same time.
app.py
import websockets
import asyncio
import warnings
import json
import datetime
import concurrent.futures
warnings.filterwarnings("ignore", category=DeprecationWarning)
PORT = 7890
print("Server listening on Port: {}".format(PORT))
async def pattern_1(websocket):
while True:
await websocket.send(str("Pattern 1"))
async def pattern_2(websocket):
while True:
await websocket.send(str("Pattern 2"))
async def handler(websocket):
try:
async for message in websocket:
message = message.replace("\'", "\"")
event = json.loads(message)
print(event)
if (event['user_id'] == 1) and (event['pattern'] == "1"):
await asyncio.create_task(pattern_1(websocket))
if (event['user_id'] == 1) and (event['pattern'] == "2"):
await asyncio.create_task(pattern_2(websocket))
except Exception as e:
print("Error: {}".format(e))
# finally:
async def main():
# Start the server
async with websockets.serve(handler, "localhost", PORT):
await asyncio.Future()
asyncio.run(main())
client.py
import websockets
import asyncio
import warnings
import json
warnings.filterwarnings("ignore", category=DeprecationWarning)
async def Candle_Pattern():
url = "ws://127.0.0.1:7890"
#connect to the server
async with websockets.connect(url) as ws:
await ws.send(json.dumps({"user_id":1, "pattern":"1"}))
#I need to connect to pattern 2 also
await ws.send(json.dumps({"user_id":1, "pattern":"2"}))
while True:
msg = await ws.recv()
print(msg)
await asyncio.sleep(2)
async def main():
await asyncio.gather(
Candle_Pattern())
asyncio.run(main())
I have an async method that I am trying to use to return data from a NodeJs application websocket interface.
When the method runs though I am just receiving a <Future pending> message within a websockets object.
What do I need to change in order for the method to receive and print the actual message from the server?
import asyncio
import json
import asyncws
import websockets
import requests
...
...
...
async def inject():
ws = await asyncws.connect(url + get_sid())
print("Sending...")
payload = base_query + f"(SELECT substring((SELECT token FROM AuthTokens WHERE UserId = 1),1,1))=3 #"
payload = string_to_json(payload, token)
payload = f'42["checkEmail", {payload}]'
ws.send(payload)
print("Sent...")
print("Receiving...")
reply = ws.recv()
if reply is None:
print("NoneType reply")
else:
print(reply)
for r in reply:
print(r)
asyncio.get_event_loop().run_until_complete(inject())
asyncio.get_event_loop().close()
Output:
Sending...
Sent...
Receiving...
<generator object Websocket.recv at 0x7fbb1813eb30>
<Future pending>
Websockets uses Python's asyncio to manage concurrency and I/O.
The websocket.send and websocket.recv are async functions. That means you need to await these to get the value.
In your code, you need to change ws.send(payload) to await ws.send(payload) and ws.recv() to await ws.recv().
You can read the documentation for Websockets here: https://websockets.readthedocs.io/en/stable/api.html#websockets.protocol.WebSocketCommonProtocol.recv
I'm making a Discord bot which send PM when it receive a Github hook.
It use Discord.py and BottlePy, the last one run in a dedicated thread.
Because both frameworks have a blocking main loop.
In BottlePy callback, I call some Discord.py async code.
I wasn't knowing what is Python async, this appear to be complicated when mixed with synchronous code...
Here's the full source code :
import discord
import bottle
import threading
import asyncio
client = discord.Client()
server = bottle.Bottle()
async def dm_on_github_async(userid,request):
print("Fire Discord dm to "+str(userid))
global client
user = client.get_user(userid)
if (user==None):
abort(500, "User lookup failed");
dm_channel = user.dm_channel
if (dm_channel==None):
dm_channel = await user.create_dm()
if (dm_channel==None):
abort(500, "Fail to create DM channel");
print("DM channel is "+str(asyncio.wait(dm_channel.id)))
await dm_channel.send("There's a Github shot !")
await dm_channel.send(str(request.body))
return
#server.post("/dm_on_github/<userid:int>")
def dm_on_github(userid):
return asyncio.run(dm_on_github_async(userid,bottle.request))
#client.event
async def on_ready():
print('We have logged in as {0.user} '.format(client))
##client.event
#async def on_message(message):
# if message.author == client.user:
# return
#
# if message.content.startswith('$hello'):
# await message.channel.send('Hello!')
# # This sample was working very well
class HTTPThread(threading.Thread):
def run(self):
global server
server.run(port=8080)
server_thread = HTTPThread()
print("Starting HTTP server")
server_thread.start()
print("Starting Discord client")
client.run('super secret key')
print("Client terminated")
server.close()
print("Asked server to terminate")
server_thread.join()
print("Server thread successful join")
I want that my Python bot send the body of the HTTP request as PM.
I get a RuntimeError: Timeout context manager should be used inside a task at return asyncio.run(dm_on_github_async(userid,bottle.request)).
I think I'm not doing this mix in the right way...
After a night, I found the way.
To call async code from sync code in another thread, we ask the loop (here this one from Discord.py) to run the callback with asyncio.run_coroutine_threadsafe(), this return a Task() and we wait for his result with result().
The callback will be run in the loop thread, in my case I need to copy() the Bottle request.
Here's a working program (as long you don't mind to stop it...) :
import discord
import bottle
import threading
import asyncio
client = discord.Client()
server = bottle.Bottle()
class HTTPThread(threading.Thread):
def run(self):
global server
server.run(port=8080)
async def dm_on_github_async(userid,request):
user = client.get_user(userid)
if (user==None):
abort(500, "User lookup failed");
dm_channel = user.dm_channel
if (dm_channel==None):
dm_channel = await user.create_dm()
if (dm_channel==None):
abort(500, "Fail to create DM channel");
# Handle the request
event = request.get_header("X-GitHub-Event")
await dm_channel.send("Got event "+str(event))
#await dm_channel.send(str(request.body)) # Doesn't work well...
return
#server.post("/dm_on_github/<userid:int>")
def dm_on_github(userid):
request = bottle.request
asyncio.run_coroutine_threadsafe(dm_on_github_async(userid,request.copy()),client.loop).result()
#client.event
async def on_ready():
print('We have logged in as {0.user} '.format(client))
# Wait for the old HTTP server
if hasattr(client,"server_thread"):
server.close()
client.server_thread.join()
client.server_thread = HTTPThread()
client.server_thread.start()
##client.event
#async def on_message(message):
# if message.author == client.user:
# return
#
# if message.content.startswith('$hello'):
# await message.channel.send('Hello!')
print("Starting Discord client")
client.run('super secret key')
print("Client terminated")
server.close()
print("Asked server to terminate")
server_thread.join()
print("Server thread successful join")
I'm trying to write an asyncio-based server. The problem is, that it stops to respond after the first request.
My code is built upon this template for echo-server and this method to pass parameters to coroutines.
class MsgHandler:
def __init__(self, mem):
# here (mem:dict) I store received metrics
self.mem = mem
async def handle(self, reader, writer):
#this coroutine handles requests
data = await reader.read(1024)
print('request:', data.decode('utf-8'))
# read_msg returns an answer based on the request received
# My server closes connection on every second request
# For the first one, everything works as intended,
# so I don't thik the problem is in read_msg()
response = read_msg(data.decode('utf-8'), self.mem)
print('response:', response)
writer.write(response.encode('utf-8'))
await writer.drain()
writer.close()
def run_server(host, port):
mem = {}
msg_handler = MsgHandler(mem)
loop = asyncio.get_event_loop()
coro = asyncio.start_server(msg_handler.handle, host, port, loop=loop)
server = loop.run_until_complete(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
On the client-side I either get an empty response or ConnectionResetError (104, 'Connection reset by peer').
You are closing the writer with writer.close() in the handler, which closes the socket.
From the 3.9 docs on StreamWriter:
Also, if you don't close the stream writer, then you would still have store it somewhere in order to keep receiving messages over that same connection.
I wrote a script for a socket server that simply listens for incoming connections and processes the incoming data. The chosen architecture is the asyncio.start_server for the socket management and the asyncio.Queues for passing the data between the producer and consumer coroutines. The problem is that the consume(q1) function is executed only once (at the first script startup). Then it is not more executed. Is the line run_until_complete(asyncio.gather()) wrong?
import asyncio
import functools
async def handle_readnwrite(reader, writer, q1): #Producer coroutine
data = await reader.read(1024)
message = data.decode()
await writer.drain()
await q1.put(message[3:20])
await q1.put(None)
writer.close() #Close the client socket
async def consume(q1): #Consumer coroutine
while True:
# wait for an item from the producer
item = await q1.get()
if item is None:
logging.debug('None items') # the producer emits None to indicate that it is done
break
do_something(item)
loop = asyncio.get_event_loop()
q1 = asyncio.Queue(loop=loop)
producer_coro = asyncio.start_server(functools.partial(handle_readnwrite, q1=q1), '0.0.0.0', 3000, loop=loop)
consumer_coro = consume(q1)
loop.run_until_complete(asyncio.gather(consumer_coro,producer_coro))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
loop.close()
handle_readnwrite always enqueues the None terminator, which causes consume to break (and therefore finish the coroutine). If consume should continue running and process other messages, the None terminator must not be sent after each message.