I'm trying to make a program that reads in data as well as sends and receives data from a server through a websocket. The goal is to create synchronized lamps where there are two client lamps and one server. When one of the lamps changes state it sends a request to the server and the server updates the other lamp. I'm currently stuck on the client code. I can establish a websocket connection to the server, read and send data to the server, and I can read in light data. I'm having an issue running both of these tasks simultaneously. I'd like to do it asynchronously to avoid race condition issues. I'm using python 3.8 and asyncio.
Here is my websocket client code so far:
async def init_connection(message):
global CONNECTION_OPEN
global CLIENT_WS
uri = WS_URI
async with websockets.connect(uri) as websocket:
CONNECTION_OPEN = True
CLIENT_WS = websocket
# send init message
await websocket.send(message)
while CONNECTION_OPEN:
await handleMessages(websocket, message)
await websocket.send(json.dumps({'type': MessageType.Close.name, 'message': USERNAME}))
await websocket.close()
Here is my read in data code so far:
async def calculate_idle(t):
global STATE
global prevColor
x_arr = []
y_arr = []
z_arr = []
while t >= 0:
x, y, z = lis3dh.acceleration
print("Current colors")
print(accel_to_color(x,y,z))
x_arr.append(x)
y_arr.append(y)
z_arr.append(z)
newColor = accel_to_color(x,y,z)
# remember prev color
do_fade(prevColor, newColor)
#strip.fill((int(a_x), int(a_y), int(a_z), 0))
#strip.show()
prevColor = newColor
time.sleep(.2)
t -= .2
is_idle = is_lamp_idle(np.std(x_arr), np.std(y_arr), np.std(z_arr))
if is_idle and STATE == "NOT IDLE" and CONNECTION_OPEN:
STATE = "IDLE"
print("Sending color")
await asyncio.sleep(1)
elif is_idle and CONNECTION_OPEN:
# Check for data
STATE = "IDLE"
print ("Receiving data")
await asyncio.sleep(1)
elif is_idle and not CONNECTION_OPEN:
print ("Idle and not connected")
rainbow_cycle(0.001) # rainbow cycle with 1ms delay per step
await asyncio.sleep(1)
else:
STATE = "NOT IDLE"
await asyncio.sleep(1)
print("Is not idle")
Here is the code that is supposed to tie them together:
async def main():
message = json.dumps({'type': "authentication", 'payload': {
'username': 'user1', 'secret': SHARED_SECRET}})
loop = asyncio.get_event_loop()
start_light = asyncio.create_task(calculate_idle(3))
await asyncio.gather(init_connection(message), start_light)
asyncio.run(main())
There's other functions, but the premise is there's a websocket connection sending and receiving data and another process reading in light data. I also need to be able to read the current state of the lights and set the current state of the lights which is why I was using global variables. Currently, it'll read the lights until it hits an await asyncio.sleep(1) in calculate idle, then switch to the websocket code and hang receiving data from the server. Ideally, it would alternate between reading the current state and checking for websocket messages. If the state changes, it would then send a websocket message.
How can I run both of these routine asynchronously and share the data between them? Any help is appreciated!
Thanks to user4815162342's comments to help narrow down the issue. My calculate idle didn't have a while true and I changed time.sleep(.2) to await asyncio.sleep(.2) and I was able to read data from the server and the lights at the same time.
Related
I have a small Server written like this:
async def handle_client(reader, writer):
request = (await reader.read()).decode('utf8') # should read until end of msg
print(request)
response = "thx"
writer.write(response.encode('utf8'))
await writer.drain()
writer.close()
loop = asyncio.get_event_loop()
loop.create_task(asyncio.start_server(handle_client, socket.gethostname(), 8886))
loop.run_forever()
and a small client written like this:
async def tcp_echo_client(message):
reader, writer = await asyncio.open_connection(
my_ip, 8886)
print(f'Send: {message!r}')
writer.write(message.encode())
await writer.drain()
data = await reader.read() # should read until end of msg
print(f'Received: {data.decode()!r}')
print('Close the connection')
writer.close()
await writer.wait_closed()
asyncio.run(tcp_echo_client(f"Hello World!"))
the client and the server start the comunication but never finish it.
why does the reader not recognise end of msg?
if i write
request = (await reader.read(1024)).decode('utf8')
instead it works, but i need to recive undefined large amount of data.
i tried to modify the code of the server like this:
while True:
request = (await reader.read(1024)).decode('utf8')
if not request:
break
it recieves all data blocks but still waits forever after the last block. why?
how do i tell the reader from the server to stop listenig and proceed in the code to send the answer?
TCP connections are stream-based, which means that when you write a "message" to a socket, the bytes will be sent to the peer without including a delimiter between messages. The peer on the other side of the connection can retrieve the bytes, but it needs to figure out on its own how to slice them into "messages". This is why reading the last block appears to hang: read() simply waits for the peer to send more data.
To enable retrieval of individual messages, the sender must frame or delimit each message. For example, the sender could just close the connection after sending a message, which would allow the other side to read the message because it would be followed by the end-of-file indicator. However, that would allow sender to only send one message without the ability to read a response because the socket would be closed.
A better option is for the writer to only close the writing side of the socket, such partial close sometimes being referred to as shutdown). In asyncio this is done with a call to write_eof:
writer.write(message.encode())
await writer.drain()
writer.write_eof()
Sent like this, the message will be followed by end-of-file and the read on the server side won't hang. While the client will be able to read the response, it will still be limited to sending only one message because further writes will be impossible on the socket whose writing end was closed.
To implement communication consisting of an arbitrary number of requests and responses, you need to frame each message. A simple way to do so is by prefixing each message with message length:
writer.write(struct.pack('<L', len(request)))
writer.write(request)
The receiver first reads the message size and then the message itself:
size, = struct.unpack('<L', await reader.readexactly(4))
request = await reader.readexactly(size)
I'm trying to create a WebSocket command line client that waits for messages from a WebSocket server but waits for user input at the same time.
Regularly polling multiple online sources every second works fine on the server, (the one running at localhost:6789 in this example), but instead of using Python's normal sleep() method, it uses asyncio.sleep(), which makes sense because sleeping and asynchronously sleeping aren't the same thing, at least not under the hood.
Similarly, waiting for user input and asynchronously waiting for user input aren't the same thing, but I can't figure out how to asynchronously wait for user input in the same way that I can asynchronously wait for an arbitrary amount of seconds, so that the client can deal with incoming messages from the WebSocket server while simultaneously waiting for user input.
The comment below in the else-clause of monitor_cmd() hopefully explains what I'm getting at:
import asyncio
import json
import websockets
async def monitor_ws():
uri = 'ws://localhost:6789'
async with websockets.connect(uri) as websocket:
async for message in websocket:
print(json.dumps(json.loads(message), indent=2, sort_keys=True))
async def monitor_cmd():
while True:
sleep_instead = False
if sleep_instead:
await asyncio.sleep(1)
print('Sleeping works fine.')
else:
# Seems like I need the equivalent of:
# line = await asyncio.input('Is this your line? ')
line = input('Is this your line? ')
print(line)
try:
asyncio.get_event_loop().run_until_complete(asyncio.wait([
monitor_ws(),
monitor_cmd()
]))
except KeyboardInterrupt:
quit()
This code just waits for input indefinitely and does nothing else in the meantime, and I understand why. What I don't understand, is how to fix it. :)
Of course, if I'm thinking about this problem in the wrong way, I'd be very happy to learn how to remedy that as well.
You can use the aioconsole third-party package to interact with stdin in an asyncio-friendly manner:
line = await aioconsole.ainput('Is this your line? ')
Borrowing heavily from aioconsole, if you would rather avoid using an external library you could define your own async input function:
async def ainput(string: str) -> str:
await asyncio.get_event_loop().run_in_executor(
None, lambda s=string: sys.stdout.write(s+' '))
return await asyncio.get_event_loop().run_in_executor(
None, sys.stdin.readline)
Borrowing heavily from aioconsole, there are 2 ways to handle.
start a new daemon thread:
import sys
import asyncio
import threading
from concurrent.futures import Future
async def run_as_daemon(func, *args):
future = Future()
future.set_running_or_notify_cancel()
def daemon():
try:
result = func(*args)
except Exception as e:
future.set_exception(e)
else:
future.set_result(result)
threading.Thread(target=daemon, daemon=True).start()
return await asyncio.wrap_future(future)
async def main():
data = await run_as_daemon(sys.stdin.readline)
print(data)
if __name__ == "__main__":
asyncio.run(main())
use stream reader:
import sys
import asyncio
async def get_steam_reader(pipe) -> asyncio.StreamReader:
loop = asyncio.get_event_loop()
reader = asyncio.StreamReader(loop=loop)
protocol = asyncio.StreamReaderProtocol(reader)
await loop.connect_read_pipe(lambda: protocol, pipe)
return reader
async def main():
reader = await get_steam_reader(sys.stdin)
data = await reader.readline()
print(data)
if __name__ == "__main__":
asyncio.run(main())
I have a streaming application that almost continuously takes the data given as input and sends an HTTP request using that value and does something with the returned value.
Obviously to speed things up I've used asyncio and aiohttp libraries in Python 3.7 to get the best performance, but it becomes hard to debug given how fast the data moves.
This is what my code looks like
'''
Gets the final requests
'''
async def apiRequest(info, url, session, reqType, post_data=''):
if reqType:
async with session.post(url, data = post_data) as response:
info['response'] = await response.text()
else:
async with session.get(url+post_data) as response:
info['response'] = await response.text()
logger.debug(info)
return info
'''
Loops through the batches and sends it for request
'''
async def main(data, listOfData):
tasks = []
async with ClientSession() as session:
for reqData in listOfData:
try:
task = asyncio.ensure_future(apiRequest(**reqData))
tasks.append(task)
except Exception as e:
print(e)
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
responses = await asyncio.gather(*tasks)
return responses #list of APIResponses
'''
Streams data in and prepares batches to send for requests
'''
async def Kconsumer(data, loop, batchsize=100):
consumer = AIOKafkaConsumer(**KafkaConfigs)
await consumer.start()
dataPoints = []
async for msg in consumer:
try:
sys.stdout.flush()
consumedMsg = loads(msg.value.decode('utf-8'))
if consumedMsg['tid']:
dataPoints.append(loads(msg.value.decode('utf-8')))
if len(dataPoints)==batchsize or time.time() - startTime>5:
'''
#1: The task below goes and sends HTTP GET requests in bulk using aiohttp
'''
task = asyncio.ensure_future(getRequests(data, dataPoints))
res = await asyncio.gather(*[task])
if task.done():
outputs = []
'''
#2: Does some ETL on the returned values
'''
ids = await asyncio.gather(*[doSomething(**{'tid':x['tid'],
'cid':x['cid'], 'tn':x['tn'],
'id':x['id'], 'ix':x['ix'],
'ac':x['ac'], 'output':to_dict(xmltodict.parse(x['response'],encoding='utf-8')),
'loop':loop, 'option':1}) for x in res[0]])
simplySaveDataIntoDataBase(id) # This is where I see some missing data in the database
dataPoints = []
except Exception as e:
logger.error(e)
logger.error(traceback.format_exc())
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
logger.error(str(exc_type) +' '+ str(fname) +' '+ str(exc_tb.tb_lineno))
if __name__ == '__main__':
loop = asyncio.get_event_loop()
asyncio.ensure_future(Kconsumer(data, loop, batchsize=100))
loop.run_forever()
Does the ensure_future need to be awaited ?
How does aiohttp handle requests that come a little later than the others? Shouldn't it hold the whole batch back instead of forgetting about it altoghter?
Does the ensure_future need to be awaited ?
Yes, and your code is doing that already. await asyncio.gather(*tasks) awaits the provided tasks and returns their results in the same order.
Note that await asyncio.gather(*[task]) doesn't make sense, because it is equivalent to await asyncio.gather(task), which is again equivalent to await task. In other words, when you need the result of getRequests(data, dataPoints), you can write res = await getRequests(data, dataPoints) without the ceremony of first calling ensure_future() and then calling gather().
In fact, you almost never need to call ensure_future yourself:
if you need to await multiple tasks, you can pass coroutine objects directly to gather, e.g. gather(coroutine1(), coroutine2()).
if you need to spawn a background task, you can call asyncio.create_task(coroutine(...))
How does aiohttp handle requests that come a little later than the others? Shouldn't it hold the whole batch back instead of forgetting about it altoghter?
If you use gather, all requests must finish before any of them return. (That is not aiohttp policy, it's how gather works.) If you need to implement a timeout, you can use asyncio.wait_for or similar.
I'm trying to write an asyncio-based server. The problem is, that it stops to respond after the first request.
My code is built upon this template for echo-server and this method to pass parameters to coroutines.
class MsgHandler:
def __init__(self, mem):
# here (mem:dict) I store received metrics
self.mem = mem
async def handle(self, reader, writer):
#this coroutine handles requests
data = await reader.read(1024)
print('request:', data.decode('utf-8'))
# read_msg returns an answer based on the request received
# My server closes connection on every second request
# For the first one, everything works as intended,
# so I don't thik the problem is in read_msg()
response = read_msg(data.decode('utf-8'), self.mem)
print('response:', response)
writer.write(response.encode('utf-8'))
await writer.drain()
writer.close()
def run_server(host, port):
mem = {}
msg_handler = MsgHandler(mem)
loop = asyncio.get_event_loop()
coro = asyncio.start_server(msg_handler.handle, host, port, loop=loop)
server = loop.run_until_complete(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
On the client-side I either get an empty response or ConnectionResetError (104, 'Connection reset by peer').
You are closing the writer with writer.close() in the handler, which closes the socket.
From the 3.9 docs on StreamWriter:
Also, if you don't close the stream writer, then you would still have store it somewhere in order to keep receiving messages over that same connection.
I wrote a script for a socket server that simply listens for incoming connections and processes the incoming data. The chosen architecture is the asyncio.start_server for the socket management and the asyncio.Queues for passing the data between the producer and consumer coroutines. The problem is that the consume(q1) function is executed only once (at the first script startup). Then it is not more executed. Is the line run_until_complete(asyncio.gather()) wrong?
import asyncio
import functools
async def handle_readnwrite(reader, writer, q1): #Producer coroutine
data = await reader.read(1024)
message = data.decode()
await writer.drain()
await q1.put(message[3:20])
await q1.put(None)
writer.close() #Close the client socket
async def consume(q1): #Consumer coroutine
while True:
# wait for an item from the producer
item = await q1.get()
if item is None:
logging.debug('None items') # the producer emits None to indicate that it is done
break
do_something(item)
loop = asyncio.get_event_loop()
q1 = asyncio.Queue(loop=loop)
producer_coro = asyncio.start_server(functools.partial(handle_readnwrite, q1=q1), '0.0.0.0', 3000, loop=loop)
consumer_coro = consume(q1)
loop.run_until_complete(asyncio.gather(consumer_coro,producer_coro))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
loop.close()
handle_readnwrite always enqueues the None terminator, which causes consume to break (and therefore finish the coroutine). If consume should continue running and process other messages, the None terminator must not be sent after each message.