I have an async method that I am trying to use to return data from a NodeJs application websocket interface.
When the method runs though I am just receiving a <Future pending> message within a websockets object.
What do I need to change in order for the method to receive and print the actual message from the server?
import asyncio
import json
import asyncws
import websockets
import requests
...
...
...
async def inject():
ws = await asyncws.connect(url + get_sid())
print("Sending...")
payload = base_query + f"(SELECT substring((SELECT token FROM AuthTokens WHERE UserId = 1),1,1))=3 #"
payload = string_to_json(payload, token)
payload = f'42["checkEmail", {payload}]'
ws.send(payload)
print("Sent...")
print("Receiving...")
reply = ws.recv()
if reply is None:
print("NoneType reply")
else:
print(reply)
for r in reply:
print(r)
asyncio.get_event_loop().run_until_complete(inject())
asyncio.get_event_loop().close()
Output:
Sending...
Sent...
Receiving...
<generator object Websocket.recv at 0x7fbb1813eb30>
<Future pending>
Websockets uses Python's asyncio to manage concurrency and I/O.
The websocket.send and websocket.recv are async functions. That means you need to await these to get the value.
In your code, you need to change ws.send(payload) to await ws.send(payload) and ws.recv() to await ws.recv().
You can read the documentation for Websockets here: https://websockets.readthedocs.io/en/stable/api.html#websockets.protocol.WebSocketCommonProtocol.recv
Related
I want my discord.py bot to send a meme from hot posts of r/memes via PRAW. After this issue, I tried searching in the web and in the documentations, but I didn't find any method to view the image. Here is my code:
import praw
import discord
from discord.ext import commands
from discord import client
reddit = praw.Reddit(client_id="d",
client_secret="d",
user_agent="automoderatoredj by /u/taskuratik")
#boot
print("il bot si sta avviando... ")
token = "token"
client = commands.Bot(command_prefix=("/"))
#bot online
#client.event
async def on_ready():
print("il bot e' ora online")
#client.command()
async def meme(submission):
if reddit:
channel = client.get_channel(722491234991472742)
submission = reddit.subreddit("memes").hot(limit=1)
await channel.send(submission.url)
client.run(token)
Your code says:
submission = reddit.subreddit("memes").hot(limit=1)
await channel.send(submission.url)
Here, you assign a listing of one post to submission. As listing is an iterable (somewhat like a list) that contains one submission, rather than the submission itself. Unlike a list, you can't use an index to access a specific item, but there are other ways to get it. One way to get the submission is
for submission in reddit.subreddit("memes").hot(limit=1):
await channel.send(submission.url)
This allows you to change the limit and send more posts if you want.
Or, you could use next() to get the next (and only) item from the post listing:
submission = next(reddit.subreddit("memes").hot(limit=1))
await channel.send(submission.url)
This will always send just one submission, even if you change the limit parameter.
PRAW is blocking, aiohttp is not and frankly discord.py comes with aiohttp. Reddit offers an endpoint to return json data that you can prase with the json.loads() method to get raw json.
This is something I wrote to fetch images from subreddits
from aiohttp import ClientSession
from random import choice as c
from json import loads
async def get(session: object, url: object) -> object:
async with session.get(url) as response:
return await response.text()
async def reddit(sub: str):
type = ['new', 'top', 'hot', 'rising']
url = f"https://www.reddit.com/r/{sub}/{c(type)}.json?sort={c(type)}&limit=10"
async with ClientSession() as session:
data = await get(session, url)
data = loads(data)
data = data['data']['children']
url = [d['data']['url'] for d in data]
return c(url)
All you need to do is await reddit(sub= 'memes') to get the required url.
Consider following code fragment.
async def f():
http_client_session = aiohttp.ClientSession()
headers = {"developerkey": "somekey"}
body = {
"password": "somepassword",
"username": "someemail#gmail.com",
}
url = "https://localhost/login"
response_body = None
async with http_client_session.post(url, json=body, headers=headers) as response:
assert response.status == 200
response_body = await response.json()
await http_client_session.close()
return response_body()
The function f is awaited in another function. aiohttp gives the warning 'Unclosed client session' but I do not understand this as I have already awaited for it to close the session.
I've seen a similar issue previously in another project. Turns out it may be just that you need to allow some time for the session to fully close. For my project I added time.sleep(5) after the close statement to let the connection end.
See this ticket on aiohttp: https://github.com/aio-libs/aiohttp/issues/1925
I'm trying to implement asynchronous client and server using pyzmq and asyncio in python3.5. I've used the asyncio libraries provided by zmq. Below is my code for client(requester.py) and server(responder.py). My requirement is to use only REQ and REP zmq sockets to achieve async client-server.
requester.py:
import asyncio
import zmq
import zmq.asyncio
async def receive():
message = await socket.recv()
print("Received reply ", "[", message, "]")
return message
async def send(i):
print("Sending request ", i,"...")
request = "Hello:" + str(i)
await socket.send(request.encode('utf-8'))
print("sent:",i)
async def main_loop_num(i):
await send(i)
# Get the reply.
message = await receive()
print("Message :", message)
async def main():
await asyncio.gather(*(main_loop_num(i) for i in range(1,10)))
port = 5556
context = zmq.asyncio.Context.instance()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:%d" % port)
asyncio.get_event_loop().run_until_complete(asyncio.wait([main()]))
responder.py:
import asyncio
import zmq
import zmq.asyncio
async def receive():
message = await socket.recv()
print("Received message:", message)
await asyncio.sleep(10)
print("Sleep complete")
return message
async def main_loop():
while True:
message = await receive()
print("back to main loop")
await socket.send(("World from %d" % port).encode('utf-8'))
print("sent back")
port = 5556
context = zmq.asyncio.Context.instance()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:%d" % port)
asyncio.get_event_loop().run_until_complete(asyncio.wait([main_loop()]))
The output that I'm getting is:
requester.py:
Sending request 5 ...
sent: 5
Sending request 6 ...
Sending request 1 ...
Sending request 7 ...
Sending request 2 ...
Sending request 8 ...
Sending request 3 ...
Sending request 9 ...
Sending request 4 ...
responder.py:
Received message: b'Hello:5'
Sleep complete
back to main loop
sent back
From the output, I assume that the requester has sent multiple requests, but only the first one has reached the responder. Also, the response sent by responder for the first request has not even reached back to the requester. Why does this happen? I have used async methods everywhere possible, still the send() and recv() methods are not behaving asynchronously. Is it possible to make async req-rep without using any other sockets like router, dealer, etc?
ZMQs REQ-REP sockets expect a strict order of one request - one reply - one request - one reply - ...
your requester.py starts all 10 requests in parallel:
await asyncio.gather(*(main_loop_num(i) for i in range(1,10)))
when sending the second request ZMQ complains about this:
zmq.error.ZMQError: Operation cannot be accomplished in current state
Try to change your main function to send one request at a time:
async def main():
for i in range(1, 10):
await main_loop_num(i)
If you need to send several requests in parallel then you can't use a REQ-REP socket pair but for example a DEALER-REP socket pair.
I have a streaming application that almost continuously takes the data given as input and sends an HTTP request using that value and does something with the returned value.
Obviously to speed things up I've used asyncio and aiohttp libraries in Python 3.7 to get the best performance, but it becomes hard to debug given how fast the data moves.
This is what my code looks like
'''
Gets the final requests
'''
async def apiRequest(info, url, session, reqType, post_data=''):
if reqType:
async with session.post(url, data = post_data) as response:
info['response'] = await response.text()
else:
async with session.get(url+post_data) as response:
info['response'] = await response.text()
logger.debug(info)
return info
'''
Loops through the batches and sends it for request
'''
async def main(data, listOfData):
tasks = []
async with ClientSession() as session:
for reqData in listOfData:
try:
task = asyncio.ensure_future(apiRequest(**reqData))
tasks.append(task)
except Exception as e:
print(e)
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
responses = await asyncio.gather(*tasks)
return responses #list of APIResponses
'''
Streams data in and prepares batches to send for requests
'''
async def Kconsumer(data, loop, batchsize=100):
consumer = AIOKafkaConsumer(**KafkaConfigs)
await consumer.start()
dataPoints = []
async for msg in consumer:
try:
sys.stdout.flush()
consumedMsg = loads(msg.value.decode('utf-8'))
if consumedMsg['tid']:
dataPoints.append(loads(msg.value.decode('utf-8')))
if len(dataPoints)==batchsize or time.time() - startTime>5:
'''
#1: The task below goes and sends HTTP GET requests in bulk using aiohttp
'''
task = asyncio.ensure_future(getRequests(data, dataPoints))
res = await asyncio.gather(*[task])
if task.done():
outputs = []
'''
#2: Does some ETL on the returned values
'''
ids = await asyncio.gather(*[doSomething(**{'tid':x['tid'],
'cid':x['cid'], 'tn':x['tn'],
'id':x['id'], 'ix':x['ix'],
'ac':x['ac'], 'output':to_dict(xmltodict.parse(x['response'],encoding='utf-8')),
'loop':loop, 'option':1}) for x in res[0]])
simplySaveDataIntoDataBase(id) # This is where I see some missing data in the database
dataPoints = []
except Exception as e:
logger.error(e)
logger.error(traceback.format_exc())
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
logger.error(str(exc_type) +' '+ str(fname) +' '+ str(exc_tb.tb_lineno))
if __name__ == '__main__':
loop = asyncio.get_event_loop()
asyncio.ensure_future(Kconsumer(data, loop, batchsize=100))
loop.run_forever()
Does the ensure_future need to be awaited ?
How does aiohttp handle requests that come a little later than the others? Shouldn't it hold the whole batch back instead of forgetting about it altoghter?
Does the ensure_future need to be awaited ?
Yes, and your code is doing that already. await asyncio.gather(*tasks) awaits the provided tasks and returns their results in the same order.
Note that await asyncio.gather(*[task]) doesn't make sense, because it is equivalent to await asyncio.gather(task), which is again equivalent to await task. In other words, when you need the result of getRequests(data, dataPoints), you can write res = await getRequests(data, dataPoints) without the ceremony of first calling ensure_future() and then calling gather().
In fact, you almost never need to call ensure_future yourself:
if you need to await multiple tasks, you can pass coroutine objects directly to gather, e.g. gather(coroutine1(), coroutine2()).
if you need to spawn a background task, you can call asyncio.create_task(coroutine(...))
How does aiohttp handle requests that come a little later than the others? Shouldn't it hold the whole batch back instead of forgetting about it altoghter?
If you use gather, all requests must finish before any of them return. (That is not aiohttp policy, it's how gather works.) If you need to implement a timeout, you can use asyncio.wait_for or similar.
So i have a command where it sends whatever the user said after the command to a website api and sends the file the site generates. However i'm changing over to aiohttp as it doesn't block like the standered requests functions
This is how i do it with normal requests and it works fine:
elif (data[0].lower() == ">signgirl"):
await bot.send_typing(message.channel)
tmp = message.content.replace(">signgirl", "")
m = hashlib.md5()
m.update(tmp.encode('utf-8'))
print(tmp, m.hexdigest())
r = requests.post("http://localhost/sign.php", stream=True, data={'text': tmp})
if (r.status_code() == 200):
await bot.send_file(destination=message.channel, filename=str(m.hexdigest()+".png"), fp=r.raw)
However when i try with aiohttp i have no idea how to actually get the raw file data..
So i made this function to get it. but it doesn't let me return an image and i cannot check the http status code without it causing an error.
async def post_data2(url, payload):
async with aiohttp.ClientSession() as session2:
async with session2.post(url, data=payload) as response2:
output = {}
output['data'] = await Image.open(BytesIO(response2.read()))
output['status'] = 200 #await str(response2.status()) #Why is this object not callable?
return output
How else could i do this? Is this possible? aiohttp doesn't seem as easy to understand.
Mister Day "V" Own from the discord.py discord server sent a perfect example of getting and sending the data
async with aiohttp.ClientSession() as session:
# or use a session you already have
async with session.get("http://example.com") as resp:
buffer = io.BytesIO(await resp.read())
# buffer is a file-like
await client.send_file(channel, fp=buffer, filename="whatever")