python asycio RuntimeWarning: coroutine was never awaited - python-3.x

Question

I am trying to send many requests to a url (~50) concurrently.
from asyncio import Queue
import yaml
import asyncio
from aiohttp import ClientSession, TCPConnector
async def http_get(url, cookie):
cookie = cookie.split('; ')
cookie1 = cookie[0].split('=')
cookie2 = cookie[1].split('=')
cookies = {
cookie1[0]: cookie1[1],
cookie2[0]: cookie2[1]
}
async with ClientSession(cookies=cookies) as session:
async with session.get(url, ssl=False) as response:
return await response.json()
class FetchUtil:
def __init__(self):
self.config = yaml.safe_load(open('../config.yaml'))
def fetch(self):
asyncio.run(self.extract_objects())
async def http_get_objects(self, object_type, limit, offset):
path = '/path' + \
'?query=&filter=%s&limit=%s&offset=%s' % (
object_type,
limit,
offset)
return await self.http_get_domain(path)
async def http_get_objects_limit(self, object_type, offset):
result = await self.http_get_objects(
object_type,
self.config['object_limit'],
offset
)
return result['result']
async def http_get_domain(self, path):
return await http_get(
f'https://{self.config["domain"]}{path}',
self.config['cookie']
)
async def call_objects(self, object_type, offset):
result = await self.http_get_objects_limit(
object_type,
offset
)
return result
async def extract_objects(self):
calls = []
object_count = (await self.http_get_objects(
'PV', '1', '0'))['result']['count']
for i in range(0, object_count, self.config['object_limit']):
calls.append(self.call_objects('PV', str(i)))
queue = Queue()
for i in range(0, len(calls), self.config['call_limit']):
results = await asyncio.gather(*calls[i:self.config['call_limit']])
await queue.put(results)
After running this code using fetch as the entry point i get the following error message:
/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/events.py:88: RuntimeWarning: coroutine 'FetchUtil.call_objects' was never awaited
self._context.run(self._callback, *self._args)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
The program that stops executing after asyncio.gather returns for the first time. I am having trouble understanding this message since I thought that I diligently made sure all functions were async tasks. The only function i didn't await was call_objects since i wanted it to run concurrently.
https://xinhuang.github.io/posts/2017-07-31-common-mistakes-using-python3-asyncio.html#org630d301
in this article gives the following explanation:
This runtime warning can happen in many scenarios, but the cause are
same: A coroutine object is created by the invocation of an async
function, but is never inserted into an EventLoop.
I believed that was what i was doing when i called the async tasks with asyncio.gather.
I should note that when i put a print('url') inside http_get it outputs the first 50 urls like i want, the problem seems to occur when asyncio.gather returns for the first time.

Answer 1

The posted code has a logic error: [i:self.config['call_limit']] should be [i:i + self.config['call_limit']].
It causes the error because the expression evaluates to an entry slice in iterations after the first one, causing some of the calls coroutines to never get passed to gather, and therefore never awaited)
i don't actually understand why it didn't just keep executing the same requests many time instead of stopping with an error.
Because your i would get incremented, making it larger than call_limit in every loop iteration except the first. For example, assuming call_limit is 10, i will be 0 in the first iteration, and you'll await calls[0:10], so far so good. But in the next iteration, i will be 10, and you'll be awaiting calls[10:10], an empty slice. In the iteration after that you'll await calls[20:10] (also an empty slice, although logically it should be an error), then calls[30:10], again empty, and so on. Only the first iteration picks up actual members of the list.