I'm trying to write tests for a method that uses "async with" statements (in this case, aioredis's connection pool), i want to mock the connection to redis, but i'm having trouble figuring out how.
Here's what i have so far:
from asyncio import Future
from unittest.mock import MagicMock
import pytest
# The thing i'm trying to test
async def set_value(redis, value):
# Do things
async with redis.get() as conn:
await conn.set("key", value)
#My Mock classes
class MockRedis():
def get(self):
return MockAsyncPool()
class MockAsyncPool(MagicMock):
async def __aenter__(self):
conn = MagicMock()
f = Future()
f.set_result(True)
conn.set = MagicMock(return_value=f)
return conn
def __aexit__(self, exc_type, exc_val, exc_tb):
pass
# The actual test
#pytest.mark.asyncio
async def test_get_token():
redis = MockRedis()
token = await set_value(redis, 'something')
assert token is not None
I run it with:
py.test path/to/file.py
And i'm getting this error:
> await conn.set("key", value)
E TypeError: object NoneType can't be used in 'await' expression
__aexit__ needs to also be asyncronous (needs to return an awaitable):
async def __aexit__(self, exc_type, exc_val, exc_tb):
pass
Without being async it is returning None instead of a coroutine so it raises an error, as for the very misleading error message I have created this issue to point out that the error message could be greatly improved.
Related
I have a class i created thats a websocket and it connects to my data endpoint with no issues. However, I wanted my socket to run forever. I am using the websockets python library. Heres a sample:
from websockets import connect
class Socket(metaclass=ABCMeta):
def __init__(self, url: str):
self.url = url
async def __aenter__(self):
self._conn = connect(self.url, ping_interval=None)
self.websocket = await self._conn.__aenter__()
return self
async def __aexit__(self, *args, **kwargs):
await self._conn.__aexit__(*args, **kwargs)
Now, i am able to write async with statement with no problems. My issue arises when i want my socket to remain connected.
reading in the library, it seems one suggested way is to do the following:
async for socket in websockets.connect(", ping_interval=None):
try:
your logic
except websockets.closedConnection as e:
continue
This allows me to keep trying to connect if there is an issue. How would i incorporate this into my class as an iterator? I tried the following but getting error:
TypeError: 'async for' received an object from __aiter__ that does not implement __anext__: coroutine
After i added the following code in the above class:
async def __aiter__(self):
return self
async def __anext__(self):
async for websocket in connect(self.url, ping_interval=None):
try:
self.websocket = await websocket
except StopIteration:
raise StopAsyncIteration
I am not posting my entire code here as the goal is to encapsulate a class around this socket class i created with the goal being
async for object in MyCustomClassSocketIterator(url):
try:
await object.send()
await object.receive()
except websockets.closedConnection as e:
etc....
where the encapsulated class has implemented receive() and send() functions. So each time program starts, object is instantiated asynchronously. If anything breaks...then it attemps to connect again if there is a socket.closedConnection. Thanks
I have switched my SQLAlchemy database code to use an async engine and am having trouble establishing basic functionality.
I have a class that starts the database like this:
class PostgresDb:
def __init__(self):
self._session = None
self._engine = None
def __getattr__(self, name):
return getattr(self._session, name)
def init(self):
self._engine = create_async_engine(
ENGINE,
echo=True,
future=True)
self._session = sessionmaker(
self._engine, expire_on_commit=False, class_=AsyncSession
)()
async def create_all(self):
async with self._engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
# Other methods...
Example of how create_all gets called:
async def init_db_tables():
self.init()
await self.create_all()
asyncio.run(init_db_tables())
When I want to achieve basic functionality, like getting all the tables, I can do something like:
def get_tables(self):
with create_engine(SYNCHRONOUS_ENGINE).connect() as conn:
meta = MetaData(conn, schema=SCHEMA)
meta.reflect(views=True)
table_list = meta.tables
return table_list
This is not ideal as I need to actually pass in a synchronous engine connection instead of the actual async engine I am using in the Class. It is also very verbose and shouldn't need to be initiated like this for every query.
I have tried doing something like this to select the table 'appuser' from the database:
async def get_tables(self):
self.init()
async with self._session() as session:
q = select('appuser')
result = await session.execute(q)
curr = result.scalars()
for i in curr:
print(i)
Which I've tried calling like this
db = PostgresDb()
asyncio.run(db.get_tables())
asyncio.get_event_loop().run_until_complete(db.get_tables())
These both give error:
async with self._session() as session:
TypeError: 'AsyncSession' object is not callable
Calling it with db.get_tables() errors RuntimeWarning: coroutine 'PostgresDb.get_tables' was never awaited db.get_tables() RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Trying to use inspector with run_sync like this:
async def get_tables(self):
await self.init()
async with self._engine.begin() as conn:
inspector = conn.run_sync(inspect(conn))
table_names = await conn.run_sync(inspector.get_table_names())
print(table_names)
Returns error
sqlalchemy.exc.NoInspectionAvailable: Inspection on an AsyncConnection is currently not supported. Please use ``run_sync`` to pass a callable where it's possible to call ``inspect`` on the passed connection.
I have read the documentation at https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html#sqlalchemy.ext.asyncio.AsyncConnection.run_sync but I am still unclear about how to work cleanly with async engines.
Thanks for any and all insight you're able to offer on how to execute a simple query get all tables in SQLAlchmey using the async engine!
I am trying to achieve fire_and_forget functionality and here are the details.
Async decorater i am trying to use:
import asyncio
import time
def fire_and_forget(f):
def wrapped(*args, **kwargs):
return asyncio.get_event_loop().run_in_executor(None, f, *args, *kwargs)
return wrapped
My Async call using above decorator:
#fire_and_forget
def call_async(self, req, body, headers_params):
logger.info("Calling ASYNC")
try:
f = urllib.request.urlopen(req)
response = f.read()
f.close()
except Exception as e:
logger.exception("api exception %s" % e)
return None
# Parse and return the response
try:
res = self._parse_response(response)
except Exception as e:
logger.exception("Exception in parsing response of %s" % e)
res = None
logger.debug("clevertap response: {}".format(res))
My Flask app calling test_xyz which in turns fire the above fire and forget call_async:
from flask import Flask, jsonify, request
from call_xyz import test_xyz
app = Flask(__name__)
#app.route('/events_dm')
def events_dm():
session_id = request.args.get('sessionId', "11111111")
test_obj = test_xyz(session_id)
test_obj.track_test()
return jsonify({"success": True})
app.run(
host='0.0.0.0',
port=8091,
debug=True,
threaded=True
)
I am not able to understand where to set my event loop correctly so that i don't get the error: "Error in xyz There is no current event loop in thread 'Thread-7'" and my events get fired correctly.
Flask is not asyncio-compatible, so you shouldn't attempt to use asyncio within it.
Besides, you're not actually using asyncio's functionality, but run_in_executor which calls into concurrent.futures, i.e. uses threads under the hood. If that's what you need, you can create an executor directly and just submit your function to it. For example:
import concurrent.futures
_pool = concurrent.futures.ThreadPoolExecutor()
def fire_and_forget(f):
def wrapped(*args, **kwargs):
return _pool.submit(lambda: f(*args, **kwargs))
return wrapped
You must probably go through once asyncio usage and its main heart event loop understanding.
Similar issue which might help you to understand.
Here is the tutorial with some good explanation
Just give some flow for how use coroutine with normal Flask app here is the sample one.
import asyncio
import datetime
from flask import Flask, jsonify, request
app = Flask(__name__)
def print_now():
print(datetime.datetime.now())
async def keep_printing(name: str="") -> None:
print(name, end=" ")
print_now()
await asyncio.sleep(1.50)
async def main(num_times):
result = await asyncio.gather(
keep_printing("first"),
keep_printing("second"),
keep_printing("third"),
)
return result
def execute_statement():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(main(2))
loop.close()
#app.route('/events_dm')
def events_dm():
execute_statement()
return jsonify({"success": True})
app.run(
host='0.0.0.0',
port=8091,
debug=True,
threaded=True
)
Output when you hit /events_dm
first 2020-07-18 00:46:26.452921
second 2020-07-18 00:46:26.452921
third 2020-07-18 00:46:26.452921
127.0.0.1 - - [18/Jul/2020 00:46:27] "GET /events_dm HTTP/1.1" 200 -
In discord.py 1.0.1 (Only version Repl.it has), the cogs are giving me a hard time.
import discord
from discord.ext import commands
class Coding(commands, Cog):
def __init__(self, client):
self.client = client
#commands.Cog.listener()
async def on_ready(self):
print("Kahoot Bot 0.1 ALPHA")
client.remove_command("help")
#commands.command()
async def clear(self, ctx, amount = 5):
await ctx.channel.purge(limit = amount + 1)
#commands.command()
async def ping(self, ctx):
await ctx.send(f"Pong! {round(client.latency * 1000)}ms.")
#client.command(pass_context = True, aliases = ["print"])
async def printing(ctx, *, what_to_print):
await ctx.send(what_to_print)
print(what_to_print)
def setup(client):
client.add_cog(Coding(client))
The gist of the errors is:
A) client is not defined
B) init() should return None, not coroutine
I've tried changing all my code to bot and back to client, but nothing's helped. No idea what's going on.
You do the inheritance wrong. You dont inherit from class: "commands" and "Cog". You inherit from class: "commands.Cog". thus changing: class Coding(commands, Cog): to class Coding(commands.Cog): will fix some of the errors.
You also do the following wrong (the "client does not exist" error):
#commands.Cog.listener()
async def on_ready(self):
print("Kahoot Bot 0.1 ALPHA")
client.remove_command("help") # this line
When we want to access a class variable we use self. in the beginning of that variable to indicate that we are using the class variable. This case you dont use self.client. but client.
As client is not defined in that function it will give an error. But it is defined as a class variable (the "init" function). To access it use: self.client.
Here's a sample code.
class Foo:
def __init__(self):
self._run_coro()
def _run_coro(self):
async def init():
bar = #some I/O op
self.bar = bar
loop = asyncio.get_event_loop()
loop.run_until_complete(init())
async def spam(self):
return await #I/O op
async def main():
foo = Foo()
await foo.spam()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
When I run this code, I get following exception:
RuntimeError: This event loop is already running
If I initialize Foo outside main, the code runs without any exception. I want to initialize Foo such that during initialization it runs a coroutine which creates a class attribute bar.
I am unable to figure how to do it correctly. How can I run a coroutine from __init__.
Any help would be highly appreciated.
class Foo:
def __init__(self):
self.session = requests.Session()
self.async_session = None
#I guess this can be done to initialize it.
s = self.init_async_session()
try:
s.send(None)
except StopIteration:
pass
finally:
s.close()
async def init_async_session(self):
#ClientSession should be created inside a coroutine.
self.async_session = aiohttp.ClientSession()
What would be the right way to initialize self.async_session
If some method uses something asynchronous it should be explicitly defined as asynchronous either. This is a core idea behind asyncio: make you write code a way you always know if some arbitrary method may do something asynchronous.
In your snippet you want to do async thing (bar I/O) inside sync method __init__ and asyncio prohibits it. You should make _run_coro async and initialize Foo asynchronously, for example, using __await__ method:
import asyncio
class Foo:
def __await__(self):
return self._run_coro().__await__()
async def _run_coro(self): # real async initializer
async def init():
await asyncio.sleep(1) # bar I/O
self.bar = 123
await init()
return self
async def spam(self):
return await asyncio.sleep(1) # I/O op
async def main():
foo = await Foo()
await foo.spam()
asyncio.run(main()) # instead of two lines in Python 3.7+
You may be interested in reading this answer to understand better how asyncio works and how to handle it.
Upd:
s = self.init_async_session()
try:
s.send(None)
Don't do such things: generator's method are only details of implementation in regard of coroutines. You can predict how coroutine will react on calling .send() method and you can rely on this behavior.
If you want to execute coroutine use await, if you want to start it "in background" use task or other functions from asyncio doc.
What would be the right way to initialize self.async_session
When it comes to aiohttp.ClientSession it should not only be created, but properly closed also. Best way to do it is to use async context manager as shown in aiohttp doc.
If you want to hide this operation inside Foo you can make it async manager either. Complete example:
import aiohttp
class Foo:
async def __aenter__(self):
self._session = aiohttp.ClientSession()
await self._session.__aenter__()
return self
async def __aexit__(self, *args):
await self._session.__aexit__(*args)
async def spam(self):
url = 'http://httpbin.org/delay/1'
resp = await self._session.get(url)
text = await resp.text()
print(text)
async def main():
async with Foo() as foo:
await foo.spam()
asyncio.run(main())
Upd2:
You can combine ways to init/close object from above to achive result you like. As long as you keep in mind both operations are asynchronous and thus should be awaited, everything should be fine.
One more possible way:
import asyncio
import aiohttp
class Foo:
def __await__(self):
return self._init().__await__()
async def _init(self):
self._session = aiohttp.ClientSession()
await self._session.__aenter__()
return self
async def close(self):
await self._session.__aexit__(None, None, None)
async def spam(self):
url = 'http://httpbin.org/delay/1'
resp = await self._session.get(url)
text = await resp.text()
print(text)
async def main():
foo = await Foo()
try:
await foo.spam()
finally:
await foo.close()
asyncio.run(main())
Here's my solution.
class Session:
def __init__(self, headers):
self._headers = headers
self._session = requests.Session()
self._async_session = None
async def _init(self):
self._session = aiohttp.ClientSession(headers=headers)
async def async_request(self, url):
while True:
try:
async with self._async_session.get(url) as resp:
resp.raise_for_status()
return await resp.text()
except aiohttp.client_exceptions.ClientError:
#retry or raise
except AttributeError:
if isinstance(self._async_session, aiohttp.ClientSession):
raise
await self._init()
def request(self, url):
return self._session.get(url).text
async def close(self):
if isinstance(self._async_session, aiohttp.ClientSession):
await self._session.close()
async def main():
session = Session({})
print(await session.async_request('https://httpstat.us/200')
await session.close()
asyncio.run(main())
I can initialize the Session class and make synchronous as well as asynchronous requests. I do not have to explicitly call await session._init() to initialize self._async_session as when session._async_request is called and self._async_session is None, then await session._init() will be called and the request will be retried.