Circular import despite absolute imports - python-3.x

I have got a problem with circular imports in Python 3.9.
I know there are numerous articles, questions and answers about this topic and I really have read a lot of them and about how to circumvent but I cannot figure out why it happens in my case. As far as I have understood one can circumvent circular imports by doing absolute imports:
This is my Stacktrace:
partially initialized module 'myproject.myapp.service.linepattern.corner' has no attribute 'Corner' (most likely due to a circular import)
File "/mnt/d/User/Programming/MyProject/myproject/myapp/service/linepattern/line.py", line 8, in Line
def __init__(self, cornerRight: cc.Corner, cornerLeft: cc.Corner) -> None:
File "/mnt/d/User/Programming/MyProject/myproject/myapp/service/linepattern/line.py", line 7, in <module>
class Line(ABC):
File "/mnt/d/User/Programming/MyProject/myproject/myapp/service/linepattern/corner.py", line 2, in <module>
import myproject.myapp.service.linepattern.line as cl
File "/mnt/d/User/Programming/MyProject/myproject/myapp/service/linepattern/cornermanager.py", line 2, in <module>
import myproject.myapp.service.linepattern.corner as cc
File "/mnt/d/User/Programming/MyProject/myproject/tests/myapp/test_cornermanager.py", line 3, in <module>
import myproject.myapp.service.linepattern.cornermanager as cm
and this is my file hierarchy
/mnt/d/User/Programming/MyProject
├── manage.py
├── myproject
│ ├── myapp
│ │ ├── __init__.py
│ │ ├── service
│ │ │ └── linepattern
│ │ │ ├── cornermanager.py
│ │ │ ├── corner.py
│ │ │ ├── line.py
│ │ │ ├── __init__.py
│ ├── settings.py
│ ├── tests
│ │ ├── myapp
│ │ │ ├── __init__.py
│ │ │ └── test_cornermanager.py
and finally my code
cornermanager.py
import myproject.myapp.service.linepattern.corner as cc
class CornerManager:
def __init__(self) -> None:
self.cornerList: List[cc.Corner]
for i in range(1, 2):
self.cornerList.append(cc.Corner(i, self))
for corner in self.cornerList:
corner.initializeLines()
def corner(self, cornerNr: int) -> cc.Corner:
return self.cornerList[cornerNr - 1]
corner.py
import myproject.myapp.service.linepattern.cornermanager as cm
import myproject.myapp.service.linepattern.line as cl
class Corner:
def __init__(self, cornerNumber: int, cornerManager: cm.CornerManager) -> None:
self.cornerNumber = cornerNumber
self.cornerManager = cornerManager
self.lineSet: Set[cl.Line]
def initializeLines(self) -> None:
#Add lines to the corner including its neighbor
if self.cornerNumber == 1:
self.lineSet.add(cl.Line(self, self.cornerManager.corner(2)))
elif self.cornerNumber == 2:
self.lineSet.add(cl.Line(self, self.cornerManager.corner(2)))
def addLine(self, line: cl.Line) -> None:
#Remove line from both corners
for corner in cl.cornerPair():
corner.lineSet.add(line)
def removeLine(self, line: cl.Line) -> None:
#Remove line from both corners
for corner in cl.cornerPair():
corner.lineSet.remove(line)
line.py
import myproject.myapp.service.linepattern.corner as cc
class Line:
def __init__(self, cornerRight: cc.Corner, cornerLeft: cc.Corner) -> None:
self.rightCorner = cornerRight
self.leftCorner = cornerLeft
def cornerPair(self) -> Tuple[cc.Corner, cc.Corner]:
#Return both corners
return (self.rightCorner, self.leftCorner)

With the code you put in the question (three separated modules), by including the following code at the top of each file should be able to fix the problems:
from __future__ import annotations
In short this code will treat annotations as strings. You can read more about this at: Type hint of the enclosing class and Evaluation of annotations.
For the errors, this actually how Python executes your modules (start from cornermanager.py):
import corner. With this Python first saves the module in its internal module (sys.modules) before running the imported module code.
In corner.py itself, the first two lines are import statement, so let's start with import cornermanager. Again, Python saves the module in its internal module then executes the code in cornermanager.
Back to cornermanager.py. As we know first line of this module is import corner, but because Python already saved this module (corner) in its internal module, it doesn't import it the second time. That is, Python skips it and executes code below this line. Interestingly, because corner hasn't executed all its code but the first line (import cornermanager), attempt to fetch its Corner class will not succeed.

Related

import inside a python package - ModuleNotFoundError

I'm trying to build a package where specific modules call other modules. Example of the structure of the package:
provapackage/
├── main.py
└── pippo
├── derivative_functions.py
├── functions_pippo.py
└── __init__.py
content of the functions_pippo module:
def add(x,y):
return x+y
content of the derivative_functions module:
from functions_pippo import add
def add_lst(l):
n=0
for i in l:
n = add(n,i)
return n
content of the main.py file:
from pippo.derivative_functions import add_lst
lst = [1,2,3,4]
print(add_lst(lst))
when I run main.py I get this error:
Traceback (most recent call last):
File "/home/g/Documents/provapackage/main.py", line 1, in <module>
from pippo.derivative_functions import add_list
File "/home/g/Documents/provapackage/pippo/derivative_functions.py", line 1, in <module>
from functions_pippo import add
ModuleNotFoundError: No module named 'functions_pippo'
It seems like python cannot find the functions_pippo module even if it's in the same folder. But when I run derivative_functions.py I don't gat any error message. Is it an import problem?
See this answer here
from .functions_pippo import add
not
from functions_pippo import add

Python call module on other directory but not find module

I learning python module , I want to put some file to one mudole dir, and then call it at the project root. but it not work. I try to solve it , but it still not success!
what is the problem ?
├── calculator
│   ├── __init__.py
│   ├── addTwo.py
│   ├── mulTwo.py
│   └── sum.py
└── main.py
# calculator/__init__.py
#nothing
# calculator/addTwo.py
def addFunc(lhs, rhs):
return lhs + rhs
# calculator/mulTwo.py
def mulFunc(lhs, rhs):
return lhs * rhs
# calculator/sum.py
import addTwo
def sumFunc(arg1, arg2, arg3):
ans = addTwo.addFunc(arg1,arg2)
ans = addTwo.addFunc(ans,arg3)
return ans
#main.py
from calculator import addTwo
from calculator import mulTwo
from calculator import sum
def test():
print(addTwo.addFunc(1,2))
print(mulTwo.mulFunc(1,2))
print(sum.sumFunc(1,2,3))
if __name__ == '__main__':
test()
run it
cong.lin#C02YN3P4LVCK  ~/Program/python/module_test  python3 main.py
Traceback (most recent call last):
File "main.py", line 4, in <module>
from calculator import sum
File "/Users/cong.lin/Program/python/module_test/calculator/sum.py", line 1, in <module>
import addTwo
ModuleNotFoundError: No module named 'addTwo'
✘ cong.lin#C02YN3P4LVCK  ~/Program/python/module_test 
You have to provide PYTHONPATH to root of your project
Try
export PYTHONPATH=$PWD
before running
python3 main.py

asyncpg - cannot perform operation: another operation is in progress

I am attempting to resolve the following error:
asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress
Here is the full traceback:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
│ │ └ 4
│ └ 7
└ <function _main at 0x109c8aca0>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
│ │ └ 4
│ └ <function BaseProcess._bootstrap at 0x109b1f8b0>
└ <SpawnProcess name='SpawnProcess-4' parent=36604 started>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
│ └ <function BaseProcess.run at 0x109b18ee0>
└ <SpawnProcess name='SpawnProcess-4' parent=36604 started>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
│ │ │ │ │ └ {'config': <uvicorn.config.Config object at 0x109cd55b0>, 'target': <bound method Server.run of <uvicorn.server.Server object...
│ │ │ │ └ <SpawnProcess name='SpawnProcess-4' parent=36604 started>
│ │ │ └ ()
│ │ └ <SpawnProcess name='SpawnProcess-4' parent=36604 started>
│ └ <function subprocess_started at 0x10a4aca60>
└ <SpawnProcess name='SpawnProcess-4' parent=36604 started>
File "/Users/ddd/Desktop/repos/xxx/.venv/lib/python3.8/site-packages/uvicorn/subprocess.py", line 61, in subprocess_started
target(sockets=sockets)
│ └ [<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 80)>]
└ <bound method Server.run of <uvicorn.server.Server object at 0x109cd56a0>>
File "/Users/ddd/Desktop/repos/xxx/.venv/lib/python3.8/site-packages/uvicorn/server.py", line 48, in run
loop.run_until_complete(self.serve(sockets=sockets))
│ │ │ │ └ [<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 80)>]
│ │ │ └ <function Server.serve at 0x10a4abca0>
│ │ └ <uvicorn.server.Server object at 0x109cd56a0>
│ └ <function BaseEventLoop.run_until_complete at 0x10a205820>
└ <_UnixSelectorEventLoop running=True closed=False debug=False>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
│ └ <function BaseEventLoop.run_forever at 0x10a205790>
└ <_UnixSelectorEventLoop running=True closed=False debug=False>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
│ └ <function BaseEventLoop._run_once at 0x10a209310>
└ <_UnixSelectorEventLoop running=True closed=False debug=False>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
handle._run()
│ └ <function Handle._run at 0x10a13ed30>
└ <Handle <TaskWakeupMethWrapper object at 0x10bb75a60>(<_GatheringFu...in progress')>)>
File "/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
│ │ │ │ │ └ <member '_args' of 'Handle' objects>
│ │ │ │ └ <Handle <TaskWakeupMethWrapper object at 0x10bb75a60>(<_GatheringFu...in progress')>)>
│ │ │ └ <member '_callback' of 'Handle' objects>
│ │ └ <Handle <TaskWakeupMethWrapper object at 0x10bb75a60>(<_GatheringFu...in progress')>)>
│ └ <member '_context' of 'Handle' objects>
└ <Handle <TaskWakeupMethWrapper object at 0x10bb75a60>(<_GatheringFu...in progress')>)>
> File "./xxx/xxx/xxx.py", line 144, in get_disclosure_data
hh_json, db_json = await asyncio.gather(*coroutines)
│ │ └ [<coroutine object xxxx at 0x10bb2cb40>, <coroutine object db_call at 0x10bb2cc40>]
│ └ <function gather at 0x10a1fad30>
└ <module 'asyncio' from '/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/__init__.py'>
File "./xxx/xxx/xxx.py", line 52, in db_call
db_json = await asyncio.gather(*coroutines, loop=asyncio.get_event_loop())
│ │ │ │ └ <built-in function get_event_loop>
│ │ │ └ <module 'asyncio' from '/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/__init__.py'>
│ │ └ [<coroutine object DBConnectionManager.fetch_item at 0x10bb434c0>, <coroutine object DBConnectionManager.fetch_item at 0x10bb...
│ └ <function gather at 0x10a1fad30>
└ <module 'asyncio' from '/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/__init__.py'>
File "./xxx/xxx/xx.py", line 97, in fetch_item
await self._connection_pool.release(self.con)
│ │ │ │ └ <PoolConnectionProxy [released] 0x10bbc9cd0>
│ │ │ └ <chd_api.data.db.DBConnectionManager object at 0x10b946a30>
│ │ └ <function Pool.release at 0x10b956a60>
│ └ <asyncpg.pool.Pool object at 0x10bb131e0>
└ <chd_api.data.db.DBConnectionManager object at 0x10b946a30>
File "/Users/ddd/Desktop/repos/xxx/.venv/lib/python3.8/site-packages/asyncpg/pool.py", line 666, in release
return await asyncio.shield(ch.release(timeout))
│ │ │ │ └ None
│ │ │ └ <function PoolConnectionHolder.release at 0x10b952e50>
│ │ └ <asyncpg.pool.PoolConnectionHolder object at 0x10bb2a5c0>
│ └ <function shield at 0x10a1faee0>
└ <module 'asyncio' from '/Users/ddd/.pyenv/versions/3.8.6/lib/python3.8/asyncio/__init__.py'>
File "/Users/ddd/Desktop/repos/xxx/.venv/lib/python3.8/site-packages/asyncpg/pool.py", line 218, in release
raise ex
File "/Users/ddd/Desktop/repos/xxx/.venv/lib/python3.8/site-packages/asyncpg/pool.py", line 208, in release
await self._con.reset(timeout=budget)
│ │ └ None
│ └ <member '_con' of 'PoolConnectionHolder' objects>
└ <asyncpg.pool.PoolConnectionHolder object at 0x10bb2a5c0>
File "/Users/ddd/Desktop/repos/xxx/.venv/lib/python3.8/site-packages/asyncpg/connection.py", line 1311, in reset
await self.execute(reset_query, timeout=timeout)
│ │ │ └ None
│ │ └ 'SELECT pg_advisory_unlock_all();\nCLOSE ALL;\nUNLISTEN *;\nRESET ALL;'
│ └ <function Connection.execute at 0x10b93f3a0>
└ <asyncpg.connection.Connection object at 0x10bc34120>
File "/Users/ddd/Desktop/repos/xxx/.venv/lib/python3.8/site-packages/asyncpg/connection.py", line 297, in execute
return await self._protocol.query(query, timeout)
│ │ │ └ None
│ │ └ 'SELECT pg_advisory_unlock_all();\nCLOSE ALL;\nUNLISTEN *;\nRESET ALL;'
│ └ <member '_protocol' of 'Connection' objects>
└ <asyncpg.connection.Connection object at 0x10bc34120>
File "asyncpg/protocol/protocol.pyx", line 321, in query
self._check_state()
File "asyncpg/protocol/protocol.pyx", line 684, in asyncpg.protocol.protocol.BaseProtocol._check_state
raise apg_exc.InterfaceError(
│ └ <class 'asyncpg.exceptions._base.InterfaceError'>
└ <module 'asyncpg.exceptions' from '/Users/ddd/Desktop/repos/chd-api/.venv/lib/python3.8/site-packages/asyncpg/exception...
asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress
where I have the following code to set up a connection pool and execute queries with connections in the pool:
class DBConnectionManager(object):
""" Class for setting up and tearing down db connection """
def __init__(self):
self.host = SETTINGS.db_host
self.database = SETTINGS.db_name
self.user = SETTINGS.db_user
self.password = SETTINGS.db_password
self.port = "5432"
self._connection_pool = None
self.con = None
async def connect(self):
if not self._connection_pool:
try:
self._connection_pool = await asyncpg.create_pool(
host=self.host,
database=self.database,
user=self.user,
password=self.password,
port=self.port,
min_size=50,
max_size=100,
)
logger.info("Database pool connection opened")
except Exception as e:
logger.exception(e)
async def fetch_item(self, query: str, *args):
if not self._connection_pool:
await self.connect()
else:
self.con = await self._connection_pool.acquire()
try:
result = await self.con.fetch(query, *args)
return result
except Exception as e:
logger.exception(e)
finally:
await self._connection_pool.release(self.con)
async def close(self):
if not self._connection_pool:
try:
await self._connection_pool.close()
logger.info("Database pool connection closed")
except Exception as e:
logger.exception(e)
and am attempting to execute some 22 database calls using the following:
async def db_call(db, lat, lng):
"""
Performs the necessary db calls given a lat, lng
Required Input:
lat::float a latitude in decimal degrees. Must be specified with `lng` (i.e. 39.2994)
lng::float a longitude in decimal degrees. Must be specified with `lat` (i.e. -122.33)
Returns:
dict
"""
coroutines = []
for table in db_map:
# SQL columns
db_fields = ",".join(
[
f"{col} AS {db_map[table]['fields'][col]}"
for col in db_map[table]["fields"]
]
)
# Output names
api_fields = [db_map[table]["fields"][col] for col in db_map[table]["fields"]]
if db_map[table]["query_type"] == "pip":
limit = db_map[table]["options"]["LIMIT"]
query = f"SELECT {db_fields} from {table} WHERE (ST_Covers(geom, GeomFromEWKT('SRID=4326;POINT({lng} {lat})'))) LIMIT {limit};"
else:
distance = db_map[table]["options"]["DISTANCE"]
geo2geo = f"geom::geography, GeomFromEWKT('SRID=4326;POINT({lng} {lat})')::geography"
query = (
f"SELECT {db_fields}, ST_Distance({geo2geo})"
f"from {table} WHERE (ST_DWithin({geo2geo}, {distance}))"
f"ORDER BY ST_Distance({geo2geo}) LIMIT 1;"
)
coroutines.append(db.fetch_item(query))
db_res = await asyncio.gather(*coroutines)
.... code for processing results
I have examined several issues on the asyncpg github concerning this error and am still not finding an appropriate solution. Note also, this call is being performed in FastAPI. Any guidance would be appreciated on why this error may be occurring/steps to resolve it.
The assignment to self.con in fetch_item causes multiple coroutines to share the same connection. While you do want them to share the connection pool, sharing the same connection doesn't make sense, as the connection is stateful.
To resolve the issue, replace usage of self.con with a local variable con.
Slightly off topic, but Google often leads to this answer. If you are using SQLAlchemy: Fixed by setting poolclass=NullPool in create_async_engine

'function' object is not subscriptable in local package

While trying to learn how Python modules and packages work, I have the following error that I can't find a way around:
$ ./myMain.py
Traceback (most recent call last):
File "./myMain.py", line 6, in <module>
print(foobar.getKey['A'])
TypeError: 'function' object is not subscriptable
My directory structure is the following:
.
├── myMain.py*
└── utils/
└── Foo/
├── __init__.py
├── __pycache__/
│   ├── __init__.cpython-36.pyc
│   └── foobar.cpython-36.pyc
└── foobar.py
and myMain.py would be the main script from where the package would be imported. myMain.py has the following code:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from utils.Foo import foobar
print(foobar.getKey['A'])
__init__.py is an empty file, and foobar.py has the function definition:
TEST_DICT = {
'A': 'This is A',
'B': 'This is B'
}
def getKey(letter):
return TEST_DICT[letter]
What am I doing wrong here?
After trying to run the code in myMain.py from within the Python interpret, I realised that the function call was writen with wrong syntax. Instead of foobar.getKey['A'] one should have instead foobar.getKey('A') because we are calling the function and not the dictionary object.

ModuleNotFoundError after packaging with pip

Why would I receive a ModuleNotFoundError: No module named 'helpers' when I execute the command prompt script after running pip3 install . from my package's root directory? The location of helpers is c:\users\username\appdata\local\programs\python\python36-32\lib\site-packages\mypackage\helpers.py so how to I get the program to look there?
PS C:\> mypackage
Traceback (most recent call last):
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\Scripts\mypackage-script.py", line 11, in <module>
load_entry_point('mypackage==0.1.2', 'console_scripts', 'mypackage')()
File "c:\users\username\appdata\local\programs\python\python36-32\lib\site-packages\pkg_resources\__init__.py", line 572, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "c:\users\username\appdata\local\programs\python\python36-32\lib\site-packages\pkg_resources\__init__.py", line 2755, in load_entry_point
return ep.load()
File "c:\users\username\appdata\local\programs\python\python36-32\lib\site-packages\pkg_resources\__init__.py", line 2408, in load
return self.resolve()
File "c:\users\username\appdata\local\programs\python\python36-32\lib\site-packages\pkg_resources\__init__.py", line 2414, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "c:\users\username\appdata\local\programs\python\python36-32\lib\site-packages\mypackage\__init__.py", line 16, in <module>
import helpers as h
ModuleNotFoundError: No module named 'helpers'
Here is the source tree:
PS C:\Users\username\source\repos\mypackage> tree /F
Folder PATH listing for volume mydisk
Volume serial number is 01234-A321
C:.
│ .gitignore
│ COPYING
│ MANIFEST.in
│ README.rst
│ setup.py
│
├───mypackage
│ │ helpers.py
│ │ __init__.py
│ │
│ └───__pycache__
│ helpers.cpython-36.pyc
│ __init__.cpython-36.pyc
│
└───tests
itsatrap.py
__init__.py
Here is the setup.py:
PS C:\Users\username\source\repos\mypackage> cat setup.py
from setuptools import setup, find_packages
from codecs import open
from os import path
path_to_here = path.abspath(path.dirname(__file__))
with open(path.join(path_to_here, 'README.rst'), encoding='utf-8') as readme_file:
readme = readme_file.read()
with open(path.join(path_to_here, 'COPYING'), encoding='utf-8') as license_file:
license = license_file.read()
setup(
name = 'mypackage',
version = '0.1.2',
license=license,
description = 'Who knows',
long_description = readme,
url = 'https://github.com/user/mypackage',
keywords = ['help!'],
packages = find_packages(exclude=['contrib', 'docs', 'tests']),
install_requires = ['matplotlib','numpy'],
extras_require = {
'dev': [''],
'test': [''],
},
entry_points = {
'console_scripts': [
'mypackage=mypackage:main'
],
},
)
Any help is appreciated. Packaging in Python is new territory for me! This program did run correctly when doing python mypackage/__init__.py from the package's root directory.
In Python 3 all imports are absolute. Use full path to import a module from a package:
from mypackage import helpers as h
To perform relative import do it explicitly:
from . import helpers as h

Resources