I was wondering if it was possible for modules in a project to be smart about their imports...
Say I have the following data structure :
/parent-directory
/package
__init__.py
main.py
/modules
__init__.py
one.py
/submodules-one
__init__.py
oneone.py
onetwo.py
two.py
Files higher in the hierarchy are supposed to import Classes from those lower in the hierarchy.
For instance main.py has
import modules.one
import modules.two
Now I'd like to be able to directly run not only main.py, but also one.py (and all the others)
Except that it doesn't work like I hoped :
If I run from main.py, I need to have in one.py
import modules.submodules-one.oneone
import modules.submodules-one.onetwo
But if I run from one.py, I'll get an error, and I need to have instead
import submodules-one.oneone
import submodules-one.onetwo
I've found a hacky way to get around it using in one.py :
if __name__ == '__main__':
import submodules-one.oneone
import submodules-one.onetwo
else:
import modules.submodules-one.oneone
import modules.submodules-one.onetwo
But isn't there a better solution?
P.S.: I also have an additional complication, I'm using pint,
which to work properly only needs to have a single instance of the unit registry, so I have in the top ____init____.py :
from pint import UnitRegistry
ur = UnitRegistry()
And obviously
from .. import ur
will fail if running from one of the files of the subfolders.
Thank you in advance for your answer.
Related
This are the files:
parent_directory:
parent_directory/ecommerce:
parent_directory/ecommerce/payments:
What I'm trying to do is to use a relative import in products.py in order to import database.py. I know that I can just write import database but I want to learn relative imports. I've also tried to import main.py from the top level directory, parent_directory, but that doesn't work either so I thought I'll just try to do the easiest relative import possible and that is, importing a file from the same package.
This is the code for products.py inside parent_directory/ecommerce:
from .database import Database
class Product:
pass
And this is the code from the database.py file inside parent_directory/ecommerce:
if __name__ == "__main__":
import products
x = products.Product()
class Database:
pass
print("file : {0:<35} || name : {1:<20} || package : {2:<20}".format(
str(__file__).split("\\")[-1],
str(__name__),
str(__package__),
))
I tried several things like:
executing products.py with -m like python -m products.py
creating a py venv
adding all and package inside the files.
Here is my init.py file from parent_directory/ecommerce:
__all__ = ["database", "products"]
__package__ = "ecommerce"
No matter what I do, I always get this error: ImportError: attempted relative import with no known parent package. I know that this question has been asked before. I've tried many things out but nothing worked. Do you have any idea how to fix this ?
I found the solution to my problem.
I have to be outside the package and use -m and instead of using / to send the path, I have to use . instead so :
python -m parent_directory.ecommerce.products
This will fix the problem but I have to execute this line outside the top-level package ( in my case, the top level package is parent_directory ).
# Inside products.py
from .database import Database # works
from ..main import SomeRandomClass # works
class Product:
pass
I've found a way to import files from parent directories, since it's very easy to import them from sub-packages by using absolute imports.
# inside parent_directory/ecommerce/products.py trying to import parent_directory/main.py
import sys
import os
current_file = os.path.realpath(__file__)
current_dir_ecommerce = os.path.dirname(current_file)
parent_dir_parent_directory = os.path.dirname(current_dir_ecommerce)
sys.path.insert(0, parent_dir_parent_directory)
import main # works
I just added the parent directory to sys.path.
I know theres heaps of questions and answers for this, I tried multitude stackoverflow links but none of these seem to help.
My project structure is:
volume_price_analysis/
README.md
TODO.md
build/
docs/
requirements.txt
setup.py
vpa/
__init__.py
database_worker.py
utils.py
test/
__init__.py
test_utils.py
input/
input_file.txt
I want to load utils.py inside test_utils.py
my test_utils.py is:
import unittest
import logging
import os
from .vpa import utils
class TestUtils(unittest.TestCase):
def test_read_file(self):
input_dir = os.path.join(os.path.join(os.getcwd()+"/test/input"))
file_name = "input_file.txt"
with open(os.path.join(input_dir+"/"+file_name)) as f:
file_contents = f.read()
f.close()
self.assertEqual(file_contents, "Hello World!\n")
if __name__ == '__main__':
unittest.main()
I want to run (say inside test folder):
python3 -m test_utils.py
I can not do that, I get a bunch of errors regarding imports of utils (tried many iterations of . , no ., from this import that etc.. etc..
Why is this so bloody complicated?
I am using Python 3.7 if that helps.
As per this answer, you can do it using importlib,
in spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py") ,instead of path/to/file, you can use ../utils.py. Also, since you are already importing a package named utils (from importlib), you should call one of them by other name, ie. dont keep module.name as utils or import importlib.utils as something else.
I figured it out, turns out python prefers you to run your code from top level folder, in my case volume_price_analysis folder, all I had to do was make a shell script that calls
python3 -m unittest vpa.test.test_utils
And inside test_utils I can import whatever I want as long as I remember that I am executing the code from main folder so loading utils.py would be
from vpa import utils inside test_utils
Let's say there are two files - mask.py and main.py.
mask.py has some function that I'm importing into main.py.
So if that function in mask.py which I'm importing has dependency like "os", where should I import os - in mask.py or main.py.
Let us consider a scenario as you have stated by using two files mask.py and main.py.
mask.py
import os
def some_function():
os.environ['a_url'] = "something.com" # using dependency as you mentioned
main.py
from mask import some_function
# do something with the function
Now, coming to your query, if you use import os in main.py but not in mask.py, you will get NameError in mask.py saying:
NameError: name 'os' is not defined
This is because you need to import any dependency in the same file where it is used. Also, if both of your file uses this dependency, you need to import it in both the files.
Hope this clarifies your query.
[Python 3.5.2, Docker container]
I have a class BCM in a file called metrics.py which I would like to use in several Jupyter notebooks which are in different directories. When they are in the same file, the following obviously works from metrics import BCM. I would like it to work with the following directory structure.
experiments\
common\
__init__.py
metrics.py
exp1\
glm.ipynb
exp2\
gbm.ipynb
It works if I do the following
import os, sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path)
from common.metrics import BCM
Based on posts that I have seen here, it would seem that the following should work from ..common.metrics import BCM. I get the following error SystemError: Parent module '' not loaded, cannot perform relative import.
Is there a way to use that class without changing the path as shown above?
I am new to python and i m having a really bad time to overcome a problem with the importing system.
Lets say i have the file system presented below:
/src
/src/main.py
/src/submodules/
/src/submodules/submodule.py
/src/submodules/subsubmodules
/src/submodules/subsubmodules/subsubmodule.py
All the folders (src, submodules, subsubmodules) have and empty __init__.py file.
In submodule.py i have:
from subsubmodules import subsubmodule
In main.py i have:
from submodules import submodule
When i run submodule.py python accepts the import. But when i run main.py python raises error for the import of subsubmodule.py because /src/submodules/subsubmodules/ folder is not in the path.
Only solution is to change the import of submodule.py to
from submodules.subsubmodules import subsubmodule
This seems to me as an awful solution because after that i cannot run submodule.py and i m sure that something else is the key to that.
An other solution is to add the following code to the __init__.py file:
import os
import sys
import inspect
cmd_subfolder = os.path.split(inspect.getfile(inspect.currentframe()))[0]
if cmd_subfolder not in sys.path:
sys.path.insert(0, cmd_subfolder)
Is there any way to do this using just the importing system of python and not other methods that do it manually using, for example sys.path or other modules like os, inspect etc..?
How can i import modules without caring about the modules they import?
You can run subsubmodule.py as
python3 -m submodule.subsubmodules.subsubmodule
If you want a shorter way to invoke it, you're free to add a shell or Python script for that on the top level of your package.
This is how imports work in Python 3; there are reasons for that.
You can avoid this issue by using sys.path in your program.
sys.path.insert(0, './lib')
import subsubmodule
For this code, you can put all your imports to a lib folder.
You can read the official documentation on Python packages where this is explained in depth.