I have a directory structure like below:
chatbot/
__init__.py
utils/
__init__.py
parser.py
nlu/
__init__.py
training/
__init__.py
module.py
I want to access parser.py from module.py.
I tried using this line from module.py:
from chatbot.utils import parser
And I got this error:
ModuleNotFoundError: No module named 'chatbot'
Any pointers to what I am doing wrong?
I am using python3 and trying to run the script as python3 nlu/training/module.py.
Thanks in advance!
I believe the right way of solving such problems is:
figure out what your top-level packages and modules are and in what directory they are
change to that directory
make all your imports absolute, i.e. always start with a top-level module or package (instead of things like from . import blah)
use the executable module (python -m) way of running your code, for example path/to/pythonX.Y -m top_level_package.executable_module (instead of path/to/pythonX.Y top_level_package/executable_module.py)
In case the top-level modules or packages are not all in the same directory:
Either work on packaging your libraries or applications correctly and install them in the site packages (eventually as editable, also known as develop mode).
Or, as a last resort, collect the other directories containing the top-level packages and modules in the PYTHONPATH environment variable like the following:
PYTHONPATH=path/to/alpha:path/to/bravo path/to/pythonX.Y -m top_level_package.executable_module
I was able to solve the issue by adding the parent directory of chatbot package to sys.path.
Say if chatbot python package is in /home/my_project/chatbot/, I added a statement like:
sys.path.append('/home/my_project') in module.py
This made all top-level python packages visible to all lower-level python packages.
Related
So I builded a python package localy:
cgi#cgires:~$ pip list | grep mads
madscgi 0.1.0
Its nice! Afterwards I can use it in Jupyter Notebook, in iPython Shell, in Python Shell and even in python scripts outside the modules code. So it works as expected 100% outside the modules code:
Thats nice, but next I want to import code from one builded module (inside the package) into another python file (inside the package). Lets name it import_test.py and try it out:
So it fails if it is getting executed in the directory, where the package is build from. And it looks like, that the python interpreter is taking the parent directory (with the same name like the module) and this is failing.
Is is possible to enforce the usage of the installed pip-package?
As #MisterMiyagi pointed out, the problem was, that there were an upper folder which had the same name as the module.
Here: mads_cons is the upper folder from import_test.py. Therefore, the upper folder is getting imported instead of the via pip installed module. Thats it.
The file you want to import should either be in the same folder or referred to with the absolute path of it.
If that doesn't suit you, you can call sys.path
import sys
sys.path
You can keep your file in any of the directories sys.path returns.
Smart would be, if you keep the file inside.
......../site-packages/
The problem
I have a directory structure for my project which follows the standard for Python packages, as it was created with this cookiecutter template:
https://github.com/audreyr/cookiecutter-pypackage#quickstart
The directory structure is
project_name
├── project_name
│ ├── __init__.py
│ └── module1.py
└── tests
└── test_module1.py
The first code line of test_module1.py is:
from project_name import module1
But I get a ModuleNotFoundError: No module named 'project_name'.
To my understanding, this should work since the folder called project_name is a package, which is ensured by presence of the __init__.py file.
I have always had trouble understanding how imports like this work. For my projects I have always just settled with having my tests in the same folder as the modules to test. I know this is bad practice, but the only way I could get the modules to actually import.
What I already tried
I have tried renaming the folder with the __init__.py file to something else and then import, as I thought it could have something to do with the parent folder and the child folder both having the name project_name. This did not work, same error.
I also tried making the test folder into a package by creating an __init.py__ file inside it, even though the Cookiecutter template does not have that.
I read in many places that making the test folder into a package is discouraged, but some suggest that structure. That did not work either.
I have searched thoroughly for solutions to this seemingly very standard problem, some of the links are here:
Python, importing modules for testing
https://gist.github.com/tasdikrahman/2bdb3fb31136a3768fac
Importing modules from parent folder
https://alex.dzyoba.com/blog/python-import/
Sibling package imports
Python imports for tests using nose - what is best practice for imports of modules above current package
My last try was to start a project with Cookiecutter, so everything would be set up properly form the beginning. However, I still get the ModuleNotFoundError.
What I don't want
I don't want to modify sys.path as many answers seem to suggest. There must be a cleaner way for such a common problem.
What am I doing wrong?
Edit for some additional info (see question from #Nicholas):
The contents of __init__.py is
# -*- coding: utf-8 -*-
"""Top-level package for project_name."""
__author__ = """my_name"""
__email__ = 'my_email'
__version__ = '0.1.0'
Which was generated by the Cookiecutter template.
Inside test_module1, I added the following before the before the ModuleNotFoundError occurs:
import sys
import os
print(sys.path)
print(os.getcwd())
sys.path prints a list, where first element is the tests directory.
['c:\\Users\\...\\project_name\\tests',
'C:\\Users\\...\\Miniconda3\\python37.zip',
'C:\\Users\\...\\Miniconda3\\DLLs', 'C:\\Users\\...\\Miniconda3\\lib',
'C:\\Users\\...\\Miniconda3',
'C:\\Users\\...\\Miniconda3\\lib\\site- packages',
'C:\\Users\\...\\Miniconda3\\lib\\site-packages\\win32',
'C:\\Users\\...\\Miniconda3\\lib\\site-packages\\win32\\lib',
'C:\\Users\\...\\Miniconda3\\lib\\site-packages\\Pythonwin']
I don't know if the lowercase 'c' in the first element matters.
os.getcwd() prints the root directory 'c:\Users\....\project_name'. Also with a lowercase 'c'.
You should create a virtual environment and install the project in order for the test modules to correctly resolve import statements.
In the project root, i.e. the directory project_name which contains a subdirectory project_name and a subdirectory tests, create a setup.py (or pyproject.toml) file for the package metadata. See here for details about that part.
From this same project root directory which is now containing the installer (setup.py), create and activate a venv and install your project:
python3 -m venv .venv
source .venv/bin/activatate # linux/macOS
# .\Scripts\activate.bat # windows
pip install --editable .
pip install pytest
pytest
If for some reason you don't want to create an installer for your project, you may run pytest like this, from the project directory:
python3 -m pytest
Unlike the bare pytest command, this will add the current working directory to sys.path allowing import statements to be resolved in tests.
I'm having some trouble with latest python version (2 & 3) to find a good common way for imports.
About the packages I pushed on PyPI, that's ok yet as I create a subfolder for the package, then use package.module in __init__.py for imports.
I would do the same in module files.
Other than that, I have some packages that I share between projects (stored on my svn server) which I import as svn external in projects.
The hierarchy is usually designed this way:
package:
|- __init__.py
|- module1.py
|- module2.py
|- ...
When I import it into projects, folders looks like this:
project:
|- package:
|- what's above
|- app.py
I usually develop the package itself as a project, meaning package is root.
When I import it into projects, they become packages, thus not at root anymore.
When using package.module or .module in standalone package project, of course, this does not work.
When using same as package in other projects, this tends to work almost fine, but I experience some issues with Python 3.6, telling that some names aren't defined (when launched directly from the command line; but working under pycharm).
Last but not least, when using py2exe to make executables (from Python 2.7), I get some undefined names (classes, etc) exceptions even if imports seems to be ok as no such exception is raised; please note I was able to see that related .pyc files are there in library.zip.
I did not go through these issues some months ago, neither python 2 nor python 3 with older versions.
I open this question as I searched on StackOverflow, python documentation (2 & 3) and a lot through the web these last days without finding something that is really relevant to the overall issue.
My question is, does anyone have any clue on a good practice for imports that would be compatible between python 2 & 3, and which would work also when package is __main__? And the bonus, still ok when packed with py2exe?
At some point, i tried some try/except ImportError blocks for these imports, but it messes things up a bit & seems to be unreliable.
Thanks a lot for your help!
Given some clues about __package__ from:
Relative imports for the billionth time
I found some more consistent solution to check before import.
if __package__ is None or __package__ == '':
from module1 import *
from module2 import *
else:
from .module1 import *
from .module2 import *
In case you run directly from the file itself (let's say module1), __package__ is None. I still had to add check for __package__ == '' as if module1 imports module2, package is not None in module2.
In this case, I prefer using relative imports even if not really recommended, as when used as a package, the package itself can have any name without changing anything.
About __init__.py file, I just use relative imports as it will be used only when importing as a package from the main app with:
import package
from package import xxx
This seems to solve things well about imports for both python 2.7.14 & 3.6, yet still causes issues with py2exe (2.7) and doesn't generate the whole poject.
Okay, the scenario is very simple. I have this file structure:
.
├── interface.py
├── pkg
│ ├── __init__.py
│ ├── mod1.py
│ ├── mod2.py
Now, these are my conditions:
mod2 needs to import mod1.
both interface.py and mod2 needs to be run independently as a main script. If you want, think interface as the actual program and mod2 as an internal tester of the package.
So, in Python 2 I would simply do import mod1 inside mod2.py and both python2 mod2.py and python2 interface.py would work as expected.
However, and this is the part I less understand, using Python 3.5.2, if I do import mod1; then I can do python3 mod2.py, but python3 interface.py throws: ImportError: No module named 'mod1' :(
So, apparently, python 3 proposes to use import pkg.mod1 to avoid collisions against built-in modules. Ok, If I use that I can do python3 interface.py; but then I can't python3 mod2.py because: ImportError: No module named 'pkg'
Similarly, If I use relative import:
from . import mod1 then python3 interface.py works; but mod2.py says SystemError: Parent module '' not loaded, cannot perform relative import :( :(
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works. But do we have to be adding the package prefix pkg to every import to other modules within that package? Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
UPDATE: I have upload those files with the (referred as "solution" above) working source code here: https://gitlab.com/Akronix/test_python3_packages. Note that I still don't like it, and looks much uglier than the python2 solution.
Related SO questions I've already read:
Python -- import the package in a module that is inside the same package
How to do relative imports in Python?
Absolute import module in same package
Related links:
https://docs.python.org/3.5/tutorial/modules.html
https://www.python.org/dev/peps/pep-0328/
https://www.python.org/dev/peps/pep-0366/
TLDR:
Run your code with python -m pkg.mod2.
Import your code with from . import mod1.
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works.
Using the -m switch is indeed the "only" solution - it was already the only solution before. The old behaviour simply only ever worked out of sheer luck; it could be broken without even modifying your code.
Going "one folder up" merely adds your package to the search path. Installing your package or modifying the search path works as well. See below for details.
But do we have to be adding the package prefix pkg to every import to other modules within that package?
You must have a reference to your package - otherwise it is ambiguous which module you want. The package reference can be either absolute or relative.
A relative import is usually what you want. It saves writing pkg explicitly, making it easier to refactor and move modules.
# inside mod1.py
# import mod2 - this is wrong! It can pull in an arbitrary mod2 module
# these are correct, they uniquely identify the module
import pkg.mod2
from pkg import mod2
from . import mod2
from .mod2 import foo # if pkg.mod2.foo exists
Note that you can always use <import> as <name> to bind your import to a different name. For example, import pkg.mod2 as mod2 lets you work with just the module name.
Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
If your package is properly installed, you can use the -m switch from anywhere. For example, you can always use python3 -m json.tool.
echo '{"json":"obj"}' | python -m json.tool
If your package is not installed (yet), you can set PYTHONPATH to its base directory. This includes your package in the search path, and allows the -m switch to find it properly.
If you are in the executable's directory, you can execute export PYTHONPATH="$(pwd)/.." to quickly mount the package for import.
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
This scenario was basically broken in python 2. While it was straightforward in many cases, it was difficult or outright impossible to fix in any other cases.
The new behaviour is more awkward in the straightforward case, but robust and reliable in any case.
I had similar problem.
I solved it adding
import sys
sys.path.insert(0,".package_name")
into the __init__.py file in the package folder.
Currently trying to work in Python3 and use absolute imports to import one module into another but I get the error ModuleNotFoundError: No module named '__main__.moduleB'; '__main__' is not a package. Consider this project structure:
proj
__init__.py3 (empty)
moduleA.py3
moduleB.py3
moduleA.py3
from .moduleB import ModuleB
ModuleB.hello()
moduleB.py3
class ModuleB:
def hello():
print("hello world")
Then running python3 moduleA.py3 gives the error. What needs to be changed here?
.moduleB is a relative import. Relative only works when the parent module is imported or loaded first. That means you need to have proj imported somewhere in your current runtime environment. When you are are using command python3 moduleA.py3, it is getting no chance to import parent module. You can:
from proj.moduleB import moduleB OR
You can create another script, let's say run.py, to invoke from proj import moduleA
Good luck with your journey to the awesome land of Python.
Foreword
I'm developing a project which in fact is a Python package that can be installed through pip, but it also exposes a command line interface. I don't have problems running my project after installing it with pip install ., but hey, who does this every time after changing something in one of the project files? I needed to run the whole thing through simple python mypackage/main.py.
/my-project
- README.md
- setup.py
/mypackage
- __init__.py
- main.py
- common.py
The different faces of the same problem
I tried importing a few functions in main.py from my common.py module. I tried different configurations that gave different errors, and I want to share with you with my observations and leave a quick note for future me as well.
Relative import
The first what I tried was a relative import:
from .common import my_func
I ran my application with simple: python mypackage/main.py. Unfortunately this gave the following error:
ModuleNotFoundError: No module named '__main__.common'; '__main__' is not a package
The cause of this problem is that the main.py was executed directly by python command, thus becoming the main module named __main__. If we connect this information with the relative import we used, we get what we have in the error message: __main__.common. This is explained in the Python documentation:
Note that relative imports are based on the name of the current module. Since the name of the main module is always __main__, modules intended for use as the main module of a Python application must always use absolute imports.
When I installed my package with pip install . and then ran it, it worked perfectly fine. I was also able to import mypackage.main module in a Python console. So it looks like there's a problem only with running it directly.
Absolute import
Let's follow the advise from the documentation and change the import statement to something different:
from common import my_func
If we now try to run this as before: python mypackage/main.py, then it works as expected! But, there's a caveat when you, like me, develop something that need to work as a standalone command line tool after installing it with pip. I installed my package with pip install . and then tried to run it...
ModuleNotFoundError: No module named 'common'
What's worse, when I opened a Python console, and tried to import the main module manually (import mypackage.main), then I got the same error as above. The reason for that is simple: common is no longer a relative import, so Python tries to find it in installed packages. We don't have such package, that's why it fails.
The solution with an absolute import works well only when you create a typical Python app that is executed with a python command.
Import with a package name
There is also a third possibility to import the common module:
from mypackage.common import my_func
This is not very different from the relative import approach, as long as we do it from the context of mypackage. And again, trying to run this with python mypackage/main.py ends similar:
ModuleNotFoundError: No module named 'mypackage'
How irritating that could be, the interpreter is right, you don't have such package installed.
The solution
For simple Python apps
Just use absolute imports (without the dot), and everything will be fine.
For installable Python apps in development
Use relative imports, or imports with a package name on the beginning, because you need them like this when your app is installed. When it comes to running such module in development, Python can be executed with the -m option:
-m mod : run library module as a script (terminates option list)
So instead of python mypackage/main.py, do it like this: python -m mypackage.main.
In addition to md-sabuj-sarker's answer, there is a really good example in the Python modules documentation.
This is what the docs say about intra-package-references:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
If you run python3 moduleA.py3, moduleA is used as the main module, so using the absolute import looks like the right thing to do.
However, beware that this absolute import (from package.module import something) fails if, for some reason, the package contains a module file with the same name as the package (at least, on my Python 3.7). So, for example, it would fail if you have (using the OP's example):
proj/
__init__.py (empty)
proj.py (same name as package)
moduleA.py
moduleB.py
in which case you would get:
ModuleNotFoundError: No module named 'proj.moduleB'; 'proj' is not a package
Alternatively, you could remove the . in from .moduleB import, as suggested here and here, which seems to work, although my PyCharm (2018.2.4) marks this as an "Unresolved reference" and fails to autocomplete.
Maybe you can do this before importing the module：
moduleA.py3
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from moduleB import ModuleB
ModuleB.hello()
Add the current directory to the environment directory
Just rename the file from where you run the app to main.py:
from app import app
if __name__ == '__main__':
app.run()
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
will solve the issue of import path issue.