I have a module with multiple files structured like this:
/bettermod/
├── __init__.py
├── api.py
├── bettermod.py
├── errors.py
└── loggers.py
From bettermod.py, I'm trying to import two things:
a class called API from api.py
the whole errors.py file
For the first thing, it is quite easy, I just have to do this:
from .api import API
However, for importing the whole errors.py file, I'm encountering a problem; I'm trying to do like this:
from . import errors
which should work, according to this python documentation, but it's raising the following error:
File "/path/to/bettermod/bettermod.py", line 10, in <module>
from . import errors
ModuleNotFoundError: No module named 'bettermod'
Edit: when debugging, I found that __name__ was equal to bettermod.bettermod
From docs:
Note that relative imports are based on the name of the current module.
I cannot tell you what is wrong with certainty, but there is a smell: bettermod package has a bettermod module. You want to do from bettermod.bettermod import MyBetterClass? I doubt it. In Python files ARE namespaces, so choosing your file and directory names is also an API design. Just keep that in mind.
I suspect the problem is masked by the name collision. Try this. If you run python in bettermod directory and say import bettermod you are importing bettermod\bettermod.py. And . is relative to the bettermod.py module. Now try to run python in directory above bettermod package directory. It will work because now . resolves to bettermod package.
Try:
import mymodule
mymodule.__file__
This will tell what mymodule is. For packages, it will show path to __init__.py. This will help you to orient yourself. Also look up PYHTONPATH and how you can use it to make sure you are importing from the right path.
Related
This question already has answers here:
Attempted relative import with no known parent package [duplicate]
(4 answers)
Relative imports in Python 3
(31 answers)
Closed 1 year ago.
I have the following directory structure:
py_test
├── __init__.py
├── dir1
│ ├── __init__.py
│ └── script1.py
└── dir2
├── __init__.py
└── script2.py
In script2 I want to "import ..\script1".
What I tried in script2:
Does not work
from ..dir1 import script1
ImportError: attempted relative import with no known parent package`
Works
import sys, os
path2add = os.path.normpath(os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, 'dir1')))
if (not (path2add in sys.path)) :
sys.path.append(path2add)
If I want to go with option 1, what is the simplest (i.e., with the least files) file/dir structure that makes it work?
I am aware of this, but I wonder if creating that directory structure can be avoided, and still use type-1 import.
I am currently using this workaround, which uses type-2 import.
Related:
How to import a Python class that is in a directory above?
Import a module from a directory (package) one level up
Getting "ImportError: attempted relative import with no known parent package" when running from Python Interpreter
Using importlib to dynamically import module(s) containing relative imports
How to import variables in a different python file
As mentioned in the comments, attempting to import modules a directory up will not work if script2.py is your entry point.
As mentioned in this link you included:
If the module's __name__ does not contain any package information (e.g., it is set to __main__), then relative imports are resolved as if the module were a top-level module, regardless of where the module is actually located on the file system.
The module's __name__ is set to __main__ if it is the entry point, or the one that you pass to the interpreter with something like python script2.py.
Any python module run as such no longer has the information needed to import files from higher directories.
Therefore you have two options:
Option 1: Keep using the workaround with sys.path.append
This will work how you want it to, but it is rather cumbersome.
Option 2: Make your entry point at the top level
Assuming your package has more than one script that needs to be run, you could create a new file that imports both script1 and script2 and then calls the functionality you want based on a command line argument. Then you will be able to keep your current directory structure and have your relative imports work just fine, without any kind of fiddling with sys.path.
I am trying to create an executable directory. It appears that the code in the init.py in one of my subpackages is executing before the main.py file in the root directory. Why is that?
Since you didn't describe any particular package structure in your question, all me to conjure up one for the sake of example. Let's say your package structure looks like the following:
package/
├── __init__.py
├── __main__.py
└── subpackage
├── __init__.py
└── submodule.py
and that package/__main__.py contains
print("before import in", __name__)
import package.subpackage.submodule
print("after import in", __name__)
while the files package/__init__.py, package/subpackage/__init__.py, and package/subpackage/submodule.py all contain
print(__name__)
(Note that __name__ is just a fancy global variable that holds the name of the current module).
If we try to run our package using the command
$ python3 -m package
we get the following output
package
before import in __main__
package.subpackage
package.subpackage.submodule
after import in __main__
This tells us that that package's top level __init__ module was the first to be loaded by the interpreter, followed by __main__. In the process of running __main__, we encounter an import statement, which causes the interpreter to briefly halt execution to load the desired module. When loading a module, Python check whether each intermediate package has already been loaded. Any packages that haven't been loaded yet will be loaded first, so importing package.subpackage.submodule results in package/subpackage/__init__.py being run, followed by package/subpackage/submodule.py. Only once all this is completed does control return back to __main__.
In your package, the __init__.py of your subpackage is not executing before __main__.py per se. Rather, you main module is (presumably) importing a module from the subpackage, which results in the subpackage's __init__ module being loaded, as demonstrated above.
My question is related to this. I am using Python 3.6 in Jupyter Notebook. My project directory is /user/project. In this directory I'm building a number of models and each has its own folder. However, there is a common functions.py file with functions that I want to use across all models. So I want to keep the functions.py file in /user/project but be able to call it from an .ipynb file in /user/project/model1, /user/project/model2, etc... How can I do this?
There is no simple way to import python files in another directory.
This is unrelated to the jupyter notebook
Here are 3 solutions to your problem
You can add the directory containing the file you want to import to your path and then import the file like this:
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import file
You can create a local module by having an empty __init__.py file in the folder you want to import. There are some weird rules regarding the folder hierarchy that you have to take into consideration.
You can create a module for the file you wish to import and install it globally.
Assuming you have a folder name Jupyter and you wish to import modules (employee) from another folder named nn_webserver.
visualizing it:
do this:
import sys
import os
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path+"\\nn_webserver")
from employee import motivation_to_work
see additional information here from #metakermit
I've been thinking about this problem because I don't like the sys.path.append() answers. A solution I propose uses the built-in Jupyter magic command to change the current working directory. Assuming you have this file structure:
project
├── model1
| └── notebook1.ipynb
├── model2
| └── notebook2.ipynb
└── functions.py
Whether you wanted to import functions from notebook1.ipynb or notebook2.ipynb, you could simply add a cell with the following line before the cell that has your package imports:
%cd ..
This changes the current working directory to the parent directory of the notebook, which then adds the path of the functions module to the default locations that Python will check for packages. To import functions:
import functions
This would work similarly if you had multiple modules in the same package directory that you wanted to import:
project
├── model1
| └── notebook1.ipynb
├── model2
| └── notebook2.ipynb
└── package
├── functions1.py
└── functions2.py
You can import both modules functions1 and functions2 from package like this:
from package import functions1, functions2
EDIT: As pointed out below, the local imports will no longer work if the cell containing the magic command is run more than once (the current working directory will be changed to the directory above at each rerun of the command). To prevent this from happening, the %cd .. command should be in its own cell (not in the same cell as the imports) at the top of the notebook and before the imports so it won't be run multiple times. Restarting the kernel and running all cells will reset the current working directory however will still return the desired imports/results.
I've solved this problem in the past by creating a symbolic link in the directory where the Jupyter notebook is located to the library it wants to load, so that python behaves as if the module is in the correct path. So for the example above, you would run the following command once per directory inside a Jupyter cell:
!ln -s /user/project/functions.py functions.py
and then you could import with
import functions
Note: I've only tried this on Linux and Mac Os, so I can't vouch for Windows.
I would suggest to install functions.py as a package in your virtual environment. There are some benefits of this:
You can access functions.py file from any iPython notebook located in any place, but at the given environment (kernel).
Once you changed any function in functions.py file, you don't need to reload your iPython notebook again and again. It will automatically reload every change.
This is the way how it can be done:
Create setup.py file (https://docs.python.org/2/distutils/setupscript.html) in your project folder
Activate your virtual environment, go to your project location, and use this command pip install -e .
Then, in your iPython notebook:
%load_ext autoreload
%autoreload 1
%aimport yourproject.functions
from functions import *
That's it!
In addition to the answer from adhg, I recommend using Pathlib, for compatibility between Linux/Windows/WSL paths formats:
Assuming the following folder structure:
.
├── work
| ├── notebook.ipynb
| └── my_python_file.py
├── py
| ├──modules
| | ├──__init__.py # empty
| | └──preparations.py
| ├──__init__.py # empty
| └── tools.py
├──.git
└── README.md
To load tools.py or preparations.py in my_python_file.py (or in notebook notebook.ipynb):
import sys
from pathlib import Path
# in jupyter (lab / notebook), based on notebook path
module_path = str(Path.cwd().parents[0] / "py")
# in standard python
module_path = str(Path.cwd(__file__).parents[0] / "py")
if module_path not in sys.path:
sys.path.append(module_path)
from modules import preparations
import tools
...
Found myself in the same exact situation as the OP, going to create several notebooks hence the wish to organise them in different subfolders
Tried this that seems to do what I need and seems cleaner to me
import os
os.chdir(os.path.dirname(os.path.dirname(os.getcwd())))
My function being two levels above so nested two os.path.dirname (with different folder structure could be only one or more)
Just implemented it and working fine, and btw I'm using JupyterLab started... two levels above where the function resides
Okay, the scenario is very simple. I have this file structure:
.
├── interface.py
├── pkg
│ ├── __init__.py
│ ├── mod1.py
│ ├── mod2.py
Now, these are my conditions:
mod2 needs to import mod1.
both interface.py and mod2 needs to be run independently as a main script. If you want, think interface as the actual program and mod2 as an internal tester of the package.
So, in Python 2 I would simply do import mod1 inside mod2.py and both python2 mod2.py and python2 interface.py would work as expected.
However, and this is the part I less understand, using Python 3.5.2, if I do import mod1; then I can do python3 mod2.py, but python3 interface.py throws: ImportError: No module named 'mod1' :(
So, apparently, python 3 proposes to use import pkg.mod1 to avoid collisions against built-in modules. Ok, If I use that I can do python3 interface.py; but then I can't python3 mod2.py because: ImportError: No module named 'pkg'
Similarly, If I use relative import:
from . import mod1 then python3 interface.py works; but mod2.py says SystemError: Parent module '' not loaded, cannot perform relative import :( :(
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works. But do we have to be adding the package prefix pkg to every import to other modules within that package? Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
UPDATE: I have upload those files with the (referred as "solution" above) working source code here: https://gitlab.com/Akronix/test_python3_packages. Note that I still don't like it, and looks much uglier than the python2 solution.
Related SO questions I've already read:
Python -- import the package in a module that is inside the same package
How to do relative imports in Python?
Absolute import module in same package
Related links:
https://docs.python.org/3.5/tutorial/modules.html
https://www.python.org/dev/peps/pep-0328/
https://www.python.org/dev/peps/pep-0366/
TLDR:
Run your code with python -m pkg.mod2.
Import your code with from . import mod1.
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works.
Using the -m switch is indeed the "only" solution - it was already the only solution before. The old behaviour simply only ever worked out of sheer luck; it could be broken without even modifying your code.
Going "one folder up" merely adds your package to the search path. Installing your package or modifying the search path works as well. See below for details.
But do we have to be adding the package prefix pkg to every import to other modules within that package?
You must have a reference to your package - otherwise it is ambiguous which module you want. The package reference can be either absolute or relative.
A relative import is usually what you want. It saves writing pkg explicitly, making it easier to refactor and move modules.
# inside mod1.py
# import mod2 - this is wrong! It can pull in an arbitrary mod2 module
# these are correct, they uniquely identify the module
import pkg.mod2
from pkg import mod2
from . import mod2
from .mod2 import foo # if pkg.mod2.foo exists
Note that you can always use <import> as <name> to bind your import to a different name. For example, import pkg.mod2 as mod2 lets you work with just the module name.
Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
If your package is properly installed, you can use the -m switch from anywhere. For example, you can always use python3 -m json.tool.
echo '{"json":"obj"}' | python -m json.tool
If your package is not installed (yet), you can set PYTHONPATH to its base directory. This includes your package in the search path, and allows the -m switch to find it properly.
If you are in the executable's directory, you can execute export PYTHONPATH="$(pwd)/.." to quickly mount the package for import.
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
This scenario was basically broken in python 2. While it was straightforward in many cases, it was difficult or outright impossible to fix in any other cases.
The new behaviour is more awkward in the straightforward case, but robust and reliable in any case.
I had similar problem.
I solved it adding
import sys
sys.path.insert(0,".package_name")
into the __init__.py file in the package folder.
Professional Python newbie here. I have created a Python module called aviation with a file in there called database.py.
I have another module called core, with a file in there called calculator.py.
I want to import aviation.database.py into my calculator.py.
The basic structure is as follows:
My project
aviation (module)
- database.py
core (module)
- calculator.py
test.py
My calculator.py file has an import such as:
from aviation import database as aviation_database
This module is not recognised and I get a red squiggly line indicating as much.
If I create another file test.py outside of aviation and core and add the above import, there are no issues in this tests.py file - the import works fine.
It appears that I need to do something so that my module can import from another module... it does allow me to import installed modules (like date), but I have no idea what I am missing.
I am using the IntelliJ IDE and my code is located in the regular C:\Users\\IdeaProjects directory.
Can someone tell me what I should do and why I am facing this problem?
Well I would recommend you to create __init__ file in the modules.
The python3 doc states that :
The __init__.py files are required to make Python treat the
directories as containing packages; this is done to prevent
directories with a common name, such as string, from unintentionally
hiding valid modules that occur later on the module search path. In
the simplest case, init.py can just be an empty file, but it can
also execute initialization code for the package or set the all
variable, described later.
So your Directory Structure becomes
My project
aviation (module)
- database.py
- __init__.py
core (module)
- calculator.py
- __init__.py
test.py
Now import all the thing in your __init__ file that you want to us from other packages. Like in aviation's __init__.py file write
import database
Similarly, for core's __init__.py
Now in calculator.py you can import them by -
from aviation import database
Let me know, if this help !
Thanks
Its as simple as this.
For instance, you have an already written code e.g myname.py and you want to import the entire module into new code e.g newwork.py
step 1. open your code newwork.py
step 2. just as you do other import, just write:
import myname
myname
Thats all.
Now, you have your entire nyname code inside newwork. When you run newwork, it runs nymame alongside.
I hope this help somebody?