I have two projects. In first one, I can import my module importme.py like:
import importme
And now I can use my function hello() in importme module without any problem. In second one, I recieve:
ImportError: No module named 'importme'
But I can import it via:
from . import importme
Why I cant import my module the same way in both projects? Should I configure some paths ?
EDIT1:
Directory structure of first project:
testproject/
├── importme.py
└── start.py
Directory structure of second project:
spiders/
├── spider.py
├── download_page.py
├── importme.py
└── __init__.py
file init.py is empty.
My favorite method of dealing with PYTHONPATH is installing package in edit mode in virtual environment.
Creating virtual environment
# create
$ python -m venv ~/virtualevns/myenv
# then activate it
$ source ~/virtualenvs/myenv/bin/activate
# you can check whether it got activated
$ which python
home/user/virtualenvs/myenv/bin/python
Writing setup.py for your project. For this refer to the official distributing packages tutorial.
Installing package in editable format.
If you install package with -e flag pip will install it in editable format meaning all changes you have in code will be present in your environment's package:
$ cd mypkg
$ pip install -e .
Finally you need to set your virtual-environment to be used in whatever IDE your editor you are using.
This is a great workflow because it's clean and reliable - you are using exactly what you'd be using in production/finished package environment.
Related
My objective is to use argparse command-line script along with a larger project. Especially I want to use some models from my project inside argparse_command.py.
Let's say the project structure is as follows:
myproject/
├── app
│ ├── db.py
│ └── __init__.py
│── management
│ └── argaparse_command.py
└── main.py
And the body of argparse_command.py:
import argparse
from app.db import engine
print('Print this when run from command-line.')
When I run the code from the command-line I get an error:
Traceback (most recent call last):
File "argaparse_command.py", line 2, in <module>
from app.db import engine
ModuleNotFoundError: No module named 'app'
And the problem is to import some stuff from within my project.
I have two questions:
Is it possible at all, and if so, how to do it?
What is the difference between running main.py with some IDE (pycharm) and argaparse_command.py with command-line (why it behaves differnt).
To be able to import app, you'd either have to install app as a python package (using a setup.py to specify how to install it for example), or be in the myproject working directory so your python interpreter has direct access to app in its path. So to answer question 1, yes it's possible.
The difference between running main.py with an IDE vs running on the command line is going to be the working directory the script is run in, which python executable is used and which virtual environment (if any) is used, and what flags are passed. You can check your IDE settings on how it invokes a script, and basically replicate that on the command line if you want.
I have a directory structure like below:
chatbot/
__init__.py
utils/
__init__.py
parser.py
nlu/
__init__.py
training/
__init__.py
module.py
I want to access parser.py from module.py.
I tried using this line from module.py:
from chatbot.utils import parser
And I got this error:
ModuleNotFoundError: No module named 'chatbot'
Any pointers to what I am doing wrong?
I am using python3 and trying to run the script as python3 nlu/training/module.py.
Thanks in advance!
I believe the right way of solving such problems is:
figure out what your top-level packages and modules are and in what directory they are
change to that directory
make all your imports absolute, i.e. always start with a top-level module or package (instead of things like from . import blah)
use the executable module (python -m) way of running your code, for example path/to/pythonX.Y -m top_level_package.executable_module (instead of path/to/pythonX.Y top_level_package/executable_module.py)
In case the top-level modules or packages are not all in the same directory:
Either work on packaging your libraries or applications correctly and install them in the site packages (eventually as editable, also known as develop mode).
Or, as a last resort, collect the other directories containing the top-level packages and modules in the PYTHONPATH environment variable like the following:
PYTHONPATH=path/to/alpha:path/to/bravo path/to/pythonX.Y -m top_level_package.executable_module
I was able to solve the issue by adding the parent directory of chatbot package to sys.path.
Say if chatbot python package is in /home/my_project/chatbot/, I added a statement like:
sys.path.append('/home/my_project') in module.py
This made all top-level python packages visible to all lower-level python packages.
The problem
I have a directory structure for my project which follows the standard for Python packages, as it was created with this cookiecutter template:
https://github.com/audreyr/cookiecutter-pypackage#quickstart
The directory structure is
project_name
├── project_name
│ ├── __init__.py
│ └── module1.py
└── tests
└── test_module1.py
The first code line of test_module1.py is:
from project_name import module1
But I get a ModuleNotFoundError: No module named 'project_name'.
To my understanding, this should work since the folder called project_name is a package, which is ensured by presence of the __init__.py file.
I have always had trouble understanding how imports like this work. For my projects I have always just settled with having my tests in the same folder as the modules to test. I know this is bad practice, but the only way I could get the modules to actually import.
What I already tried
I have tried renaming the folder with the __init__.py file to something else and then import, as I thought it could have something to do with the parent folder and the child folder both having the name project_name. This did not work, same error.
I also tried making the test folder into a package by creating an __init.py__ file inside it, even though the Cookiecutter template does not have that.
I read in many places that making the test folder into a package is discouraged, but some suggest that structure. That did not work either.
I have searched thoroughly for solutions to this seemingly very standard problem, some of the links are here:
Python, importing modules for testing
https://gist.github.com/tasdikrahman/2bdb3fb31136a3768fac
Importing modules from parent folder
https://alex.dzyoba.com/blog/python-import/
Sibling package imports
Python imports for tests using nose - what is best practice for imports of modules above current package
My last try was to start a project with Cookiecutter, so everything would be set up properly form the beginning. However, I still get the ModuleNotFoundError.
What I don't want
I don't want to modify sys.path as many answers seem to suggest. There must be a cleaner way for such a common problem.
What am I doing wrong?
Edit for some additional info (see question from #Nicholas):
The contents of __init__.py is
# -*- coding: utf-8 -*-
"""Top-level package for project_name."""
__author__ = """my_name"""
__email__ = 'my_email'
__version__ = '0.1.0'
Which was generated by the Cookiecutter template.
Inside test_module1, I added the following before the before the ModuleNotFoundError occurs:
import sys
import os
print(sys.path)
print(os.getcwd())
sys.path prints a list, where first element is the tests directory.
['c:\\Users\\...\\project_name\\tests',
'C:\\Users\\...\\Miniconda3\\python37.zip',
'C:\\Users\\...\\Miniconda3\\DLLs', 'C:\\Users\\...\\Miniconda3\\lib',
'C:\\Users\\...\\Miniconda3',
'C:\\Users\\...\\Miniconda3\\lib\\site- packages',
'C:\\Users\\...\\Miniconda3\\lib\\site-packages\\win32',
'C:\\Users\\...\\Miniconda3\\lib\\site-packages\\win32\\lib',
'C:\\Users\\...\\Miniconda3\\lib\\site-packages\\Pythonwin']
I don't know if the lowercase 'c' in the first element matters.
os.getcwd() prints the root directory 'c:\Users\....\project_name'. Also with a lowercase 'c'.
You should create a virtual environment and install the project in order for the test modules to correctly resolve import statements.
In the project root, i.e. the directory project_name which contains a subdirectory project_name and a subdirectory tests, create a setup.py (or pyproject.toml) file for the package metadata. See here for details about that part.
From this same project root directory which is now containing the installer (setup.py), create and activate a venv and install your project:
python3 -m venv .venv
source .venv/bin/activatate # linux/macOS
# .\Scripts\activate.bat # windows
pip install --editable .
pip install pytest
pytest
If for some reason you don't want to create an installer for your project, you may run pytest like this, from the project directory:
python3 -m pytest
Unlike the bare pytest command, this will add the current working directory to sys.path allowing import statements to be resolved in tests.
My question is related to this. I am using Python 3.6 in Jupyter Notebook. My project directory is /user/project. In this directory I'm building a number of models and each has its own folder. However, there is a common functions.py file with functions that I want to use across all models. So I want to keep the functions.py file in /user/project but be able to call it from an .ipynb file in /user/project/model1, /user/project/model2, etc... How can I do this?
There is no simple way to import python files in another directory.
This is unrelated to the jupyter notebook
Here are 3 solutions to your problem
You can add the directory containing the file you want to import to your path and then import the file like this:
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import file
You can create a local module by having an empty __init__.py file in the folder you want to import. There are some weird rules regarding the folder hierarchy that you have to take into consideration.
You can create a module for the file you wish to import and install it globally.
Assuming you have a folder name Jupyter and you wish to import modules (employee) from another folder named nn_webserver.
visualizing it:
do this:
import sys
import os
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path+"\\nn_webserver")
from employee import motivation_to_work
see additional information here from #metakermit
I've been thinking about this problem because I don't like the sys.path.append() answers. A solution I propose uses the built-in Jupyter magic command to change the current working directory. Assuming you have this file structure:
project
├── model1
| └── notebook1.ipynb
├── model2
| └── notebook2.ipynb
└── functions.py
Whether you wanted to import functions from notebook1.ipynb or notebook2.ipynb, you could simply add a cell with the following line before the cell that has your package imports:
%cd ..
This changes the current working directory to the parent directory of the notebook, which then adds the path of the functions module to the default locations that Python will check for packages. To import functions:
import functions
This would work similarly if you had multiple modules in the same package directory that you wanted to import:
project
├── model1
| └── notebook1.ipynb
├── model2
| └── notebook2.ipynb
└── package
├── functions1.py
└── functions2.py
You can import both modules functions1 and functions2 from package like this:
from package import functions1, functions2
EDIT: As pointed out below, the local imports will no longer work if the cell containing the magic command is run more than once (the current working directory will be changed to the directory above at each rerun of the command). To prevent this from happening, the %cd .. command should be in its own cell (not in the same cell as the imports) at the top of the notebook and before the imports so it won't be run multiple times. Restarting the kernel and running all cells will reset the current working directory however will still return the desired imports/results.
I've solved this problem in the past by creating a symbolic link in the directory where the Jupyter notebook is located to the library it wants to load, so that python behaves as if the module is in the correct path. So for the example above, you would run the following command once per directory inside a Jupyter cell:
!ln -s /user/project/functions.py functions.py
and then you could import with
import functions
Note: I've only tried this on Linux and Mac Os, so I can't vouch for Windows.
I would suggest to install functions.py as a package in your virtual environment. There are some benefits of this:
You can access functions.py file from any iPython notebook located in any place, but at the given environment (kernel).
Once you changed any function in functions.py file, you don't need to reload your iPython notebook again and again. It will automatically reload every change.
This is the way how it can be done:
Create setup.py file (https://docs.python.org/2/distutils/setupscript.html) in your project folder
Activate your virtual environment, go to your project location, and use this command pip install -e .
Then, in your iPython notebook:
%load_ext autoreload
%autoreload 1
%aimport yourproject.functions
from functions import *
That's it!
In addition to the answer from adhg, I recommend using Pathlib, for compatibility between Linux/Windows/WSL paths formats:
Assuming the following folder structure:
.
├── work
| ├── notebook.ipynb
| └── my_python_file.py
├── py
| ├──modules
| | ├──__init__.py # empty
| | └──preparations.py
| ├──__init__.py # empty
| └── tools.py
├──.git
└── README.md
To load tools.py or preparations.py in my_python_file.py (or in notebook notebook.ipynb):
import sys
from pathlib import Path
# in jupyter (lab / notebook), based on notebook path
module_path = str(Path.cwd().parents[0] / "py")
# in standard python
module_path = str(Path.cwd(__file__).parents[0] / "py")
if module_path not in sys.path:
sys.path.append(module_path)
from modules import preparations
import tools
...
Found myself in the same exact situation as the OP, going to create several notebooks hence the wish to organise them in different subfolders
Tried this that seems to do what I need and seems cleaner to me
import os
os.chdir(os.path.dirname(os.path.dirname(os.getcwd())))
My function being two levels above so nested two os.path.dirname (with different folder structure could be only one or more)
Just implemented it and working fine, and btw I'm using JupyterLab started... two levels above where the function resides
Okay, the scenario is very simple. I have this file structure:
.
├── interface.py
├── pkg
│ ├── __init__.py
│ ├── mod1.py
│ ├── mod2.py
Now, these are my conditions:
mod2 needs to import mod1.
both interface.py and mod2 needs to be run independently as a main script. If you want, think interface as the actual program and mod2 as an internal tester of the package.
So, in Python 2 I would simply do import mod1 inside mod2.py and both python2 mod2.py and python2 interface.py would work as expected.
However, and this is the part I less understand, using Python 3.5.2, if I do import mod1; then I can do python3 mod2.py, but python3 interface.py throws: ImportError: No module named 'mod1' :(
So, apparently, python 3 proposes to use import pkg.mod1 to avoid collisions against built-in modules. Ok, If I use that I can do python3 interface.py; but then I can't python3 mod2.py because: ImportError: No module named 'pkg'
Similarly, If I use relative import:
from . import mod1 then python3 interface.py works; but mod2.py says SystemError: Parent module '' not loaded, cannot perform relative import :( :(
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works. But do we have to be adding the package prefix pkg to every import to other modules within that package? Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
UPDATE: I have upload those files with the (referred as "solution" above) working source code here: https://gitlab.com/Akronix/test_python3_packages. Note that I still don't like it, and looks much uglier than the python2 solution.
Related SO questions I've already read:
Python -- import the package in a module that is inside the same package
How to do relative imports in Python?
Absolute import module in same package
Related links:
https://docs.python.org/3.5/tutorial/modules.html
https://www.python.org/dev/peps/pep-0328/
https://www.python.org/dev/peps/pep-0366/
TLDR:
Run your code with python -m pkg.mod2.
Import your code with from . import mod1.
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works.
Using the -m switch is indeed the "only" solution - it was already the only solution before. The old behaviour simply only ever worked out of sheer luck; it could be broken without even modifying your code.
Going "one folder up" merely adds your package to the search path. Installing your package or modifying the search path works as well. See below for details.
But do we have to be adding the package prefix pkg to every import to other modules within that package?
You must have a reference to your package - otherwise it is ambiguous which module you want. The package reference can be either absolute or relative.
A relative import is usually what you want. It saves writing pkg explicitly, making it easier to refactor and move modules.
# inside mod1.py
# import mod2 - this is wrong! It can pull in an arbitrary mod2 module
# these are correct, they uniquely identify the module
import pkg.mod2
from pkg import mod2
from . import mod2
from .mod2 import foo # if pkg.mod2.foo exists
Note that you can always use <import> as <name> to bind your import to a different name. For example, import pkg.mod2 as mod2 lets you work with just the module name.
Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
If your package is properly installed, you can use the -m switch from anywhere. For example, you can always use python3 -m json.tool.
echo '{"json":"obj"}' | python -m json.tool
If your package is not installed (yet), you can set PYTHONPATH to its base directory. This includes your package in the search path, and allows the -m switch to find it properly.
If you are in the executable's directory, you can execute export PYTHONPATH="$(pwd)/.." to quickly mount the package for import.
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
This scenario was basically broken in python 2. While it was straightforward in many cases, it was difficult or outright impossible to fix in any other cases.
The new behaviour is more awkward in the straightforward case, but robust and reliable in any case.
I had similar problem.
I solved it adding
import sys
sys.path.insert(0,".package_name")
into the __init__.py file in the package folder.