I have the following project structure:
path_to_dir/
│
├── a_notebook.ipynb
└── testCases_v2.py
In the .py file I have defined several functions.
I'm trying to import them in the notebook like this:
from testCases_v2 import layer_sizes_test_case
However, I am getting:
ImportError: cannot import name 'layer_sizes_test_case' from 'testCases_v2' (/path_to_dir/testCases_v2.py)
I even tried adding the directory to the system path:
import os
import sys
module_path = os.path.abspath(os.path.join('path_to_dir'))
if module_path not in sys.path:
sys.path.append(module_path)
But the problem persists.
How to solve it? (And, yes, I checked that the name of the function I am importing is spelled correctly)
Let's say you have this:
path_to_dir/
│
├── a_notebook.ipynb
└── tests
|
└── testCases_v2.py
And in testCases_v2 you have a class with the same name.
You have to manually import the modules you want Jupyter take into
account in sys.path In this case, the module is 'tests' and the path
you have to include is 'path_to_dir'
import sys
sys.path.insert(0, 'path/to/dir')
In the module where your py files are, add an empty __init__.py
path_to_dir/
│
├── a_notebook.ipynb
└── tests
|
├──testCases_v2.py
└── __init__.py
You should be able to import it now by doing:
from tests.testCases_v2 import testCases_v2
The python doc about modules might come in handy
Related
I have a folder structure like
app
├── modules
│ └── __init__.py
| └── helper.py
|
├── tests
| └── __init__.py
| └── unittest.py
|
__init__.py
In the unittest.py I have a module imported like
from modules import helper
helper.do_something()
Unfortunately if I run unittest.py I get ImportError: cannot import name 'helper' from 'modules' (unknown location). I'd appreciate a hint.
I have a similar project structure (main module folder containing source code folder as well as tests folder) and this is how imports are used in my tests:
from app.modules import helper
You could also use relative imports:
from ..modules import helper
It really depends on your needs and tastes.
I am trying to build a Python (3.7) module that has a structure similar to this:
module_dir/
├─ setup.py
├─ my_module/
│ ├─ utils/
│ │ ├─ common.py
│ │ ├─ __init__.py
│ ├─ __init__.py
│ ├─ dao/
│ │ ├─ base_dao.py
│ │ ├─ dwh/
│ │ │ ├─ dwh_dao.py
│ │ │ ├─ __init__.py
│ │ ├─ __init__.py
But I am having some issues:
when I import from this module in other projects, I am getting an ImportError.
In particular, my dwh_dao.py file contains the following import:
from dao.base_dao import BaseDAO
And it seems to make the import fail, but if I replace this import with a relative one, it works:
from ..dao.base_dao import BaseDAO
So far so good, but from dwh_dao.py I am also trying to reach utils.common and I cannot do that with a relative import AFAIK.
Therefore my questions are:
Is there any way to go up more than one level in a module using relative imports?
Why is my library failing to resolve absolute imports and I need to replace them with relative ones?
EDIT1:
Including my setup.py as well
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
with open('requirements.txt') as f:
requirements = f.read().splitlines()
setuptools.setup(
name="my_module",
version="1",
author="",
author_email="",
description="ACMECorp Internal Shared Library",
long_description=long_description,
long_description_content_type="text/markdown",
url="",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
],
install_requires=requirements
)
EDIT2:
If I use absolute imports also in my library's dwh_dao.py file, like
from my_module.dao.base_dao import BaseDAO
Then it works, but why in a common project it would have been fine to use
from dao.base_dao import BaseDAO
and in a library project I also have to include module name in the import itself?
why in a common project it would have been fine to use
from dao.base_dao import BaseDAO and in a library project I also have
to include module name in the import itself?
Python have a packages hierarchy. When you package a project in a standard way (this is why I wanted to see your setup.py) the main package is mymodule and the pip installer makes sure it will be found in the python sys.path. This is why importing mymodule or mymodule.dao is working.
Now when you run your code directly from the command line or IDE things might be different. This is mostly dependent on your working directory or PYTHONPATH
If you want to simulate exactly what will happen after pip install, do the following:
Open a python on the project folder (module_dir in your case)
Make sure your PYTHON_PATH is empty (it should be).
Try to import from the python.
You will see that import dao is not working.
However, if you will do it from the my_module folder (guess this is probably what you did) import dao will work. This is happening because the current folder is implicitly in the sys.path
I have a Python project that uses the MicroKernel pattern where I want each of the modules to be completely independent. I import each of the modules into the kernel and that works fine. However, when I am in a module I want the root of the module to be the module dir. This is the part that is not working.
Project structure;
.
├── requirements.txt
├── ...
├── kernel
│ ├── config.py
│ ├── main.py
│ ├── src
│ │ ├── __init__.py
│ │ ├── ...
│ └── test
│ ├── __init__.py
│ ├── ...
├── modules
│ └── img_select
│ ├── __init__.py
│ ├── config.py
│ ├── main.py
│ └── test
│ ├── __init__.py
│ └── test_main.py
If I import from main import somefunction in modules/img_select/test/test_main.py I get the following error:
ImportError: cannot import name 'somefunction' from 'main' (./kernel/main.py)
So it clearly does not see the modules/img_select as the root of the module, which leads to the following question:
How can I set the root for imports in a module?
Some additional info, I did add the paths with sys.path in the config files;
kernel/config.py;
import os
import sys
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
MODULES_DIR = os.path.join(ROOT_DIR, '../modules')
sys.path.insert(0, os.path.abspath(MODULES_DIR))
modules/img_select/config.py;
import os
import sys
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.abspath(ROOT_DIR))
And my python version is 3.7.3
I do realise that there are a lot of excellent resources out there, but I have tried most approaches and can't seem to get it to work.
I'm not sure what main you are trying to import from. I think python is confused from the pathing as well. How does test_main.py choose which main to run? Typically when you have a package (directory with __init__.py) you import from the package and not individual modules.
# test_main.py
# If img_select is in the path and has __init__.py
from img_select.main import somefunction
If img_select does not have __init__.py and you have img_select in the path then you can import from main.
# test_main.py
# If img_select is in the path without __init__.py
from main import somefunction
In your case I do not know how you are trying to indicate which main.py to import from. How are you importing and calling the proper config.py?
You might be able to get away with changing the current directory with os.chdir. I think your main problem is that img_select is a package with __init__.py. Python doesn't like to use from main import ... when main is in a package. Python is expecting from img_select.main import ....
Working Directory
If you are in the directory modules/img_select/test/ and call python test_main.py then this directory is known as your working directory. Your working directory is wherever you call python. If you are in the top level directory (where requirements.txt lives) and call python modules/img_select/test/test_main.py then the top level directory is the working directory. Python uses this working directory as path.
If kernel has an __init__.py then python will find kernel from the top level directory. If kernel is not a package then you need add the kernel directory to the path in order for python to see kernel/main.py. One way is to modify sys.path or PYTHONPATH like you suggested. However, if your working directory is modules/img_select/test/ then you have to go up several directories to find the correct path.
# test_main.py
import sys
TEST_DIR = os.path.dirname(__file__) # modules/img_select/test/
IMG_DIR = os.path.dirname(TEST_DIR)
MOD_DIR = os.path.dirname(IMG_DIR)
KERNEL_DIR = os.path.join(os.path.dirname(MOD_DIR), 'kernel')
sys.path.append(KERNEL_DIR)
from main import somefunction
If your top level directory (where requirements.txt lives) is your working directory then you still need to add kernel to the path.
# modules/img_select/test/test_main.py
import sys
sys.path.append('kernel')
As you can see this can change depending on your working directory, and you would have to modify every running file manually. You can get around this with abspath like you are doing. However, every file needs the path modified. I do not recommend manually changing the path.
Libraries
Python pathing can be a pain. I suggest making a library.
You just make a setup.py file to install the kernel or other packages as a library. The setup.py file should be at the same level as requirements.txt
# setup.py
"""
setup.py - Setup file to distribute the library
See Also:
* https://github.com/pypa/sampleproject
* https://packaging.python.org/en/latest/distributing.html
* https://pythonhosted.org/an_example_pypi_project/setuptools.html
"""
from setuptools import setup, Extension, find_packages
setup(name='kernel',
version='0.0.1',
# Specify packages (directories with __init__.py) to install.
# You could use find_packages(exclude=['modules']) as well
packages=['kernel'], # kernel needs to have __init__.py
include_package_data=True,
)
The kernel directory needs an __init__.py. Install the library as editable if you are still working on it. Call pip install -e . in the top level directory that has the setup.py file.
After you install the library python will have copied or linked the kernel directory into its site-packages path. Now your test_main.py file just needs to import kernel correctly
# test_main.py
from kernel.main import somefunction
somefunction()
Customizing init.py
Since kernel now has an __init__.py you can control the functions available from importing kernel
# __init__.py
# The "." indicates a relative import
from .main import somefunction
from .config import ...
try:
from .src.mymodule import myfunc
except (ImportError, Exception):
def myfunc(*args, **kwargs):
raise EnvironmentError('Function not available. Missing dependency "X".')
After changing the __init__.py you can import from kernel instead of kernel.main
# test_main.py
from kernel import somefunction
somefunction()
If you delete the NumPy (any library) from the site manager and save that folder in another location then use:
import sys
sys.path.append("/home/shubhangi/numpy") # path of numpy dir (which is removed from site manager and paste into another directory)
from numpy import __init__ as np
import numpy as np
arr = np.array([1, 2, 3, 4, 5])
print(arr)
print(type(arr))
VSCode Version: 1.41.1
OS Version:Ubuntu 18.04
Steps to Reproduce:
# tree:
.
├── demo1
│ ├── __init__.py
│ └── test.py
├── __init__.py
├── auto.py
# auto.py
def func():
print("1")
# test.py
from auto import func
func()
Use examples to solve problems that arise in a project
Run the test.py file, and I get "ModuleNotFoundError: No module named 'func'"
I used 'CTRL '+ left mouse button in test.py to jump to func
The same code can be run in pycharm
If you run test.py directly then you need to add the parent folder to PYTHONPATH. Try:
import sys
sys.path.append("..\<parent_folder>")
from auto import func
Otherwise, if you merely want to import test.py in another .py file, you can use relative import of python
from . import auto #another dot '.' to go up two packages
auto.func()
Reference
Add this in test.py, before import:
import sys
sys.path.insert(0, "/path/to/project/root/directory")
For me it's not a good file organization. A better practice might be as below:
Let your project file tree be like:
.
├── __init__.py
├── lib
│ ├── auto.py
│ └── __init__.py
└── test.py
And write test.py like:
from lib.auto import func
func()
Simple one-line solution
from ... import auto
and call the function by using auto.func().
I'm having trouble with relative imports, but I can't seem to figure out what's wrong in this case. It seems like a straightforward relative import from another module in the same package, so I'm at a loss for how to debug this.
My project is set up like so:
.
├── ckmeans
│ ├── __init__.py
│ ├── _ckmeans.pxd
│ ├── _ckmeans_wrapper.pyx
│ ├── _ckmeans.py
│ ├── _evaluation.py
│ └── _utils.py
└── setup.py
At the top of __init__.py:
from ._ckmeans import ckmeans # _ckmeans.py
And at the top of _ckmeans.py:
from . import _ckmeans_wrapper # _ckmeans_wrapper.pyx
And at the top of _ckmeans_wrapper.pyx:
cimport _ckmeans # _ckmeans.pxd
I run pip install --ignore-installed --upgrade -e ., and everything seems to go smoothly. Then when I try to run my test suite, or import ckmeans in the interpreter, I get the error:
ImportError: cannot import name '_ckmeans_wrapper'
When I comment out the import statement from __init__.py and then import ckmeans in the interpreter, it does indeed seem to be missing the _ckmeans_wrapper module. I suspect that something is failing silently in the Cython build, but I don't have any idea how to debug.
Here's the setup.py:
import numpy as np
from Cython.Build import cythonize
from setuptools import setup, Extension
extension = Extension(
name='_ckmeans_wrapper',
sources=['ckmeans/_ckmeans_wrapper.pyx'],
language="c++",
include_dirs=[np.get_include()]
)
setup(
name='ckmeans',
version='1.0.0',
packages=['ckmeans'],
ext_modules = cythonize(extension),
install_requires=['numpy', 'Cython']
)
The name argument to Extension was incorrect. It should be name='ckmeans._ckmeans_wrapper'.