Several questions have been posted on the topic of relative paths in python, and it looks like this continues to be a fairly confusing topic, and no final word mentioned on it. I had found the most illustrative example in https://github.com/wweiradio/pkg-relative-import
However, when I made a few changes to those files, I ended up again with some error messages. After some amount of debugging, I came up with the following files and directory structure which works. (In the hierarchy mentioned below, I have mentioned the imports in each file, and which imports work and which dont.)
NOTE: (1) the code works, whether or not __init__.py files exist. So I have deleted all those files. (2) the files mentioned contain only the mentioned import lines, and maybe some print statements, and nothing else.
I run the code as:
cd top_level; python3 start1.py --> this works
cd top_level/src; python3 start2.py --> this does not work, as expected
I learnt the following lessons from this exercise:
(1) no relative paths on the invocation directory is allowed, or anywhere outside it.
(2) any further imports should be explicitly relative (eg .sub3, and not sub3),
or absolute with reference to the top level (eg src.sub2.sub3)
(3) relative paths have nothing to do with packages, nor the presence or absence of __init__.py.
eg, in the below code I have deleted all init.py, and it still works
(4) current dir is ., its parent is .., its grandparent is ..., etc
Can you confirm if these lessons are correct, and are there some other factors that I need to remember while using relative paths?
directory structure:
- top_level/
- top.py
- start1.py --> import src.sub.relative
- src/
- parent.py
- start2.py --> import sub.relative
- sub/
- relative.py --> from .. import parent
from ..sub2 import relative2
# these imports give error if start2.py is called
# this is understood, so we call only start1.py
- sub2/
- relative2.py --> import top
from .. import parent
#from sub3 import relative3 # --> doesnt work
from .sub3 import relative3 # --> works
from src.sub2.sub3 import relative3 # --> works
#import sub3.relative3 # --> doesnt work
import src.sub2.sub3.relative3 # --> works
- sub3/
- relative3.py --> import top
from ... import parent
Related
My project structure
mt-kart
|
--> src/data_kart
| |
| -->apis
| |
| -->__init__.py
| |
| -->main.py
--> tests
|
-->__init__.py
|
-->conftest.py
|
-->test_others.py
conftest.py has the following line.
from data_kart.main import fastapi_app
When run pytest from command line from project root(mt-kart). I get the following error
ModuleNotFoundError No module named 'data_kart'
How to make data_kart visible for conftest.py .
Option 1
Use relative imports:
from ..src.data_kart.main import fastapi_app
fastapi_app()
Please note, the above requires running conftest.py outside the project's directory, like this:
python -m mt-kart.tests.conftest
Option 2
Use the below in conftest.py (ref):
import sys
import os
# getting the name of the directory where the this file is present.
current = os.path.dirname(os.path.realpath(__file__))
# Getting the parent directory name where the current directory is present.
parent = os.path.dirname(current)
# adding the parent directory to the sys.path.
sys.path.append(parent)
# import the module
from src.data_kart.main import fastapi_app
# call the function
fastapi_app()
And if it happens for a module being outside the parent directory of tests, then you can get the parent of the parent direcory and add that one to the sys.path. For example:
parentparent = os.path.dirname(parent)
sys.path.append(parentparent)
Option 3
Same as Option 2, but shorter.
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from src.data_kart.main import fastapi_app
fastapi_app()
Option 4
Options 2 and 3 would work for smaller projects, where only a couple of files need to be updated with the code above. However, for larger projects, you may consider including the directory containing your package directory in PYTHONPATH, as described here (Option 3).
I have following folder structure
|
|---notebook
| |
| --- MyNoteBook.ipnyb
|
|---python
| |
| --- image_rotation_utils.py
I have following content in python.image_rotation_utils.py
class ImageRotationUtils:
""" Utils for rotating images."""
def __init__(self, image_path=None, image_name=None, output_path=None, output_annotations_file=None ):
""" Initializes the class.
Args:
image_path: the relative path to the image
image_name: image file name
output_path: the relative path where rotated output file is stored.
output_annotations_file: annotations file where rotated anotations are stored.
"""
self.image_path = image_path
self.image_name = image_name
self.output_path = output_path
self.output_annotations_file = output_annotations_file
#other functions defined
Now in MyNoteBook.ipnyb I have following cells
Cell 1:
import sys
sys.path.insert(0, '..')
from python.image_rotation_utils import *
Above is executed successfully now in cell 2 I have below
myUtils = ImageRotationUtils()
I have getting below error
----> 1 ImageRotationUtils()
TypeError: __init__() missing 4 required positional arguments: 'image_path', 'image_name', 'output_path', and 'output_annotations_file'
whye I am getting this error when I have default values in constructor has None in class definition.
I am new to python and trying to put code in productive manner. What mistake I am making. Kindly provide inputs
What you pasted looks ok to me, and your code works for me without errors.
Perhaps something before or after that code is what is spoiling the class definition?
Perhaps you are running this in Jupyter or in IPython and you imported ImageRotationUtils previously, but then changed the code, and tried to reimport again -- and something was wrong with the new reimport so your definition did not get over-written? If that's the case, restart the environment (or the kernel in Jupyter) and rerun the code.
I would suggest putting a simple initialization code, like that constructor line, into the same source file and executing it as a separate process to test if that is the code or the environment issue.
As a matter of convenience and to avoid tweaking sys.path in your code, I would suggest adding your python directory to the PYTHONPATH environment variable before you load your environment, so you can just import.
I am using Python 3.9.2 with sphinx-build 3.5.2. I have created a project with the following directory and file structure
core_utilities
|_core_utilities
| |_ __init__.py
| |_read_files.py
|_docs
|_ sphinx
|_Makefile
|_conf.pg
|_source
| |_conf.py
| |_index.rst
| |_read_files.rst
|_build
The read_files.py file contains the following code. NOTE: I simplified this example so it would not have distracting information. Within this code their is one class that contains one member function to read a text file, look for a key word and read the variable to the right of the keyword as a numpy.float32 variable. The stand-alone function written to read in a csv file with specific data types and save it to a pandas dataframe.
# Import packages here
import os
import sys
import numpy as np
import pandas as pd
from typing import List
class ReadTextFileKeywords:
def __init__(self, file_name: str):
self.file_name = file_name
if not os.path.isfile(file_name):
sys.exit('{}{}{}'.format('FATAL ERROR: ', file_name, ' does not exist'))
# ----------------------------------------------------------------------------
def read_double(self, key_words: str) -> np.float64:
values = self.read_sentence(key_words)
values = values.split()
return np.float64(values[0])
# ================================================================================
# ================================================================================
def read_csv_columns_by_headers(file_name: str, headers: List[str],
data_type: List[type],
if not os.path.isfile(file_name):
sys.exit('{}{}{}'.format('FATAL ERROR: ', file_name, ' does not exist'))
dat = dict(zip(headers, data_type))
df = pd.read_csv(file_name, usecols=headers, dtype=dat, skiprows=skip)
return df
The conf.py file as the following information in it.
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('../../../core_utilities'))
# -- Project information -----------------------------------------------------
project = 'Core Utilities'
copyright = 'my copyright'
author = 'my name'
# The full version, including alpha/beta/rc tags
release = '0.1.0'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.todo', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc',
'sphinx.ext.autosummary', 'sphinx.ext.githubpages']
autodoc_member_order = 'groupwise'
autodoc_default_flags = ['members', 'show-inheritance']
autosummary_generate = True
autodock_mock_imports = ['numpy']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'nature'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
The index.rst file has the following information.
. Core Utilities documentation master file, created by
sphinx-quickstart
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Core Utilities's documentation!
==========================================
.. toctree::
:maxdepth: 2
:caption: Contents:
read_files
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
and the read_files.rst file has the following information
**********
read_files
**********
The ``read_files`` module contains methods and classes that allow a user to read different
types of files in different ways. Within the module a user will find functionality that
allows them to read text files and csv files by columns and keywords. This module
also contains functions to read sqlite databases, xml files, json files, yaml files,
and html files
.. autoclass:: test.ReadTextFileKeywords
:members:
.. autofunction:: test.read_csv_columns_by_headers
In addition, I am using a virtual environment that is active when I try to compile this with command line. I run the following command to compile html files with sphinx.
sphinx-build -b html source build
When I run the above command it fails with the following error.
WARNING: autodoc: failed to import class 'ReadTextFileKeywords' from module 'read_files'; the following exception was raised; No module named pandas.
If I delete the line from pandas import pd and then delete the function read_csv_columns_by_headers along with the call to the function in the read_files.rst file, everything compiles fine. It appears that for some reason sphinx is able to find numpy, but it for some reason does not seem to recognize pandas, although both exist in the virtual environment and were loaded with a pip3 install statement. Does anyone know why sphinx is able to find other modules, but not pandas?
Add pandas to autodoc
autodoc_mock_imports = ['numpy','pandas']
I have the following directory structure
code
|-----|--experiments
| |---v0
| |---agcnn.py
|-----|--pn
| |---bag
| |--run.py
run.py uses a module inside the agcnn.py. However when I try the import as following:
from experiments.v0.agccn import AG
it gives a relative import error saying no module. I'm using VScode and the the working dir is taken care by Vscode as root which is /code/ in this case.
How can I change the settings to accommodate the import ?
I have the following directory structure:
/home/pi
- project/
- p1v1.py
- tools1/
- __init__.py
- tools1a/
- __init__.py
- file1.py
- file2.py
- tools1a1/
- __init__.py
- file3.py
- file4.py
- tools1a2/
- __init__.py
- file5.py
- file6.py
I am trying to import all the modules from the file1.py into my project file p1v1.py
from file1 import *
but end up with either an
ImportError: attempted relative import with no known parent package
or an
ValueError: Attempted relative import in non-package
depending on what I use in p1v1.py because the functions in file1.py depend on file3.py and file4.py. I would like to use explicit imports (for clarity), but I'm not sure how to do this. Any advice would be appreciated.
Thank you!
Through trial and error eventually figured out how to solve this:
import sys
sys.path.insert(0,'..')
from tools1.tools1a.file1 import function as f1
Note: For this to work, I needed to be editing and executing my script p1v1.py out of the working directory /home/pi/project/. Hope this helps others with a similar problem!