I have the following file structure:
bot
├── LICENSE.md
├── README.md
├── bot.py # <-- file that is executed from command line
├── plugins
│ ├── __init__.py
│ ├── debug.py
│ └── parsemessages.py
├── helpers
│ ├── __init__.py
│ ├── parse.py
│ └── greetings.py
└── commands
├── __init__.py
└── search.py
bot.py, when executed from the command line, will load in everything in the plugins directory.
I want plugins/parsemessages.py to import parse from the helpers directory, so I do that:
# parsemessages.py
from ..helpers import parse
parse.execute("string to be parsed")
I run python3 bot.py from the command line.
I get the following error:
File "/home/bot/plugins/parsemessages.py", line 2, in <module>
from ..helpers import parse
ValueError: attempted relative import beyond top-level package
So I change two dots to one:
# parsemessages.py
from .helpers import parse
parse.execute("string to be parsed")
...but I get another error:
File "/home/bot/plugins/parsemessages.py", line 2, in <module>
from .helpers import parse
ImportError: No module named 'plugins.helpers'
How can I get this import to work?
It's worth noting that I'm not attempting to make a package here, this is just a normal script. That being said, I'm not willing to mess around with sys.path - I want this to be clean to use.
Additionally, I want parse to be imported as parse - so for the example above, I should be typing parse.execute() and not execute().
I found this post and this post, but they start with a file that's quite deep in the file structure (mine is right at the top). I also found this post, but it seems to be talking about a package rather than just a regular .py.
What's the solution here?
You could remove the dots, and it should work:
# parsemessages.py
from helpers import parse
parse.execute("string to be parsed")
That's probably your best solution if you really don't want to make it a package. You could also nest the entire project one directory deeper, and call it like python3 foo/bot.py.
Explanation:
When you're not working with an actual installed package and just importing stuff relative to your current working directory, everything in that directory is considered a top-level package. In your case, bot, plugins, helpers, and commands are all top-level packages/modules. Your current working directory itself is not a package.
So when you do ...
from ..helpers import parse
... helpers is considered a top-level package, because it's in your current working directory, and you're trying to import from one level higher than that (from your current working directory itself, which is not a package).
When you do ...
from .helpers import parse
... you're importing relative to plugins. So .helpers resolves to plugins.helpers.
When you do ...
from helpers import parse
... it finds helpers as a top-level package because it's in your current working directory.
If you want to execute your code from the root, my best answer to this is adding to the Path your root folder with os.getcwd().
Be sure your sibling folder has a init.py file.
import os
os.sys.path.insert(0, os.getcwd())
from sibling import module
Related
I'm trying to figure out file hierarchy and local imports in python3. I've run into a problem with a file structure below:
└── project
├── run.py
└── scripts
├── __init__.py
├── misc.py
├── script1.py
└── script2.py
I use relative imports for all files in the "scripts" directory.
from misc import x, y, z
I use absolute imports in run.py.
from scripts.misc import a, b, c
from scripts.script1 import more_stuff
My goal is to have the python files independent and fully functional. However, when I attempt to execute run.py, I encounter an error.
from misc import x, y, z
ModuleNotFoundError: No module named 'misc'
I would expect relative paths to be relative to the original file and not adopt the path of the executed file. Can I fix this by modifying the imports or file structure?
It also appears I don't understand how __init__.py works. I want to re-use generic package names (like "scripts"). I had assumed that __init__.py files would be read immediately downstream relative to the executed file: if run.py is executed, only the scripts directory at the same level should be considered. I have found that a distant (unrelated?) "scripts" directory receives priority. Is this something that can be addressed with more careful absolute paths? Example below.
└── dir
└── project1
| ├── run.py
| └── scripts
| ├── __init__.py
| └── settings.py
└── subdir
└── project2
├── run.py
└── scripts
├── __init__.py
└── settings.py
Executing run.py from "project1" will attempt to import the "scripts" directory from project2.
cannot import name 'variable' from 'scripts.settings' (/Users/.../dir/subdir/project2/scripts/settings.py)
Removing __init__.py from project2/scripts no longer produces the error when executing run.py from "project1".
You are saying:
I use relative imports for all files in the "scripts" directory.
from misc import x, y, z
But this is not relative. For relative you need to have
from .misc import x, y, z
To understand why the unrelated scripts is taking precedence, look on your sys.path and verify if indeed it comes before your scripts package. I can assume the is some leftover for ide path manipulation.
I have a Python project that uses the MicroKernel pattern where I want each of the modules to be completely independent. I import each of the modules into the kernel and that works fine. However, when I am in a module I want the root of the module to be the module dir. This is the part that is not working.
Project structure;
.
├── requirements.txt
├── ...
├── kernel
│ ├── config.py
│ ├── main.py
│ ├── src
│ │ ├── __init__.py
│ │ ├── ...
│ └── test
│ ├── __init__.py
│ ├── ...
├── modules
│ └── img_select
│ ├── __init__.py
│ ├── config.py
│ ├── main.py
│ └── test
│ ├── __init__.py
│ └── test_main.py
If I import from main import somefunction in modules/img_select/test/test_main.py I get the following error:
ImportError: cannot import name 'somefunction' from 'main' (./kernel/main.py)
So it clearly does not see the modules/img_select as the root of the module, which leads to the following question:
How can I set the root for imports in a module?
Some additional info, I did add the paths with sys.path in the config files;
kernel/config.py;
import os
import sys
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
MODULES_DIR = os.path.join(ROOT_DIR, '../modules')
sys.path.insert(0, os.path.abspath(MODULES_DIR))
modules/img_select/config.py;
import os
import sys
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.abspath(ROOT_DIR))
And my python version is 3.7.3
I do realise that there are a lot of excellent resources out there, but I have tried most approaches and can't seem to get it to work.
I'm not sure what main you are trying to import from. I think python is confused from the pathing as well. How does test_main.py choose which main to run? Typically when you have a package (directory with __init__.py) you import from the package and not individual modules.
# test_main.py
# If img_select is in the path and has __init__.py
from img_select.main import somefunction
If img_select does not have __init__.py and you have img_select in the path then you can import from main.
# test_main.py
# If img_select is in the path without __init__.py
from main import somefunction
In your case I do not know how you are trying to indicate which main.py to import from. How are you importing and calling the proper config.py?
You might be able to get away with changing the current directory with os.chdir. I think your main problem is that img_select is a package with __init__.py. Python doesn't like to use from main import ... when main is in a package. Python is expecting from img_select.main import ....
Working Directory
If you are in the directory modules/img_select/test/ and call python test_main.py then this directory is known as your working directory. Your working directory is wherever you call python. If you are in the top level directory (where requirements.txt lives) and call python modules/img_select/test/test_main.py then the top level directory is the working directory. Python uses this working directory as path.
If kernel has an __init__.py then python will find kernel from the top level directory. If kernel is not a package then you need add the kernel directory to the path in order for python to see kernel/main.py. One way is to modify sys.path or PYTHONPATH like you suggested. However, if your working directory is modules/img_select/test/ then you have to go up several directories to find the correct path.
# test_main.py
import sys
TEST_DIR = os.path.dirname(__file__) # modules/img_select/test/
IMG_DIR = os.path.dirname(TEST_DIR)
MOD_DIR = os.path.dirname(IMG_DIR)
KERNEL_DIR = os.path.join(os.path.dirname(MOD_DIR), 'kernel')
sys.path.append(KERNEL_DIR)
from main import somefunction
If your top level directory (where requirements.txt lives) is your working directory then you still need to add kernel to the path.
# modules/img_select/test/test_main.py
import sys
sys.path.append('kernel')
As you can see this can change depending on your working directory, and you would have to modify every running file manually. You can get around this with abspath like you are doing. However, every file needs the path modified. I do not recommend manually changing the path.
Libraries
Python pathing can be a pain. I suggest making a library.
You just make a setup.py file to install the kernel or other packages as a library. The setup.py file should be at the same level as requirements.txt
# setup.py
"""
setup.py - Setup file to distribute the library
See Also:
* https://github.com/pypa/sampleproject
* https://packaging.python.org/en/latest/distributing.html
* https://pythonhosted.org/an_example_pypi_project/setuptools.html
"""
from setuptools import setup, Extension, find_packages
setup(name='kernel',
version='0.0.1',
# Specify packages (directories with __init__.py) to install.
# You could use find_packages(exclude=['modules']) as well
packages=['kernel'], # kernel needs to have __init__.py
include_package_data=True,
)
The kernel directory needs an __init__.py. Install the library as editable if you are still working on it. Call pip install -e . in the top level directory that has the setup.py file.
After you install the library python will have copied or linked the kernel directory into its site-packages path. Now your test_main.py file just needs to import kernel correctly
# test_main.py
from kernel.main import somefunction
somefunction()
Customizing init.py
Since kernel now has an __init__.py you can control the functions available from importing kernel
# __init__.py
# The "." indicates a relative import
from .main import somefunction
from .config import ...
try:
from .src.mymodule import myfunc
except (ImportError, Exception):
def myfunc(*args, **kwargs):
raise EnvironmentError('Function not available. Missing dependency "X".')
After changing the __init__.py you can import from kernel instead of kernel.main
# test_main.py
from kernel import somefunction
somefunction()
If you delete the NumPy (any library) from the site manager and save that folder in another location then use:
import sys
sys.path.append("/home/shubhangi/numpy") # path of numpy dir (which is removed from site manager and paste into another directory)
from numpy import __init__ as np
import numpy as np
arr = np.array([1, 2, 3, 4, 5])
print(arr)
print(type(arr))
I'm trying to figure out if this is an error in my design, or an error in the redis-py library. Essentially, my understanding of namespace in Python is that packages should be designed such that all components are under the package namespace namespace. Meaning, if I have a queue in packageA and a queue in packageB, there should be no collision since they are namespaced (packageA.queue and packageB.queue). However, I'm running into an error in a package I am building.
This is the directory structure for the package I am building:
○ → tree
.
├── __init__.py
├── net
│ ├── __init__.py
│ ├── rconn.py
└── test.py
The __init__.py files are all empty. Here's the code of my test.py file:
○ → cat test.py
from net import rconn
and here's the code from my net/rconn.py file:
○ → cat net/rconn.py
import redis
Running test.py, everything works, no errors. However, if I add a queue directory in here and create an empty init.py within, here's the new tree:
○ → tree
.
├── __init__.py
├── net
│ ├── __init__.py
│ ├── rconn.py
├── queue
│ ├── __init__.py
└── test.py
Running test.py results in the following error:
Traceback (most recent call last):
File "test.py", line 1, in <module>
from net.rconn import ass
File "/Users/yosoyunmaricon/python_test/net/rconn.py", line 1, in <module>
import redis
File "/usr/local/lib/python3.7/site-packages/redis/__init__.py", line 1, in <module>
from redis.client import Redis, StrictRedis
File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 10, in <module>
from redis._compat import (b, basestring, bytes, imap, iteritems, iterkeys,
File "/usr/local/lib/python3.7/site-packages/redis/_compat.py", line 139, in <module>
from queue import Queue
ImportError: cannot import name 'Queue' from 'queue' (/Users/yosoyunmaricon/python_test/queue/__init__.py)
So, I get what's happening. The Redis code says from queue import Queue, and when I create an empty queue directory (i.e., no Queue), it breaks the package. My question is this: Is that good design? Should the Redis package be more explicit and say something along the lines of from redis.queue import Queue, or is this simply an error in my own design?
It's not the Redis package that should adjust here, because it cannot know or cannot handle the different ways users could integrate the Redis package into their own applications, like how you have a similarly named queue package. Furthermore, there is no redis.queue because that queue, is not part of redis, but the built-in Python queue package. You can go to /usr/local/lib/python3.7/site-packages/redis/_compat.py and print out the queue.__file__, which would give you the path to Python's queue. It expects importing the built-in queue package.
Unfortunately for you, when Python builds the module search paths for resolving imports, it builds it in the following order:
The directory containing the input script (or the current directory when no file is specified).
PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
The installation-dependent default.
...which puts your own queue at the start of the list and that's what gets imported. So, yes, getting an ImportError because you shadowed the built-in queue is more of an error in your own design.
You could probably do some tricks here with sys.path or PYTHONPATH, but why bother when you can just rename your queue to something else. Or, what I usually do is to group my own packages into a parent folder, named after the acronym for the project ("abcdlibs") or some app identier or something like "mylibs":
.
├── __init__.py
├── mylibs
│ └── queue
│ ├── __init__.py
├── mynet
│ ├── __init__.py
│ └── rconn.py
└── test.py
That way, you could make it clear that mylibs.queue is different from queue.
I have the following project structure:
x/
a.py
b.py
main.py
a.py:
from b import *
class A:
.....
main.py
from x.a import A
.....
I want to be able to run a.py independently as well as access its functionality through main.py
I'm able to run a.py but when I try to import it as shown in main.py, the module is unable to be found. I can fix this problem by adding the following line to a.py:
sys.path.append(os.path.join(os.path.dirname(__file__)))
but this feels hacky. Is there a better way to achieve the desired behavior?
You need to mark the directory "x" as a package to be able to load anything off it.
As stated in the official documentation of Python, you have to create an empty "__init__.py" file in the root of "x" to mark it off as a package.
Then your directory structure should look something like this:
.
└── x
├── __init__.py
├── a.py
└── b.py
└── main.py
You may want to edit "a.py" to load the modules relative to the package it is in using a period to represent the current package:
# x/a.py
from .b import *
class A:
# rest of your code
directory structure
test/
__init__.py
line.py
test.py
test.py
from . import line
output:
Traceback (most recent call last):
File "test.py", line 1, in <module>
from . import line
ImportError: cannot import name 'line'
I know I can just import line. but it may import standard library in python3.
why did this error happen? does python3 support this syntax?
ps: I'm not in interactive console and test directory has already been a package
I'm not sure if you can import module with the same name as library utilities directly. However, you could consider putting your modules inside a package, so your directory structure would look like:
.
├── main.py
├── package
│ ├── __init__.py
│ ├── line.py
│ ├── test.py
Where __init__.py might have some setup commands (you can omit this file if you have none) and in main.py you can do the following:
from package import line
...
Within the directory package if you'd like to say, import line.py in test.py you can use the syntax:
from . import line
Note relative imports (using the from . notation) will only work inside the script and not in the interactive console. Running the test.py ( if it's using relative imports) directly will also not work, although importing it from main.py will work.