I've been working on a python3 application and I ran into a strange problem that picked my curiosity after annoying me greatly.
My file structure is something like this:
root/
| __init__.py
| main.py
| fuzzy/
| __init__.py
| foo.py
| dreamy/
| __init__.py
| bar.py
| meh.py
So I need to use a method of the foo module in both bar and meh.
In bar I wrote:
from fuzzy.foo import foo_function
And that worked perfectly fine.
Now in meh I used the exact same syntax but ended up with an import error:
ModuleNotFoundError: No module named 'fuzzy'
I managed to solve the problem using this method, but I would like to understand why this happens.
According to the Python3 documentation my import syntax is correct:
An alternative way of importing the submodule is:
from sound.effects import echo
This seems to also be the syntax provided by answered question on this forum
Moreover, both bar and meh are in the same directory and their code starts in exactly the same way.
Does anyone know why this error occurs in one file but not in the other ?
Is there something I did wrong ?
If I did do something wrong, what would be the correct way (or good practices) to import local packages / methods ?
Thanks for your insight.
---EDIT---
I did not fiddle with my PYTHONPATH in any way, and I am running Python3 from a (vanilla) Conda venv. For good measures, I also created a new file structure and copied the code in new, blank files. This was to make sure that nothing "funny" would have happened to my directory.
This question seems to be terribly simple, however I did notice that other people struggled with it.
Though I found the answer to my question thanks to Baum mit Augen.
Baum mit Augen took the time to write this answer which is very instructive and well written.
So sum my problem up:
In development, I tried to call my functions by launching my bar.py from the terminal. This means bar is the main, but located inside a package subdirectory. This causes an Import Error because "its name does not reveal that it is in a package".
I just want to highlight 2 things unrelated to my question:
Considering the post on which I have found this answer this is NOT an easy problem. FYI: I had a quick chat with a Python developer who couldn't find the words to explain this to me.
I find it incredible that those who seek help with valid questions are basically being answered to "RTFM", arch-linux style. I really find it a pity.
Related
Not exactly a newbie but not an expert either, obviously. Could not find this problem elsewhere.
This is a simple layout created with Atom, and it used to work just fine from the terminal, but no more, since the module c.py has just been added. This layout does work as a PyCharm project.
Is this layout inherently flawed, or should it work in theory, assuming the simplest possible "hello world" functionality?
Thank you in advance.
--blah
|__blah
|__a.py
|__b.py
|__c.py
#a.py
import b
import c
#do stuff
(venv)me blah $ python3
>>>import blah.a
ModuleNotFoundError: No module named 'c'
In PyCharm, blah.blah is "marked as sources root". Searching that led me to this post: What does "Mark directory as sources root" really do?
Realizing that PyCharm did some things behind the scenes that I took for granted, I tried the following in the Atom version of the project:
import blah.c
which solved the problem.
Thus I surmise that the strict answer to the original question is, "no".
I've read PEP 328 and similar questions like this, but they still didn't answer my questions about python 3 import. My question is:
why does a single dot . relative import ever produce an error like the following?
code: (__init__.py exists in the same directory)
from . import module_x
error:
ImportError: attempted relative import with no known parent package
Yes, there are workarounds to get around of this error for example by running python -m from the parent directory, but why?
Shouldn't . always refer to the same directory of the current module and the example always works as long as the module_x exists in the same directory?
Particularly PyCharm tries to run unittest test cases from the directory where the test file is, and would always fail the . import. Some may argue it's PyCharm's problem, but I think a simple import like this should really always work regardless where the interpreter runs.
There is a detailed answer here, also here. Sorry I cannot write this as a comment.
I am trying to create a Python package. You can have a look at my awful attempt here.
I have a module called imguralbum.py. It lives in a directory called ImgurAlbumDownloader which I understand is the name of the package -- in terms of what you type in an import statement, e.g.
import ImgurAlbumDownloader
My module contains two classes ImgurAlbumDownloader and ImgurAlbumException. I need to be able to use both of these classes in another module (script). However, I cannot for the life of me work out what I am supposed to put in my __init__.py file to make this so. I realize that this duplicates a lot of previously answered questions, but the advice seems very conflicting.
I still have to figure out why (I have some ideas), but this is now working:
from ImgurAlbumDownloader.imguralbum import ImgurAlbumDownloader, ImgurAlbumException
The trick was adding the package name to the module name.
It sure sounds to me like you don't actually want a package. It's OK to just use a single module, if your code just does one main thing and all its parts are closely related. Packages are useful when you have distinct parts of your code that might not all be needed at the same time, or when you have so much code that a single module would be very large and hard to find things in.
Suppose we have the following project structure:
E:\demo_proj\
utilities_package\
__init__.py
util_one.py
util_two.py
demo_package\
__init__.py
demo_sub_package\
__init__.py
demo_sub_sub_package\
demo_file.py
What is a sensible way for demo_file.py to import from utilities_package?
The utilities_package is in a much higher level directory than demo_file.py.
Although it is not shown in the little diagram I gave above, suppose that utilities_package is used everywhere throughout the project. It is not sensible to make utilities_package a sub-directory of demo_sub_sub_package
Some similar questions have been asked here and here.
The asker of the first question neglected to include a __init__.py in their packages. That is not an issue in my case.
The answers given to the the second question linked-to above are problematic:
Most of the answers are quite old. For example, Loader.load_module was deprecated after Python 3.4
Most of the answers are specific to the question in that they only go up one directory, whereas, I would like to go up several levels.
Some of the answers are high-level descriptions without code. Other answers have the opposite problem: they contain ugly-looking code, but don't bother to explain it.
I am open to solutions which require restructuring the project. I'm also interested in solutions which leave the project structure intact, but which insert the relevant code into demo_file.py
It already has the very epic solution for your problems. Please refer there: Relative imports for the billionth time. Just as a notation, the python3 and python2 have the totally different definitions for the relative and absolute import. For version 3+, any modules import directly using import module_name are treated as the absolute importation, which means this module must be included in the sys.path. Otherwise, for any modules import like from one_module import something or from . import something are treated like the relative importation. The dot-notation represent the current directory. However, any module is executed directly by itself, it will lose its namespace, instead it will always be treated as __main__, and all the files in the same directory will automatically add to sys.path by default. Hope it will help you.
I'll first begin by stating that I have been coming back to this problem over and over again for the last several months. No matter how much I research into it, I cannot seem to find a satisfactory answer. I'm hoping that the community here can help me out.
Basic problem - consider this structure of python packages and python modules.
|- src
|- pkg_1
|- foo_1.py
|- foo_2.py
|- __init__.py
|- pkg2
|- bar_1.py
|- bar_2.py
|- __init__.py
|- do_stuff.py
|- __init__.py
Suppose that module bar_2 needs to import from module foo_1. My options are myriad, but a few present themselves quickly.
(My preferred way) Module bar_2 can do from ..pkg_1 import foo_1. This is great because it doesn't require what amounts to hard-coding a path into the module, thereby allowing flexibility, ease of maintenance, all that. In do_stuff.py if I then write from src.pkg_2 import bar_2 and then run, I am golden. Here is an example setup:
foo_1.py:
class Foo_1():
def __init__(self):
print('Hello from foo_1!')
bar_2.py:
from ..pkg_1 import foo_1
class Bar_2():
def __init__(self):
print('Hello from bar_2!')
foo_1.Foo_1() #Prints foo_1 message!
do_stuff.py:
from src.pkg_2 import bar_2
bar_2.Bar_2()
Console prints:
Hello from bar_2!
Hello from foo_1!
All is well. However, consider the next scenario.
Suppose now that I want to run bar_2 as __main__, as follows:
from ..pkg_1 import foo_1
class Bar_2():
def __init__(self):
print('Hello from bar_2!')
foo_1.Foo_1()
if __name__ == '__main__':
Bar_2()
A SystemError is raised: from ..pkg_1 import foo_1
SystemError: Parent module '' not loaded, cannot perform relative import
For far longer than I care to admit, I did not understand the reason for this. The solution, though, lies in the fact that when you run a module directly, its __name__ variable is set to __main__. Since the relative imports establish position in the hierarchy with __name__, this means that there is no directory information to parse to figure things out. This makes loads of sense, and I feel very dumb for having not realized it before.
So, thus began my quest (yeah, just getting started). Eventually I learned of the __package__ variable. Reading about it in the PEP notes, it seemed as though it would solve all my problems! So I tried the following boilerplate code before the import statements in bar_2:
if __name__ == '__main__':
__package__ = 'src.pkg_2'
This did not work. :(
I since have come to find out that Guido has addressed this very issue and that he regards the whole notion of running a module from within a package as anti-pattern.
See this link:
https://mail.python.org/pipermail/python-3000/2007-April/006793.html
This makes sense, as I will be the first to admit that I only do it for on the fly testing... which should never be done! (Right??) Therefore, as best as I understand, there are no elegant ways to run the module from within a package UNLESS you do absolute imports... which I would like to avoid.
So, after all of that, here is my question: Should I use one of the many hacky ways out there to get around this problem and do unholy things with the system path so that I can have my relative import cake and eat it (i.e. test on the fly by running as __main__) too?? I think I already know the answer (I just would like some wizened Yoda-like person to confirm).
Possible Answer:
Use relative imports, because hard-coding paths (and hard-coding in general if avoidable) is ant-pattern.
Do not bother with running modules nested in packages as __main__... instead, run them from your testing module (which you wrote first... right??).
Thank you for taking the time to read this question. I realize there are many other questions regarding this topic... but I wanted to 'share my journey' and see if my current course of action is correct.
Running the following in src/:
python -m pkg2.bar_2
will have pkg2/bar_2.py be your main script while still being inside a package, meaning that relative imports will work.
I follow these rules which prevent any issues:
Always use full imports, no relative imports.
Always start a Python process from the root of the project or using the full absolute path.
It's rather rare that you benefit from relative imports when you have to move files around (and when you do, it's not that much work to rename a few import lines).
Full paths from the root of a project or using the full path on disk of a file removes any ambiguity and will run the desired file.
My answer (and that of many CPython core developers) is essentially the same as Simeon's. The only thing not hard-coded by relative imports is the name of the package ('src', in this case). But you hard-coded it in do_stuff.py anyway. Within-subpackage relative imports (unlike the cross-subpackage imports you exhibit) let you copy the subpackage to another package (with a different name), and possibly change the subpackage name as you do so. Do you really need that particular flexibility? Is it really worth more than the very real cost of relative imports?
Why the subpackages, instead of putting everything in the main package, src? Idlelib has about 60 run-time modules in idlelib/ itself. The only subpackage is idle_test, for test modules. All imports start with idlelib.
I am a big fan of being able to run the test for one module (rather than the package test suite) by running non-cli modules as the main module. It encourages incremental TDD. So I have 'if name.. ' clauses in both run-time and test modules in idlelib.