ImportError with from . import x on simple python files - python-3.x

I wanted to port some code from python 2 to python 3 and it failed on an import error. So I tried to get rid of the porting itself and focus on the import by creating 2 basic python files to test with. However I can't even get those to work.
So I have 2 files
test.py:
print('Test works')
and test2.py:
from . import test
The result however is this error in Pycharm:
ImportError: cannot import name 'test' from '__main__'
(C:/Users/Username/test2.py)
In Ubuntu Shell:
Traceback (most recent call last): File "test2.py", line 1, in
from . import test1 SystemError: Parent module '' not loaded, cannot perform relative import
How can I solve it?

This "folder structure matters" is a large problem in python3. Your folder structure should NOT matter when coding, but should be referenced properly.
I've just resorted to using if/else depending on if ran locally or as part of a module:
if __name__ == "__main__": # Local Run
import test
else: # Module Run, When going production - delete if/else
from . import test

In Python3
test2.py:
import test
test.py:
if __name__ == "__main__":
print('Test works')
If you want to print "Test works" in other file
test2.py:
import test
test.main()
test.py:
def main():
print('Test works')
if __name__ == "__main__":
main()

The folder structure matters. You didn't name your module; I will call it foo. Put things in the right place:
$ mkdir foo
$ touch foo/__init__.py
$ mv /some/place/{test1,test2}.py foo/
Notice that python -c 'import test' works already, even before you wrote your test.py file. To avoid confusion I recommend naming your file test1.py.
The way you call your code matters. Verify that . dot is in sys.path:
$ export PYTHONPATH=.
$ python -m foo.test1
Alternatively this should work, if you prefer:
$ python foo/test1.py
EDIT:
I answered question #1, and OP now asks question #2, about this diagnostic:
ImportError: cannot import name 'test' from 'main' (C:/Users/Username/test2.py)
Please organize your files in the proper structure. You need to put test2.py within a foo/ directory (or whatever you care to call it). You need to create an empty foo/__init__.py file, as that is important for the import machinery.
Also, the PYTHONPATH env var in the calling environment matters. In addition to the command line environment, you have now introduced a PyCharm environment. Take care to configure PyCharm correctly for your project. Click on Preferences -> Project Structure and ensure that foo shows up as a Source Folder. You can debug this in either environment by executing these lines:
import sys
import pprint
pprint.pprint(sys.path)
Since import test would succeed even without your project being in place, I recommend you rename to test1.py and use import test1 so you can be sure you're pulling in your code.

I had the same problem, and ended up removing the "from . " that was added from the 2to3 conversion script.

Related

How to make the path work to run tests outside of pytest

I have a test that passes with pytest. I also have an adjacent source that's a console app (not pytest) that exercises the same function that barfs with ModuleNotFoundError for the same function call.
mylib/mylib.py:
def adder(num1, num2):
return num1 + num2
test/my_test.py:
from mylib.mylib import adder
def test_adder():
assert adder(2, 2) == 4
test/mytest_outside_pytest.py:
from mylib.mylib import adder
print(str(adder(2, 2)))
As I suggest pytest completes and indicates the test passed, but the following does not:
$ python3 test/mytest_outside_pytest.py
Traceback (most recent call last):
File "test/mytest_outside_pytest.py", line 1, in <module>
from mylib.mylib import adder
ModuleNotFoundError: No module named 'mylib'
Nor does:
$ cd test/
$ python3 mytest_outside_pytest.py
Traceback (most recent call last):
File "mytest_outside_pytest.py", line 1, in <module>
from mylib.mylib import adder
ModuleNotFoundError: No module named 'mylib'
I have empty __init__.py in both folders. I don't know what pytest does to path loading that I can't replicate in my "main" console app. Of course I don't really care about an adder lib, I'm just aiming at the smallest reproducible problem.
pytest has a quite complex mechanism to add some directories to the PYTHONPATH and allow tests to run automagically (see the doc)
When you do not use pytest you do not have this mechanism and you have to manage the PYTHONPATH by yourself.
Why the PYTHONPATH is not correct in your case?
When you run a module using python path/to/module.py, the interpreter adds the directory containing the module (absolute path of path/to in this example) in the PYTHONPATH.
In your case, you run python test/mytest_outside_pytest.pyand the interpreter adds the absolute path of test directory in the PYTHONPATH.
You can check it by adding import pdb; pdb.set_trace() at the very beginning of your mytest_outside_pytest.py module to enter a debug session and run it. Then import sys and display the sys.path to see that test is the first directory in your PYTHONPATH.
You can also see that the root directory is not listed and thus the packages it contains cannot be imported.
How to solve it?
It can be done in multiple ways.
You can manually define the PYTHONPATH in your command:
PYTHONPATH=. python test/mytest_outside_pytest.py
or you can use python -m ... as suggested in comment (do not forget that in this case, you have to specify a module, not a path, and remember that it will work only from the directory containng the mylib and test packages):
python -m test.mytest_outside_pytest
In both cases, a debug session can show you that the root directory and the test directory are in the PYTHONPATH

Problem importing python file from folder above

I know theres heaps of questions and answers for this, I tried multitude stackoverflow links but none of these seem to help.
My project structure is:
volume_price_analysis/
README.md
TODO.md
build/
docs/
requirements.txt
setup.py
vpa/
__init__.py
database_worker.py
utils.py
test/
__init__.py
test_utils.py
input/
input_file.txt
I want to load utils.py inside test_utils.py
my test_utils.py is:
import unittest
import logging
import os
from .vpa import utils
class TestUtils(unittest.TestCase):
def test_read_file(self):
input_dir = os.path.join(os.path.join(os.getcwd()+"/test/input"))
file_name = "input_file.txt"
with open(os.path.join(input_dir+"/"+file_name)) as f:
file_contents = f.read()
f.close()
self.assertEqual(file_contents, "Hello World!\n")
if __name__ == '__main__':
unittest.main()
I want to run (say inside test folder):
python3 -m test_utils.py
I can not do that, I get a bunch of errors regarding imports of utils (tried many iterations of . , no ., from this import that etc.. etc..
Why is this so bloody complicated?
I am using Python 3.7 if that helps.
As per this answer, you can do it using importlib,
in spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py") ,instead of path/to/file, you can use ../utils.py. Also, since you are already importing a package named utils (from importlib), you should call one of them by other name, ie. dont keep module.name as utils or import importlib.utils as something else.
I figured it out, turns out python prefers you to run your code from top level folder, in my case volume_price_analysis folder, all I had to do was make a shell script that calls
python3 -m unittest vpa.test.test_utils
And inside test_utils I can import whatever I want as long as I remember that I am executing the code from main folder so loading utils.py would be
from vpa import utils inside test_utils

ModuleNotFoundError: No module named in python 3 even after hacking sys.path()

I have this file structure:
/home/test
├── dirA
│   └── ClassA.py
└── dirB
└── Main.py
With the following content in the files:
ClassA.py:
class ClassA:
def __str__(self):
return 'Hi'
Main.py:
from dirA.ClassA import ClassA
class Main:
def main():
a = ClassA()
if __name__ == '__main__':
Main.main()
I change the current dir to:
$ cd /home/test/dirB
This works:
$ PYTHONPATH=/home/test python Main.py
This doesn't:
$ python Main.py
Traceback (most recent call last):
File "Main.py", line 1, in <module>
from dirA.ClassA import ClassA
ModuleNotFoundError: No module named 'dirA'
Adding this lines in Main.py has no effect:
import os, sys
# Get the top level dir.
path = os.path.dirname(os.path.dirname(__file__))
sys.path.append(path)
The module still can't be found! There are plenty of similar questions but I couldn't make this work programmatically (skipping the PYTHONPATH env var.) I understand that dirs are not modules, files are but this works in PyCharm (is the IDE fixing PYTHONPATH?)
You need to make sure that you've altered your sys.path before you attempt to load any package that might depend on the altered path - otherwise your script will fail the moment it encounters and import statement. In other words, make sure your Main.py begins as:
import os
import sys
path = os.path.join(os.path.dirname(__file__), os.pardir)
sys.path.append(path)
from dirA.ClassA import ClassA
To ensure that the last import statement operates on the altered path.
Thanks for all the help. Per suggestion, I added the appended path as the first statement. That still didn't work. But then I used:
sys.path.append(sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
And this suddenly worked. Still not sure why the abspath makes a difference as the printed path I got was already absolute for __file__.
Building on the answer given by #zwer, we need to check if the script is being run in interactive mode or not. If it is, then we can use the approach mentioned by #zwer. If it is not, we can use the hard coded way (which is not always foolproof.
# https://stackoverflow.com/questions/16771894/python-nameerror-global-name-file-is-not-defined
if '__file__' in vars():
print("We are running the script non interactively")
path = os.path.join(os.path.dirname(__file__), os.pardir)
sys.path.append(path)
else:
print('We are running the script interactively')
sys.path.append("..")

import package from an other package

I have a very simple project architecture:
project/
__init__.py
dira/
__init__.py
a.py
dirb/
__init__.py
b.py
All the __init__.py are empty.
a.py only contains a "hello" function:
def hello(): return "hello world"
I want b.py to use the hello function from a.py.
What I tried so far:
Following answers from this question, I tried:
import ..dira.a as aa
aa.hello
But when running b.py I get a syntax error on ..a.a
I also tried to manually add the project directory to sys.path and then run:
import dira.a as aa
aa.hello
But when running b.py I get:
ImportError: No module named 'a'
What should I do? Why the solution I found doesn't work?
Short answer:
import ..dira is invalid syntax and only from ..dira.a import hello works
Working configuration: b.py:
from ..dira.a import hello
print(hello())
Also, from docs:
"Note that relative imports are based on the name of the current
module. Since the name of the main module is always "main",
modules intended for use as the main module of a Python application
must always use absolute imports."
So if you do relative imports you cannot call b.py directly. Apperently, you cannot do it from project level either. (Python interpreter in project folder, call import dirb.b)
An error occurs:
ValueError: attempted relative import beyond top-level package
But calling import project.dirb.b makes proper output:
hello world
EDIT: Why import project.dirb.b can be called only outside of project module?
Let's look at the problem from interpreter's side.
You call import project.dirb.b. It runs dirb/init.py, 'current module' is project.dirb
Interpreter finds command from ..dira.a import hello
It treats double dot ..dira.a as "move one module up and go to dira and import a.hello". 'Current module' is project.dirb, so parent module is project, thus it finds parent.dira
What happens when you call import dirb.b directly? It fails on trying to go level up in module hierarchy and reports error.
Thus, relative import makes project.dirb.b importable only outside of project.

Nosetests gives ImportError when __init__.py is included (using cython)

I just came across a very strange error while using nose and cython inside my virtualenv with python3. For some reason nosetests started giving me an ImportError even though python -m unittest basic_test.py was working. I made a new directory to reproduce the error to make sure there wasn't something weird in that directory.
Here are the three files: fileA.pyx, setup.py, and basic_test.py
file1.pyx
class FileA:
def __init__(self):
self.temp = {}
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension('fileA', ['fileA.pyx'],)
]
setup(
name='test',
ext_modules=ext_modules,
cmdclass={'build_ext': build_ext},
)
basic_test.py:
def test():
from fileA import FileA
FileA()
assert True
Fresh directory. I run python setup.py build_ext --inplace. It compiles. I run nosetests the single test passes.
Then I do touch __init__.py and then run nosetests again and it fails with this error:
ImportError: No module named 'fileA'
Is this a bug or do I not understand how init affects imports?
Update:
I found this post about import traps and read something that might explain how adding init breaks it. I still don't get exactly how it fits in though.
This is an all new trap added in Python 3.3 as a consequence of fixing
the previous trap: if a subdirectory encountered on sys.path as part
of a package import contains an init.py file, then the Python
interpreter will create a single directory package containing only
modules from that directory, rather than finding all appropriately
named subdirectories as described in the previous section.

Resources