I just came across a very strange error while using nose and cython inside my virtualenv with python3. For some reason nosetests started giving me an ImportError even though python -m unittest basic_test.py was working. I made a new directory to reproduce the error to make sure there wasn't something weird in that directory.
Here are the three files: fileA.pyx, setup.py, and basic_test.py
file1.pyx
class FileA:
def __init__(self):
self.temp = {}
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension('fileA', ['fileA.pyx'],)
]
setup(
name='test',
ext_modules=ext_modules,
cmdclass={'build_ext': build_ext},
)
basic_test.py:
def test():
from fileA import FileA
FileA()
assert True
Fresh directory. I run python setup.py build_ext --inplace. It compiles. I run nosetests the single test passes.
Then I do touch __init__.py and then run nosetests again and it fails with this error:
ImportError: No module named 'fileA'
Is this a bug or do I not understand how init affects imports?
Update:
I found this post about import traps and read something that might explain how adding init breaks it. I still don't get exactly how it fits in though.
This is an all new trap added in Python 3.3 as a consequence of fixing
the previous trap: if a subdirectory encountered on sys.path as part
of a package import contains an init.py file, then the Python
interpreter will create a single directory package containing only
modules from that directory, rather than finding all appropriately
named subdirectories as described in the previous section.
Related
I'm trying to create an EXE from a Python project I built and I'm running into some issues. I've done this before with PyInstaller for simpler tools and cx-freeze for tools where I use custom modules but the way I used to set it up doesn't seem to be working anymore.
In this particular case I've tried:
cx-freeze==6.10
pyinstaller==4.10
py2exe==0.11.1.0
auto-py-to-exe==2.18.0
and a few others with no luck
Everything works perfectly fine through the Python 3.8 interpreter. I'm assuming it's either because of the way I'm importing PySide2 here (which I don't normally do but did for this project to see if it would speed up my programming time) or that the EXE modules can't find my custom modules. Here is a mock version of my program (names/paths changed for simplicity):
Example folder of my project ("C:\a\MyProjects\Project1"):
Example folder of the custom module I'm using. Each using various other built-in and/or 3rd party python modules. ("C:\a\path\to\external\modules"):
Example of my main file (C:\a\MyProjects\Project1\ui.py) I want to turn into an EXE:
import os
import sys
import colorsys
from PySide2.QtWidgets import *
from PySide2.QtCore import *
from PySide2.QtGui import *
import utils # module in project ("C:\a\MyProjects\Project1\utils.py")
sys.path.append(r"C:\a\path\to\external\modules") # custom module location for MyModule
from MyModule.foo import module1 as foo_mod1
from MyModule.foo import module2 as foo_mod2
from MyModule.bar import module1 as bar_mod1
from MyModule.bar import module2 as bar_mod2
from MyModule.baz import module1 as baz_mod1
from MyModule.baz import module2 as baz_mod2
class MainDialog(QDialog):
[...code...]
[...use of "dark.stylesheet"...]
[...use of "images\image 1.png"...]
[...use of "images\image 2.png"...]
def main():
global win
try: # try to close existing instances
win.close()
except NameError:
pass
win = MainDialog()
win.show()
if __name__ == "__main__":
app = QApplication.instance()
if not app:
app = QApplication(sys.argv)
main()
sys.exit(app.exec_())
Can someone tell me the best module and method to make this into a windowed (console-less), if possible, single file application. This is one of the more complicated tools I've had to make into a desktop app so if I can figure this out I should be good for most of the other things I have to do.
Thanks in advance.
EDIT:
Example of cx-freeze setup.py file in project I tried (C:\a\MyProjects\Project1\setup.py):
from cx_Freeze import setup, Executable
import sys
setup(
name="MyProject",
version="1.0",
options={'build_exe': {
'excludes': ["Tkinter"],
'includes': ["utils", "MyModule.foo", "MyModule.bar", "MyModule.baz"],
'path': sys.path + [r"C:\a\path\to\external\modules"],
'include_files': ["images\image 1.png", "images\image 2.png"],
'packages': ["PySide2.QtWidgets", "PySide2.QtCore", "PySide2.QtGui"]}},
executables=[Executable("ui.py", base="Win32GUI")]
Example of pyinstaller cmd command I tried:
cd C:\\a\MyProjects\Project1
pyinstaller -p "C:\a\path\to\external\modules" --onefile -w ui.py
# even tried this with a custom spec file like:
# pyinstaller --onefile -w ui.spec
There is a command in PyInstaller to include hidden modules call --hidden-module (You can use multile times). So, you can just do
pyinstaller -p "C:\a\path\to\external\modules" --onefile -w --hidden-module="example module 1" -hidden-module="example module 2" ui.py
This seems to have fixed itself after I got a new workstation and installed the latest Python (3.10) and Pyinstaller (5.1) versions.
Also think having your custom Python Libraries in the PYTHONPATH and/or PATH Environment variables may be important for allowing Pyinstaller to find them.
Trying to write tests in Python3 with the folder structure
package/
__init.py__
Foo.py
Bar.py
test/
__init.py__
test_Foo.py
Foo.py
import Bar
test_Foo.py
import Foo
Running python3 -m unittest results in No module named Foo
If I changed it to
from package import Foo
It will resolve, but in Foo.py it'll have the error No module named 'Bar'
If I change Foo.py to
from package import Bar
The test will run without error, but running python3 Foo.py will result in No module named 'package'
I've gone through various answers about this kind of problems, like this one
Running unittest with typical test directory structure
and this
Python3 import modules from folder to another folder
and none of them address this issue
your project root (and also the working directory!) need to be the parent of package.
Then every import will be in the form import package.module or from package import module.
if you run python packgae/Foo.py or python -m package.Foo it should work.
Also take a look at pypa - sample project to see the recommended Python project structure.
The goal is to use the pytest unit test framework for a Python3 project that uses Cython. This is not a plug-and-play thing, because pytest by default is not able to import the Cython modules.
One unsuccessful solution would be to use the pytest-cython plugin, but it simply does not work for me:
> py.test --doctest-cython
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --doctest-cython
inifile: None
rootdir: /censored/path/to/my/project/dir
To verify that I have the package installed:
> pip freeze | grep pytest-cython
pytest-cython==0.1.0
UPDATE:
I'm using PyCharm and it seems that it is not using my pip-installed packages but rather uses a custom(?) pycharm repository for packages used by my project. Once I added pytest-cython to that repository, the command runs but strange enough it doesn't recognize the Cython module anyway, although the package/add-on is specifically designed for that purpose:
> pytest --doctest-cython
Traceback:
tests/test_prism.py:2: in <module>
from cpc_naive.prism import readSequence, processInput
cpc_naive/prism.py:5: in <module>
from calculateScore import calculateScore, filterSortAlphas,
calculateAlphaMatrix_c#, incrementOverlapRanges # cython code
E ImportError: No module named 'calculateScore'
Another unsuccessful solution I got here is to use pytest-runner, but this yields:
> python3 setup.py pytest
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'pytest'
UPDATE:
I first had forgotten to add setup_requires=['pytest-runner', ...] and tests_require=['pytest', ...] to the setup script. Once i did that, I got another error:
> python3 setup.py pytest
Traceback (most recent call last):
File "setup.py", line 42, in <module>
tests_require=['pytest']
(...)
AttributeError: type object 'test' has no attribute 'install_dists'
UPDATE 2 (setup.py):
from distutils.core import setup
from distutils.extension import Extension
from setuptools import find_packages
from Cython.Build import cythonize
import numpy
try: # try to build the .c file
from Cython.Distutils import build_ext
except ImportError: # if the end-user doesn't have Cython that's OK; you should have shipped the .c files anyway.
use_cython = False
else:
use_cython = True
cmdclass = {}
ext_modules = []
if use_cython:
ext_modules += [
Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.pyx"],
extra_compile_args=['-g'], # -g for debugging
define_macros=[('CYTHON_TRACE', '1')]),
]
cmdclass.update({'build_ext': build_ext})
else:
ext_modules += [
Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.c"],
define_macros=[('CYTHON_TRACE', '1')]), # compiled C files are stored in /home/pdiracdelta/.pyxbld/
]
setup(
name='cpc_naive',
author=censored,
author_email=censored,
license=censored,
packages=find_packages(),
cmdclass=cmdclass,
ext_modules=ext_modules,
install_requires=['Cython', 'numpy'],
include_dirs=[numpy.get_include()],
setup_requires=['pytest-runner'],
tests_require=['pytest']
)
UPDATE 3 (partial fix):
As suggested by #hoefling I downgraded pytest-runner to a version <4 (in fact 3.0.1) and this resolves the error in update 1, but now I get the same Exception as with the pytest-cython solution:
E ImportError: No module named 'calculateScore'
It just doesn't seem to recognize the module. Perhaps this is due to some absolute/relative import mojo I don't understand.
How can I use pytest with Cython? How can I discover why these methods aren't working and then fix it?
FINAL UPDATE:
After taking both the original problem and the question Updates into consideration (thanks #hoefling for solving these issues!), this question is now reduced to the question of:
why can pytest no import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine?
As #hoefling suggested, one should use pytest-runner version <0.4 to avoid the
AttributeError: type object 'test' has no attribute 'install_dists'
To then answer the actual and final question (in addition to partial, off-topic, user-specific fixes added to the question post itself) of why pytest cannot import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine:
that remaining issue is solved here.
Currently trying to work in Python3 and use absolute imports to import one module into another but I get the error ModuleNotFoundError: No module named '__main__.moduleB'; '__main__' is not a package. Consider this project structure:
proj
__init__.py3 (empty)
moduleA.py3
moduleB.py3
moduleA.py3
from .moduleB import ModuleB
ModuleB.hello()
moduleB.py3
class ModuleB:
def hello():
print("hello world")
Then running python3 moduleA.py3 gives the error. What needs to be changed here?
.moduleB is a relative import. Relative only works when the parent module is imported or loaded first. That means you need to have proj imported somewhere in your current runtime environment. When you are are using command python3 moduleA.py3, it is getting no chance to import parent module. You can:
from proj.moduleB import moduleB OR
You can create another script, let's say run.py, to invoke from proj import moduleA
Good luck with your journey to the awesome land of Python.
Foreword
I'm developing a project which in fact is a Python package that can be installed through pip, but it also exposes a command line interface. I don't have problems running my project after installing it with pip install ., but hey, who does this every time after changing something in one of the project files? I needed to run the whole thing through simple python mypackage/main.py.
/my-project
- README.md
- setup.py
/mypackage
- __init__.py
- main.py
- common.py
The different faces of the same problem
I tried importing a few functions in main.py from my common.py module. I tried different configurations that gave different errors, and I want to share with you with my observations and leave a quick note for future me as well.
Relative import
The first what I tried was a relative import:
from .common import my_func
I ran my application with simple: python mypackage/main.py. Unfortunately this gave the following error:
ModuleNotFoundError: No module named '__main__.common'; '__main__' is not a package
The cause of this problem is that the main.py was executed directly by python command, thus becoming the main module named __main__. If we connect this information with the relative import we used, we get what we have in the error message: __main__.common. This is explained in the Python documentation:
Note that relative imports are based on the name of the current module. Since the name of the main module is always __main__, modules intended for use as the main module of a Python application must always use absolute imports.
When I installed my package with pip install . and then ran it, it worked perfectly fine. I was also able to import mypackage.main module in a Python console. So it looks like there's a problem only with running it directly.
Absolute import
Let's follow the advise from the documentation and change the import statement to something different:
from common import my_func
If we now try to run this as before: python mypackage/main.py, then it works as expected! But, there's a caveat when you, like me, develop something that need to work as a standalone command line tool after installing it with pip. I installed my package with pip install . and then tried to run it...
ModuleNotFoundError: No module named 'common'
What's worse, when I opened a Python console, and tried to import the main module manually (import mypackage.main), then I got the same error as above. The reason for that is simple: common is no longer a relative import, so Python tries to find it in installed packages. We don't have such package, that's why it fails.
The solution with an absolute import works well only when you create a typical Python app that is executed with a python command.
Import with a package name
There is also a third possibility to import the common module:
from mypackage.common import my_func
This is not very different from the relative import approach, as long as we do it from the context of mypackage. And again, trying to run this with python mypackage/main.py ends similar:
ModuleNotFoundError: No module named 'mypackage'
How irritating that could be, the interpreter is right, you don't have such package installed.
The solution
For simple Python apps
Just use absolute imports (without the dot), and everything will be fine.
For installable Python apps in development
Use relative imports, or imports with a package name on the beginning, because you need them like this when your app is installed. When it comes to running such module in development, Python can be executed with the -m option:
-m mod : run library module as a script (terminates option list)
So instead of python mypackage/main.py, do it like this: python -m mypackage.main.
In addition to md-sabuj-sarker's answer, there is a really good example in the Python modules documentation.
This is what the docs say about intra-package-references:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
If you run python3 moduleA.py3, moduleA is used as the main module, so using the absolute import looks like the right thing to do.
However, beware that this absolute import (from package.module import something) fails if, for some reason, the package contains a module file with the same name as the package (at least, on my Python 3.7). So, for example, it would fail if you have (using the OP's example):
proj/
__init__.py (empty)
proj.py (same name as package)
moduleA.py
moduleB.py
in which case you would get:
ModuleNotFoundError: No module named 'proj.moduleB'; 'proj' is not a package
Alternatively, you could remove the . in from .moduleB import, as suggested here and here, which seems to work, although my PyCharm (2018.2.4) marks this as an "Unresolved reference" and fails to autocomplete.
Maybe you can do this before importing the moduleļ¼
moduleA.py3
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from moduleB import ModuleB
ModuleB.hello()
Add the current directory to the environment directory
Just rename the file from where you run the app to main.py:
from app import app
if __name__ == '__main__':
app.run()
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
will solve the issue of import path issue.
I tried to run a python code, say myfile.py (also tried to rename it as myfile.pyx) as follows:
import pyximport
pyximport.install(setup_args={"script_args":["--compiler=mingw32"]},
reload_support=True)
import myfile
myfile.mycode()
I am using PyCharm. The code seems to have run fine without any error and even gave me correct results on the Python Console within PyCharm.
However no pyd (or pxd) files were generated. How can I know if my code (myfile.mycode()) ran via Cython or via regular Python?
I am using Python 3.4, Cython 0.21.2.
Thanks
pyximport generates a temporary pyd file that is not in the working directory. You probably want to build a setup.py that looks something like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension('myfile',
sources=['myfile.pyx'],
language='c++',
)]
setup(
name = 'myfile',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
which you can compile using:
python setup.py build_ext -i clean