I'm writing a package that wraps PyQt5 functionality and trying to put the documentation on readthedocs. Since PyQt5 is an extension module I mock the module and its classes (manually, because using unittest.mock causes metaclass conflicts):
class PyQt5:
class QtCore:
#staticmethod
def qVersion():
return '5.0.0'
class QObject:
pass
# etc
sys.modules['PyQt5'] = PyQt5
This works fine locally. But although the builds pass without error on readthedocs, there is no autodoc output. What am I missing ?
The project on BitBucket: https://bitbucket.org/fraca7/qtypy/
On ReadTheDocs: https://readthedocs.org/projects/qtypy/
Despite it "passing" the build, if you look carefully at your logs, you will see there are errors like ImportError: No module named 'qtypy' when it starts invoking sphinx.
When I've done this successfully in the past, I've always had a setup.py file at the top level of the repository for installing the package, which I believe is the only way that readthedocs can install the package.
I've then enabled, on readthedocs project admin -> advanced settings,
Install your project inside a virtualenv using setup.py install"
This ensures your module is available to be imported when sphinx runs, so that it can automatically generate the documentation (provided you have successfully mocked PyQt5).
Related
I am having trouble with importing python modules from two git repos as the code was not designed as a package, and there are name collisions which disabled me from being able to use sys.path.append in a straight-forward (albeit hacky?) manner.
Note that the problem is not particular of git-repos however I have encountered this problem only in this scenario because of trying to interface with existing work.
The folder structure looks like following:
project
- repo1
- foo.py
- bar.py
- run.py
- repo2
- foo.py
- bar.py
- run.py
test.py
Both foo.py import bar.py with import bar statements, which works fine as long as you are running the run.py scripts in the repo, however if I try to import repo1.foo and repo2.foo I run into multiple issues.
On attempting the following code
# test.py
import repo1.foo
import repo2.foo
I get the error ModuleNotFoundError: No module named 'bar', this makes some sense as nothing in the path makes either bar.py accessible.
However if I try to append paths 'repo1' and 'repo2' then I can only import either repo1.bar or repo2.bar depending on order of appends, which is not an acceptable solution.
As the repositories are not my code I would prefer to not change them, but instead wrap them into a namespaced package if possible, however I am unable to find any solutions for the problem.
TL;DR
Update:: As I was depending on this work extensively I published it as a Python package available here: https://pypi.org/project/packagify/ . You can install it instead of copying the old gist.
Add the class from this gist: https://gist.github.com/rijulg/3ea372bef35adb68e27080c949c942af in your code and you can use it as following
from packagify import Packagify
package = Packagify("/home/workspace/my_package")
object = package.import_module("module", ["object"])
object1, object2 = package.import_module("module", ["object1", "object2"])
Methods Tried
Installing the directory as a module using setuptools, this did not work as on loading the module various parts inside it threw ModuleNotFoundError as they were trying to access siblings without namespaces.
Various importlib functions, there are a few that allow loading python modules from .py files directly. This might have worked if the modules I wanted to import were contained within a file, but then perhaps there would be other solutions as well. As the problem stood I wasn't able to use this method as the module upon loading wasn't able to locate it's siblings.
Adding sys.path as mentioned in the question, but I also tried appending the path, loading module and then removing the path before loading the other module. This also did not work because presumably the modules were loaded into python cache and subsequent loads from different directory were ignored because of name clashes.
Finally, I noticed something weird happening when I changed the import level using the __import__ function and I tried the following snippet which worked fine on my toy example. However this still had various issues that I encountered while trying to load the actual repos I wanted to work with. As the solution appeared to be going in the right direction I made the class linked in the gist mentioned a the beginning of the post.
import builtins
original_import = __import__
def my_import(name, globals, locals, fromlist, level):
print("my_import", name, globals['__package__'], level)
try:
return original_import(name, globals, locals, fromlist, level)
except:
level = level + 1 if globals['__package__'] is not None else level
return original_import(name, globals, locals, fromlist, level)
builtins.__import__ = my_import
from repo1.foo import main as main_a
from repo2.foo import main as main_b
main_a()
main_b()
I have a directory structure like below:
chatbot/
__init__.py
utils/
__init__.py
parser.py
nlu/
__init__.py
training/
__init__.py
module.py
I want to access parser.py from module.py.
I tried using this line from module.py:
from chatbot.utils import parser
And I got this error:
ModuleNotFoundError: No module named 'chatbot'
Any pointers to what I am doing wrong?
I am using python3 and trying to run the script as python3 nlu/training/module.py.
Thanks in advance!
I believe the right way of solving such problems is:
figure out what your top-level packages and modules are and in what directory they are
change to that directory
make all your imports absolute, i.e. always start with a top-level module or package (instead of things like from . import blah)
use the executable module (python -m) way of running your code, for example path/to/pythonX.Y -m top_level_package.executable_module (instead of path/to/pythonX.Y top_level_package/executable_module.py)
In case the top-level modules or packages are not all in the same directory:
Either work on packaging your libraries or applications correctly and install them in the site packages (eventually as editable, also known as develop mode).
Or, as a last resort, collect the other directories containing the top-level packages and modules in the PYTHONPATH environment variable like the following:
PYTHONPATH=path/to/alpha:path/to/bravo path/to/pythonX.Y -m top_level_package.executable_module
I was able to solve the issue by adding the parent directory of chatbot package to sys.path.
Say if chatbot python package is in /home/my_project/chatbot/, I added a statement like:
sys.path.append('/home/my_project') in module.py
This made all top-level python packages visible to all lower-level python packages.
I'm in the process of learning how to package a python library using the official guide. I've started cloning the minimal sample package suggested in the guide here. I've then added the file my_module.py inside the folder sampleproject storing a simple power function. Another function is also stored in /sampleproject/sampleproject/__init__.py. The resulting structure of the library is the following
Finally, I've used pip to successfully install the package in the interpreter. The only thing left is to make sure that I'm able to run the functions stored in the subfolder sampleproject.
import sampleproject
sampleproject.main()
# Output
"Call your main application code here"
This is great. The package is able to run the function in __init__.py. However, the package is not able to find module.py:
import sampleproject
sampleproject.module
# Output
AttributeError: module 'sampleproject' has no attribute 'module'
I've tried to add __init__.py in the main folder and to change the settings in entry_points in setup.py without success. What should I let sampleproject to be able to find the function in module.py?
Your sampleproject.module is a function you would like to execute?
In this case, do as for the sampleproject, add () to execute it:
sampleproject.module()
Otherwise, you can import your package like this:
import sampleproject.module
or:
from sampleproject import module
To be clearer, you would have to import module in your sampleproject __init__.py. Then, when you want to use the package, import it (is some py file at root):
import sampleproject # is enough as it's going to import everything you stated in __init__.py
After that, you can start to use what's in the package you imported with maybe module() if you have a function called module in your package.
init.py discussions
it seems,
you are in sampleproject->module.py
so you need to try,
from sampleproject import module
Currently trying to work in Python3 and use absolute imports to import one module into another but I get the error ModuleNotFoundError: No module named '__main__.moduleB'; '__main__' is not a package. Consider this project structure:
proj
__init__.py3 (empty)
moduleA.py3
moduleB.py3
moduleA.py3
from .moduleB import ModuleB
ModuleB.hello()
moduleB.py3
class ModuleB:
def hello():
print("hello world")
Then running python3 moduleA.py3 gives the error. What needs to be changed here?
.moduleB is a relative import. Relative only works when the parent module is imported or loaded first. That means you need to have proj imported somewhere in your current runtime environment. When you are are using command python3 moduleA.py3, it is getting no chance to import parent module. You can:
from proj.moduleB import moduleB OR
You can create another script, let's say run.py, to invoke from proj import moduleA
Good luck with your journey to the awesome land of Python.
Foreword
I'm developing a project which in fact is a Python package that can be installed through pip, but it also exposes a command line interface. I don't have problems running my project after installing it with pip install ., but hey, who does this every time after changing something in one of the project files? I needed to run the whole thing through simple python mypackage/main.py.
/my-project
- README.md
- setup.py
/mypackage
- __init__.py
- main.py
- common.py
The different faces of the same problem
I tried importing a few functions in main.py from my common.py module. I tried different configurations that gave different errors, and I want to share with you with my observations and leave a quick note for future me as well.
Relative import
The first what I tried was a relative import:
from .common import my_func
I ran my application with simple: python mypackage/main.py. Unfortunately this gave the following error:
ModuleNotFoundError: No module named '__main__.common'; '__main__' is not a package
The cause of this problem is that the main.py was executed directly by python command, thus becoming the main module named __main__. If we connect this information with the relative import we used, we get what we have in the error message: __main__.common. This is explained in the Python documentation:
Note that relative imports are based on the name of the current module. Since the name of the main module is always __main__, modules intended for use as the main module of a Python application must always use absolute imports.
When I installed my package with pip install . and then ran it, it worked perfectly fine. I was also able to import mypackage.main module in a Python console. So it looks like there's a problem only with running it directly.
Absolute import
Let's follow the advise from the documentation and change the import statement to something different:
from common import my_func
If we now try to run this as before: python mypackage/main.py, then it works as expected! But, there's a caveat when you, like me, develop something that need to work as a standalone command line tool after installing it with pip. I installed my package with pip install . and then tried to run it...
ModuleNotFoundError: No module named 'common'
What's worse, when I opened a Python console, and tried to import the main module manually (import mypackage.main), then I got the same error as above. The reason for that is simple: common is no longer a relative import, so Python tries to find it in installed packages. We don't have such package, that's why it fails.
The solution with an absolute import works well only when you create a typical Python app that is executed with a python command.
Import with a package name
There is also a third possibility to import the common module:
from mypackage.common import my_func
This is not very different from the relative import approach, as long as we do it from the context of mypackage. And again, trying to run this with python mypackage/main.py ends similar:
ModuleNotFoundError: No module named 'mypackage'
How irritating that could be, the interpreter is right, you don't have such package installed.
The solution
For simple Python apps
Just use absolute imports (without the dot), and everything will be fine.
For installable Python apps in development
Use relative imports, or imports with a package name on the beginning, because you need them like this when your app is installed. When it comes to running such module in development, Python can be executed with the -m option:
-m mod : run library module as a script (terminates option list)
So instead of python mypackage/main.py, do it like this: python -m mypackage.main.
In addition to md-sabuj-sarker's answer, there is a really good example in the Python modules documentation.
This is what the docs say about intra-package-references:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
If you run python3 moduleA.py3, moduleA is used as the main module, so using the absolute import looks like the right thing to do.
However, beware that this absolute import (from package.module import something) fails if, for some reason, the package contains a module file with the same name as the package (at least, on my Python 3.7). So, for example, it would fail if you have (using the OP's example):
proj/
__init__.py (empty)
proj.py (same name as package)
moduleA.py
moduleB.py
in which case you would get:
ModuleNotFoundError: No module named 'proj.moduleB'; 'proj' is not a package
Alternatively, you could remove the . in from .moduleB import, as suggested here and here, which seems to work, although my PyCharm (2018.2.4) marks this as an "Unresolved reference" and fails to autocomplete.
Maybe you can do this before importing the moduleļ¼
moduleA.py3
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from moduleB import ModuleB
ModuleB.hello()
Add the current directory to the environment directory
Just rename the file from where you run the app to main.py:
from app import app
if __name__ == '__main__':
app.run()
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
will solve the issue of import path issue.
I froze a python 3.6 program with cx_freeze, and it worked just fine. But as soon as I tried to run it, I got this error message.
Does anyone know what to do? Please help!
try using
import tkinter
(small 't' instead of capital)
Try checking the dir names in the 'lib' folder (in my case it was 'build\exe.win-amd64-3.6\lib'). I had a similar issue (without the 'ImportError: DLL load failed...') and found that the 'lib' directory contained a "Tkinter" folder. Changing its name to lowercase 'tkinter' did the trick and made the .exe run just fine.
If you read the error more clearly, it stated in a comment that if importing _tkinter failed, your computer isn't configured to use tkinter. You should download tk and ttk to make it work.
And According to http://wiki.python.org/moin/TkInter :
If it fails with "No module named _tkinter" or "Tkinter", your Python configuration
needs to be modified to include this module (which is an extension
module implemented in C). Do not edit Modules/Setup (it is out of
date). You may have to install Tcl and Tk (when using RPM, install the
-devel RPMs as well) and/or edit the setup.py script to point to the right locations where Tcl/Tk is installed. If you install Tcl/Tk in
the default locations, simply rerunning "make" should build the
_tkinter extension.