I'm trying to package a module built with pybind11, which has a stub .pyi file associated with it.
The project structure is something like this:
my-pkg/
setup.py
my_pkg.pyi
my_pkg/
src/
my_cpp_files.cpp
The setup.py file for the package looks something like this
# setup.py
from glob import glob
from setuptools import setup
from pybind11.setup_helpers import Pybind11Extension
ext_modules = [
Pybind11Extension("my_pkg", sorted(glob("src/*.cpp")), include_dirs=['./'], cxx_std=17),
]
setup(
name='my-pkg',
description='This is my package',
version='1.0.0',
ext_modules=ext_modules,
)
This successfully installs the my_pkg module into site-packages like so
site-packages/
my_pkg-1.0.0.dist-info/
my_pkg.cpython-38-x86_64-linux-gnu.so
However the my_pkg.pyi is not in included. I have manually copy and pasted the my_pkg.pyi stub file into site-packages directory to see if my IDE (vscode) recognises it, and it does.
Therefore I need a way for setup.py to install my_pkg.pyi into site-packages.
I have tried including the following lines into the setup() call without success:
data_files=[('.', ['my_pkg.pyi'])]
package_data={'.': ['my_pkg.pyi']}
package_data={'my_pkg': ['my_pkg.pyi']}
Related
I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them.
Example : from src.kuzzle import KuzzleQuery
Structure:
main.py
src
kuzzle.py
I have imported my project folder in data storage and when i refreshed the UI of airflow composer i've got this error:
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/airflow/gcs/dags/quality-control/main.py", line 8, in <module>
from src.kuzzle import KuzzleQuery, Tag
ModuleNotFoundError: No module named 'src' ```
Try adding an __init__.py file inside src directory to make it a module.
GCP Composer is loading the dags folder as a python module, so you can just put your code there in a separate file or folder and it will work as usual referencing it from the dags.
dags/
dag1.py
util.py
Then you can reference util and import it from dag1 as:
from util import custom_function
If dags folder does not work for you composer also loads more directories like the ones for loading modules.
You can also check this question since it is also related.
Airflow dag dependencies not available to dags when running Google's Cloud Compose
You could check these options:
The path of the module is not correct
Probably, you would want to import a module file, but this module is not in the same directory.
For example:
Project Structure
core.py
folder_1
---module.py
We want to import the module.py
Core.py
import module.py #incorrect
Output:
ModuleNotFoundError: No module named 'module'
Core.py
import folder_1.module.py #correct
Output:
...Program finished with exit code 0
The library not Installed
If you want to import a module of a library which is not installed in your virtual environment before importing a library's module, you need to install it with the pip command. You could see this documentation:
For example, import Beautifulsoup4 library it is not installed
>>> from bs4 import BeautifulSoup
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'bs4'
Now, let's install the library and try to re-import it
Installing:
pip install beautifulsoup4
Collecting beautifulsoup4
Using cached https://files.pythonhosted.org/packages/d1/41/e6495bd7d3781cee623ce23ea6ac73282a373088fcd0ddc809a047b18eae/beautifulsoup4-4.9.3-py3-none-any.whl
Requirement already satisfied: soupsieve>1.2; python_version >= "3.0" in /home/py/Desktop/seo_pro/seo_env/lib/python3.6/site-packages (from beautifulsoup4) (1.9.5)
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.9.3
Re-importing:
>>> from bs4 import BeautifulSoup
>>>
Setup your files as packages and modules
You need to properly set up your files as packages and modules.
Packages are a way of structuring Python’s module namespace by using “dotted module names”. For example, the module name A.B designates a submodule named B in a package named A. Just like the use of modules saves the authors of different modules from having to worry about each other’s global variable names, the use of dotted module names saves the authors of multi-module packages like NumPy or Pillow from having to worry about each other’s module names.
sound/ Top-level package
__init__.py Initialize the sound package
formats/ Subpackage for file format conversions
__init__.py
wavread.py
wavwrite.py
aiffread.py
aiffwrite.py
auread.py
auwrite.py
...
effects/ Subpackage for sound effects
__init__.py
echo.py
surround.py
reverse.py
...
filters/ Subpackage for filters
__init__.py
equalizer.py
vocoder.py
karaoke.py
You could see more information.
I'm creating a python 3.9 program and want to install packages locally. So the way my project is set up is this:
__main__.py
test.py
requirements.txt
lib/
__init__.py
In my requirements.txt file I have 3 lines:
colorama==0.2.2
click==8.0.3
pendulum==2.1.2
Then I run: python -m pip install -r requirements.txt -t ./lib
This installs all the packages and dependencies inside of the lib directory.
Then I import the modules at the top of my test.py file:
from lib import colorama
from lib import click
from lib import pendulum
In doing some testing, I've found that colorama works fine. I'll use it in a simple test:
print(colorama.Fore.BLUE + "Hello, World!"). The text is blue in the console and everything is working.
I then try to use the other packages and I get ModuleNotFoundError exception:
print(pendulum.now('Europe/Paris'))
Exception has occurred: ModuleNotFoundError - No module named 'pendulum'
This is coming from one of its own files.
The same thing happens when I use Click, but it's a little different. I'll get the same ModuleNotFound exception, but it's for its own dependency on Colorama. I don't think it's related to the fact that I'm also importing Colorama because if I uninstall I get the same error.
I've also tried this with the python-docx package. I added python-docx==0.8.11 to the requirements.txt file, then issued the same command as above to install to my local lib directory. It seems to install fine. I see the docx directory and all its dependencies. Then I import from lib import docx then do something simple in test.py:
doc = docx.Document()
Then get ModuleNotFound error: File "C:\Users\name\Development\python\test-local-package\lib\docx_init_.py", line 3, in (Current frame) No Module named 'docx'
Does anyone know what I'm doing wrong?
When you put those libraries into your lib folder and import them the way you are doing, you're changing their package names. No longer is colorama a top-level package, it's now lib.colorama. Some libraries might be fine with that, but for others, they expect to be able to import their own code using their normal names. If colorama.some_submodule tries to import colorama, it will fail.
It's important to realize that a statement like from lib import colorama doesn't change how colorama can be found everywhere. It only changes the local namespace. The package is still lib.colorama, we've just bound it to the name colorama in the current module.
As JonSG has suggested in comments, a better solution is to put the lib folder into the Python search path so that import colorama will find the package with its normal name. Modifying sys.path is one way to do that, another is the PYTHONPATH environment variable (probably not ideal for your current issue, but sometimes useful in other situations).
I have a project such as:
myproject
setup.py
-myproject
-package1
-package2
I am using a setup.py as:
NAME='myproject'
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
long_description=long_description,
long_description_content_type='text/markdown',
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
packages=find_packages(exclude=('tests',)),
package_data={NAME: ['VERSION']},
install_requires=require(),
extras_require={},
include_package_data=True)
When I install (pip install -e .) this I can access the packages as import myproject.package1. However, I want to change this so I instead import it as import mynewname.package1. In the example above when changing NAME=mynewname and then installing, the packages become no longer visible, and import mynewname gives a ModuleNotFoundError.
I don't want to change the name of the project or structure, just the top level name under which the package is installed. Something like import mynewname.myproject.package1 would also work, but i'm unsure of how to do this.
Thanks
I installed recordclass via pip3 install recordclass which installs it under /Users/timothee/homebrew/lib/python3.7/site-packages/recordclass; I'm trying to use it from a sublimetext package but it fails:
import sys
sys.path.append('/Users/timothee/homebrew/lib/python3.7/site-packages')
import recordclass
ImportError: No module named 'recordclass.mutabletuple'
If I instead use sys.path.append('/Users/timothee/homebrew/lib/python3.7/site-packages/recordclass') I get: SystemError: Parent module '' not loaded, cannot perform relative import
Note that it does work for some other pip3 installed modules, eg jstyleson, simplejson.
Here is contents of recordclass:
\ls
__init__.py arrayclass.py datatype.py mutabletuple.cpython-37m-darwin.so recordobject.cpython-37m-darwin.so test utils.py
__pycache__ dataobject.cpython-37m-darwin.so litelist.cpython-37m-darwin.so recordclass.py structclass.py typing
Not sure whether the reason is that sublime bundles its own python interpreter (3.3 IIRC) and my system python is 3.7, and it installs as recordobject.cpython-37m-darwin.so so can't be seen by sublime's python; but temporarily renaming to recordobject.cpython-33m-darwin.so didn't help.
Note: I've read other similar questions but these didn't help, eg:
* Install python module without PIP => not relevant; I can install it; just not load it from sublime
* Installing python modules like "Web3" without pip/pip3? ditto
* https://github.com/SublimeText/UnitTesting/issues/67
This is the most relevant post: Sublime Plugin: How can I import wx? but the solution there doesn't seem to work; since the problem seems to be these darwin.so files in my case
I just came across a very strange error while using nose and cython inside my virtualenv with python3. For some reason nosetests started giving me an ImportError even though python -m unittest basic_test.py was working. I made a new directory to reproduce the error to make sure there wasn't something weird in that directory.
Here are the three files: fileA.pyx, setup.py, and basic_test.py
file1.pyx
class FileA:
def __init__(self):
self.temp = {}
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension('fileA', ['fileA.pyx'],)
]
setup(
name='test',
ext_modules=ext_modules,
cmdclass={'build_ext': build_ext},
)
basic_test.py:
def test():
from fileA import FileA
FileA()
assert True
Fresh directory. I run python setup.py build_ext --inplace. It compiles. I run nosetests the single test passes.
Then I do touch __init__.py and then run nosetests again and it fails with this error:
ImportError: No module named 'fileA'
Is this a bug or do I not understand how init affects imports?
Update:
I found this post about import traps and read something that might explain how adding init breaks it. I still don't get exactly how it fits in though.
This is an all new trap added in Python 3.3 as a consequence of fixing
the previous trap: if a subdirectory encountered on sys.path as part
of a package import contains an init.py file, then the Python
interpreter will create a single directory package containing only
modules from that directory, rather than finding all appropriately
named subdirectories as described in the previous section.