I'm trying to configure my poetry pyproject.toml in a way that we can choose between a cpu and a gpu version of the dependencies. The reason for that is that the library torch needs to be installed by specifying the hardware you want to use (see poetry's Instructions for installing PyTorch). My final goal is to be able to install the dependencies on my local computer (Mac with M1 ship) as well as to be able to generate a wheel to send to some distant server and then pip install it (I'm actually using Databricks dbx to do that). Here is the configuration file:
[tool.poetry]
name = "my_project"
version = "0.1.0"
description = ""
authors = ["Me"]
readme = "README.md"
packages = [
{include = "my_project"},
{include = "scripts"}
]
[tool.poetry.dependencies]
python = "3.8.10"
torch = [
{url = "https://download.pytorch.org/whl/cpu/torch-1.12.1-cp38-none-macosx_11_0_arm64.whl", markers="extra == 'cpu'"},
{url = "https://download.pytorch.org/whl/cu102/torch-1.12.1%2Bcu102-cp38-cp38-linux_x86_64.whl", markers="extra == 'gpu'"}
]
[tool.poetry.extras]
cpu=["torch"]
gpu=["torch"]
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
When I run:
poetry install --extras="cpu"
I get the following error:
Because my_project depends on both torch (1.12.1) # https://download.pytorch.org/whl/cpu/torch-1.12.1-cp38-none-macosx_11_0_arm64.whl and torch (1.12.1+cu102) # https://download.pytorch.org/whl/cu102/torch-1.12.1%2Bcu102-cp38-cp38-linux_x86_64.whl, version solving failed.
Related
I'm trying to learn how to package scripts with poetry to be later installed via pip.
Here's a sample minimal pyproject.toml
name = "scripttest"
version = "0.1.0"
description = ""
authors = []
packages = [
{ include = "src" }
]
[tool.poetry.dependencies]
python = "^3.10"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.poetry.scripts]
scripttest = 'src.scripttest:main'
Repository has following structure
├── pyproject.toml
├── README.md
└── src
├── importantimport.py
└── scripttest.py
scripttest.py only has an import and a print statement
from importantimport import data
def main():
print(data)
while importantimport.py contains a list inside
data = [0, 1, 0, 1]
Running python src/scripttest.py yields expected output of
[0, 1, 0, 1], but once installed via pip install ., running scripttest returns ModuleNotFoundError: No module named 'importantimport', despite importantimport.py being right next to it in ~/.local/lib/python3.10/site-packages/src
How do I ensure that imports work locally after pip install?
Thank you.
Delete these lines:
packages = [
{ include = "src" }
]
Your project has no packages, only lone modules. Poetry's default discovery patterns will find the importantimport.py file.
Change these lines:
[tool.poetry.scripts]
scripttest = 'src.scripttest:main'
Into this:
[tool.poetry.scripts]
scripttest = 'scripttest:main'
I think you will want
from scripttest.importantimport import data
This is because importantimport is package with your library, scripttest
I'm writing my first Rust-based Python module, and it kills the Python process on import. I've got it down to a pretty minimal example, based loosely on the html-py-ever example (which does run for me without crashing).
I'm running Python 3.8 on an M1 macbook, Python is compiled for arm64.
% python -c "import platform;print(platform.machine())"
arm64
My output, reproducer command using the files pasted below. The install should take care of any python requirements:
(rust) jeremytemp#Jeremy-McGibbons-MacBook-Pro minimal % pip install -e . && python test.py
Obtaining file:///Users/jeremytemp/rust/minimal
Installing collected packages: minimal
Attempting uninstall: minimal
Found existing installation: minimal 0.1.0
Uninstalling minimal-0.1.0:
Successfully uninstalled minimal-0.1.0
Running setup.py develop for minimal
Successfully installed minimal-0.1.0
zsh: killed python test.py
src/lib.rs:
use pyo3::{prelude::*, wrap_pyfunction};
#[pyfunction]
fn foo() -> PyResult<u64>{
let u: u64 = 1;
Ok(u)
}
#[pymodule]
fn minimal(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
m.add_function(wrap_pyfunction!(foo, m)?)?;
Ok(())
}
Cargo.toml:
[package]
name = "minimal"
version = "0.1.0"
edition = "2021"
[dependencies]
pyo3 = { features = ["extension-module"] }
[lib]
name = "minimal"
crate-type = ["cdylib"]
setup.py:
from setuptools import setup
from setuptools_rust import RustExtension
setup(
rust_extensions=[RustExtension("minimal.minimal")],
)
setup.cfg:
[metadata]
name = minimal
version = 0.1.0
license = MIT
[options]
packages = minimal
zip_safe = False
setup_requires = setuptools-rust >= 0.12.1;
python_requires = >=3.8
include_package_data = True
minimal/__init__.py:
from .minimal import *
test.py:
import minimal
pip freeze output is
(rust) jeremytemp#Jeremy-McGibbons-MacBook-Pro minimal % pip freeze
attrs==21.4.0
beautifulsoup4==4.11.1
certifi==2021.10.8
iniconfig==1.1.1
# Editable Git install with no remote (minimal==0.1.0)
-e /Users/jeremytemp/rust/minimal
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.8
pytest==7.1.2
semantic-version==2.9.0
setuptools-rust==1.3.0
soupsieve==2.3.2.post1
tomli==2.0.1
typing_extensions==4.2.0
What am I doing wrong? Are there any steps I can take to get more helpful debugging output than "killed"?
Not a very satisfying answer, but the example executes fine on my windows machine. I'm assuming this is an issue with pyo3 on M1 Macs.
I have two python packages. One is a util library, and one is an application that will use the util library (eventually I will have more apps sharing the library.
I am using poetry for both, and the app specifies the common library as a dependency using the path and develop properties.
For example, my layout looks something like this:
- common/
- common/
- __init__.py
- py.typed
- pyproject.toml
- myapp/
- myapp/
- __init__.py
- py.typed
- pyproject.toml
And the myapp\pyproject.toml looks something like this:
[tool.poetry]
name = "myapp"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
[tool.poetry.dependencies]
python = "^3.9"
common = { path = "../common", develop = true }
[tool.poetry.dev-dependencies]
mypy = "^0.910"
flake8 = "^4.0.1"
black = {version = "^21.12b0", allow-prereleases = true}
pytest = "^6.2.5"
pytest-cov = "^3.0.0"
pytest-mock = "^3.6.1"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
When I run mypy on myapp I get something like:
myapp/__init__.py:1:1: error: Cannot find implementation or library stub for module named "common" [import]
Assuming the "util" package has type hints, you'll want to add a py.typed file to it (in the root directory of the package) so mypy understands it comes with type hints. py.typed should be empty, it's just a flag file.
If your package does not have type hints, then you'd have to add stubs files (.pyi).
More details: https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages
Building an Python application with cx_freeze.
from cx_Freeze import setup, Executable
_packages = []
_excludes = []
_include_files = [...]
buildOptions = dict(packages = _packages, enter code here`excludes = _excludes, include_files = _include_files, build_exe = '<app name>')
setup(name = '<app name>',
version = <version>,
description = '<description>',
options = dict(build_exe = buildOptions),
executables = [Executable('<app name>.py',
targetName = '<app name>',
icon = '<app name>.png')])
Attempting to install an application build with cx_freeze on Linux in /usr/bin/ with application resources in /usr/share/.
Of course this results in:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module name 'encodings'
Current thread 0x0... (most recent call first):...
I think I need to set/change the default location of the lib folder but I have been unable to figure out how to do that. It's entirely possible that I might be on the completely wrong track.
I'm trying to avoid using bbfreeze.
After reviewing the code I believe what I want to do is not possible. The 'lib' directory is hard coded in cx_freeze.
The problem is:
trying to use cmd to for setup.py sdist
cmd returns the message:
The expected result is that my module should have been transformed into a distribution and installed to my Python.
Below is the setup.py:
from distutils.core import setup
setup(
name = 'nester',
version = '1.0.0',
py_modules = ['nester'],
author = 'hfpython',
author_email = 'hfpython#headfirstlabs.com',
url = 'http://www.headfirstlabs.com',
description = 'A simple printer of nested lists',
)