Docker: Move LibXMLSec settings / dependencies into executable directory - python-3.x

So. I'm trying to get my Dockerfile to run.
Dockerfile is complicated, and doing things somewhat wrong for Python. The important part, though, is that it was working before I added XMLSec to the dependencies. (via the onelogin [https://github.com/onelogin/python-saml] package)
On our non-Dockerized systems, we run:
apt install -y libxml2-dev libxmlsec1-dev libxmlsec1-openssl pkg-config
and it installs the dependencies. My containerized attempts, however, run into this issue:
Running setup.py install for xmlsec: started
Running setup.py install for xmlsec: finished with status 'error'
Complete output from command /usr/local/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-wd6h548m/xmlsec/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ufxw9hor/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
package init file 'src/xmlsec/__init__.py' not found (or not a regular file)
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/xmlsec
copying src/xmlsec/py.typed -> build/lib.linux-x86_64-3.6/xmlsec
copying src/xmlsec/constants.pyi -> build/lib.linux-x86_64-3.6/xmlsec
copying src/xmlsec/template.pyi -> build/lib.linux-x86_64-3.6/xmlsec
copying src/xmlsec/__init__.pyi -> build/lib.linux-x86_64-3.6/xmlsec
copying src/xmlsec/tree.pyi -> build/lib.linux-x86_64-3.6/xmlsec
running build_ext
error: Unable to invoke pkg-config.
The solution seems to be to use the XMLSec library as a compiled Wheel. (Per something I read in the XMLSec github) I can't find any documentation on how to add this compiled wheel into an executable location within the container via the Dockerfile.
I've done quite a bit of research on this, and can't find sample code or the exact answer. I it may be obvious, but I'm missing core knowledge.

Related

Unable to execute 'gcc' : No such file or directory when installing apache superset on IBM i 7.3

I am trying to install apache superset from scratch and following recommendation to use virtual environment (venv) and then ran command python3 -m pip install apache-superset. Getting the following error:
Complete output from command /home/XXX/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-z7
by2m2a/cffi/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code
, file, 'exec'))" install --record /tmp/pip-record-fqli9xso/install-record.txt --single-version-externally-managed --prefix /
tmp/pip-build-env-u0f6md7n --compile --install-headers /home/SHAWNW/venv/include/site/python3.6/cffi:
unable to execute 'gcc': No such file or directory
unable to execute 'gcc': No such file or directory
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.freenode.net.)

google buildpack psycopg2-binary Error: pg_config executable not found

This is in my requirements.txt
psycopg2-binary==2.8.3 # via -r requirements/base.in
that I am building inside docker image. I was under impression that if I install psycopg2-binary vs psycopg2 I should not install additional postgresql devel libs. Am I wrong?
pack build --builder=gcr.io/buildpacks/builder:v1 test-python
Collecting psycopg2-binary==2.8.3
Downloading psycopg2-binary-2.8.3.tar.gz (378 kB)
ERROR: Command errored out with exit status 1:
command: /layers/google.python.runtime/python/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z7z_l56l/psycopg2-binary_ddfc5ed05bb44cf4b7e2f14d634bd6ae/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z7z_l56l/psycopg2-binary_ddfc5ed05bb44cf4b7e2f14d634bd6ae/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vsc_2hlv
cwd: /tmp/pip-install-z7z_l56l/psycopg2-binary_ddfc5ed05bb44cf4b7e2f14d634bd6ae/
Complete output (23 lines):
running egg_info
creating /tmp/pip-pip-egg-info-vsc_2hlv/psycopg2_binary.egg-info
writing /tmp/pip-pip-egg-info-vsc_2hlv/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-vsc_2hlv/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-pip-egg-info-vsc_2hlv/psycopg2_binary.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-vsc_2hlv/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).
No, you are not wrong; I had the same issue today, and after a bit of googling I found this (solved) github issue: https://github.com/psycopg/psycopg2/issues/699#
The comments there (https://github.com/psycopg/psycopg2/issues/699#issuecomment-377188700) indicate one possible cause for the issue is using an outdated version of pip.
I was using version 18.x, and after upgrading it (python -m pip install -U pip, which got me pip 21.1.3), the installation of psycopg2-binary finished successfully (as expected) even though I don't have any postgres dev libraries installed.

Failed building wheel for GoogleAppEngineCloudStorageClient

I get this error when trying to install google cloud:
Failed building wheel for GoogleAppEngineCloudStorageClient
Running setup.py clean for GoogleAppEngineCloudStorageClient
Failed to build GoogleAppEngineCloudStorageClient
Installing collected packages: GoogleAppEngineCloudStorageClient
Running setup.py install for GoogleAppEngineCloudStorageClient ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jZCvYX/GoogleAppEngineCloudStorageClient/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-LgTihN-record/install-record.txt --single-version-externally-managed --compile --user --prefix= --home=/tmp/tmprq5a80:
running install
error: can't combine user with prefix, exec_prefix/home, or install_(plat)base
this is installed on the /lib folder, I also made a requirements.txt
as well as the appengine_config.py.
Also I am using python 2.7, on linux
use:
pip install --user --install-option="--prefix=" <package_name>
However, better to install the whole Google Cloud SDK as it includes: command line tools (gcloud, gsutil, bq) in addition to client libraries.

pip install in Cygwin cannot find file error

I am on a Mac OS El Capitan, running a Windows 10 64-bit VM inside Parallels. I have Cygwin installed and Anaconda3. I would like to install two packages (pyrsistent and rpy2) using pip. Both of them throw an error "error: [WinError 2] The system cannot find the file specified" without specifying the file it can't find.
Here's the output:
$ pip install pyrsistent
Collecting pyrsistent
Using cached pyrsistent-0.11.9.tar.gz
Requirement already satisfied (use --upgrade to upgrade): six in c:\anaconda3\lib\site-packages (from pyrsistent)
Building wheels for collected packages: pyrsistent
Running setup.py bdist_wheel for pyrsistent
Complete output from command C:\Anaconda3\python.exe -c "import setuptools;__file__='C:\\cygwin64\\tmp\\pip-build-sqcinj9m\\pyrsistent\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d C:\cygwin64\tmp\tmpn25raothpip-wheel-:
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-3.5
copying _pyrsistent_version.py -> build\lib.win-amd64-3.5
creating build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_checked_types.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_field_common.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_helpers.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_immutable.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_pbag.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_pclass.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_pdeque.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_plist.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_pmap.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_precord.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_pset.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_pvector.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\_transformations.py -> build\lib.win-amd64-3.5\pyrsistent
copying pyrsistent\__init__.py -> build\lib.win-amd64-3.5\pyrsistent
running build_ext
building 'pvectorc' extension
error: [WinError 2] The system cannot find the file specified
----------------------------------------
Failed building wheel for pyrsistent
Failed to build pyrsistent
Installing collected packages: pyrsistent
Running setup.py install for pyrsistent
Complete output from command C:\Anaconda3\python.exe -c "import setuptools, tokenize;__file__='C:\\cygwin64\\tmp\\pip-build-sqcinj9m\\pyrsistent\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\cygwin64\tmp\pip-_wbdiief-record\install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
running build_ext
building 'pvectorc' extension
error: [WinError 2] The system cannot find the file specified
----------------------------------------
Command "C:\Anaconda3\python.exe -c "import setuptools, tokenize;__file__='C:\\cygwin64\\tmp\\pip-build-sqcinj9m\\pyrsistent\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\cygwin64\tmp\pip-_wbdiief-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\cygwin64\tmp\pip-build-sqcinj9m\pyrsistent
Both python3 and pip are in the path:
$ which pip
/cygdrive/c/Anaconda3/Scripts/pip
$ which python
/cygdrive/c/Anaconda3/python
The error is identical when trying to install rpy2 so it's not something particular to the package I'm trying to install. Does anyone have ideas of the problem or ideas for troubleshooting? I've tried debugging into the install.py but the code throwing the error is in C. I've tried doing the install in verbose mode, but it still doesn't say what file it cannot find in that case.
(and if you're wondering why I haven't used conda, it's because it doesn't install pyrsistent, and for the rpy2 install, it insists on linking it to its own install of R, and even when specifying my own install, something doesn't link correctly. I've already been down that road and would like to stick to pip).
Here is a workaround for anyone who is stuck installing python packages on Windows. There are several sites that have Windows binaries for python packages. I was able to find a binary for rpy2 here but there are several other sites too, if you just google something like "python Windows binaries". Then I did:
pip install rpy2-2.7.4-cp35-none-win_amd64.whl
pyrsistent was not in that website or any of several other binaries websites I found. But someone else on my dev team sent me the /pyrsistent directory inside his Anaconda3/Lib/site-packages directory and after copying it to the same directory on my machine, I was able to import pyrsistent as well.
It's not pretty, but it worked.

Cannot find vcvarsall.bat

NOTE: I have seen multiple answers to no avail, so please look at the solution and my failed solutions list before showing as duplicate.
Hey,
So I got a new computer (Old Windows w/ Python 3.4), so I installed Python 3.5 on my new one. I started to install modules until I came across an error like this:
C:\Users\Samuel>pip install noise
Collecting noise
Using cached noise-1.2.2.tar.gz
Installing collected packages: noise
Running setup.py install for noise
Complete output from command "c:\program files\python 3.5\python.exe" -c "import setuptools, tokenize;__file__='C:\\Users\\Samuel\\AppData\\Local\\Temp\\pip-build-e9dlwpm2\\noise\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\Samuel\AppData\Local\Temp\pip-7tgvnyfe-record\install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.5
creating build\lib.win-amd64-3.5\noise
copying perlin.py -> build\lib.win-amd64-3.5\noise
copying setup.py -> build\lib.win-amd64-3.5\noise
copying shader.py -> build\lib.win-amd64-3.5\noise
copying shader_noise.py -> build\lib.win-amd64-3.5\noise
copying test.py -> build\lib.win-amd64-3.5\noise
copying __init__.py -> build\lib.win-amd64-3.5\noise
running build_ext
building 'noise._simplex' extension
error: Unable to find vcvarsall.bat
----------------------------------------
Command ""c:\program files\python 3.5\python.exe" -c "import setuptools, tokenize;__file__='C:\\Users\\Samuel\\AppData\\Local\\Temp\\pip-build-e9dlwpm2\\noise\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\Samuel\AppData\Local\Temp\pip-7tgvnyfe-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\Samuel\AppData\Local\Temp\pip-build-e9dlwpm2\noise
I tried to use easy_install as a substitute, but that came with 1) an access denied error (Overcome w/ admin privelages) and 2) Not being able to find vcvarsall.bat
Because of this, I installed Visual Studio 2015 and Visual C++ 2008.
No difference.
I added the path to vcvarsall.bat to the PATH.
No difference.
I added the path to vcvarsall.bat to VS100COMNTOOLS, VS140COMNTOOLS and VS90COMNTOOLS.
No difference.
Running pip and easy_install again w/ admin permission, wearing my lucky socks.
No difference.
Installed and ran Miniconda as suggested by one of the Stack Overflow answers.
Didn't find a single package I needed. Turns out it's not looking for PyPI anyway.
I have no idea how to solve this. Do I need to revert to Python 3.4?
SPECS
Windows 10 x86 64-bit
Python 3.5
VS 2015, VC++ 2008 installed
I think you have the wrong C++: VC++ 2008 is for Python before 3.3.
The VC++ 2015 that you need comes when you install Visual Studio 2015 but you need to explicitly specify the C++ (check the box) for the C++ (2015) in the custom installation options. Try changing that via a re-installation.
Just tried it:
Windows 10 x86 64-bit
Python 3.5
VS 2015 But with the Visual C++ install option checked
Then pip install noise worked fine.

Resources