libcrypto not found using oscrypto library in Alpine-based Docker image - python-3.x

I am getting exactly the same error as mentioned in ->
The library libcrypto could not be found
I understood the issue, however, I could not figure out the resolution. Do I need to update my lambda configuration or do I need to upgrade my Python libraries?
PFB my requriements.txt files
cryptography==36.0.2
botocore==1.20.0
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
boto3==1.17.0
asn1crypto==1.5.1
certifi==2022.9.14
cffi==1.15.1
charset-normalizer==2.1.1
filelock==3.8.0
idna==3.4
oscrypto==1.3.0
pycparser==2.21
pycryptodomex==3.15.0
PyJWT==2.5.0
pyOpenSSL==22.0.0
pytz==2022.2.1
requests==2.28.1
typing_extensions==4.3.0
urllib3==1.26.12
My docker file -
FROM python:3.9-alpine3.16
COPY requirements.txt requirements.txt
RUN apk --update --no-cache add --virtual build-dependencies gcc python3-dev musl-dev libc-dev linux-headers libxslt-dev libxml2-dev py-pip ca-certificates wget libffi-dev openssl-dev python3-dev expat==2.4.9-r0 py-pip build-base zlib zlib-dev libressl libressl-dev \
&& apk add python3 make g++ \
&& pip install --upgrade pip \
&& pip install --upgrade pip setuptools \
&& pip install -r requirements.txt \
&& apk del build-dependencies
RUN pip install snowflake-connector-python==2.8.0 --no-use-pep517
RUN python -c 'from oscrypto import asymmetric'
Attempting docker build with the Dockerfile above results in a failure with:
Step 4/4 : RUN python -c 'from oscrypto import asymmetric'
---> Running in dc8f8b8920ac
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python3.9/site-packages/oscrypto/asymmetric.py", line 19, in <module>
from ._asymmetric import _unwrap_private_key_info
File "/usr/local/lib/python3.9/site-packages/oscrypto/_asymmetric.py", line 27, in <module>
from .kdf import pbkdf1, pbkdf2, pkcs12_kdf
File "/usr/local/lib/python3.9/site-packages/oscrypto/kdf.py", line 9, in <module>
from .util import rand_bytes
File "/usr/local/lib/python3.9/site-packages/oscrypto/util.py", line 14, in <module>
from ._openssl.util import rand_bytes
File "/usr/local/lib/python3.9/site-packages/oscrypto/_openssl/util.py", line 6, in <module>
from ._libcrypto import libcrypto, libcrypto_version_info, handle_openssl_error
File "/usr/local/lib/python3.9/site-packages/oscrypto/_openssl/_libcrypto.py", line 9, in <module>
from ._libcrypto_cffi import (
File "/usr/local/lib/python3.9/site-packages/oscrypto/_openssl/_libcrypto_cffi.py", line 27, in <module>
raise LibraryNotFoundError('The library libcrypto could not be found')
oscrypto.errors.LibraryNotFoundError: The library libcrypto could not be found
The command '/bin/sh -c python -c 'from oscrypto import asymmetric'' returned a non-zero code: 1

The exception is taking place at https://github.com/wbond/oscrypto/blob/1.3.0/oscrypto/_openssl/_libcrypto_cffi.py
Tracking through oscrypto._ffi, the problem comes down to an issue opening libcrypto with the ctypes library:
>>> import ctypes.util
>>> ctypes.util.find_library('crypto')
>>>
Why? Because /usr/lib has only libcrypto.so.1.1, and not a libcrypto.so symlink pointing to it. Easily fixed by adding an extra line to your Dockerfile:
RUN ln -s libcrypto.so.1.1 /usr/lib/libcrypto.so
...after which Python's ctypes behaves:
>>> from ctypes.util import find_library
>>> find_library('crypto')
'libcrypto.so.1.1'
...and so does oscrypto:
>>> from oscrypto import asymmetric
>>>

Related

Unable to find libcrypt.so.1 while importing ibm_db

I am getting following traceback while importing ibm_db in python -
Traceback (most recent call last):
File "/working/Script.py", line 5, in <module>
import ibm_db
ImportError: Error loading shared library libcrypt.so.1: No such file or directory (needed by
/usr/local/lib/python3.9/site-packages/clidriver/lib/libdb2.so.1)
Please find below my docker image and requirements.txt file -
FROM python:3.9-alpine3.16
COPY requirements.txt requirements.txt
RUN apk --update add --virtual build-dependencies python3 py-pip openssl ca-certificates py-openssl wget libffi-dev openssl-dev python3-dev py-pip py3-pandas build-base \
&& apk add python3 make g++ \
&& pip install --upgrade pip \
&& pip install -r requirements.txt \
&& apk del build-dependencies
ENTRYPOINT [ "python", "Script.py" ]
Requirements.txt -
cryptography==2.9
botocore==1.12.253
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
snowflake-connector-python==1.9.1
snowflake-sqlalchemy==1.2.4
SQLAlchemy==1.3.19
ibm_db
s3fs
The IBM clidriver (on which the python ibm_db module depends) is not supported on Alpine-Linux - that is to say, it will not work on Alpine-Linux.
That driver (clidriver) is closed source, and for Linux x64 is built with gcc.
You must choose a supported base Linux , Ubuntu, Red Hat, SuSe.

using package compiled with nvcc and installed with setup.py with remote interpreter

I use the remote interpreter of my docker container.
The Dockerfile contents are:
FROM nvidia/cuda:8.0-cudnn7-devel-ubuntu16.04
RUN apt-get update && apt-get install -y \
rsync \
htop \
git \
openssh-server \
nano \
cmake \
python-opencv \
python2.7 \
unzip
RUN apt-get -qq -y install curl bzip2 && \
curl -sSL https://repo.continuum.io/miniconda/Miniconda2-4.6.14-Linux-x86_64.sh -o /tmp/miniconda.sh && \
bash /tmp/miniconda.sh -bfp /usr/local && \
rm -rf /tmp/miniconda.sh && \
conda install -y python=2 && \
conda update conda && \
apt-get -qq -y autoremove && \
apt-get autoclean && \
rm -rf /var/lib/apt/lists/* /var/log/dpkg.log && \
conda clean --all --yes
#RUN . usr/local/etc/profile.d/conda.sh #didnt work
#RUN conda create -n pwcnet_test python=2.7
#RUN chmod u+r+x usr/local/etc/profile.d/conda.sh
#RUN conda env list
#RUN usr/local/etc/profile.d/conda.sh activate pwcnet_test
# install pytorch and other dependencies
#RUN pip install http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl
#RUN pip install torchvision visdom dominate opencv-python cffi
# install external packages
ADD torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl
RUN pip install torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl
#RUN conda install pytorch=0.2.0 cuda80 -c soumith
RUN pip install torchvision==0.2.2 visdom==0.1.8.8 dominate==2.3.5 opencv-python==4.1.0.25 cffi==1.12.2
ADD external_packages external_packages
RUN bash /external_packages/correlation-pytorch-master/make_cuda.sh
#RUN mkdir /usr/local/lib/python2.7/site-packages/correlation
#RUN cp /usr/local/lib/python2.7/site-packages/correlation_package-0.1-py2.7-linux-x86_64.egg /usr/local/lib/python2.7/site-packages/correlation_package-0.1-py2.7-linux-x86_64.zip
#RUN unzip /usr/local/lib/python2.7/site-packages/correlation_package-0.1-py2.7-linux-x86_64.zip -d /usr/local/lib/python2.7/site-packages/correlation
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
The content of make_cuda.sh is:
#!/usr/bin/env bash
CUDA_PATH=/usr/local/cuda-8.0
#cd correlation-pytorch/correlation_package/src
echo "Compiling correlation layer kernels by nvcc..."
# TODO (JEB): Check which arches we need
nvcc -c -o /external_packages/correlation-pytorch-master/correlation-pytorch/correlation_package/src/corr_cuda_kernel.cu.o /external_packages/correlation-pytorch-master/correlation-pytorch/correlation_package/src/corr_cuda_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52
nvcc -c -o /external_packages/correlation-pytorch-master/correlation-pytorch/correlation_package/src/corr1d_cuda_kernel.cu.o /external_packages/correlation-pytorch-master/correlation-pytorch/correlation_package/src/corr1d_cuda_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52
#cd ../../
python /external_packages/correlation-pytorch-master/correlation-pytorch/setup.py build install
The image is created successfully and the interpreter works great (in run and debug).
if I run pip list, I can see the package correlation-package 0.1 appears in the list of installed modules.
If I look at the the packages-site content I see and egg file: correlation_package-0.1-py2.7-linux-x86_64.egg.
I can also notice that if I run 'python -m site', in the output appears '/usr/local/lib/python2.7/site-packages/correlation_package-0.1-py2.7-linux-x86_64.egg'.
But if I try to imp.find_module('correlation-package')[1], it write to me that:
Traceback (most recent call last):
File "<input>", line 1, in <module>
ImportError: No module named correlation-package
It is also doesn't recognize it if I try to simply import a class from the module:
from correlation_package.modules.corr import Correlation
returns:
usr/local/bin/python2.7 /opt/.pycharm_helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 172.17.0.1 --port 45203 --file /opt/project/models/PWCNet.py
Connected to pydev debugger (build 222.3739.30)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/vista/.cache/JetBrains/PyCharm2022.2/remote_sources/-609766439/-183260405/correlation/correlation_package/_ext/corr/_corr.py (will have no effect)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/vista/.cache/JetBrains/PyCharm2022.2/remote_sources/-609766439/-183260405/correlation/correlation_package/_ext/corr/_corr.py (will have no effect)
Traceback (most recent call last):
File "/opt/.pycharm_helpers/pydev/pydevd.py", line 1496, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/opt/project/models/PWCNet.py", line 13, in <module>
from correlation_package.modules.corr import Correlation
ImportError: No module named correlation_package.modules.corr
python-BaseException
and of course it doesn't see it in the file itself:
You can see in the Dockerfile in the commented commands that I tried to unzip the files in the .egg archive of the module and import it using imp.load_source as following:
Correlation = imp.load_source('Correlation', '/usr/local/lib/python2.7/site-packages/correlation/correlation_package/_ext/corr/_corr.py')
but this ended with the error:
raceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/correlation/correlation_package/_ext/corr/_corr.py", line 7, in <module>
__bootstrap__()
File "/usr/local/lib/python2.7/site-packages/correlation/correlation_package/_ext/corr/_corr.py", line 6, in __bootstrap__
imp.load_dynamic(__name__,__file__)
ImportError: /usr/local/lib/python2.7/site-packages/correlation/correlation_package/_ext/corr/_corr.so: undefined symbol: __gxx_personality_v0
if I run this command from the python console and a different error pop up from debugging this line in the debugger of pycharm:
/usr/local/bin/python2.7 /opt/.pycharm_helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 172.17.0.1 --port 41629 --file /opt/project/models/PWCNet.py
Connected to pydev debugger (build 222.3739.30)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/vista/.cache/JetBrains/PyCharm2022.2/remote_sources/-609766439/-183260405/correlation/correlation_package/_ext/corr/_corr.py (will have no effect)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/vista/.cache/JetBrains/PyCharm2022.2/remote_sources/-609766439/-183260405/correlation/correlation_package/_ext/corr/_corr.py (will have no effect)
Traceback (most recent call last):
File "/opt/.pycharm_helpers/pydev/pydevd.py", line 1496, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/opt/project/models/PWCNet.py", line 17, in <module>
Correlation = imp.load_source('Correlation', '/usr/local/lib/python2.7/site-packages/correlation/correlation_package/_ext/corr/_corr.py')
File "/usr/local/lib/python2.7/site-packages/correlation/correlation_package/_ext/corr/_corr.py", line 7, in <module>
__bootstrap__()
File "/usr/local/lib/python2.7/site-packages/correlation/correlation_package/_ext/corr/_corr.py", line 6, in __bootstrap__
imp.load_dynamic(__name__,__file__)
ImportError: dynamic module does not define init function (initCorrelation)
python-BaseException
as the contents of the corr.py file are the following:
def __bootstrap__():
global __bootstrap__, __loader__, __file__
import sys, pkg_resources, imp
__file__ = pkg_resources.resource_filename(__name__, '_corr.so')
__loader__ = None; del __bootstrap__, __loader__
imp.load_dynamic(__name__,__file__)
__bootstrap__()
and the contents of the unzipped egg /correlation/correlation_package/_ext/corr/ are:
_corr.py
_corr.pyc
_corr.so
the debug of this load_source operation shows that the imp.load_dynamic in the bootstrap function runs as follows:
What is the correct way to import a package which is on the docker container environment ?
Thanks in advance.
P.S:
If I add to the Dockerfile the line RUN python /external_packages/correlation-pytorch-master/correlation-pytorch/test/test.py which is made to test the package installation, I get the error:
Step 9/14 : RUN python /external_packages/correlation-pytorch-master/correlation-pytorch/test/test.py
---> Running in c57c673c66c1
Traceback (most recent call last):
File "/external_packages/correlation-pytorch-master/correlation-pytorch/test/test.py", line 4, in <module>
from correlation_package.modules.corr import Correlation, Correlation1d
ImportError: No module named correlation_package.modules.corr
This is confusing because as was said above pip list does shows it int he installed packages list. This is totally executed from inside the container, as you can see below:
Step 9/15 : RUN pip list
---> Running in adbe3ba085b2
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 wont be maintained after that date. A future version of pip will drop support for Python 2.7.
Package Version
---------------------- -----------
asn1crypto 1.4.0
backports-abc 0.5
certifi 2020.6.20
cffi 1.12.2
chardet 3.0.4
conda 4.8.4
conda-package-handling 1.6.0
correlation-package 0.1
cryptography 2.6.1
dominate 2.3.5
enum34 1.1.6
futures 3.3.0
idna 2.8
ipaddress 1.0.23
numpy 1.16.6
opencv-python 4.1.0.25
Pillow 6.2.2
pip 19.0.3
pycosat 0.6.3
pycparser 2.19
pyOpenSSL 19.0.0
PySocks 1.7.1
PyYAML 5.4.1
pyzmq 19.0.2
requests 2.21.0
ruamel-yaml 0.15.46
scandir 1.10.0
scipy 1.2.3
setuptools 41.0.0
singledispatch 3.7.0
six 1.16.0
torch 0.2.0.post3
torchfile 0.1.0
torchvision 0.2.2
tornado 5.1.1
tqdm 4.19.9
urllib3 1.24.1
visdom 0.1.8.8
websocket-client 0.59.0
wheel 0.37.1
Removing intermediate container adbe3ba085b2
---> 120baf3ba68a
Step 10/15 : RUN python /external_packages/correlation-pytorch-master/correlation-pytorch/test/test.py
---> Running in e28c04ca8d12
Traceback (most recent call last):
File "/external_packages/correlation-pytorch-master/correlation-pytorch/test/test.py", line 4, in <module>
from correlation_package.modules.corr import Correlation, Correlation1d
ImportError: No module named correlation_package.modules.corr

While installing nltk package getting ModuleNotFoundError: No module named '_sqlite3'

Getting below error while running a code snippet which has nltk dependencies.
Manually installed python 3.6.5 and using it in the virtual enviornment
The OS (Red Hat Linux 7.4-Maipo) has python 2.7.4 installed in it.
Traceback (most recent call last):
File "FLASK_RECOMMENDER_VER_1_1.py", line 15, in <module>
import nltk
File "/home/ec2-user/RecommenderEngine/re_env/lib/python3.6/site-packages/nltk/__init__.py", line 137, in <module>
from nltk.stem import *
File "/home/ec2-user/RecommenderEngine/re_env/lib/python3.6/site-packages/nltk/stem/__init__.py", line 29, in <module>
from nltk.stem.snowball import SnowballStemmer
File "/home/ec2-user/RecommenderEngine/re_env/lib/python3.6/site-packages/nltk/stem/snowball.py", line 32, in <module>
from nltk.corpus import stopwords
File "/home/ec2-user/RecommenderEngine/re_env/lib/python3.6/site-packages/nltk/corpus/__init__.py", line 66, in <module>
from nltk.corpus.reader import *
File "/home/ec2-user/RecommenderEngine/re_env/lib/python3.6/site-packages/nltk/corpus/reader/__init__.py", line 105, in <module>
from nltk.corpus.reader.panlex_lite import *
File "/home/ec2-user/RecommenderEngine/re_env/lib/python3.6/site-packages/nltk/corpus/reader/panlex_lite.py", line 15, in <module>
import sqlite3
File "/usr/local/lib/python3.6/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/usr/local/lib/python3.6/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
Tried below solution but it didn't help (used yum and pip instead) :
solution 1
sudo apt-get install libsqlite3-dev
(Or you can install more packages as suggested on the pyenv wiki:
apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev
Now in the downloaded python source rebuild and install python with the following command: ./configure --enable-loadable-sqlite-extensions && make && sudo make install.
My Attempt throwing below error :
Collecting libsqlite3-dev
Could not find a version that satisfies the requirement libsqlite3-dev (from versions: )
No matching distribution found for libsqlite3-dev
Answering my own question.
Finally fixed the issue by Installing sqlite-devel package :
yum install sqlite-devel
and re-installing the python as follows:
sudo wget http://python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz
tar xf Python-3.6.5.tar.xz
cd Python-3.6.5
sudo ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
sudo ./configure --enable-optimizations
sudo make && sudo make altinstall

Spark on Bluemix : !pip install --user shapely == Could not find library geos_c or load any of its variants

!pip install --user shapely
results in
Collecting shapely
Using cached Shapely-1.5.16.tar.gz
Complete output from command python setup.py egg_info:
Failed `CDLL(libgeos_c.so.1)`
Failed `CDLL(libgeos_c.so)`
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/gpfs/global_fs01/sym_shared/YPProdSpark/user/s390-a2b719b24f799b-80b3d8dd92c1/notebook/tmp/pip-build-PYROZS/shapely/setup.py", line 38, in <module>
from shapely._buildcfg import geos_version_string, geos_version, \
File "shapely/_buildcfg.py", line 167, in <module>
fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])
File "shapely/_buildcfg.py", line 161, in load_dll
libname, fallbacks or []))
OSError: Could not find library geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so']
How can I overcome this issue on Bluemix Spark service where I don't have admin permission to install libraries.
There are lots of similar questions, but none of them are about the Bluemix Spark service.
This worked for me:
!wget http://download.osgeo.org/geos/geos-3.5.0.tar.bz2
!tar jxf geos-3.5.0.tar.bz2
!cd geos-3.5.0 && ./configure --prefix=$HOME/geos-bin && make && make install
!GEOS_CONFIG=$HOME/geos-bin/bin/geos-config pip install --user shapely
...
output omitted for brevity
...
Collecting shapely
Using cached Shapely-1.5.16.tar.gz
Installing collected packages: shapely
Running setup.py install for shapely ... [?25l- \ | / - \ | / - done
[?25hSuccessfully installed shapely-1.5.16

zope.interface.registry ImportError for Pyramid on Travis

I'm having trouble working with Pyramid on travis; the zope.interfaces dependency doesn't work.
This has to be in system python or --system-site-packages because I'm using binary scientific packages (scipy).
File "/usr/local/lib/python2.7/dist-packages/pyramid/config/__init__.py", line 20, in <module>
from pyramid.authorization import ACLAuthorizationPolicy
File "/usr/local/lib/python2.7/dist-packages/pyramid/authorization.py", line 9, in <module>
from pyramid.security import (
File "/usr/local/lib/python2.7/dist-packages/pyramid/security.py", line 13, in <module>
from pyramid.threadlocal import get_current_registry
File "/usr/local/lib/python2.7/dist-packages/pyramid/threadlocal.py", line 3, in <module>
from pyramid.registry import global_registry
File "/usr/local/lib/python2.7/dist-packages/pyramid/registry.py", line 5, in <module>
from zope.interface.registry import Components
ImportError: No module named registry
This is my current travis.yml:
language: c
install:
- sudo apt-get install python-scipy python-pip
- sudo pip install -r requirements.txt
- sudo pip install .
script:
- nosetests
after_success:
- coveralls
The build log indicates that zope.interface-4.1.2 has been installed.

Resources