download python library to create deployment package - python-3.x

I am trying to download a python library (requests) into a folder in order to create a deployment package.
from terminal (I am on mac)
i use
pip3 install requests -t .
I get the following error.
Collecting requests
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.wh
➜ AWS apt-get install python3-pip
zsh: command not found: apt-get
➜ AWS ln -s /Library/Frameworks/Python.framework/Versions/3.6/bin/pip /usr/bin/pip3
ln: /usr/bin/pip3: Operation not permitted
➜ AWS ln -s /Library/Frameworks/Python.framework/Versions/3.6/bin/pip /usr/local/bin/pip3
ln: /usr/local/bin/pip3: File exists
➜ AWS pip3 install requests -t .
Collecting requests
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none
-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-a
ny.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.wh
l
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-
any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-no
ne-any.whl
Installing collected packages: chardet, idna, urllib3, certifi, requests
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 179, in main
status = self.run(options, args)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 393, in run
use_user_site=options.use_user_site,
File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 57, in install_given_reqs
**kwargs
File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 913, in install
use_user_site=use_user_site, pycompile=pycompile,
File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 445, in move_wheel_files
warn_script_location=warn_script_location,
File "/usr/local/lib/python3.7/site-packages/pip/_internal/wheel.py", line 313, in move_wheel_files
prefix=prefix,
File "/usr/local/lib/python3.7/site-packages/pip/_internal/locations.py", line 180, in distutils_scheme
i.finalize_options()
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/install.py", line 248, in finalize_options
"must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both
any idea how to fix this?
the requests packages in already installed and I have made a test script that uses it and it works.
I just would like to understand what's wrong.

I solved by running
pip3 install -U pip
and then re-runnig
pip3 install requests -t .

Related

Why does pyarrow installation fail with cython not found when cython is included in requirements.txt?

So, I have a docker file in which one of the instructions is :
RUN pip3 install -r requirements.txt
And in my requirements.txt:
...
uwsgi==2.0.19.1
cython==0.29
dependency-injector==4.37.0
pyyaml==6.0
apscheduler==3.7.0
pyarrow==5.0.0
...
When I run the docker build, I see that Cython is installed but pyarrow still fails. I found this link - https://github.com/apache/arrow/issues/2163 - which mentions that cmake & cython are required and I added that in my requirements.txt but it still does not help. Do I have to add additional statements in my Dockerfile to install cython?
Output from Dockerfile build:
Collecting cython==0.29 (from -r requirements.txt (line 8))
Downloading https://files.pythonhosted.org/packages/64/3f/cac281f3f019b825bbc03fa8cb7eb03d9c355f4aa9eef978279a4966cb21/Cython-0.29-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)
...
Collecting pyarrow==5.0.0 (from -r requirements.txt (line 12))
Downloading https://files.pythonhosted.org/packages/68/7c/0e38bfb949ededdd9b648d54cba47972835704543d7409d6f853504d0581/pyarrow-5.0.0.tar.gz (739kB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-395casa1/pyarrow/setup.py", line 39, in <module>
from Cython.Distutils import build_ext as _build_ext
ModuleNotFoundError: No module named 'Cython'
Python 3.6 installed on this base image
Ugh, this is probably one of my dumb-dumb moments but here's what worked:
RUN pip3 install --upgrade pip && pip install -r requirements.txt
and I didn't even have to specify cython in requirements.txt

No module named 'request' even after it is already there while running "Facebook AI Performance Evaluation Platform"

I searched a lot and tried many of the SQ solutions but nothing helped me here... can someone check what exactly I messed up with?
My Python3 is installed in the directory here /usr/local/Cellar/python/3.7.0/
In my .bashrc I have added
export PYTHONPATH=$PYTHONPATH:/Users/swapnil.kotwal/Library/Python/3.7/lib/python/site-packages
Also, .bash_profile have
source ~/.bashrc
alias python=python3
export PATH="/usr/local/bin:/usr/local/opt/node#10/bin:$PATH"
My Below command also works fine
$ python -c "import requests; print (requests.__version__)"
2.20.0
And other things are as below
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ /usr/bin/python --version
Python 2.7.10
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ which python
/usr/bin/python
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ whereis python
/usr/bin/python
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ python3 --version
Python 3.7.0
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ which pip3
/usr/local/bin/pip3
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ vi benchmarking/run_bench.py
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json
Traceback (most recent call last):
File "benchmarking/run_bench.py", line 18, in <module>
from utils.utilities import getPythonInterpreter, getString
File "/Users/swapnil.kotwal/Swapnil/FAI-PEP/benchmarking/utils/utilities.py", line 16, in <module>
import requests
ImportError: No module named requests
SwapnilsMacBook:FAI-PEP swapnil.kotwal$
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ pip3 install six
Collecting six
Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.11.0
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ pip3 install requests
Collecting requests
Using cached https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://files.pythonhosted.org/packages/56/9d/1d02dd80bc4cd955f98980f28c5ee2200e1209292d5f9e9cc8d030d18655/certifi-2018.10.15-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests)
Using cached https://files.pythonhosted.org/packages/8c/4b/5cbc4cb46095f369117dcb751821e1bef9dd86a07c968d8757e9204c324c/urllib3-1.24-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests)
Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Installing collected packages: chardet, certifi, urllib3, idna, requests
Successfully installed certifi-2018.10.15 chardet-3.0.4 idna-2.7 requests-2.20.0 urllib3-1.24
pip list
SwapnilsMacBook:FAI-PEP swapnil.kotwal$ pip list
Package Version
---------------- ----------
certifi 2018.10.15
chardet 3.0.4
idna 2.7
pip 18.1
pipenv 2018.10.13
requests 2.20.0
setuptools 40.4.3
six 1.11.0
urllib3 1.24
virtualenv 16.1.0
virtualenv-clone 0.4.0
wheel 0.32.0
SwapnilsMacBook:FAI-PEP swapnil.kotwal$
But, when I ran FAI-PEP
https://github.com/facebook/FAI-PEP
using command benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json
I got error as below
$ benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json
Traceback (most recent call last):
File "benchmarking/run_bench.py", line 18, in <module>
from utils.utilities import getPythonInterpreter, getString
File "/Users/swapnil.kotwal/Swapnil/FAI-PEP/benchmarking/utils/utilities.py", line 16, in <module>
import requests
ImportError: No module named requests
Please check where requests gets installed. Then check whether python is searching for the library in this directory.
In order to see where requests is installed, you can use the following pip command
pip list
compare How do I find the location of my Python site-packages directory? . In case the Location column is empty, the library was installed to the default directory.
And to see the directories in which python is searching for libraries you can use
print(__import__('sys').path)
In case requests is installed in a directory python is not looking in, I suggest reading these question: Permanently add a directory to PYTHONPATH , Install a Python package into a different directory using pip?
Also check that your python script is not executed in a virtual environment (virtualenv) where the package is not installed/available https://docs.python-guide.org/dev/virtualenvs/.

Installing Python modules on MacOSX

I'm trying to install the http library through the command "pip install http".
It give me this error:
Command "python setup.py egg_info" failed with error code 1
I just installed setuptools but it doesn't work anyway.
EDIT:
when I launch the command 'pip3 install http' I get this:
pip3 install http
Collecting http
Using cached https://files.pythonhosted.org/packages/e3/91/a9260805e532e33df273b8f7dffad5c51693f8f9ba5f86bedcf42a7f22eb/http-0.02.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/1v/xm2qv7l132sghltfqm3q15wm0000gn/T/pip-install-g44pke5e/http/setup.py", line 3, in <module>
import http
File "/private/var/folders/1v/xm2qv7l132sghltfqm3q15wm0000gn/T/pip-install-g44pke5e/http/http/__init__.py", line 17, in <module>
from request import Request
ImportError: cannot import name 'Request'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/1v/xm2qv7l132sghltfqm3q15wm0000gn/T/pip-install-g44pke5e/http/
I tried many times to check and upload requests package but when I do it I get this:
pip3 install requests
Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2.19.1)
Requirement already satisfied: certifi>=2017.4.17 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (3.0.4)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (1.23)
Requirement already satisfied: idna<2.8,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from requests) (2.7)
Somebody knows what I made wrong with it?
http version 0.02 was released in 2012. The homepage returns error 404.
I think the package is simply abandoned, broken and outdated. Avoid it or fork it and update it yourself.

Installing dropbox fails and corrupts pip3

I'm running Python 3.4.2 on Raspbian Jessie on a Pi3
pip3 works just fine until I try this:
sudo pip3 install dropbox
Which yields:
Downloading/unpacking dropbox
Requirement already satisfied (use --upgrade to upgrade): urllib3 in /usr/lib/python3/dist-packages (from dropbox)
Requirement already satisfied (use --upgrade to upgrade): six>=1.3.0 in /usr/lib/python3/dist-packages (from dropbox)
Downloading/unpacking requests>=2.5.1,!=2.6.1 (from dropbox)
Downloading/unpacking idna>=2.5,<2.6 (from requests>=2.5.1,!=2.6.1->dropbox)
Downloading/unpacking certifi>=2017.4.17 (from requests>=2.5.1,!=2.6.1->dropbox)
Downloading/unpacking chardet>=3.0.2,<3.1.0 (from requests>=2.5.1,!=2.6.1->dropbox)
Installing collected packages: dropbox, requests, idna, certifi, chardet
Found existing installation: requests 2.4.3
Not uninstalling requests at /usr/lib/python3/dist-packages, owned by OS
Found existing installation: chardet 2.3.0
Not uninstalling chardet at /usr/lib/python3/dist-packages, owned by OS
Successfully installed dropbox requests idna certifi chardet
Cleaning up...
After which, pip3 (but not pip) is now corrupt and simply running:
sudo pip3
gives the following error message:
/usr/local/lib/python3.4/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.9.1) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Traceback (most recent call last):
File "/usr/bin/pip3", line 9, in <module>
load_entry_point('pip==1.5.6', 'console_scripts', 'pip3')()
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 356, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2476, in load_entry_point
return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2190, in load
['__name__'])
File "/usr/lib/python3/dist-packages/pip/__init__.py", line 74, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/lib/python3/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
from pip.download import path_to_url
File "/usr/lib/python3/dist-packages/pip/download.py", line 22, in <module>
import requests, six
File "/usr/local/lib/python3.4/dist-packages/requests/__init__.py", line 90, in <module>
from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'
I've installed a lot of other packages from source including ffmpeg, gstreamer 1.10.5, opencv 3.2.0, so I suspect that process is at fault, but only trying to install dropbox using pip3 corrupts pip3.
Any help in solving this would be appreciated.
#mdasari Thanks a ton for your help. I'm running Python 3.4.2, so I used easy_install3 to remove and reinstall pip as you suggested. This time dropbox appears to have installed correctly (haven't tested it yet), but pip3 is once again corrupted and giving me the same error as before. However, after rebooting, the error went away and pip3 appears to be fine. For others, this is exactly what I did to correct the problem (once again, I'm using Python 3.4.2):
sudo apt-get remove python-pip
sudo apt-get autoremove
sudo easy_install3 -U pip
sudo pip3 install dropbox
reboot
I think the problem is with pip, not dropbox. Did you try uninstalling current version pip and reinstall it?
apt-get remove python-pip
easy_install-3.5 -U pip
You can look at previous issues here and here

Spark on Bluemix : !pip install --user shapely == Could not find library geos_c or load any of its variants

!pip install --user shapely
results in
Collecting shapely
Using cached Shapely-1.5.16.tar.gz
Complete output from command python setup.py egg_info:
Failed `CDLL(libgeos_c.so.1)`
Failed `CDLL(libgeos_c.so)`
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/gpfs/global_fs01/sym_shared/YPProdSpark/user/s390-a2b719b24f799b-80b3d8dd92c1/notebook/tmp/pip-build-PYROZS/shapely/setup.py", line 38, in <module>
from shapely._buildcfg import geos_version_string, geos_version, \
File "shapely/_buildcfg.py", line 167, in <module>
fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])
File "shapely/_buildcfg.py", line 161, in load_dll
libname, fallbacks or []))
OSError: Could not find library geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so']
How can I overcome this issue on Bluemix Spark service where I don't have admin permission to install libraries.
There are lots of similar questions, but none of them are about the Bluemix Spark service.
This worked for me:
!wget http://download.osgeo.org/geos/geos-3.5.0.tar.bz2
!tar jxf geos-3.5.0.tar.bz2
!cd geos-3.5.0 && ./configure --prefix=$HOME/geos-bin && make && make install
!GEOS_CONFIG=$HOME/geos-bin/bin/geos-config pip install --user shapely
...
output omitted for brevity
...
Collecting shapely
Using cached Shapely-1.5.16.tar.gz
Installing collected packages: shapely
Running setup.py install for shapely ... [?25l- \ | / - \ | / - done
[?25hSuccessfully installed shapely-1.5.16

Resources