pip install in virtualenv on EC2 missing packages in requirements.txt file on install - python-3.x

I am pip installing a variety of packages inside a virtual environment on an EC2 instance (2018-03). Pip successfully installs some of the packages, but not all. I am using pip install --force-reinstall -r requirements.txt on the following requirements file, which was created on a system with a working instance with Python 3.7.4 on Macos:
alembic==1.1.0
bcrypt==3.1.7
boto3==1.9.156
botocore==1.12.249
certifi==2019.9.11
cffi==1.12.3
chardet==3.0.4
Click==7.0
coverage==4.5.4
docutils==0.15.2
Flask==1.0.2
Flask-Bcrypt==0.7.1
Flask-Login==0.4.1
Flask-Migrate==2.5.2
Flask-Script==2.0.6
Flask-Session==0.3.1
Flask-SQLAlchemy==2.4.0
Flask-WTF==0.14.2
idna==2.8
itsdangerous==1.1.0
Jinja2==2.10.1
jmespath==0.9.4
lxml==4.3.2
Mako==1.1.0
MarkupSafe==1.1.1
numpy==1.16.2
pandas==0.24.2
patsy==0.5.1
psycopg2==2.8.3
pycparser==2.19
python-dateutil==2.8.0
python-editor==1.0.4
pytz==2019.2
requests==2.21.0
s3transfer==0.2.1
scikit-learn==0.20.3
scipy==1.2.1
six==1.12.0
SQLAlchemy==1.3.8
statsmodels==0.9.0
trimesh==3.1.11
urllib3==1.24.3
Werkzeug==0.15.5
WTForms==2.2.1
And pip gives me the following message:
Successfully installed Click-7.0 Flask-1.0.2 Flask-Bcrypt-0.7.1 Flask-Login-0.4.1 Flask-Migrate-2.5.2 Flask-SQLAlchemy-2.4.0 Flask-Script-2.0.6 Flask-Session-0.3.1 Flask-WTF-0.14.2 Jinja2-2.10.1 Mako-1.1.0 MarkupSafe-1.1.1 SQLAlchemy-1.3.8 WTForms-2.2.1 Werkzeug-0.15.5 alembic-1.1.0 bcrypt-3.1.7 boto3-1.9.156 botocore-1.12.249 certifi-2019.9.11 cffi-1.12.3 chardet-3.0.4 coverage-4.5.4 docutils-0.15.2 idna-2.8 itsdangerous-1.1.0 jmespath-0.9.4 lxml-4.3.2 numpy-1.16.2 pandas-0.24.2 patsy-0.5.1 psycopg2-2.8.3 pycparser-2.19 python-dateutil-2.8.0 python-editor-1.0.4 pytz-2019.2 requests-2.21.0 s3transfer-0.2.1 scikit-learn-0.20.3 scipy-1.2.1 six-1.12.0 statsmodels-0.9.0 trimesh-3.1.11 urllib3-1.24.3
However when I do a pip list I only see the following packages installed. Note that coverage and psycopg2, for example, were listed as successfully installed, but don't show. Consequently, a quick python -c "import coverage" will fail.
Package Version
---------------- ---------
alembic 1.1.0
boto3 1.9.156
botocore 1.12.249
certifi 2019.9.11
chardet 3.0.4
Click 7.0
docutils 0.15.2
Flask 1.0.2
Flask-Bcrypt 0.7.1
Flask-Login 0.4.1
Flask-Migrate 2.5.2
Flask-Script 2.0.6
Flask-Session 0.3.1
Flask-SQLAlchemy 2.4.0
Flask-WTF 0.14.2
idna 2.8
itsdangerous 1.1.0
Jinja2 2.10.1
jmespath 0.9.4
Mako 1.1.0
patsy 0.5.1
pip 19.3.1
pycparser 2.19
python-dateutil 2.8.0
python-editor 1.0.4
pytz 2019.2
requests 2.21.0
s3transfer 0.2.1
setuptools 41.6.0
six 1.12.0
trimesh 3.1.11
urllib3 1.24.3
Werkzeug 0.15.5
wheel 0.33.6
WTForms 2.2.1
My questions are:
Are there ways to resolve with pip options other than --force-reinstall and --no-cache-dir, which I have already tried?
Could my issue relate to the fact that the latest supported package of Python I can grab on EC2 is 3.6.8, but my requirements.txt file was created on a 3.7.4 system? (Yes yes, I know, next time I'll be wiser and just use docker from the start.)

After much hand-wringing and nearly doing Docker bootcamp, I decided to go back for another round and found the underlying problem. Inside my virtual environment, I noticed that pip list was only listing packages as listed in the $MYENV/lib/Python3.6/data/dist-packages, whereas the offending packages were all located in $MYENV/lib64/Python3.6/data/dist-packages. This led to some additional searching and this response, which suggests symlinking the lib64 version of dist-packages to the lib version, which did the trick and resolved my issue. I'm using pip version 19.3.1, and the referenced article is from over 2 years ago, so I will definitely be talking to some folks.
However, for those who want the quick solution, here are the steps.
Fresh virtual environment: Set up a fresh virtual environment as you normally would.
Symlink BEFORE pip: Do your symlinking before you start installing any packages. Generally this will look something like:
ln -s $YOURENV/lib/Python3.x/data/dist-packages $YOURENV/lib64/Python3.x/data/dist-packages
Go ahead and pip: No you can pip install -r requirements.txt as you normally would. Double-check with a pip list and you should see all your packages listed.

Related

Pandas build fail M1 Big Sur 11.1

I am using MacBook m1 which is running Big Sur 11.1 ,and I have installed Xcode-commandline-tools version 12.3 and it has installed python3.8.5 and pip3 .python3 and pip are native apps,that is they run in ARM
pip3 version 20.3.3
python3 version 3.8.5
setuptools version 51.0.0
wheel version 0.36.2
when I type python3 -m pip install pandas
The output is :
Defaulting to user installation because normal site-packages is not writeable
Collecting pandas
Downloading pandas-1.2.0.tar.gz (5.4 MB)
|████████████████████████████████| 5.4 MB 150 kB/s
Installing build dependencies ... error
And with a very long list of error about 30,000 lines (only last few lines)
and
pip3 list output is
cppy 1.1.0
kiwisolver 1.3.1
numpy 1.20.0rc1
pip 20.3.3
pyparsing 2.4.7
python-dateutil 2.8.1
setuptools 51.0.0
six 1.15.0
wheel 0.36.2
Is pandas not yet supported or I am doing it wrong and same goes with Matplotlib.
Install Miniforge for arm64 (Apple Silicon)latest installer here: installer here
Now create a conda environment for whatever version you have (I'm running 3.9.2)
conda create -n cenv python=3.9.2
conda activate cenv
conda install pandas
You will drive yourself nuts trying to get all the different packages working if you try to go from wheels/pip, at time of writing.
I think I got pandas working but couldn't get matplotlib working due to kiwi solver issues. Use miniforge/conda. This is the way.
You can try installing pandas version 0.25.3. This is the most stable release of Pandas which brings in numpy 1.19.1 which is the most stable version of NUMPY.
try running pip install pandas==0.25.3, and it should install the required packages.

Why does pip list not work correctly within a virtual environment created by pipx?

I'm on a Linux/Ubuntu system, in a folder designated by pyenv as Python 3.8.1. I've install pipx using pip install pipx. If I install an application (cowsay), and activate the virtual environment for the application, pip lists my globally installed packages. Why I'm I not seeing the site packages for the cowsay install when the virtual environment is activated?
You can see from the example below that pip list gives me the same result regardless of the activation status of the virtualenv? I know that during project development, this is not the observed response. When inside a venv, pip list gives me the site packages installed in the venv.
pipx --verion
0.15.5.1
pip --version
pip 20.2.3 from /home/steve/.pyenv/versions/3.8.1/lib/python3.8/site-packages/pip (python 3.8)
cd /home/steve/.local/pipx/venvs/
source activate cowsay/bin/activate
(cowsay):~/.local/pipx/venvs$ pip list
Package Version
----------- --------
argcomplete 1.12.1
click 7.1.2
distro 1.5.0
packaging 20.4
pip 20.2.3
pipx 0.15.5.1
pyparsing 2.4.7
setuptools 41.2.0
six 1.15.0
userpath 1.4.1
(cowsay) :~/.local/pipx/venvs$ deactivate
:~/.local/pipx/venvs$ pip list
Package Version
----------- --------
argcomplete 1.12.1
click 7.1.2
distro 1.5.0
packaging 20.4
pip 20.2.3
pipx 0.15.5.1
pyparsing 2.4.7
setuptools 41.2.0
six 1.15.0
userpath 1.4.1
Your pip command is not actually using the venv. Checking type pip will tell you where it is really resolved. pipx won't create a console script for pip within the venv, because there is no need.
The console scripts created for the venv, including a cowsay executable and a python symlink on $PATH, will be under:
~/.local/pipx/venvs/cowsay/bin
But the console script for the pip installation will have been created under a location which is not on $PATH:
~/.local/pipx/shared/bin
You should still be able to see cowsay lib within the actual venv like this:
python -m pip list
You might wonder why python -m pip is working when pip is not? In fact, the pip installation is shared between venvs, and you should be able to see the path file hack which sets that up here:
~/.local/pipx/venvs/cowsay/lib/python3.8/site-packages/pipx_shared.pth

Failed to install jupyter using pip

I tried to install jupyter using pip: pip install jupyter (even with admin privilege)
But for some reason it could not install it successfully. Here is the error:
Could not find a version that satisfies the requirement
widgetsnbextension~=3.3.0 (from ipywidgets->jupyter) (from versions:
1.0.0b2, 1.0.0b3, 1.0.0b4, 1.0.0b5, 1.0.0, 1.1.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 2.0.0b1, 2.0.0b2, 2.0.0b3, 2.0.0b4, 2.0.0b5, 2.0.0b6, 2.0.0b7, 2.0.0b8, 2.0.0b9, 2.0.0b10, 2.0.0b11, 2.0.0b12, 2.0.0b13, 2.0.0b14, 2.0.0b15, 2.0.0b16, 2.0.0b17, 2.0.0b18, 2.0.0b19, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.0.1, 3.0.0a0, 3.0.0a1, 3.0.0a2, 3.0.0a3, 3.0.0a4, 3.0.0a5, 3.0.0a6, 3.0.0a7, 3.0.0a8, 3.0.0a9, 3.0.0a10, 3.0.0a11, 3.0.0a12, 3.0.0a13, 3.0.0a14, 3.0.0a15, 3.0.0b0, 3.0.0b1, 3.0.0b2, 3.0.0b3, 3.0.0b4, 3.0.0b5, 3.0.0b6, 3.0.0b7, 3.0.0b8, 3.0.0b9, 3.0.0b10, 3.0.0b11, 3.0.0b12, 3.0.0b13, 3.0.0rc0, 3.0.0rc1, 3.0.0rc2, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.0.8, 3.1.0rc0, 3.1.0rc1, 3.1.0rc2, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.2.0, 3.2.1, 3.3.0b0) No matching distribution found for widgetsnbextension~=3.3.0 (from ipywidgets->jupyter)
Note: I also did pip install widgetsnbextension and there was no error. Yet I still could not install jupyter.
How shall I proceed?
you must first step:
pip install widgetsnbextension --upgrade
second step:
pip3 install --upgrade pip
pip3 install jupyter
jupyter notebook
Follow the commands:
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade jupyter
To run the notebook:
python3 -m pip notebook
or
jupyter notebook
A short demo:
https://www.youtube.com/watch?v=gIRYMfkpsy4
This demo illustrates the steps from scratch, if the above steps don't work
https://www.youtube.com/watch?v=BVS3U1OQzO4
If you are objective is to use Jupyter notebook, install Anaconda, it comes as a package where you have anaconda navigator and anaconda prompt to manage IDE and packages respectively.
You can straight away type "Jupyter Notebook" in Anaconda Prompt to use it.
Let's say, I need to use Jupyter notebook in a virtual environment
conda create --name venv
Now I want Jupyter in this virtual environment,
Go to Anaconda Navigator, select the desired environment, Use the Jupyter Notebook or any other IDEs.

AWS CLI not installing all the way

Trying to get a nodeJS app up on AWS, but when installing the Command Line Interface(CLI) with the command that aws it's telling me: pip install awsebcli --upgrade --user
Unfortunately I get Requirement not upgraded as not directly required:
matplotlib 1.3.1 requires nose, which is not installed.
matplotlib 1.3.1 requires tornado, which is not installed.
python-dateutil 2.7.2 has requirement six>=1.5, but you'll have six 1.4.1 which is incompatible.
blessed 1.14.2 has requirement six>=1.9.0, but you'll have six 1.4.1 which is incompatible
I am not to familiar with Python, but I do have 2.7.2 on my system and installed pip as well. Not sure what to do? I am guessing a command to get nose and tornado and blessed installed?

Can someone help me in installing python package "Prophet" on windows 10

Can someone help me in installing python package "Prophet" on windows 10 .
I tried installing python 3.5 and the dependency 'pystan' but yet I get below error.
"The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted.This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand.Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.
Command "python setup.py egg_info" failed with error code 1 in c:\users\suman\appdata\local\temp\pip-build-aqoiqs\fbprophet\"`
I spent a good two days on this issue, finally, I found the following solution
1) Download and install Anaconda 3 (for python 3.6)
2) Create an environment and call it whatever you like
3) Run that environment (which will open the command prompt)
4) follow the following steps in command prompt : (commands are shown in bold)
python -m pip install -U pip
Requirement already up-to-date: pip in c:\users\william\anaconda3\envs\testtime\lib\site-packages
conda install -c conda-forge pystan
Fetching package metadata ...............
Solving package specifications: .
Package plan for installation in environment C:\Users\William\Anaconda3\envs\Testtime:
The following NEW packages will be INSTALLED:
backports: 1.0-py36_1 conda-forge
backports.functools_lru_cache: 1.4-py36_1 conda-forge
ca-certificates: 2017.11.5-0 conda-forge
cycler: 0.10.0-py36_0 conda-forge
cython: 0.27.3-py36_0 conda-forge
freetype: 2.7-vc14_1 conda-forge [vc14]
icc_rt: 2017.0.4-h97af966_0
icu: 58.2-vc14_0 conda-forge [vc14]
intel-openmp: 2018.0.0-hd92c6cd_8
jpeg: 9b-vc14_2 conda-forge [vc14]
libpng: 1.6.34-vc14_0 conda-forge [vc14]
matplotlib: 2.1.0-py36_0 conda-forge
mkl: 2018.0.1-h2108138_4
numpy: 1.13.3-py36ha320f96_0
openssl: 1.0.2m-vc14_0 conda-forge [vc14]
pyparsing: 2.2.0-py36_0 conda-forge
pyqt: 5.6.0-py36_4 conda-forge
pystan: 2.17.0.0-py36_vc14_0 conda-forge [vc14]
python-dateutil: 2.6.1-py36_0 conda-forge
pytz: 2017.3-py_2 conda-forge
qt: 5.6.2-vc14_1 conda-forge [vc14]
sip: 4.18-py36_1 conda-forge
six: 1.11.0-py36_1 conda-forge
tornado: 4.5.2-py36_0 conda-forge
zlib: 1.2.11-vc14_0 conda-forge [vc14]
pip install fbprophet
I also had several issues installing prophet on my Windows 10 machine using conda and Python 3.7.
The following steps made it work:
Create a fresh conda environment
Install pystan and other dependencies using conda install numpy cython matplotlib scipy pandas -c conda-forge (see https://pystan2.readthedocs.io/en/latest/windows.html#steps)
Install prophet using conda install prophet -c conda-forge
In then still had a numpy error Importing the numpy c-extensions failed... as described here: https://numpy.org/devdocs/user/troubleshooting-importerror.html But this was due to my IDE which is VS Code as I could successfully run my prohet model using the command line with the activated conda environment.
So please also make sure to test the installation from the command line in order to rule out that the error stems from you IDE.
Not sure if the same trick would work on Windows, anyways posting w.r.t what just worked on RHEL 7, f.w.i.w
Motivation was installing fbprophet using conda:
conda install -c conda-forge fbprophet
and, it could install pystan==2.17.1.0 and fbprophet==0.6
So, the trick was:
uninstall the pystan package when you had earlier run pip3 install fbprophet, basically one needs to execute the following:
pip3 install pystan==2.17.1.0
pip3 install fbprophet==0.6
And one can thereby import fbprophet in python3 terminal, so working :)
I currently have a fairly vanilla install of Python 3.5 on my Windows 10 machine that I setup using Anaconda.
I was able to install Prophet using the below command, that might work for you. Depending on how your installed Python you may need to run the below command in a cmd shell that was opened as administrator.
pip install prophet
Whenever I have tried to install fbprophet package multiple issues have come.I will suggest a few steps, hopefully they should work.
Close all existing Python / R running instances. If possible restart the machine. I have seen to work as few libraries / dependencies which I want to update might be getting used by existing running instances.
Use anaconda prompt.
First install pystan -
`conda install pystan`
Now install fbprophet
'conda install -c conda-forge fbprophet
Hopefully it should work for you. Also, ensure you have admin rights.
I faced the same issue and my solution was to:-
Create a new environment with Python3.5
conda create -n pht python=3.5 anaconda
Install Prophet using the command.
conda install -c conda-forge fbprophet
I didn't install 'gcc' although this was advised before installing Prophet.

Resources