On my AWS ec2 cloud server(ubuntu) when I run the server.py file to run my server then is showed me the below error:
File "server.py", line 31, in <module>
util.load_saved_artifacts()
File "/home/ubuntu/HousePricePrediction/server/util.py", line 39, in load_saved_artifacts
__model = pickle.load(f)
ModuleNotFoundError: No module named 'sklearn.linear_model._base'
The model was created on a different system with a different version of sklearn.
After finding this error then I checked the scikit-learn version.
Check the current version:
sudo pip3 list scikit
Note: If your os isn't ubuntu then remove 'sudo'. And if you are
using 'conda' then use 'conda' instead of pip3.
Then I found the current version scikit-learn (0.20.3) and realized that maybe this version is backdated or not suitable for the system. Then I upgrade scikit-learn.
Upgrade scikit-learn:
sudo pip3 install -U scikit-learn
After that, I again run my server.py file to run the server and my server successfully runs.
Note: If your system needs a specific version of the 'scikit-learn' library for your system
then use this command:
sudo pip3 install scikit-learn=0.17 - (for example)
I believe I have successfully install TA-lib in Ubuntu VM (it is using ARM64) as when I type pip list, it shows in my python 3.8 packages together with all other modules. Unfortunatley, when I call import talib, an error as below exists
>>> import talib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ubuntu/.local/lib/python3.8/site-packages/talib/__init__.py", line 52, in <module>
from ._ta_lib import (
ImportError: libta_lib.so.0: cannot open shared object file: No such file or directory
https://imgur.com/1ZU4wZn.png
I also use the command export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH but the same error is shown.
The talib folders as per screenshot
https://imgur.com/ogIslEp.png
https://imgur.com/BIqXi37.png
It is a problem specific to ARM64 or AArch64 as I manage to install talib in X64. I successfully installed talib in X64 with the same Ubuntu version , the only difference is only it is in X64
Both ARM64 and X64 talib has the libta_lib.so.0 in ./ta-lib/src/.libs/libta_lib.so.0
It's normal, you shouldn't use pip to install python3 libraries, use pip3 instead, pip is for python2. Probably what happened is that you installed ta-lib for python2(that use pip) instead of python3(that use pip3).
Please, let us know if you still have problem after you executing the following command in a shell:
pip3 install ta-lib
The issue is specific to ARM/Aarch64 Virtual Machine in Ubuntu 20.04.
In your directory of Ubuntu, type
sudo ldconfig
To confirm if TA-lib able to import with error, reboot your VM.
This is the simplest way to fix the issue permanently without export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
I need to install a Python package in a custom Docker container that I'm building from the official 'ubuntu' Docker image, so I want to minimize how much space this uses. Python3 installs fine and runs, but for some reason, pip is not included.
So I installed via apt install python3-pip, this works but it is a whopping 300 megs and takes a couple of minutes to install (apparently because it installs a sh*load of stuff to build binary packages from gcc etc).
Of course I could uninstall python3-pip from the image after installing the dependencies I want, and additionally use apt autoremove to get rid of 299 megs. However this takes another minute.
So although the above works, it significantly increases the build time of my Docker image. So I tried to see if there was a way of installing the dependency without pip:
I tried downloading the dependency's .tar.gz from PyPI, extracted, and tried python3 setup.py install, but this gets me an odd error:
Traceback (most recent call last):
File "setup.py", line 59, in <module>
from distutils import log
ImportError: cannot import name 'log'
I thought perhaps I need to install setuptools, or upgrade distutils.
I tried to use get-pip.py from the official site but that failed too:
Traceback (most recent call last):
File "get-pip.py", line 20890, in <module>
main()
File "get-pip.py", line 197, in main
bootstrap(tmpdir=tmpdir)
File "get-pip.py", line 82, in bootstrap
import pip._internal
File "/tmp/tmpjpa5gs_x/pip.zip/pip/_internal/__init__.py", line 40, in <module>
File "/tmp/tmpjpa5gs_x/pip.zip/pip/_internal/cli/autocompletion.py", line 8, in <module>
File "/tmp/tmpjpa5gs_x/pip.zip/pip/_internal/cli/main_parser.py", line 8, in <module>
File "/tmp/tmpjpa5gs_x/pip.zip/pip/_internal/cli/cmdoptions.py", line 17, in <module>
File "/tmp/tmpjpa5gs_x/pip.zip/pip/_internal/locations.py", line 10, in <module>
ImportError: cannot import name 'sysconfig'
which is very weird because if I start python3, import sysconfig works fine.
I also tried apt install python-pyyaml (the dependency I need in my Docker image) but that doesn't seem to exist.
So I'm out of options.
I ran into a similar issue and wanted to give an alternative solution.
On Ubuntu 20.04 build-essential and python3-dev are recommended packages for python3-pip, therefore you can use the --no-install-recommends option to skip them:
RUN apt update -y && \
apt install python3 python3-pip --no-install-recommends -y && \
apt clean
This took my image from 420MB to 165MB, and obviously the build time was also quicker.
Note: this will work fine for pure-Python packages, but you will likely need build-essential and python3-dev if you want to compile anything
Useful links
https://packages.ubuntu.com/focal/python3-pip
https://manpages.ubuntu.com/manpages/xenial/man8/apt-get.8.html
The Debian and Ubuntu package for PyYAML is called python-yaml, not python-pyyaml.
sudo apt install python-yaml
or
sudo apt install python3-yaml
respectively.
(It seems to be common in Debian/Ubuntu package names to drop any additional "py" prefix that a Python package might have: it's apt install python-tz instead of python-pytz, too. They don't seem to like the py-redundancy.)
I have been installing pip from the pip-get.py in Docker (Ubuntu) containers for a few years without problem. For me it is the best way to not-get pip-out-of-date warnings or (at some point, some time ago) SSL related errors.
So the second part of your answer is close, but your python install seems a bit too minimal, you need sysconfig as provided by python-distutils. You can try this rather minimal Dockerfile:
FROM ubuntu:latest
MAINTAINER Anthon
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y \
python3 \
python3-distutils \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# this gets you the latest pip
COPY pip/get-pip.py /tmp/get-pip.py
RUN python3 /tmp/get-pip.py
RUN pip3 install pyyaml
which I ran using this Makefile:
doit: pip/get-pip.py
docker build .
pip/get-pip.py:
-#mkdir pip
curl https://bootstrap.pypa.io/get-pip.py -o pip/get-pip.py
(those need to be TAB characters on the indented lines) to make sure pip-get.py is available from the context (you can of course download it from within the Dockerfile, but that is not necessary). That ends in a succesful PyYAML install, but it will be slow.
I recommend you start using ruamel.yaml (disclaimer: I am the author of that package), by changing the last line of the Dockerfile to read:
RUN pip3 install ruamel.yaml
Apart from many bugfixes in the original PyYAML code it is based upon, ruamel.yaml supports YAML 1.2 and YAML 1.1 (replaced in 2009 and the version PyYAML supports), and installs the appropriate version from a .whl file, so you'll have the fast C loader available in your container (PyYAML doesn't do that).
You can load a YAML file using the C-loader in ruamel.yaml using:
from pathlib import Path
from ruamel.yaml import YAML
path = Path('yourfile.yaml')
yaml = YAML(typ='safe')
data = yaml.load(path)
When there is no apt install something that is available for a python package, here is how to do it. Thanks to #Anthon and #digitalarbeiter as their answers provided important information to arrive at solution.
To install via a setup.py file (esp. useful in Ubuntu Docker container):
it was sufficient for me to
apt install python3-distutils
<download package, tar xvf, cd to folder>
python3 setup.py install
This method of installation only works for pure Python packages (should not not be a surprise), which means that Python packages that have non-pure Python dependencies may not install or, if they do, will have some functionality unavailable.
Note that even before installing python3-distutils, python3 -m distutils worked; this implies that the builtin distutils, that comes with Python3 via apt install python3, is not the full distutils; I did not know that, is this fact ever mentioned anywhere?
To install pip without the gcc toolchain: it was sufficient for me to
apt install python3-distutils
wget https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
Then pip install pyyaml completed. It seemed to install from a .tar.gz so it is the pure Python implementation too. Not surprising. This technique is useful if a package is not installable via apt install python3-<package>
The above methods required only a few megs of disk space.
A couple other missing pieces of the puzzle for me were:
apt install python3-<something>:
I missed the fact that many Python packages are distributed this way in Debian, handy for packages that have a C implementation (or C dependencies) since no compilation necessary.
AND I didn't know that for packages that are called py<something> on PyPI, the apt install is python3-<something> NOT python3-py<something>. Unfortunately apt search pyyaml was no help here.
apt search <something>: I had sort of forgotten about it because bash on (desktop) Ubuntu automatically suggests the right package to download when a command is not found.
In particular apt search yaml yields 81 packages that provide YAML read/write in several languages (Python 2, Python 3, nodejs, Java, Go, Ruby, Erlang, Lua, Perl, C, C++, Clojure), linter, schema validator, etc.
Multiple search terms are AND'ed so apt search yaml python3 showed the python3-yaml that I missed.
Unfortunately apt search pyyaml produces no results, eventhough the Source and Homepage fields of apt show python3-yaml contain the word "pyyaml". I could not find a way to make search include those fields.
My question is two-fold, but I believe they are related:
I have recently realized that I have a permission issue with my python3 installation. If I run python3, I cannot import any packages. However, if I run sudo python3, all packages import flawlessly. Similarly, I must run jupyter notebooks with sudo as well. From reading, I believe I have made an error by installing many packages with sudo pip. Is there a straightforward way to fix this?
I am trying to install ipython extensions without luck. Using
sudo pip install https://github.com/ipython-contrib/IPython-notebook-extensions/archive/master.zip --user
I receive an error error: command 'x86_64-linux-gnu-gcc' failed with exit status 1.
If I download the zip file, and run sudo python3 setup.py install, the error is:
IPython-notebook-extensions/configure_nbextensions.py", line 80, in <module>
except (psutil.ZombieProcess, psutil.AccessDenied):
AttributeError: 'module' object has no attribute 'ZombieProcess'
If I run python setup.py install, the resulting error is:
File "/usr/lib/python2.7/dist-packages/copyreg/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.
This is the same error that I get when importing a package from python (when opened with python, rather than sudo python) in the terminal.
I am using Ubuntu 14.04.
except (psutil.ZombieProcess, psutil.AccessDenied):
AttributeError: 'module' object has no attribute 'ZombieProcess'
From the message,The old version of psutil is missing these module.
you should update the package of psutil use:
sudo pip3 install --upgrade psutil
To install pymongo for pypy3 2.1 Beta 1, I installed pip using the following commands:
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
./pypy ez_setup.py --user
where pypy is the executable of pypy3 2.1 Beta 1. After that, pip and pip-3.2 will come to the current directory. But when running pip or pip-3.2, I get the error:
Traceback (most recent call first):
File "pip-3.2", line 5, in <module>
from pkg_resources import load_entry_point
zipimport.ZipImportError: pip==1.4.1
It seems that the problem comes from from pkg_resources import load_entry_point of the pip/pip-3.2. But this statement runs OK when I put it in pypy or python3 IDLE. What's the matter? How to solve this problem and proceed to install pymongo for pypy3 2.1 Beta 1? Thank you.
PS: I'm using Ubuntu. python3 is installed in the system. If u need any other information pertaining to solving the question, please comment.