In my environments created with anaconda, the same packages installed with conda are not compatible when I try to install with pip.
Is there a difference how pip and conda handle dependencies?
Here an example of requirements.txt
# Python version 3.9.13
django==2.2.5
djangorestframework==3.14.0
gensim==4.1.2
joblib==1.1.1
nltk==3.7
numpy==1.21.5
openpyxl==3.0.9
pandas==1.4.4
pickleshare==0.7.5
scikit-learn==1.1.3
seaborn==0.12.0
spacy==3.3.1
tensorflow==2.9.1
unidecode==1.2.0
conda allows you to create the environment, pip reports incompatibility between django and djangorestframework.
Conda checks, if all packages that will end up in the environment are compatible with each other and tries to find the optimal solution - considering all package versions.
Pip is less strict and only checks, if the new package is compatible with the existing ones. It doesn't change versions of the previously installed packeages.
Pip installs packages from from pypi.org, while conda installs from anaconda.org. The packages are not exactly the same, since the Anaconda staff authors new packages and tries to increase their compatibility with the older ones.
However, sometimes you are not interested in 100% compatibility but just want to use the latest features. Then pip is good enough because your unit tests will tell you if something goes wrong.
Related
I am trying to create a conda environment that will enable me to use JupyterLab extensions as well as matplotlib (among other things). When installing JupyterLab extensions in the past, I have been prompted that I require nodejs>=12.0. When I try installing both nodejs and matplotlib in a new environment, I receive a conflict. Any suggestions for how to proceed?
I do the following:
conda create -n test python=3.10
conda activate test
conda install jupyterlab
conda install nodejs
conda install matplotlib
Installation proceeds without a problem until the final step, at which point I am informed:
The following NEW packages will be INSTALLED:
blas anaconda/linux-64::blas-1.0-mkl None
...
zstd anaconda/linux-64::zstd-1.5.2-ha4553b6_0 None
The following packages will be DOWNGRADED:
icu 68.1-h2531618_0 --> 58.2-he6710b0_3 None
nodejs 16.13.1-hb931c9a_0 --> 10.13.0-he6710b0_0 None
Proceed ([y]/n)?
Of course, if I allow it to downgrade nodejs, then I won't be able to use Jupyter Lab extensions, which require nodejs>=12.0. Any suggestions for how to proceed?
(Other packages want to downgrade nodejs to 10.3 as well, e.g. conda install jupyter, but for clarity, I've focused on matplotlib.)
I ran into a similar issue, and I got around it by not installing node via conda. I think node just needs to be on your path so you can install with https://nodejs.org/en/download/ or if you need to support multiple node versions you can use nvm.
I am working on an existing Python 3 code-base that provides a setup.py so the code is installed as a Python library. I am trying to get this internal library installed with its own dependencies (the usual data science ones e.g. pandas, pyodbc, sqlalchemy etc).
I would like to have this internal library to deal with these dependencies and assume that if that library is installed, then all the transitive dependencies are assumed to be installed. I also would like to have the Anaconda (conda) version of the package rather than the pip version.
I started with a requirements.txt, but moved quickly to this field in setup.py:
install_requires=[
"pyodbc>=4.0.27",
"sqlalchemy>=1.3.8",
"pandas>=0.25.1",
"requests>=2.22.0",
"assertpy>=0.14",
"cycler>=0.10.0",
]
However when I run the installation process:
either with python setup.py install --record installed_files.txt
or with pip install .
I see that there is some gcc / C++ compilation going on that shows logs about Python wheels (I don't completely understand the implications of Python eggs and Python wheels, but AFAIK if conda is available then I should go with the conda version rather than eggs/wheels because then I don't have to take care of the C++ code underneath the Python code).
I really would prefer having conda to install these C++ blobs wrapped in some Python code as a libraries e.g. pandas.
is it possible at all to have conda driving the installation process described in setup.py so I am not dealing with gcc?
how can I make sure that other Python code depending on this internal library (installed via setup.py) is using the same (transitive) dependencies defined in that setup.py?
Regardless the installation method, how can I make sure that the dependencies for e.g. pandas are installed as well? Sometimes I see that numpy as a dependency of pandas is not installed when running setup.py, but I would like to avoid doing this manually (e.g. with some requirements.txt file).
pip doesn't know about conda, so you cannot build a pip-installable package that pulls in its dependencies from conda channels.
conda doesn't care about setup.py, it uses a different format for recording dependencies.
To install your code with conda, you should create a conda package, and specify your dependencies in a meta.yaml file. Refer to the documentation of "conda build" for details.
https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
2 pip are installed one is v 9 and other is v 19
how to update the pip to the latest version 19?
I m unable to download some packages due to this version conflict.
and I want only one latest pip version should be instaled, and it will be used in all the packages.
You have two different versions of python and pip installed. This is common.
/usr/local/lib/python2.7/dist-packages/pip was most likely installed by the operating system, and allows you to sudo pip install Python 2.7 packages.
/home/manish/.local/lib/python3.5/site-packages/pip is in your home folder, which means you shouldn't have to elevate your user privileges to pip install a package. This is for Python 3.5.
Managing Python versions and dependencies can be massively confusing, especially if you're just trying to follow along with a tutorial.
The best tool I can recommend to make sense of this is pyenv. It lets you install any version of python in your home folder, and allows you to switch between on a per-project (folder) basis by creating a .python-version file. pyenv.
When installing Theano anaconda automatically tries to install pygpu despite this being an optional dependency. I have deleted the .theanorc file from my windows user directory.
Also when running my application Theano tries to load from the GPU. It's like it remembers somehow?
conda install theano
Fetching package metadata .............
Solving package specifications: .
Package plan for installation in environment
C:\Users\zebco\Miniconda3\envs\py35:
The following NEW packages will be INSTALLED:
libgpuarray: 0.6.9-vc14_0
pygpu: 0.6.9-py36_0
theano: 0.9.0-py36_0
Proceed ([y]/n)?
As you can see I've only specified to install theano yet conda wants to install everything including optional dependancies.
Your assumption that pygpu is optional is dependent on the package manager you are using.
Regular Python (pip)
If you are using a direct Python install (obtained using brew or Python site) then you would be using pip to install theano. This basically comes from
https://pypi.python.org/pypi/Theano/1.0.0
If you download the file and unzip it. Open setup.py, you will see below lines
install_requires=['numpy>=1.9.1', 'scipy>=0.14', 'six>=1.9.0'],
So they are set as the dependencies for this package. Which means when you install theano you will also get numpy, scipy and six.
Anaconda Python (conda)
Now coming to Anaconda python. Anaconda doesn't use a package format that PyPI or pip uses. It uses its own format. In case of Anaconda you should be using conda to install the packages you need and not pip.
Conda has channels which is nothing but a repository which has some packages available. You can install a package from any channel using below
conda install -c <channel-name> <package-name>
The default channel is conda-forge. If you look at the theano package over there
https://anaconda.org/conda-forge/theano/files
And download and extract it. There will be a info/recipe/meta.yml file. You will notice below content in the same
requirements:
build:
- ca-certificates 2017.7.27.1 0
- certifi 2017.7.27.1 py36_0
- ncurses 5.9 10
- openssl 1.0.2l 0
- python 3.6.2 0
- readline 6.2 0
- setuptools 36.3.0 py36_0
- sqlite 3.13.0 1
- tk 8.5.19 2
- xz 5.2.3 0
- zlib 1.2.11 0
run:
- python
- setuptools
- six >=1.9.0
- numpy >=1.9.1
- scipy >=0.14
- pygpu >=0.6.5,<0.7
Which specifies that if you want to run this package then pygpu is also on of its dependencies. So conda downloads pygpu as a dependency which you though was optional (which is probably true if you were using regular python and pip)
Update:
Usually, 'Optional Dependency' is an oxymoron. Something optional is not a dependency, a dependency is a software package another piece of software depends on to function for features.
One may get by without a dependency if the dependency does not interact with the package except for one atomized feature which is not being used. As a beginner I would suggest you not take this path.
I am not super familiar with Theano, but Theano can use the system's GPU to speed up its computations, and it seems to me pygpu and gpulibarray are what enable this functionality. Which means it is not optional.
I believe pygpu is 'optional' if you do not wish to use the GPU for speeding up computation (only done if the GPU is powerful enough to be useful for this).
The --no-deps command above allows you to install a package without its dependencies but that is rarely wise, unless one really knows what they are doing. As a beginner I would not recommend you go down this path yet. Conda was designed specifically to ensure scientific packages are easily managed with all necessary stuff installed without any fuss or muss. pip is a general python package manager, but is not built specifically for scientific packages.
If you wish to install theano without installing its dependencies, then you have one of three options:
use conda install theano --no-deps.
Install it using pip instead of conda, using pip install theano. This will install theano, numpy, scipy and six but not pygpu and libgpuarray.
Create a custom conda build file for Theano. Documentation is at:
https://conda.io/docs/user-guide/tasks/build-packages/index.html
Original Answer:
You probably know this already but, use this command instead:
conda install theano --no-deps
This does not install dependencies of the package. If you already have the essential dependencies installed, as it would seem, this should work out for you.
libgpuarray is a dependency of pygpu. With this command switch neither will be installed.
Can you share the .yaml file that you edited?
Sometimes pip install launches a lengthy compilation process. Sometimes it does not. This was most notable with numpy, because it takes significant time to compile but is negligible time when installing binaries. I have an Ubuntu 14 machine where it always compiles numpy, and an Ubuntu 16 machine where it never compiles.
I assumed that Ubuntu 14 packages were no longer available or something. But then I launched a brand new VM with this same older OS, and pip install numpy, went super fast (no compiling). So clearly it is not simply the OS version impacting me. What is going on here?
It's probably a difference in the version of pip you're using. Both binary and source wheels exist in pypi so the question is which will pip choose. I found that on the same machine, pip 1.5.4 would choose the source wheel but pip 9.0.1 would choose the binary one.
Newer versions of pip (ex. 9.0.1) have options for controlling this behaviour: https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-no-binary
I'm guessing maybe older versions of pip don't even support binary wheels.
So try upgrading pip and then it should install without compiling.