I have a 2-part question about conda vs. pip virtual environments. I found great information on the answers What is the difference between pip and conda? and Does Conda replace the need for virtualenv? but still have something unclear.
I have a given python project (say PR) that I need to install and further develop on a linux server (say S) where python is installed with anaconda. Now, the usage/installation instructions of PR tell me to use python to create virtual environment and pip to install all packages. That is,
python3 -m venv PR
pip install --editable . (the dot included at the end)
According to "pip install --editable ./" vs "python setup.py develop" the latter reads the file setup.py (included in PR) which contains a function setup(...) with option install_requires listing all the required packages and installs them automatically. I have tested this on my own computer (which does not have conda) and it works fine. At least no error messages.
Now I need to further develop PR on S. My question Part 1: can I use conda instead of pip to create and update virtual environment? If yes, what would be the conda command replacing pip install --editable . ? I'm positive I will later need to install other packages as well. I'm worried about conflicts between conda/pip.
On S, I have Spyder and no other python IDEs. I have never used Spyder but I'm very familiar with PyCharm (Windows) and VS Code (Linux) so I assume debugging with Spyder will be similar to those. My question Part 2 (tied to Part 1): if I have to use pip to install packages, does Spyder see those? Or can it only see conda-installed packages?
(Edit/update): Thank you Carlos for comments. I continue my question:
I created and activated the virtual environment (VE) with conda
conda create PR_venv
conda activate PR_venv
Installed pip with
conda install pip
(this upgraded pip and installed several other packages too, including newer version of python). Installed PR and its required packages with pip
pip install -e .
Now, if I run the PR package inside this active VE interactively from the terminal, everything works fine. I would like to do the same from within spyder, to get the IDE debugging abilities in my hand.
When I start spyder, open a python file to be run, click "Run" button, it crashes in the import statements.
Spyder cannot see the installed packages. It can see only the local package PR but none of the packages installed by pip for this VE.
I am not sure what is the correct question here; I'm confused how are conda VEs related to spyder/jupyter/ipython ? I cannot find information in the conda documents about this.
I cannot find from spyder documents anything about VEs. Do I have to somehow re-install the packages (how?) inside Spyder? It seems pointless because the packages are installed already.
(Edit/Update 2): The information on https://docs.spyder-ide.org/current/installation.html makes me even more confused: Spyder is presented as both a stand-alone program and as a python package. So do I have to re-install Spyder inside the VE(?!) with
conda activate PR_venv
conda install spyder
Any clarification would be appreciated. I have always thought that the IDEs are stand-alone programs and that's it. This Spyder setup twists my brains into pretzel.
(Spyder maintainer here) About your questions:
can I use conda instead of pip to create and update virtual environment?
Yes, you can. Please see here to learn about the functionality offered by conda for managing environments.
If yes, what would be the conda command replacing pip install --editable . ?
Conda doesn't offer a good replacement for that command. However, you can still use it in a conda environment, as long as all you've installed all your package dependencies with conda before running it. That would avoid mixing conda and pip packages, which usually leads to really bad results.
if I have to use pip to install packages, does Spyder see those? Or can it only see conda-installed packages?
Spyder can work with pip and conda packages without problems. Just make sure of not mixing them (as I said above) and you'll be fine. In addition, please read our documentation to learn how to connect a local Spyder instance to a remote server.
Part 1: yes I can use conda to create VE and pip to install packages
conda create PR_venv
conda activate PR_venv
conda install pip
pip install --editable .
conda list
The last line shows which packages are installed by conda and which by pip (shown as pypi)
Part 2: spyder by default cannot see the packages. Need to do two things:
conda install spyder-kernels
Open Spyder and Tools > Preferences > Python Interpreter > Use the following interpreter > [full path to VE python command]
Restart Spyder. Now it can see the packages.
(Edit:) this link is great: https://github.com/spyder-ide/spyder/wiki/Working-with-packages-and-environments-in-Spyder
Recently, I upgraded to Anaconda3 2020.07 which uses python 3.8. In past versions of anaconda, tensorflow was installed successfully. Tensorflow failed to be installed successfully in this version.
I ran the command below;
conda install tensorflow-gpu
The error message that I received is shown below;
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- tensorflow-gpu -> python[version='3.5.*|3.6.*|3.7.*|>=3.7,<3.8.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0|>=2.7,<2.8.0a0']
Your python: python=3.8
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
The following specifications were found to be incompatible with your CUDA driver:
- feature:/win-64::__cuda==11.0=0
Your installed CUDA driver is: 11.0
Is there a conda command with the right parameters to get tensorflow installed successfully?
UPDATE:
TF is now compatible with Python 3.8
Tensorflow is not compatible with Python 3.8. See https://www.tensorflow.org/install/pip
You need to downgrade your python version :
conda install python=3.7
Create an environment with python 3.7 and then activate it:
conda create -n p37env python=3.7
conda activate p37env
And install tensorflow.
This worked for me, and found out the answer from the Anaconda user guide (under how to use a different python version: https://conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-python )
From the requirement page:
Python 3.8 support requires TensorFlow 2.2 or later.
So there is a verison of Tensorflow compatible with python 3.8.
The problem is that TensorFlow 2.2.0 is not available through conda on Windows, this should be the reason why you get PackagesNotFoundError when running
conda install tensorflow=2.2
EDIT 15/03/21
Tensorflow 2.3.0 is compatible with Windows
i think we have two options here
pip install tensorflow
or we can use another env of anaconda such as like this below
conda create -n tf tensorflow pydotplus jupyter
conda activate tf
Actually you can directly use pip inside anaconda prompt, after I tested it, I found the conda is capable with pypi, first run the anaconda prompt with administrator permission (in windows), then enter "conda update --all" to make sure all the packages are latest, finally enter "pip install tensorflow" to install (the new version of tensorflow already includes tensorflow-gpu).
Then using VS code to open an ipynb and run
import tensorflow as tf
tf.test.gpu_device_name()
everything looks good.
For more info please refer to Anaconda official docs: https://docs.anaconda.com/anaconda/ .
Latest development for tensorflow installation on anaconda.
https://anaconda.org/anaconda/tensorflow
https://anaconda.org/anaconda/tensorflow-gpu
9 days ago, Anaconda uploaded a new tensorflow v2.3 package. Anaconda3 2020.07 (uses python v3.8) users can easily upgrade to tensorflow v2.3 with the following commands;
conda install -c anaconda tensorflow
conda install -c anaconda tensorflow-gpu
I have personally tested that the installation worked successfully.
The other answers for this question have now become obsolete.
Expanding upon William's answer here with more explicit instructions and caveats. Pip is the recommended way to install latest version of tensorflow as per tensorflow's installation instructions -- "While the TensorFlow provided pip package is recommended, a community-supported Anaconda package is available."
Here is the code that uses pip to do the installation in a Conda environment:
conda create -n env_name python=3.8
conda activate env_name
conda install pandas scikit-learn matplotlib notebook ##installing usual Data Science packages that does include numpy and scipy
pip install tensorflow
python -c "import tensorflow as tf;print(tf.__version__)" ##checks tf version
In general, we should be careful while mixing two package managers (conda and pip). So, it is suggested that:
Only after conda has been used to install as many packages as possible
should pip be used to install any remaining software. If modifications
are needed to the environment, it is best to create a new environment
rather than running conda after pip.
For an example, if we would like to install seaborn in the just created env_name environment, we should:
conda create --name cloned_env --clone env_name
conda activate cloned_env
conda install seaborn
Once we check the cloned_env environment is working fine, we can delete the env_name environment.
I was running into the same issue in conda prompt for Python 3.8.5 and fixed it using a Python wheel instead. Here are the steps:
Open conda prompt and install pip if you don't have it already: python -m pip install --upgrade pip
python -m pip install --upgrade https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.4.0-cp38-cp38-win_amd64.whl
Note: If you need a CPU specific tensorflow, use this wheel: https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow_cpu-2.4.0-cp38-cp38-win_amd64.whl
I just downgraded python to 3.7 as tf is not avialable to 3.8 version also I cannot use virtualenv for code that's why
The only working answer for me is:
conda install -c conda-forge tensorflow
It appears that tensorflow 2.5 on GPU has issues with spyder. So, I made new environment and installed tensorflow gpu as suggested by anaconda. Now I have to use either prompt or jupyter . At least it works
For macos users I suggest create an environment with python 3.7 and install tensorflow there.
You can run these commands too:
conda create -n new_env_name python=3.7
conda activate new_env_name
I had a similar problem in Anaconda Spyder. Here was my solution (In the Anaconda Console):
conda install pip
pip install tensorflow ==2.2.0
I'm currently running Python 3.7.6 (64bit) in a few Conda envs, all of them I use to work in jupyter.
I added Jupyter outside of conda for my qgis python 3.7 (32bit) and all my conda envs now point to this kernel and refuse to work for everything except qgis.
I've worked around this by deleting the entire path and all the conda kernels launch again, which then breaks the qgis kernel.
Is there a way I can easily remove the python 3.7 paths from the conda envs so they ignore it?
Or should I just not use the python that comes with qgis and build a conda env from scratch?
I've found the solution:anaconda - graphviz - can't import after installation
I want to use graphviz and follow the commend in https://anaconda.org/anaconda/graphviz
run following in terminal
conda install -c anaconda graphviz
However no matter in Jupyter Notebook, python or Pycharm to import graphviz, it always shows
ModuleNotFoundError: No module named 'graphviz'
How to solve this problem? Thank you.
PS:
when run which python in terminal: it return /opt/anaconda3/bin/python, therefore I use anaconda environment by default. And I have only one environment in anaconda that is root.
when I run conda list in terminal, I can find this line :
graphviz 2.40.1 hefbbd9a_2
I found a weird thing:
my pip and conda use the same environment:
run :which pip
get : /opt/anaconda3/bin/pip
run : which conda
get : /opt/anaconda3/bin/conda
However when I run pip list, I cannot find graphviz and many other packages which shows in conda list. For these packages show in conda list but not in pip list, I also cannot import them no matter in Jupyter notebook, python, pycharm etc. Why this happens?
After using "conda install attrs", other package installations are working fine without any http connection or ModuleNotFoundError errors. Please try and let me know.
I am trying to run some code using ortools on a python environment. I did not have troubles on a windows machine but I am having problems on mac (10.12.6). if in my virtual environment I run
pip freeze
or
conda list
ortools appears in my list of installed packages. But if I try to use it
ipython
from ortools.linear_solver import pywrapplp
I get an error saying that there's no module named ortools. If I go to
mac/anaconda3/envs/nameenv/lib/python3.6/site-packages I do have a folder called ortools with some python files including pywrapplp. Do you know what I am doing wrong ?
EDIT
following request from coments:
import os
os.getcwd()
returns '/Users/imac'
which ipython
/anaconda3/bin/ipython
Installing ortools is a bit of a headache. It was some days ago, I think I finally made it with
easy_install ortools
I think it is a problem with the path. I guess because I did not install it with conda it does not find the package. I got around writting:
sys.path.append('/anaconda3/envs/env_name/lib/python3.6/site-packages/')
at the begining of my ipynb. That way I can run ortools.
You could have several python interpreter installed (python2 and python3)
So if you want to use it with ipython, which seems to be bind on python3 in your case.
First check if you have pypi package ortools installed
ipython -m pip show ortools
If you got an error it means the package is not installed.
so you can easily install it using:
ipython -m pip install --user ortools
note: We provide Pypi ortools package (64 bits) for Manylinux, Windows and MacOS.
You can also rebuild it from source https://developers.google.com/optimization/introduction/installing/source