I am using conda version 4.5.11, python 3.6.6, and Windows 10.
I create a virtual environment using conda
conda create --name venv
When I check for installed packages
conda list
it is (as expected), empty.
But
pip list
is quite long.
Question #1: Why? - when I create a virtual environment using
python -m venv venv
the pip list is empty.
When I am not in an activated virtual environment, then
conda list
is also quite long, but it isn't the same as the pip list (* see follow up below)
In general, the pip list is a subset of the conda list. There is at least one exception ('tables' in the pip list, not in conda list) but I haven't analysed too closely. The conda list changes/displays some (all?) hyphens to underscores (or pip does the reverse). There also a few instances of versions being different.
Question #2: Why? (and follow up questions - can they be? and should I care?)
I was hoping to have a baseline conda 'environment' (that may not be the right word) -ie, the packages I have installed/updated into Ananconda/conda and then all virtual environments would be pulled from that. If I needed to install something new, it would be first installed into the baseline. Only when I need to create an application using different versions of packages from the baseline (which I don't envision in the foreseeable future) would I need to update the virtual environments differently.
Question #3: Am I overthinking this? I am looking for consistency and hoping for understanding.
-- Thanks.
Craig
Follow Up #1: After installing some packages to my empty conda venv, the results of conda list and pip list are still different, but the pip list is much shorter than it was, but is a subset of the conda list (it does not include two packages I don't use, so I don't care)
Follow Up #2: In the empty environment, I ran some code
python my-app.py
and was only mildly surprised that it ran without errors. As expected, when I installed a package (pytest), it failed to run due to the missing dependencies. So ... empty is not empty.
1. conda list vs pip list
If all you did was create the environment (conda create -n venv), then nothing is installed in there, including pip. Nevertheless, the shell is still going to try to resolve pip on using the PATH environment variable, and is possibly finding the pip in the Anaconda/Miniconda base environment.
2. pip list is subset of conda list outside env
This could simply be a matter of conda installing things other than Python packages, which pip doesn't have the option to install. Conda is a more generic package manager and brings in all the dependencies (e.g., shared libraries) necessary to run each package - by very definition this is a broader range than what is available from the PyPI.
3. Overthinking
I think this is more of a workflow style question, and generally outside the scope of StackOverflow because it's going to get opinionated answers. Try searching around for best practice recommendations and pick a style suited to your goals.
Personally, I would never try to install everything into my base/root Conda environment simply because the more one installs, the more one has dependency requirements pulling in different directions. In the end, Conda will centralize all packages anyway (anaconda/pkgs or miniconda3/pkgs), so I focus on making modular environments that serve specific purposes.
Related
I am using Conda on a shared compute cluster where numerical and io libraries have been tune for the system.
How can I tell Conda to use these and only worry about the libraries and packages which are not already there on path?
For example:
There is a openmpi library installed and the package which I would like to install and mange with Conda has it also as a dependency.
How can I tell Conda to just worry about what is not there?
One trick is to use a shell package - an empty package whose only purpose is to satisfy constraints for the solver. This is something that Conda Forge does with mpich, as mentioned in this section of the documentation. Namely, for every version, they include an external build variant, that one could install like
conda install mpich=3.4.2=external_*
signaling that it will be supplied by the host. One can consult the recipe's meta.yaml for a concrete example.
I don't think this is great (seems like a lot of work), but I also don't know of a better alternative.
Given the pending retirement of python 2 support in January 2020, we should be thinking about migrating from Miniconda2, which is python 2 based, to Miniconda3, which is python 3 based.
As of the date of asking this question, the Miniconda web pages don't address this topic.
Is there a good procedure for converting from Miniconda2 to Miniconda3 that will preserve the existing environments created under Miniconda2?
You could try to upgrade your python version as suggested in this answer:
conda install python=3.7
But I'm not sure how safe that would be. (unsafe)
A safe approach is to simply install miniconda3 into a new path on your machine reproduce identically your current environments on the new miniconda installation.
To do that, you'll have to create a spec list for each of your environments in miniconda2 by:
conda activate myenv
conda list --explicit > myenv-spec-file.txt
Then under your miniconda3 installation, do:
conda create --name myenv --file myenv-spec-file.txt
The conda docs have detailed instructions on this process.
Keep in mind that when you install miniconda3, it will add an entry into your .*rc file (e.g. .bashrc, if using bash) and the new conda based on python 3 will be used when running any conda command. So you should create your spec files prior to installing miniconda3.
Edit: As pointed out by merv and nekomatic, upgrading conda in-place is not safe.
I use Pycharm to create and manage my virtualenvs in my projects.
The problem is that after adding a library with pycharm, when I type the command (pip3 freeze --user), the library does not appear in the command result.
I have to manually type the pip install command each time so that the library is visible.
What manipulation should I do in PyCharm to solve this problem?
For what you are saying, the first thing that comes to mind is that you should use:
pip freeze
And not
pip3 freeze
Because the command mapped to the pip version when you have virtualenv activated is the first. Note that for installing you seem to use pip, and not pip3
Moreover, the --user option afaik is related to the packages installed in the user folder:
--user Install to the Python user install directory for your platform. Typically
~/.local/, or %APPDATA%\Python on
Windows. (See the Python documentation for site.USER_BASE for full details.)
If your packages are installed in the virtualenv folder, I would tell you to not use that option.
Also please make sure you have your virtualenv activated. In linux you can do so by source path/to/virtualenv/activate
Edit
I understand that the reason you are using pip3 is because you may have different versions of Python in your machine. Let me explain you a bit further how it works, because version management is usually a headache for many programmers and it is common to find problems when doing so.
If you install different versions of Python in your linux machine, and you do that as root, then the installation will proceed for the whole system. Usually Python2 installation folder for Linux machines is /usr/bin/python. However, I am uncertain of which directory is used for Python3 installations. You can check that easily by doing whereis python3. You can serach the path to binary of any command by doing whereis command. Note that this works also for whereis python as far as you don't have virtualenv activated.
Aditionally, the link to the binary of a command (or the set of instructions to be exectued, more broadly) is defined in certain folders in Linux, depending on whether you created the command as root or as a user, and possibly also on the distro. This works differently in Windows, that uses the Registry Edit utility to handle command mappings. When you enable your virtualenv, what you are doing is creating an environment that enables mapping system commands such as python to the Python installation in your virtualenv folder.
When you disable the virtualenv, the command points again to the default installation path. Same happens with pip, so incorrect usage of this tool may result in different packages being installed in different locations, and therefore not appearing available for the right Python version at any given circumstance.
In Linux, environment variables are shell dependent, though you can write them out with echo $variable and set them with variable=value (from bash). The search path is simply called PATH and you can get yours by typing echo $PATH.
Source: https://askubuntu.com/a/262073/426469
I encourage you to check other questions in SE network such as this: https://unix.stackexchange.com/a/42211/96121, to learn more about this.
Addendum
Quick tip: it is common to use the pip freeze command as follows:
pip freeze > requirements.txt
It is a standard that leads to understanding that modules in such file are required for the correct functioning of your application. That lets you easily exclude the virtualenv folder when you install the program in another computer, since you can readily know the requriments for a fresh installation. However, you can use the command as you want.
Precisely the same as here, which hasn't been resolved.
Followed the sequential directions here; all channels added.
Tried:
Adding to .bash_profile export PKG_CONFIG_PATH=$PKG_CONFIG_PATH://anaconda/pkgs accordingly the directory where there is cairomm installed..
./configure -with--CAIROMM_CFLAGS -with--CAIROMM_LIBS
Can someone kindly make sure at least I have implemented alternate solutions correctly?
And of course I've tried the simplest conda install graph-tool after adding channels from ostrokach-forge and the like.
Instead of success, I get the following:
PackagesNotFoundError: The following packages are not available from current channels:
Whoa, so unpopular post!
For myself and other novices' use in the future ->
(after all, conda install-ed python is easier)
Slightly different but same in that certain library is not reached, acc. to this query:
conda needs to be able to find all the dependencies at once.
The -c flag only adds that channel for that one command.
conda install -c vgauthier rwest graph-tool. But an easier way is to add those channels to your configuration
conda config --add channels vgauthier --add channels rwest
And then perform
conda install graph-tool
But when I used conda install -c http://conda.anaconda.org/vgauthier graph-tool command, it worked instantly.
Before that, nothing worked. (when I only used the user name vguathier or ostrokach-forge or else)
If I type vi .condarc I see
channels:
- ostrokach-forge
- conda-forge
- defaults
And because I remain quite ignorant in using brew-installed packages with conda-installed python, I started out by following the directions to install all the necessary dependencies using brew. (including pixman)
Wonder how the command found everything though.. plus python upgraded to 3.6 from 3.5.
God this is how I am left with solving os issues.. i'm not 100 % clear how I made the computer connect the dots.
And still, nonetheless, I am still left with how to figure the install with ./configure. I want to understand the error msg that was returned and how to address it.
I'm using Windows 7 and am up to date on patches. I was using Python 3.5.2 and wanted to upgrade to 3.6, so I went to the Python site, downloaded 3.6.4.exe for Windows, and ran it. The Python seems to work fine and is 3.6.4, but trying to import any of the packages I was using (pandas, numpy, tensorflow, etc) now gives me ModuleNotFoundError: No module named <whichever module it was>. Also, pip list now shows only pip and setuptools.
It appears to be a known issue, for example this from nearly a year ago, which suggests that I should "uninstall the python bindings and install again", but I can't figure out what that means. Fortunately I can still access 3.5.2 by using py -3.5, and then my imports work. Can anyone tell me how to fix this for 3.6 without having to reinstall all my modules manually?
I was thinking possibly I should have upgraded through pip; it seems like that's possible but maybe a bad idea for some reason? On further investigation it looks like what I should have done was save my requirements with pip freeze > requirements.txt, and then after installing the new Python restore them with pip install -r requirements.txt. Is this right?
Hard to say if you have an install problem, but this is what I would try if I were in your place.
Create a virtual environment as per; docs
c:>c:\Python36\python -m venv c:\path\to\myenv
Activate your virtual environment
C:> \Scripts\activate.bat
Run your application from within your activated environment. Each time you get an import error, do a pip-install from within the active environment. (For your own modules, you may need to modify PYTHONPATH in 'activate.bat')
Once you have your application running again, do your pip freeze > requirements.txt, and keep that with your project.
Each time you run your application, do so from within the activated virtual environment.
This will give you a clean requirements.txt that doesn't include a bunch of junk from other projects. Then, when you go to 3.7, just create the virtualenv and run your requirements.txt and wala!
I suspect your issue is simply not running against the correct interpreter, running from within a virtual environment should at least rule it out.